Add support for nanosecond timestamps

[urso]

The @timestamp field only has microsecond precisions. Internally, or when collecting from external sources, timestamps might have higher precisions. Elasticsearch supports the date_nano type, and allows to store timestamps with higher precision even if date is used.

The feature-timestamp-nano branch is used for development.
In progress PR merging the feature branch into master: #15872

• Support for parsing timestamp with different permissions. [WIP]beats @timestamp support nanoseconds #9818
• Update dtfmt to support higher precision. [WIP]beats @timestamp support nanoseconds #9818
  • Add n formatter to always print nanoseconds. [WIP]beats @timestamp support nanoseconds #9818
  • Allow S formatter to be used for higher precisions. [WIP]beats @timestamp support nanoseconds #9818
  • Add f formatter. Timestamp fractions pattern #15911
• Update unit tests. [WIP]beats @timestamp support nanoseconds #9818
• Update integration tests. [WIP] Minor improvements and test fixes for timestamp nanosecond support #16005
• Update system tests. [WIP] Minor improvements and test fixes for timestamp nanosecond support #16005,
• Check timestamp parsing in individual beats/modules/inputs:
  • Can the input/module parse the timestamp with higher precision than is currently the case?
  • Metricbeat modules
  • Filebeat inputs
  • Filebeat modules
  • Heartbeat monitors
  • Auditbeat modules
  • Packetbeat
• Check whether we can change the template in a backwards compatible way.
  • Do we need to use date_nano, or can keep date for timestamps?
• Check Kibana works correctly with nano types (according to Support date_nanos in Kibana kibana#31424 it works but we would need to do some testing):
  • do visualizations work without updates?
  • do visualizations/dashboards work when mixing data from indices with millisecond and microsecond precision.

Other related issues:

• Support log4j format for timestamps (comma-milliseconds) Support log4j format for timestamps (comma-milliseconds) #17794
• filebeat docker-json timestamp support nanoseconds? filebeat docker-json timestamp support nanoseconds? #17660

[LotoQ]

Hi guys, was just wondering if there was a planned release date for this feature? I was just noticing that it looked like most of the tasks were complete, so really looking forward to being able to use it soon! :^)

[pytimer]

filebeat docker-json timestamp support nanoseconds? #17660

Like issue #17660, libbeat docker-json reader not support nanoseconds, it maybe occur some problem, because i have a container and this container logs like below, i have no way to sort by the correct @timestamp. So filebeat support nanoseconds is necessary for me.

{"log": "Starting web...", "stream":"stderr", "time":"2020-06-15T05:26:22.521012484Z"}
{"log": "Starting web middleware...", "stream":"stderr", "time":"2020-06-15T05:26:22.521012490Z"}

[pytimer]

filebeat docker-json timestamp support nanoseconds? #17660

Like issue #17660, libbeat docker-json reader not support nanoseconds, it maybe occur some problem, because i have a container and this container logs like below, i have no way to sort by the correct @timestamp. So filebeat support nanoseconds is necessary for me.

{"log": "Starting web...", "stream":"stderr", "time":"2020-06-15T05:26:22.521012484Z"}
{"log": "Starting web middleware...", "stream":"stderr", "time":"2020-06-15T05:26:22.521012490Z"}

I read the source code about docker_json.go, i found parse docker log via time.RFC3339, it can parse 2020-06-15T05:26:22.521012484Z to 2020-06-15 05:26:22.521012484 +0000 UTC . I also test time.Parse() it return nanoseconds, so i think docker-json reader can support nanoseconds. If something wrong, please point out.

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[mtheck]

Running on Kubernetes and still limited to milliseconds precision for log timestamps is a bit embarrassing!

[jlind23]

8.3 goal - understand where we are at and take a decision based on it.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[jlind23]

@kvch as you will be working on the analysis of that one, could you please provide us as an outcome:

• The estimated time left to merge this PR into master [WIP] Add for timestamps with nanoseconds precisio #15872 (It seems to be the only PR left)
• The list of inputs on which there is still work to do.

cc @ph @cmacknz in case you see something else to add

[kvch]

I have opened a new PR #31553 in favour of the old one (#15872). At the moment it is a draft because the tests are still failing.

I tested a few modules in Filebeat and Metricbeat manually. Pipelines and visualizations seem to work. I am confident once the tests pass on the CI we could get this in a few days. However, I would like to ask for additional manual testing from either QA source or others on our team.

[jlind23]

@dikshachauhan-qasource could you please consider it for the 8.3 testing scope?

[dikshachauhan-qasource]

Hi @jlind23 Thanks for the update on requirement for testing. We will check this out and let you know progress in comments.

Thanks
Diksha

[amolnater-qasource]

Hi @jlind23 @kvch
As per our understanding we have validated this by installing Metricbeat with System module enabled on latest 8.3 Snapshot Kibana cloud environment.
Further, we have observed the data from Metricbeat under Discover tab.

• Currently data is available with @timestamp: 10:09:03.365Z

Screenshots:

Queries:

Could you please confirm:

• If these changes are just for Beats and no impact on Elastic-Agent logs.
• Validation requires to be covered only Discover tab.

Please let us know if we are missing any other scenario to be covered.
Thanks

[kvch]

The feature hasn't been merged to main yet. Or did you try test it with my feature branch?

[amolnater-qasource]

Hi @kvch
Thanks for looking into this, we haven't validated this upon feature branch. We attempted to test current behaviour of @timestamp on cloud-staging kibana for metricbeat and wanted to confirm if this is the testing area.

Further we had few queries related to this feature which we have shared under #15871 (comment)
We will be retesting this once the related PR will be merged.

Thanks

[amolnater-qasource]

Hi @kvch
We have attempted to validate this on 8.3 Snapshot Kibana cloud environment and followed below steps:

1. Setup Kibana cloud environment.
2. Installed Metricbeat with system module enabled and Auditbeat.
3. Observed data with @timestamp under Discover tab.
   
4. We observed that data is still available with timestamp: @ 17:22:57.955

Build details:
BUILD: 52935
COMMIT: 5d5603a57237d8fe9cf186916c713b9ddddf039d
(MAY 19, 2022 3:42 PM)

Question

Could you please share some more testing details or could confirm if we are missing anything here?

cc: @jlind23
Thanks

[kvch]

By default Beats still use millisecond precision. You have to set timestamp.precision in the configuration, see the reference:

# Configure the precision of all timestamps in Metricbeat.
# Available options: millisecond, microsecond, nanosecond
#timestamp.precision: millisecond

[joshdover]

Were metricbeat mappings changed in this release? If not, I believe the @timestamp field will still show at millisecond precision in Discover. You'll need to look at the _source.@timestamp field in the JSON tab of the "Expanded document" flyout.

[amolnater-qasource]

Hi @kvch @joshdover
Thanks for the feedback.
We have revalidated this on latest 8.3 Snapshot kibana cloud environment and followed below steps:

• We have added timestamp.precision: nanosecond under metricbeat-reference.yml and winlogbeat-reference.yml.
• We then observed the data under Discover>JSON tab of the "Expanded document" flyout.
• We observed nanosecond data under _source: @timestamp.

OS& Beats covered:
Platform: Windows 10 x64
Winlogbeat
Metricbeat

Screenshots:

Question:

Please let us know if we need to validate this on any other beats or platforms.

Thanks

[stephan-erb-by]

Would it make sense to change the default mappings as part of this release as well? Or is this out of scope in any case?

In my mind beats could generate templates where @timestamp uses date_nanos if timestamp.precision is set to nanosecond.

[joshdover]

@cmacknz should we do this mappings change?

[cmacknz]

It sounds reasonable to me to switch to date_nanos in the templates. Likely we can do this with a follow up issue.

@kvch does this sound reasonable to you?

[axw]

There are still some open issues regarding date_nanos in Kibana. Shouldn't we wait for them to be addressed before switching field types?

• [Logs UI] The UI throws error when @timestamp format is date_nanos kibana#88290
• 64 bit number/integer support in Kibana kibana#40183

[cmacknz]

Thanks Andrew, I've created an issue to support date_nanos in the future: #31838

[kvch]

I agree with @axw. Also, the default @timestamp granularity is still milliseconds, so we do not have to hurry with the change.
Also, all of the vizualizations work with the current template even if the timestamp precision is set to nanoseconds.