Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

De-dot Micrometer metric names #1700

Merged
merged 6 commits into from
Mar 16, 2021
Merged

Conversation

felixbarny
Copy link
Member

@felixbarny felixbarny commented Mar 15, 2021

What does this PR do?

This is a breaking change. However, it seems to be a necessary one:
The consequences could be quite severe
(mapping conflicts may only be fixable by deleting the index).
Also, the micrometer integration is still in beta so breaking changes
are probably ok.

The alternatives, such as flattened fields have downsides, such as
lack of Kibana support and worse performance.
The Elasticsearch team is not considering flattened fields the
endgame and are investigating other alternatives.

With that, I don't see the need for a config option to opt-out of the
de-doting yet.

relates to elastic/apm#347

Checklist

  • This is an enhancement of existing features, or a new feature in existing plugins
    • I have updated CHANGELOG.asciidoc
    • I have added tests that prove my fix is effective or that my feature works
    • Added an API method or config option? Document in which version this will be introduced
    • I have made corresponding changes to the documentation
  • This is a bugfix
  • This is a new plugin
    • I have updated CHANGELOG.asciidoc
    • My code follows the style guidelines of this project
    • I have made corresponding changes to the documentation
    • I have added tests that prove my fix is effective or that my feature works
    • New and existing unit tests pass locally with my changes
    • I have updated supported-technologies.asciidoc
    • Added an API method or config option? Document in which version this will be introduced
    • Added an instrumentation plugin? Describe how you made sure that old, non-supported versions are not instrumented by accident.
  • This is something else

This is a breaking change. However, it seems to be a necessary one:
The consequences could be quite severe
(mapping conflicts may only be fixable by deleting the index).
Also, the micrometer integration is still in beta so breaking changes
are probably ok.

The alternatives, such as flattened fields have downsides, such as
lack of Kibana support and worse performance.
The Elasticsearch team is not considering flattened fields the
endgame and are investigating other alternatives.

With that, I don't see the need for a config option to opt-out of the
de-doting yet.
@felixbarny felixbarny marked this pull request as ready for review March 15, 2021 08:49
@apmmachine
Copy link
Contributor

apmmachine commented Mar 15, 2021

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Branch indexing

  • Start Time: 2021-03-16T08:39:18.266+0000

  • Duration: 53 min 10 sec

  • Commit: de36946

Test stats 🧪

Test Results
Failed 0
Passed 1853
Skipped 14
Total 1867

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 1853
Skipped 14
Total 1867

@codecov-io
Copy link

Codecov Report

Merging #1700 (07bde74) into master (10c322b) will decrease coverage by 0.02%.
The diff coverage is 95.23%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master    #1700      +/-   ##
============================================
- Coverage     58.86%   58.84%   -0.03%     
  Complexity       92       92              
============================================
  Files           403      403              
  Lines         18475    18478       +3     
  Branches       2570     2572       +2     
============================================
- Hits          10875    10873       -2     
- Misses         6832     6835       +3     
- Partials        768      770       +2     
Impacted Files Coverage Δ Complexity Δ
.../micrometer/MicrometerMeterRegistrySerializer.java 94.78% <95.23%> (-0.76%) 0.00 <0.00> (ø)
...tic/apm/agent/report/ssl/TLSFallbackSSLSocket.java 39.13% <0.00%> (-2.18%) 0.00% <0.00%> (ø%)
...ic/apm/agent/profiler/collections/LongHashSet.java 17.06% <0.00%> (-0.40%) 0.00% <0.00%> (ø%)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 10c322b...07bde74. Read the comment docs.

@SylvainJuge SylvainJuge self-assigned this Mar 15, 2021
@cyrille-leclerc
Copy link

Thanks @felixbarny, I'm excited to see this alignment across our products. Did you implement a configuration flag for backward compatibility?

@felixbarny
Copy link
Member Author

I've added a dedot_custom_metrics flag. The docs contain a warning that explains the potential for mapping conflicts. While we wouldn't recommend setting the flag to false, it may help to mitigate backward compatibility issues.

@cyrille-leclerc
Copy link

Thanks @felixbarny . I'm not familiar enough with asciidoc and I didn't see De-dotting be disabled via <<config-dedot-custom-metrics, ``dedot_custom_metrics``>>

private static void serializeValueStart(String key, String suffix, JsonWriter jw, StringBuilder replaceBuilder, boolean dedotMetricName) {
replaceBuilder.setLength(0);
if (dedotMetricName) {
DslJsonSerializer.sanitizeLabelKey(key, replaceBuilder);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] maybe rename method as it's now not only used for label keys, but more keys in general.


@Test
void testDedotMetricName() {
meterRegistry.counter("foo.bar").increment(42);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] maybe add extra assertion to ensure MetricsConfiguration#isDedotCustomMetrics returns true by default.

@felixbarny
Copy link
Member Author

@felixbarny
Copy link
Member Author

@cyrille-leclerc: I think I misinterpreted your last statement. I just implemented that flag after you asked for it.

@lucabelluccini
Copy link
Contributor

Hello @felixbarny - I wanted to propose the feature flag but you did in the meanwhile 👏

I would be tempted to switch its default to false: we do not have a breaking change and we keep getting mapping rejections only for users of such metrics which contain a . in the field name and for which they have the scalar/object problem.
If instead the way forward is de-dotting all the metrics in the future, then we can switch to true.

@felixbarny
Copy link
Member Author

Hey Luca, I was thinking of that but the current behavior (dedot_custom_metrics=false) is actually quite dangerous. It affects all Spring Boot users. By default, Spring Boot exports jdbc connection pool metrics that result in a mapping conflict. I think it's better to be safe than sorry and de-dot by default, which is what Metricbeat does, too.

I know that this is a breaking change and thus can be painful for some users. Those users can use the flag to opt-out (and live in constant danger of mapping conflicts). Note that the Micrometer integration is still in beta which is why I think introducing a breaking change is reasonable.

@cyrille-leclerc
Copy link

Thanks @lucabelluccini . As @felixbarny explained, we looked at the pros and cons and we prefer to introduce this backward incompatible change on this beta version with a configuration param to go back to the previous mode:

  • Pros
    • Dedotted works out of the box with most popular use cases such as Spring Boot
    • Behaviour aligned with metricbeat
  • Cons
    • backward compatibility problem mitigated by the configuration flag to re enable the old behaviour. The beta phase is intended to support backward incompatible changes if we unfortunately have to introduce some.

@lucabelluccini
Copy link
Contributor

Totally agree and I didn't recall it was in beta. 👏 Thank you for addressing this!

@felixbarny felixbarny merged commit 71213cd into elastic:master Mar 16, 2021
@felixbarny felixbarny deleted the micrometer_dedot branch March 16, 2021 09:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants