[ML] Extend population preview chart to show actual and typical value

[qn895]

Summary

This PR addresses #33490 extends the y axis domain to the population preview chart to also account for the anomaly actual and typical, not just the data.

This makes it easier to contextualize anomalies where markers look to be in the same range as other data points, but its typical value is actually much higher. I ran this for focusData inside

Original (red dot doesn't seem so anomalous)

Fixed

Originally I tried setting it based on the chart's whole range instead of just the highlighted data and they looked quite off.

Checklist

Delete any items that are not applicable to this PR.

• This was checked for keyboard-only and screenreader accessibility
• This renders correctly on smaller devices using a responsive layout. (You can test this in your browser
• This was checked for cross-browser compatibility, including a check against IE11

[qn895]

I used yScaleDomainMin < 0 ? yScaleDomainMin : 0 here because I wasn't sure if we should always expect the data set to have positive values.

[peteharverson]

Gave this a good test using a range of population jobs on apache2, nginx and the gallery data sets, and generally the change to the way the y axis range is calculated is working well - a big improvement when the anomaly actual is a lot lower than the typical.

@qn895 @walterra I wonder if there is anything we can do to improve the sampling used to plot the background population of gray dots? The example shown above for the 'Fixed' case looks good, in that you see a lot of gray dots around the typical value of 4271, but a lot of the cases I worked with, you don't see (m)any clustered around the population typical. e.g. below most of the gray dots are plotted between 0 and 50000, which is somewhat confusing when the typical for the anomaly is reported as 145000:

Note though that this fix is a big step forward from the current behavior:

[pheyos]

Looking at your screenshots, I think it's way better than before, where charts for unusual low value didn't show enough context.
But I think having the typical value as an upper bound for the chart might still not give enough context as everything that's be a bit above the typical isn't shown.
Not sure if this makes sense, but I would like to try to have the typical around the middle of the chart. In that case the upper chart bound could be something like typical + (typical - actual) - for low anomalies (actual < typical) only of course as the high anomalies already show enough context.
What do you think @peteharverson ?

[peteharverson]

@pheyos (correct me if I'm wrong @qn895 !) the new calculation for the max is now doing _max = Math.max(_max, metricValue, actualValue, typicalValue); so I would expect the upper bound to be above the anomaly typical, assuming some of the charts points have metric values above typical. Although the comment I made above in #67569 (comment) is somewhat related, as from my testing, I often found very few points plotted around the typical value where the anomalies are for unusually low values. Unless we can can change the sampling used to populate the 'background' population, we might just end up with even more blank space at the top of the chart.

[walterra]

Did some local tests, and I think it's a nice improvement and a quick win. We can think of and do additional improvements, but I'd vote for doing those in follow-ups. (Anything that would touch how we query the data is more peculiar for example, because they are already quite heavy).

[walterra]

Do we need a check here to not push the limit below 0 for detectors based on count for example where we can be sure there will never be a value below 0 and we don't want to push the axis domain below 0?

[peteharverson]

LGTM.

I added an extra item to the population chart meta issue #18428 for investigating if we can improve the sampling of the population used to draw the background gray dots.

[elasticmachine]

Pinging @elastic/ml-ui (:ml)

[walterra]

Latest changes LGTM

[qn895]

@elasticmachine merge upstream

[kibanamachine]

💛 Build succeeded, but was flaky

• continuous-integration/kibana-ci/pull-request
• Commit: 6b7787e
• Flaky suites:
  • kibana-firefoxSmoke

---

Test Failures

Kibana Pipeline / kibana-oss-agent / Firefox UI Functional Tests.test/functional/apps/home/_home·js.homepage app Kibana takes you home clicking on kibana logo should take you to home page

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

[00:00:00]       │
[00:12:16]         └-: homepage app
[00:12:16]           └-> "before all" hook
[00:12:16]           └-> "before all" hook
[00:12:16]           └-: Kibana takes you home
[00:12:16]             └-> "before all" hook
[00:12:16]             └-> clicking on kibana logo should take you to home page
[00:12:16]               └-> "before each" hook: global before each
[00:12:16]               │ debg navigating to settings url: http://localhost:6111/app/management
[00:12:16]               │ debg navigate to: http://localhost:6111/app/management
[00:12:16]               │ debg browser[log] "^ A single error about an inline script not firing due to content security policy is expected!"
[00:12:16]               │ debg ... sleep(700) start
[00:12:17]               │ debg ... sleep(700) end
[00:12:17]               │ debg returned from get, calling refresh
[00:12:19]               │ debg browser[info] "INFO: 2020-06-03T20:34:33Z
[00:12:19]               │        Adding connection to http://localhost:6111/elasticsearch
[00:12:19]               │
[00:12:19]               │      "
[00:12:19]               │ERROR browser[error] (new TypeError("NetworkError when attempting to fetch resource.", ""))
[00:12:19]               │ debg browser[log] "^ A single error about an inline script not firing due to content security policy is expected!"
[00:12:19]               │ debg currentUrl = http://localhost:6111/app/management
[00:12:19]               │          appUrl = http://localhost:6111/app/management
[00:12:19]               │ debg TestSubjects.find(kibanaChrome)
[00:12:19]               │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:12:24]               │ debg browser[info] "INFO: 2020-06-03T20:34:39Z
[00:12:24]               │        Adding connection to http://localhost:6111/elasticsearch
[00:12:24]               │
[00:12:24]               │      "
[00:12:25]               │ debg ... sleep(501) start
[00:12:25]               │ debg ... sleep(501) end
[00:12:25]               │ debg in navigateTo url = http://localhost:6111/app/management
[00:12:25]               │ debg TestSubjects.exists(statusPageContainer)
[00:12:25]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="statusPageContainer"]') with timeout=2500
[00:12:28]               │ debg --- retry.tryForTime error: [data-test-subj="statusPageContainer"] is not displayed
[00:12:28]               │ debg TestSubjects.click(headerGlobalNav > logo)
[00:12:28]               │ debg Find.clickByCssSelector('[data-test-subj="headerGlobalNav"] [data-test-subj="logo"]') with timeout=10000
[00:12:28]               │ debg Find.findByCssSelector('[data-test-subj="headerGlobalNav"] [data-test-subj="logo"]') with timeout=10000
[00:12:28]               │ERROR browser[error] (new Error("ScopedHistory instance has fell out of navigation scope for basePath: /app/management", "http://localhost:6111/33573/bundles/core/core.entry.js", 76))
[00:12:28]               │ERROR browser[error] (new Error("ScopedHistory instance has fell out of navigation scope for basePath: /app/management", "http://localhost:6111/33573/bundles/core/core.entry.js", 76))
[00:12:28]               │ debg isGlobalLoadingIndicatorVisible
[00:12:28]               │ debg TestSubjects.exists(globalLoadingIndicator)
[00:12:28]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:12:28]               │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:12:28]               │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:14:08]               │ info Taking screenshot "/dev/shm/workspace/kibana/test/functional/screenshots/failure/homepage app Kibana takes you home clicking on kibana logo should take you to home page.png"
[00:14:09]               │ info Current URL is: http://localhost:6111/app/home#/
[00:14:09]               │ info Saving page source to: /dev/shm/workspace/kibana/test/functional/failure_debug/html/homepage app Kibana takes you home clicking on kibana logo should take you to home page.html
[00:14:09]               └- ✖ fail: "homepage app Kibana takes you home clicking on kibana logo should take you to home page"
[00:14:09]               │

Stack Trace

Error: expected testSubject(globalLoadingIndicator-hidden) to exist
    at TestSubjects.existOrFail (test/functional/services/common/test_subjects.ts:62:15)
    at process._tickCallback (internal/process/next_tick.js:68:7)

---

History

• 💔 Build #50195 failed 1f1974b

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream