[Lens] Create a filter with field:value when last value metric is used on a datatable

[mbondyra]

Fixes #152883

The task says this is the behavior that we want for datatable, but with the way data plugin interacts with Lens, it's actually impossible to modify only for datatable. In my opinion that's an improvement though, we probably want the same behavior for all supported visualization (so also metric).

Scenarios to test:

1,2,3 returns the price:${value} filter
4. returns price:exists filter
5. returns category.keyword:${value} coming from filter by
6. OR filter for array values

[elasticmachine]

Pinging @elastic/kibana-visualizations @elastic/kibana-visualizations-external (Team:Visualizations)

[drewdaemon]

I tested all the cases you suggested and everything worked well.

I did find a confusing scenario, but I'm not sure if there's anything we can do.

When you're dealing with an array value but you have "Show array values" turned off (using top_metrics agg instead of top_hits), the filter that is created excludes all the documents in the time range, making the metric go blank (ignore the terrible formatting you see in the video, I created an issue for that: #160815).

Screen.Recording.2023-06-28.at.2.41.38.PM.mov

This is definitely because of elastic/elasticsearch#82281 (original issue in Lens #102956).

I did some experiments on top_metrics and, given an array of numbers, top_metrics returns a single number that seems to be somehow computed from the numbers in the array, but through no predictable calculation I could figure out.

DELETE abc

POST abc/_doc
{
  "arr": [10, 50],
  "timestamp": "2020-02-02T00:00:00.000Z"
}

GET abc/_search
{
  "size": 0,
  "aggs": {
    "a": {
      "top_metrics": {
        "metrics": {"field": "arr"},
        "sort": {"timestamp": "desc"}   
      }
    }
  }
}

Result:

"aggregations": {
    "a": {
      "top": [
        {
          "sort": [
            "2020-02-02T00:00:00.000Z"
          ],
          "metrics": {
            "arr": 30
          }
        }
      ]
    }

So in this scenario the filter would get created as arr : 30 which doesn't match any docs. (Also, why 30? Docs say "A top_metric aggregation on array values may return inconsistent results.")

Anyway, not caused in any way by your PR, but perhaps something we could connect with ES about as a follow-up.

[mbondyra]

Thanks @drewdaemon for the thorough review! So the mentioned problem happens because of precision? Eg day_of_week_i doesn't have this effect and creates a correct filter. I am wondering if we should just block key:value filter for last values when value is of type number and is not integer (so has a fraction). Not sure though. @stratoula WDYT?

[stratoula]

@mbondyra I am trying to decide which option is going to cause less confusion. I tend to believe that we should keep it as it is on your PR. I feel that if numeric fields work differently it will cause more confusion.
It makes sense to talk to ES about it. I feel that if it creates confusion we can explain easily why it happens.

[mbondyra]

Scratch that, it's because of the last value actually being an array, not because of the precision so it wouldn't help anyway. Yeah, it's probably on elasticsearch, let's talk during the sync.

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: a5323d9

Metrics [docs]

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
data	406.5KB	406.9KB	+385.0B

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
enterpriseSearch	14	16	+2
securitySolution	413	417	+4
total			+6

Total ESLint disabled count

id	before	after	diff
enterpriseSearch	15	17	+2
securitySolution	492	496	+4
total			+6

History

• 💚 Build #138871 succeeded 16aaabb
• 💚 Build #138048 succeeded 2c58e23

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream