Lens showing empty field when there is data

[alexfrancoeur]

For some reason I feel like I've opened a bug for this before, so it may already be addressed. I'm looking back 30 days where there is relevant data, but Lens is telling me the field is empty. I would expect this field to be available under all fields. I'm not sure what timeframe we're looking back for data, but should we be matching the global time picker?

This is on version 7.10.1. Happy to share an example if needed

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[flash1293]

We are sourcing 500 documents for determining whether a field is "available". If the field is only present on some documents, you can be unlucky and run into this case where Lens is reporting the wrong thing. Does it work correctly if you add a filter "x does exist" (x being the field name)?

[flash1293]

If that's the case, there is an issue for this already: #58330

Also, the rest of Lens should still work (if you pull in the field, you should get a chart)

[alexfrancoeur]

We are sourcing 500 documents for determining whether a field is "available". If the field is only present on some documents, you can be unlucky and run into this case where Lens is reporting the wrong thing. Does it work correctly if you add a filter "x does exist" (x being the field name)?

Yes, if I filter on that field, it shows up in the list of available fields. I'll close out in favor of #58330 but it feels like we'll want a better experience here as we begin to make Lens the default experience. It doesn't seem like Lens handles sparse fields that well. I can understand the performance impact for optimizing for sparse fields, but there has to be a better UX.

The scenario I find the most unsettled is when the preview comes empty and when dropped it shows data, it makes Lens seem unreliable. I understand it is due to sparse data, maybe we should try to optimize only this use case.

I have similar thoughts to @AlonaNadler's in that issue, that was exactly what I thought and immediately went to the older visualize application as a result.