Data view field conflicts resolving by filtering data

[philippkahr]

**Kibana version: 8.14.0 **

Describe the bug:
I know that field conflicts are an issue and the data view is telling me about it. I like to use the generic data views such as logs-* which enable this problem.

What I would expect is if I have to data streams:

demo-kibana.log
demo-kibana.audit

where one maps the source.ip as ip and one maps it as keyword. That the data view: demo-kibana.* shows a conflict. That is working as expected.
Now when I add a KQL and filter down using constant keyword: demo-kibana.log I would expect the conflict to go away and for Kibana to let me use the field. Because within the same dataset the field is mapped the same. There is no conflict within demo-kibana.log there is a conflict between kibana.log and kibana.audit
The issue is that this also forbids me to use in controls since the data view “errors”.
I would expect Kibana to reevaluate the fields once I add a KQL that filters me away entire shards that create this issue, such as using constant keyword for data_stream.dataset.

Steps to reproduce:

PUT _index_template/demo-kibana.audit
{
  "template": {
    "mappings": {
      "properties": {
        "data_stream": {
          "type": "object",
          "properties": {
            "dataset": {
              "type": "constant_keyword"
            }
          }
        },
        "source": {
          "type": "object",
          "properties": {
            "ip": {
              "type": "keyword"
            }
          }
        }
      }
    }
  },
  "index_patterns": [
    "demo-kibana.audit-*"
  ],
  "data_stream": {}
}

PUT _index_template/demo-kibana.log
{
  "template": {
    "mappings": {
      "properties": {
        "data_stream": {
          "type": "object",
          "properties": {
            "dataset": {
              "type": "constant_keyword"
            }
          }
        },
        "source": {
          "type": "object",
          "properties": {
            "ip": {
              "type": "ip"
            }
          }
        }
      }
    }
  },
  "index_patterns": [
    "demo-kibana.log-*"
  ],
  "data_stream": {}
}

PUT _data_stream/demo-kibana.log-default
PUT _data_stream/demo-kibana.audit-default

POST demo-kibana.log-default/_doc
{
  "@timestamp": "2024-07-09T12:00:00.000Z",
  "data_stream": {
    "dataset": "kibana.log"
  },
  "source": {
    "ip": "192.168.0.1"
  }
}

POST demo-kibana.audit-default/_doc
{
  "@timestamp": "2024-07-09T12:00:00.000Z",
  "data_stream": {
    "dataset": "kibana.audit"
  },
  "source": {
    "ip": "192.168.0.1"
  }
}

Expected behavior:

When filtering down to a subset the fields shold be re-evaluated. Making sure that there are only conflicts within the actual queried indices.

Screenshots (if relevant):

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[kertal]

Closing this because it's not planned to be resolved in the foreseeable future. It will be tracked in our Icebox and will be re-opened if our priorities change. Feel free to re-open if you think it should be melted sooner.