Pattern based ECS

[ruflin]

As part of the effort to bring ECS closer to Elasticsearch #1869 elastic/elasticsearch#89743 one of the challenges was how not having to map all the fields. The number of fields in ECS has grown a lot over the past years and keeps growing. One way I found to avoid having to map all the fields is using patterns (see elastic/elasticsearch#89743 for all the patterns). For example all *.ip are type:ip, all *.bytes are type:long etc.

I don't think this is a coincidence but an unwritten (or is written down?) rule in ECS that the same name of fields should be the same type. I wonder if we could take this further with ECS and make this a rule. The result of this would not only be to simplify the mapping in Elasticsearch which is a nice side effect, but also ensure whenever someone uses a field .ip it should be of type ip.

Most of the datasets using ECS have their own additional fields. The main rule so far is to not conflict with ECS. This could be expanded to "use these conventions".

[kgeller]

@ruflin I don't believe this rule is currently written-down, but would be at the least, a great candidate for the documented ECS conventions.

[ruflin]

The part I would like to discuss is making this more then just a convention. It would be great to add it to the convention too but I worry as soon as we have it there, we stay away from discussing making it a rule.