feat: include alias path when generating template

[rgmz]

This fixes #876

Done quickly via GitHub web, but let me know if there are any additional changes required (e.g. tests or changelog).

[cla-checker-service]

💚 CLA has been signed

[rgmz]

x Author of the following commits did not sign a Contributor Agreement:
9864f19

Please, read and sign the above mentioned agreement if you want to contribute to this project

I signed the CLA prior to creating this PR, so maybe it takes a bit of time to be recognized.

[webmat]

I re-ran our tool to check the CLA, and you signed it correctly indeed.

Thanks for submitting this, we will review shortly

[ebeahan]

Thanks again for submitting @rgmz! I see value in adding support for the alias type per your description in #876 .

These changes would introduce an optional path field set attribute to the schema, which is required when type is specified as alias? Is the following example snippet an accurate example?

    - name: id
      level: extended
      type: alias
      path: custom.host.id

We'll need to consider other aspects for supporting this additional path field and the alias type (likely others - these were two that came to mind so far):

• The beats generator, ./scripts/generators/beats.py should also support path when type is alias.
• The target of path is subject to restrictions which wouldn't be checked or enforced by the ECS tooling (at least initially). We could mark the support for alias as experimental and note the limitation, but it could cause some hidden complexity when attempting to apply the mappings.

[webmat]

Yeah I support this addition.

This only fixes it for the Go generator. I'd like it if we failed the run if path is missing on an alias field, in cleaner.py / field_mandatory_attributes. Check it out here

@rgmz Are you good with adding this? If not, perhaps @ebeahan could pick this up?

[ebeahan]

This only fixes it for the Go generator

ES generator? Or is there a relationship to the Go generator I'm overlooking?

I did see a possible need for introducing a new type in the Go generator, but since how we're aiming to remove it (#628) in the near future I wasn't concerned. I also did some light testing of the branch and didn't see any immediate issues.

[webmat]

Yes the alias datatype itself should make it there no problem. But it's a good point to ensure the path attribute makes it there too, and that it's not being filtered out.

Note that ECS doesn't make use of the alias type, so we should make sure the ES script has test coverage for this as well.

[rgmz]

@webmat @ebeahan

Apologies for letting this stagnate, it's been a busy few months! :)

These changes would introduce an optional path field set attribute to the schema, which is required when type is specified as alias? Is the following example snippet an accurate example?

    - name: id
      level: extended
      type: alias
      path: custom.host.id

Correct. The intention is to match the schema to the Elasticsearch usage of alias, which you've linked.

We'll need to consider other aspects for supporting this additional path field and the alias type (likely others - these were two that came to mind so far):

• The beats generator, ./scripts/generators/beats.py should also support path when type is alias.

• The target of path is subject to restrictions which wouldn't be checked or enforced by the ECS tooling (at least initially). We could mark the support for alias as experimental and note the limitation, but it could cause some hidden complexity when attempting to apply the mappings.

I've updated ./scripts/generators/beats.py and scripts/schema/cleaner.py to support path for the alias type.

I agree that validating the generated artifacts would be ideal, however, that's probably out of scope for this PR. e.g., you can also generate an invalid template by defining type: invalid_type123.

[ebeahan]

Thanks for the continue work here @rgmz ! This will be a nice feature addition.

Outside of the few notes I made below, can you also add:

• A brief entry in the Supported keys to describe fields section of schemas/README.md for path
• A changelog entry to CHANGELOG.next.md

[rgmz]

Thanks for the continue work here @rgmz ! This will be a nice feature addition.

Outside of the few notes I made below, can you also add:

* [ ]  A brief entry in the `Supported keys to describe fields` section of  `schemas/README.md` for `path`

* [ ]  A changelog entry to `CHANGELOG.next.md`

Done!

(Working from the GitHub UI once again, so let me know if I missed anything.)

[ebeahan]

This looks great @rgmz ! Thanks for making those changes.

I'm approving and will handle merging and then backporting 🚢

[webmat]

Thanks @rgmz and @ebeahan for getting this to the finish line :-)