Introducing preprocessors on the module input override

[mjmbischoff]

Introducing preprocessors on the module input override which are run after the input but before the processors in the module so some pre transformation can occur.

What does this PR do?

Allows preprocessors to be defined on the module input override which are run after the input but before the processors in the module as opposed to the processors defined at this level which are run after the module. The preprocessors are normal processors and after merging the config don't exist as a separate property but are merged with the processors.

Why is it important?

This PR introduces flexibility to filebeat and allows more module reuse vs local forking (breaking updates) and can simplify the ingest pipeline requiring fewer components

Checklist

• My code follows the style guidelines of this project
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

• Competes with run processors, defined in the block overriding the input, before the module. #26862

Use cases

When intermediate transport is used, for example syslog shipped over syslog, leading to double syslog headers which the module doesn't expect.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2021-07-30T14:51:01.199+0000

• Duration: 92 min 46 sec

• Commit: dc5772b

Test stats 🧪

Test	Results
Failed	0
Passed	14912
Skipped	2313
Total	17225

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	14912
Skipped	2313
Total	17225

[kvch]

First I would refrain from merging this. I would like to see if parsers are good enough for inputs. In general, we do not want to have much processing in Beats. Everything should be done at the Ingest Node, if possible.

Also, I will bring this up in internal discussion with other devs on the team.

[mjmbischoff]

I'm afraid when the parsers can adres all the use-cases it will be a duplicated effort of the processors.
And with regard to processing, as long as as there is any processing in beats without this the data will be in the wrong structure and fail.

Let me know what the discussion yields.

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[mergify]

This pull request does not have a backport label. Could you fix it @mjmbischoff? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[botelastic]

Hi!
We just realized that we haven't looked into this PR in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it in as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

[botelastic]

Hi!
This PR has been stale for a while and we're going to close it as part of our cleanup procedure.
We appreciate your contribution and would like to apologize if we have not been able to review it, due to the current heavy load of the team.
Feel free to re-open this PR if you think it should stay open and is worth rebasing.
Thank you for your contribution!