[Metricbeat] Make integration tests more generic for modules

[ruflin]

The goal is that each module on its own has less and less code and is more config based. The basic fetch testing we have for each module is moved into a general integration testing and it uses the configuration to figure out which services to start, ports to talk to etc.

This is still WIP and needs quite a bit of cleanup.

[ruflin]

I would like to get #7957 first merged before moving forward with this as I think it will look even simpler with it (and otherwise conflicts with it).

[fearful-symmetry]

I really like this idea.

[ruflin]

@jsoriano As discussed, assigning to you.

[ruflin]

@jsoriano It would be really nice to have this "unified" but as it went stale for quite some time I suggest I close it for now. Please reopen if you still plan to work on it.

[jsoriano]

This (or something similar) was implemented in #11981.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b general-integration-tests upstream/general-integration-tests
git merge upstream/master
git push upstream general-integration-tests

[mergify]

This pull request does not have a backport label. Could you fix it @ruflin? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.