[Security Solution] Adds logging and performance fan out API for threat/Indicator matching

[FrankHassanabad]

Summary

• Adds logging output for trouble shooting
• Adds an API to be able to configure how many concurrent searches and how many items per search to use

API additions are these two switches:

concurrent_searches
items_per_search

When you create a rule. You can use the following example to post one or to change the settings to see the performance impact:

./post_rule.sh ./rules/queries/query_with_threat_mapping_perf.json

Without using these two experimental API settings, the functionality is the same as the existing algorithm and only advanced users will be able to set the additional REST settings through this API. If you use the front end after setting the settings, the settings will be reset as that's how the forms code currently works and this will not preserve the settings if afterwards a rule is edited/changed.

Both these API settings should be considered experimental and potentially breakable as we figure out the best performance strategies for indicator matching.

Checklist

• Unit or functional tests were updated or added to match the most common scenarios

[yctercero]

Should the wording here match the UI when it comes to threatIndex, threatMapping, threatQuery or at least let the user know that's what they correspond to? Or I guess that's particular to the field name right? Would this be an error the user sees in the front end ever?

[yctercero]

I take back my comment, that would add a lot of overhead any time we decide to switch what we call it in the UI.

[FrankHassanabad]

I view log files as something that is for both the users and us to look through so when UI wording changes I might not be easily able to keep files and REST up to date (or REST API would break) but for things like log file messages I try to update them to match the UI and the wording in people's heads.

I try to take extra time to make the logs more human readable for users more so than even myself even though the majority of the time they end up in the forums. I kind of would like to make them more clear.

We don't i18n them though so at the moment they're not as accessible as we would like them to be.

[yctercero]

Yea, I was just thinking about whether they'd get confused with the threat vs indicator lingo in the error message. i18n eventually would be super nice too.

[yctercero]

This looks great!! I pulled down, ran tests and played around with changing the concurrent search number. The extra logs added are super helpful.

Added some nit comments and we chatted offline about the upper bound limits of these fields that I'm super curious to find out more about. This is really cool stuff!

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 3a5c86e

Metrics [docs]

async chunks size

id	before	after	diff
securitySolution	7.8MB	7.8MB	+303.0B

distributable file count

id	before	after	diff
default	42729	42730	+1

History

• 💚 Build #86286 succeeded 068f6f2
• 💚 Build #86021 succeeded e7c1423
• 💔 Build #85947 failed 013d5bf
• 💔 Build #85938 failed 733d9c6
• 💚 Build #85719 succeeded 901a1ce

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream