Add x-opaque-id to access logs and slow logs

[tomcallahan]

Out of FixItFriday, we decided to close #19778 in favor of this issue, which is to add the x-opaque-id to our access logs and slow logs. This will permit troublesome queries from Kibana to be tracked more accurately.

[elasticmachine]

Pinging @elastic/es-core-infra

[imotov]

I implemented it for search slow logs and spent some time today trying to apply it to the index slow logs. Unfortunately, with indexing it's not that simple. The main issue here is that indexing slow log can be reached in several different ways - original indexing on primary, as a result of replication and as a result of the recovering from the translog. And because indexing doesn't use a context object, it requires a significant plumbing effort to get id delivered to the slow log and preserved during replication and translog recovery. Since the original issue was about search, I am unsure if implementing something like this would for indexing slow log would be worth the effort.

[imotov]

The opaque_id is now available in audit logging as well #31878. We should discuss if want to undertake adding it to index slow log or we can close this issue.

[imotov]

We discussed this and decided that there is really no compelling use case for logging opaque-id in the index slow log and a similar effect can be achieved by simply adding the id as a field to the indexed document. Therefore we are not planning to implement slow indexing log logging as part of this issue.