[APM] Inject agent config directly into APM Fleet policies

[axw]

As part of the move to Fleet we will need to move away from fetching agent config directly from Kibana, as the privileges APM Server is given do not cover this. Instead, agent config will be pushed down to APM Server via the server's policy.

When agent config is created/updated/removed in Kibana, APM Fleet policies should be updated to include the new agent config directly. For each config block, Kibana will need to supply: the criteria (service name and/or service environment), settings, and an Etag value.

As APM Server will not be communicating directly with Kibana, we will have to come up with a new way of identifying that config has been applied. I can think of two main options, as described in elastic/apm-server#5018:

1. APM Server will index a document whenever a config block is first known to have been applied to an agent. This is similar to what we do today.
2. Agents will periodically send an event to APM Server including agent statistics and the currently applied config Etag, and this will be indexed by the server. This would have the benefit of enabling us to see how many and which agents have applied the config.

Proposed user flow:

1. When superuser visits agent configuration settings, get the list of agent policies -> check each for the apm integration package policy
2. If an APM fleet integration exists, then display a checkbox in agent configuration settings to synchronize settings with fleet management
3. When checked, agent configs will be copied to policy and can only be modified by the super user (other users will see a ready-only page)

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[simitt]

Adding the reference to @axw 's POC here.

[simitt]

Every user with APM app privileges can configure agent central configuration in the APM app. The Fleet app requires superuser privileges for every operation, therefore also for pushing down configuration changes to any APM integrations policy.

Two ways of solving this come to mind:

• Check a users privileges and show a warning in the UI that their changes will only be passed to the APM integrations whenever a superuser logs into Kibana and navigates to the APM config management settings page. OR
• require superuser privileges for changing APM config management settings as soon as at least one Fleet policy with an APM integration is dedected. This is the least favorable, as it would break existing behavior if someone plays around with adding an APM integration.

Potential flow:

• A non-superuser creates/deletes/updates an apm agent central configuration setting. On save the APM app shows a warning that the changes will only be applied to APM integration policies (if any exist) whenever a superuser next navigates to the apm agent central configuration page. There is no way to only show this if an apm integration exists, as for querying this information superuser privileges are required.
• A superuser navigates to the apm agent central configuration UI OR creates/deletes/updates an apm agent central configuration setting. The APM app fetches all existing agent policies that contain an APM integration and updates the complete apm-server.agent_config.* configuration section. The complete section needs to be updated, to also contain the changes potentially made earlier by non-superusers.

Generally, the current apm agent central configuration logic needs to keep being supported as long as running an APM server standalone is supported.

[sorenlouv]

Potential flow:

The outlined flow doesn't communicate to the user whether a change has been sync'd to the fleet policy or not which I think is a big drawback.

@nehaduggal mentioned that it was acceptable to not have agent config support in 7.13. From my understanding fleet is working on a ui for agent configs that we can use in 7.14, so perhaps it's better to defer this effort until then?

[simitt]

@sqren brought up the idea of having a check box that can be activated to also apply agent central config options fo apm server integrations. The check box could only be enabled/disabled by superusers. When enabled, the UI is disabled for non-superusers. Changes could always immediately be applied to all apm integrations as it is ensured that the user triggering the changes has enough privileges. To keep the effort simple, the check box would be shown in the APM app.
This is certainly not ideal as it requires manual interaction, but would allow us to move forward with the central config changes.

[formgeist]

The Fleet app requires superuser privileges for every operation, therefore also for pushing down configuration changes to any APM integrations policy

I might have missed this, but is this just an accepted limitation for now until we expand the role permissions into Fleet, or will it always be the case that only the superuser can apply config changes to the agents via the policy management? If that's the case, I think we have to consider what user experience we're imagining longer term to mitigate this limitation for APM users who AFAIK wants the ability to change an agent config through the APM app not only pertaining to a superuser.

[sorenlouv]

is this just an accepted limitation for now until we expand the role permissions into Fleet, or will it always be the case that only the superuser can apply config changes to the agents via the policy management

The former. It is a (painful) limitation that we have to deal with for now but not permanently. It might make sense to talk to the fleet team about when they're planning to add more granular permissions.