[Ingest Manager] Agent unenroll

[michalpristas]

What does this PR do?

This PR works with draft from here: elastic/kibana#70031
Accepts UNENROLL actions and stop all the processes it has started.

Why is it important?

Removing endpoint on unenroll.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

[elasticmachine]

💚 Build Succeeded

Expand to view the summary

Build stats

• Build Cause: [Pull request #19507 updated]

• Start Time: 2020-07-07T10:23:34.460+0000

• Duration: 31 min 39 sec

[michalpristas]

Tested with latest kibana branch unpdates, agent ACKs UNENROLL and disapears from the list as expected

[blakerouse]

Overall the code looks good. Seems you still need to add more to clean the action store, assume that is coming in a later branch.

I have more of a question on what happens to Agent after the completion of unenroll. Will it still be able to communicate with the Fleet gateway? I assume that the API keys will be invalidated after ack of unenroll, so then communication will be broken.

If that is the case, should Agent set a configuration setting into the Fleet config so it knows that it is unenrolled? We don't want Fleet to go to standalone mode (that would be weird/wrong). We also don't want it to keep reaching out to Fleet and failing (that would add load both the the client and server, for no benefit). We also don't want to just stop, as the service manager of the system will just keep restarting it.

If I understand unenroll correctly we need some kinda of idle state, that just logs Agent is unenrolled and it just really does nothing (but stays running).

[michalpristas]

@blakerouse I was thinking we will use the same mechanism needed for reexec, cancelling everything down and stopping all the loops/processes. for now cancellation is not that good. but it is a good point. in the meantime i will just stop fleet gateway

[michalpristas]

Reworked it a bit going to idle state on restart after being unenrolled.