[Fleet] Handle initialization timeout better

[mostlyjason]

I just launched a new deployment on staging with 7.10 and I got this error:

Unable to initialize Fleet
[process_cluster_event_timeout_exception] failed to process cluster event (put-pipeline-logs-endpoint.events.library-0.16.0) within 30s response from /_ingest/pipeline/logs-endpoint.events.library-0.16.0: {"error":{"root_cause":[{"type":"process_cluster_event_timeout_exception","reason":"failed to process cluster event (put-pipeline-logs-endpoint.events.library-0.16.0) within 30s"}],"type":"process_cluster_event_timeout_exception","reason":"failed to process cluster event (put-pipeline-logs-endpoint.events.library-0.16.0) within 30s"},"status":503}

It worked when I refreshed the page a few minutes later, indicating it was a timeout to a background process that resolved on its own.

This looks like bad error because it happens during the getting started flow, its ugly and doesn't tell the user what to do to overcome the problem. It'd be better if we could extend the timeout to make it less likely users will encounter it. Alternatively, we could update our error message for 503s to say "please try again in a few minutes".

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[Aqualie]

Facing this issue after upgrading from 7.11.2 to 7.12.0:

Unable to initialize Fleet
[process_cluster_event_timeout_exception] failed to process cluster event (put-pipeline-logs-system.system-0.10.9) within 30s response from /_ingest/pipeline/logs-system.system-0.10.9: {"error":{"root_cause":[{"type":"process_cluster_event_timeout_exception","reason":"failed to process cluster event (put-pipeline-logs-system.system-0.10.9) within 30s"}],"type":"process_cluster_event_timeout_exception","reason":"failed to process cluster event (put-pipeline-logs-system.system-0.10.9) within 30s"},"status":503}

Does not work when refreshing the page minutes or hours later.

[jen-huang]

This might be alleviated with the recent changes we made to not have /setup errors block the UI (#97404). We need to double check.

[jlind23]

Closing as not relevant anymore