Add global checkpoint polling to cross-cluster replication

[jasontedor]

Today cross-cluster replication works by submitting read requests to the leader shard. When there are no operations in the response, we schedule a follow-up and repeatedly poll on a scheduled delay until we receive operations in the response. Our goal is add an endpoint for a remote cluster to listen for changes in the global checkpoint on the leader. To do this, we will make the following changes:

• the replication tracker should make visible when the global checkpoint is updated Expose whether or not the global checkpoint updated #32659
• index shard should be modified for global checkpoint listeners to be registered Introduce global checkpoint listeners #32696
• shard follow tasks should be updated to register a listener on a remote cluster to go into a long poll mode Introduce long polling for changes #33683

We will make the first of these two changes in master/6.x, and the third change will be made on the ccr feature branch.

Relates #30086

[elasticmachine]

Pinging @elastic/es-distributed

[bleskes]

@jasontedor my original thinking was to build this into the read changes requests, looking forward to a changes API. Reading this description, it seems your thinking is to have a dedicated action. Can you clarify?

[jasontedor]

@bleskes Yes, I was thinking a separate action. Are you thinking another flag on the shard changes request that designates the request can go into a poll mode?

[bleskes]

@jasontedor My not-so-well-thought-out idea was to initially go into "wait mode" iff the local shard can't serve any ops for the incoming request. This amounts to the fromSeqNo be equal to the global checkpoint + 1. Later we can introduces better flags (which imo will be more relevant for a changes api) like "wait for at least x ops to arrive but no longer than x time".

[jasontedor]

I will need to look into this but this concerns me because I think that it means we can end up with multiple outstanding polls on the follower side and I think that we should avoid that (mostly for simplicity, the callbacks are not particularly expensive to hold on to for the life of a poll).