[Structured Streaming] Exactly-once guarantee with ILM/Rollover

[danielyahn]

What kind an issue is this?

• Feature Request.

Feature description

When using ILM (index lifecycle management) and its roll over API implementation, your ingest job needs to point at the write alias. ILM, especially with its ability to roll over by size, has benefits on operation.

However, it's not possible to have exactly-once guarantee when using write alias. Current checkpoint implementation for ES Sink doesn't capture the actual index that write alias points to.

Therefore, when if you're replaying some batches (whether one batch that failed halfway or multiple batches for any operational reason), you can't guarantee that your records are going to same index.

[jbaiera]

I'm marking this as discuss for the team since this is a general problem across multiple streaming write workloads that are using ILM for managing indices.

[jakelandis]

Related: elastic/elasticsearch#44794