[META] Insufficient guardrails leading to disk going full on nodes

[RS146BIJAY]

Is your feature request related to a problem? Please describe.
Currently, we are observing multiple instances where data volume is getting 100% filled up on one or more node. We have guardrails like flood stage watermark in place which ensures that OpenSearch put blocks at the right time and enough amount of space is available for OpenSearch to perform internal operations (like segment merge, cluster state update etc.). Still, we sometime observe that available space on few or all data nodes of a domain goes to 0. This can cause node from getting removed from cluster (by either FSHealthService checks or due to cluster state update) which may ultimately result in red clusters (if it contains active primary).

Describe the solution you'd like
OpenSearch should ensure that guardrails like FloodStage watermarks are applied correctly and enough amount of space is available for OpenSearch to perform internal operations (like segment merge, cluster state update etc.).

OpenSearch Subtasks

• [BUG] OpenSearch should clean up temporary files created during segment merge incase segment merge fails #5710
• [BUG] OpenSearchMergePolicy should not allow segment merge if enough amount of space is not available for the segment merge to go through #5711
• [BUG] DiskThresholdDecider should consider shard size of all relocating shards on the target node while making relocation decisions #5386
• Block new index creation #4603
• Apply index create block when all the nodes of the cluster has breached high disk watermark #4456
• DiskThresholdDeciderIT.testIndexCreateBlockWithAReadOnlyBlock is flaky #5956
• Add a guardrail to limit maximum number of shard on the cluster #6050
• [BUG] Add a setting to control auto release of OpenSearch Monitors managed block #6269