[Segment Replication + Remote Store] SegRep with Remote Store should not have index level override and should check for the criteria below

[nandi-github]

Configuring a cluster to SegRep with Remote Store MUST be allowed only when the following criteria are met

1. Seg Rep should be set at cluster level with no option to disable at index level (when the SegRep with Remote Store is enabled)
2. Doc Rep or Seg Rep with Node-to-Node index MUST not be in the system (clean)
3. The system MUST be able to validate the item (2) and prevent enablement of Remote Store
4. Prevent switching to Remote Store in SegRep with N2N (Note to Node) and DocRep mode from a cluster in Remote Store with SegRep
5. Validate that both Remote Storage and Remote translog are enabled.

The user should be notified of the failure and a proper error message with details should be provided.

• Configuring "SegRep with Remote Store" failed for this "reason"

Additional context
These limitations/details should be documented in our docs.

Terminology details
MUST --> If the stated requirement is not met, fail.(Condition stated are mandatory)
SHOULD/SHALL --> Recommend not to be allowed as good practice but if the coder wants to proceed even after that, it is fine. Please add the justification.
MAY --> At the discretion of the implementor to provide or not to.

[anasalkouz]

Thanks @nandi-github. Here are few comments:

Configuring a cluster to SegRep with Remote Store should be allowed only when the following criteria are met

Could you elaborate more what are setting names and values that users should/allowed to use. Currently, we have setting to enable SegRep, setting to enable remote storage and a setting to enable remote translog. Shall those be combined? do the users to have flexibility to enable remote storage and remote translog separately

Configuring "SegRep with Remote Store" failed for this "reason"
what is the reason? could you provide more detailed error message.

[nandi-github]

Thanks @nandi-github. Here are few comments:

Configuring a cluster to SegRep with Remote Store should be allowed only when the following criteria are met
Good comment Anas. Let me update the requirements to reflect
MUST --> If the stated requirement is not met, fail.

SHOULD/SHALL --> Recommends not to be allowed as good practice but if the coder wants to proceed even after that, it is fine. Please add the justification.

MAY --> At the discretion of the implementor to provide or not to.

Could you elaborate more what are setting names and values that users should/allowed to use. Currently, we have setting to enable SegRep, setting to enable remote storage and a setting to enable remote translog. Shall those be combined?

Please combine them.

do the users to have flexibility to enable remote storage and remote translog separately

I am recommending to combine it in the first release. We can revisit in future release whether providing that flexibility has merits.

Configuring "SegRep with Remote Store" failed for this "reason"
what is the reason?
An error message with details of the Not permitted/Error combination will help users to fix their API call.

could you provide more detailed error message.
Example error message (Feel free to modify as long the gist of the message remains the same)
"The requested configuration with x,y and z failed as this is not a valid/permitted configuration when the current cluster state is a, b,c

[anasalkouz]

@nandi-github, Could you please update the description with the suggested details

[sachinpkale]

This is done as part of #8812 by @linuxpi