[Segment Replication] Revisit network timeout settings to avoid timeout exceptions

[dreamer-89]

Coming from discussion, as part of peer recovery a round of segment replication is performed to ensure the target is upto date with primary. For long running recovery process, it may be possible that segment replication request may time out. This is applicable for both primary and replica recovery process.

Related

#5242
#5313

[mch2]

I think all settings in RecoverySettings should be looked at, but a few other node level settings in particular where I think Segrep should have its own config & defaults:

INDICES_RECOVERY_MAX_BYTES_PER_SEC_SETTING - current default is 40MB/s
INDICES_RECOVERY_MAX_CONCURRENT_FILE_CHUNKS_SETTING - current default is 2.

We currently reuse these from recovery given the recovery code to send files was refactored to be reusable for segrep.

[dreamer-89]

Segment Replication uses the recovery settings for its operations. It is difficult to segregate these settings from Recovery because existing recovery settings are defined at NodeScope level and some are used to ensure restricted usage of Node's resources. E.g. max_bytes_per_sec (INDICES_RECOVERY_MAX_BYTES_PER_SEC_SETTING) is shared b/w recovery and segment replication (default 40MB/s) via a RateLimiter object. I think adding a similar new setting for segment replication will be problematic as:

1. Combined node level max_bytes_per_sec value will be higher than 40MB/s. It is also true the other way round where both continue to use the same setting and complete 40MB/s wouldn't be reserved for recovery alone. This needs to be called out in documentation issue [DOC] Segment Replication - GA release documentation-website#2255
2. It is confusing to have separate settings controlling same underlying resource (Node here).

The other settings can be refactored into a separate SegmentReplicationSettings class but due to Recovery & SegRep both using common classes & pattern, using single settings makes sense.

Previous Issue opened for timeout: #4392

[dreamer-89]

Addressed concern of using hard-coded constant for identifying the request timeout for fetching segment files in #6523.

This issue was opened to ensure segrep timeouts doesn't result in persistent recovery failures. Setting the max time for fetching files should address the timeout concern.

[dreamer-89]

Closing this issue as known timeout related issue is addressed in #6523
We can come back to this issue if there are any issues identified during benchmarking to be done post feature complete, tracked in #5147