Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Backport 2.x] [Remote Store] Backport remote segment store changes #4380

Merged

Conversation

sachinpkale
Copy link
Member

Description

Backport remote segment store changes to 2.x

Original PRs

  • Use RemoteSegmentStoreDirectory instead of RemoteDirectory (#4240)
  • [Remote Store] Add RemoteSegmentStoreDirectory to interact with remote segment store (#4020)
  • [Remote Store] Change remote_store setting validation message to make it more clear (#4199)
  • [Remote Store] Add validator that forces segment replication type before enabling remote store (#4175)
  • [Remote Store] Add rest endpoint for remote store restore (#3576)
  • [Remote Store] Add support to add nested settings for remote store (#4060)
  • [Remote Store] Add remote store restore API implementation (#3642)
  • [Remote Store] Inject remote store in IndexShard instead of RemoteStoreRefreshListener (#3703)
  • [Remote Store] Upload segments to remote store post refresh (#3460)

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@sachinpkale sachinpkale requested review from a team and reta as code owners September 1, 2022 16:39
@github-actions
Copy link
Contributor

github-actions bot commented Sep 1, 2022

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Sep 2, 2022

Gradle Check (Jenkins) Run Completed with:

sachinpkale and others added 9 commits September 2, 2022 09:53
…ch-project#3460)

* Add RemoteDirectory interface to copy segment files to/from remote store

Signed-off-by: Sachin Kale <[email protected]>

Co-authored-by: Sachin Kale <[email protected]>

* Add index level setting for remote store

Signed-off-by: Sachin Kale <[email protected]>

Co-authored-by: Sachin Kale <[email protected]>

* Add RemoteDirectoryFactory and use RemoteDirectory instance in RefreshListener

Co-authored-by: Sachin Kale <[email protected]>
Signed-off-by: Sachin Kale <[email protected]>

* Upload segment to remote store post refresh

Signed-off-by: Sachin Kale <[email protected]>

Co-authored-by: Sachin Kale <[email protected]>
Signed-off-by: Sachin Kale <[email protected]>
…reRefreshListener (opensearch-project#3703)

* Inject remote store in IndexShard instead of RemoteStoreRefreshListener

Signed-off-by: Sachin Kale <[email protected]>

* Pass supplier of RepositoriesService to RemoteDirectoryFactory

Signed-off-by: Sachin Kale <[email protected]>

* Create isRemoteStoreEnabled function for IndexShard

Signed-off-by: Sachin Kale <[email protected]>

* Explicitly close remoteStore on indexShard close

Signed-off-by: Sachin Kale <[email protected]>

* Change RemoteDirectory.close to a no-op

Signed-off-by: Sachin Kale <[email protected]>

Co-authored-by: Sachin Kale <[email protected]>
…h-project#3642)

* Add remote restore API implementation

Signed-off-by: Sachin Kale <[email protected]>
…pensearch-project#4060)

* Add support to add nested settings for remote store

Signed-off-by: Sachin Kale <[email protected]>
…-project#3576)

* Add rest endpoint for remote store restore

Signed-off-by: Sachin Kale <[email protected]>
…ore enabling remote store (opensearch-project#4175)

* Add validator that forces segment replication type before enabling remote store

Signed-off-by: Sachin Kale <[email protected]>
… it more clear (opensearch-project#4199)

* Change remote_store setting validation message to make it more clear

Signed-off-by: Sachin Kale <[email protected]>
…e segment store (opensearch-project#4020)

* Add RemoteSegmentStoreDirectory to interact with remote segment store

Signed-off-by: Sachin Kale <[email protected]>
…h-project#4240)

* Use RemoteSegmentStoreDirectory instead of RemoteDirectory

Signed-off-by: Sachin Kale <[email protected]>
@github-actions
Copy link
Contributor

github-actions bot commented Sep 2, 2022

Gradle Check (Jenkins) Run Completed with:

@codecov-commenter
Copy link

Codecov Report

Merging #4380 (e5cb89f) into 2.x (ea5b946) will increase coverage by 0.17%.
The diff coverage is 72.86%.

@@             Coverage Diff              @@
##                2.x    #4380      +/-   ##
============================================
+ Coverage     70.56%   70.73%   +0.17%     
- Complexity    56923    57143     +220     
============================================
  Files          4572     4583      +11     
  Lines        273817   274413     +596     
  Branches      40152    40216      +64     
============================================
+ Hits         193207   194102     +895     
+ Misses        64450    64059     -391     
- Partials      16160    16252      +92     
Impacted Files Coverage Δ
.../org/opensearch/client/support/AbstractClient.java 34.96% <0.00%> (+0.07%) ⬆️
...pensearch/common/settings/IndexScopedSettings.java 100.00% <ø> (ø)
.../java/org/opensearch/common/util/FeatureFlags.java 50.00% <ø> (ø)
...rc/main/java/org/opensearch/index/IndexModule.java 82.03% <ø> (ø)
.../java/org/opensearch/plugins/IndexStorePlugin.java 100.00% <ø> (ø)
.../java/org/opensearch/snapshots/RestoreService.java 57.19% <0.00%> (-3.13%) ⬇️
...opensearch/cluster/routing/ShardRoutingHelper.java 95.65% <ø> (ø)
...ore/restore/TransportRestoreRemoteStoreAction.java 25.00% <25.00%> (ø)
...c/main/java/org/opensearch/index/IndexService.java 74.03% <28.57%> (-1.03%) ⬇️
...on/admin/cluster/RestRestoreRemoteStoreAction.java 33.33% <33.33%> (ø)
... and 518 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@Bukhtawar Bukhtawar merged commit 03273b5 into opensearch-project:2.x Sep 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants