[Remote Segment Store] Add another instance of Store to InternalEngine that uses RemoteDirectory

[sachinpkale]

Is your feature request related to a problem? Please describe.

• The RemoteDirectory interface is defined as part of task.
• This task creates an instance of Store, named remoteStore, which uses RemoteDirectory.
• remoteStore is injected into InternalEngine. Goal is to use remoteStore to upload newly created segments as part of refresh flow.

[sachinpkale]

As I think more on this, there are 2 ways to access RemoteDirectory in InternalEngine:

• Inject another instance of Store, called remoteStore into InternalEngine. This remoteStore will have instance of RemoteDirectory. With this approach, we have clear separation of store with remoteStore and don't have to change implementation of Store as the methods in Store are built against the Directory instance that the Store encapsulates.
• Store to have instance of RemoteDirectory as well.
  • In this approach, instance of Store is created with 2 directories: first directoy is the same way it is created today, second instance would be instance of RemoteDirectory.
  • So, the concept is: Store will have 2 directories: primaryDirectory and secondaryDirectory. As both are directory instances, Store will not assume any implementation details of these directories. So, both the directories could be:
    • instance of FSDirectory,
    • instance of RemoteDirectory
    • primaryDirectory is instance of RemoteDirectory and secondaryDirectory is instance of FSDirectory
  • All the methods of store will continue to work on primaryDirectory and will have corresponding overloaded method which will take a boolean to indicate if to use primaryDirectory or secondaryDirectory.
  • In V1 of remote store:
    • we will add secondaryDirectroy to the store
    • primaryDirectory will still be called directory (we may take a call to change the name to primaryDirectory if required)
    • Store methods will only support primaryDirectory (methods will not be overloaded in V1)
    • Store will add a method to return an instance of secondaryDirectory (similar to directory() method).

With these two approaches, we are basically discussing the place to handle the RemoteDirectory instance. Whether it should be 1. in InternalEngine, as a part of RemoteStore or 2. in Store, as a secondaryDirectory.

[sachinpkale]

PR link: #3285