feat: AEP 6: Sharding and refactoring of MongoDB deployment inside ArmoniK #48
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
ngruelaneo
reviewed
Jul 3, 2024
ngruelaneo
reviewed
Jul 3, 2024
ngruelaneo
reviewed
Jul 3, 2024
ngruelaneo
reviewed
Jul 4, 2024
tschneider-aneo
commented
Jul 12, 2024
ngruelaneo
reviewed
Jul 12, 2024
ngruelaneo
reviewed
Sep 26, 2024
ngruelaneo
reviewed
Sep 27, 2024
This was referenced Sep 30, 2024
ngruelaneo
reviewed
Sep 30, 2024
AEP/aep-00004.md
Outdated
| - Since the server is now preferably accessed with SRV strings, the port output is still present for retrocompatibility concerns but hardcoded with a value of 27017 (MongoDB default port). | ||
|
|
||
| **What would be specific to the sharded MongoDB Helm chart ?** | ||
| - It would be possible to deploy an out-of-the-box sharded MongoDB instance, with a configurable, but rather static, number of shards. We are currently working on a way to implement autoscaling. The number of config servers replicas and mongos "replicas" would also be configurable. |
There was a problem hiding this comment.
je ne sais pas si cela devrait etre dans cette AEP. Cela me semble out of scope (un nouveau sujet)
There was a problem hiding this comment.
Je pense que l'autoscaling (horizontal) est une conséquence plus que logique au sharding de base de données, donc le mentionner me parait pertinent
aneojgurhem
added a commit
to aneoconsulting/ArmoniK.Core
that referenced
this pull request
Oct 1, 2024
# Motivation Following aneoconsulting/ArmoniK.Infra#168, this PR aims to enable the deployment of a sharded MongoDB as ArmoniK's database. A more complete motivation is provided in AEP 004 (aneoconsulting/ArmoniK.Community#48). This PR is meant to treat a previsible bottleneck on ArmoniK's database by enabling a sharded architecture to it. For now, when a sharded database is specified, the following collections will be sharded : - Result - TaskData - SessionData # Description This PR adds : - A method `ShardCollectionAsync` to the `IMongoDataModelMapping` interface. - 2 new `MongoOptions` : a string `AuthSource` and a boolean `Sharding`. - A new class `ShardingExt` containing an extension method `shardCollection` for the `IClientSessionHandle` interface. When the `MongoOption` `Sharding` is true, the `MongoCollectionProvider` calls `ShardCollectionAsync`. Then the implementation depends on whether the collection is wanted to be sharded : - If the collection has to be sharded, the `shardCollection` extension method will be called. - If the collection isn't wanted to be sharded, the method directly returns a complete `Task`. The new `MongoOption` `AuthSource` is required because the `MongoClient` has to authenticate as an administrator to be able to shard a collection. # Testing It has been tested manually. Since these developmentsts are very coupled with ArmoniK's database, it is difficult to write unit tests. Anyhow, it is still possible to have automatized integration tests that would verify in a first time if a deployment of ArmoniK using these deployments succeeds, and in a second time if sharding is indeed enabled on the right collections. Workflows testing a deployment of ArmoniK with a sharded MongoDB are currently being studied. # Impact In the end, it is expected to enhance ArmoniK's strong scalability. # Additional Information To better understand how MongoDB sharding works see [MongoDB's documentation](https://www.mongodb.com/docs/manual/sharding/). To understand why this new MongoOption is required, see : MongoDB's documentation on the [shardCollection database command](https://www.mongodb.com/docs/manual/reference/command/shardCollection/#mongodb-dbcommand-dbcmd.shardCollection) and on [connection string's authSource option](https://www.mongodb.com/docs/manual/reference/connection-string-options/#mongodb-urioption-urioption.authSource) # Checklist - [X] My code adheres to the coding and style guidelines of the project. - [X] I have performed a self-review of my code. - [X] I have commented my code, particularly in hard-to-understand areas. - [ ] I have made corresponding changes to the documentation. - [X] I have thoroughly tested my modifications and added tests when necessary. - [X] Tests pass locally and in the CI. - [ ] I have assessed the performance impact of my modifications.
tschneider-aneo
changed the title
tschneider-aneo
changed the title
tschneider-aneo
changed the title
tschneider-aneo
changed the title
|
You need to rename the file to aep5 |
It is indeed going to be deleted ! I left the file named aep4 to keep the conversations on it ! |
tschneider-aneo
changed the title
tschneider-aneo
changed the title
ngruelaneo
previously approved these changes
Oct 2, 2024
ngruelaneo
previously approved these changes
Oct 3, 2024
|
I would use svg images for the illustrations. This would avoid storing binary files within the repo |
We have a public bucket to store this kind of image. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.