This repository has been archived by the owner on Apr 26, 2024. It is now read-only.
-
-
Notifications
You must be signed in to change notification settings - Fork 2.1k
Additional functionality for declaring worker types in Complement #14921
Merged
reivilibre
merged 33 commits into
matrix-org:develop
from
realtyem:comp-worker-shorthand
Mar 14, 2023
Merged
Additional functionality for declaring worker types in Complement #14921
reivilibre
merged 33 commits into
matrix-org:develop
from
realtyem:comp-worker-shorthand
Mar 14, 2023
Commits on Jan 23, 2023
-
Allow for a multipler to be added to a requested worker.
This allows for appending a ':' and a positive integer to a requested worker type for the SYNAPSE_WORKER_TYPES environment variable(and likewise, the WORKER_TYPES) used in Complement. For example: 'event_persister:2' would be yield identical results of 'event_persister, event_persister'
Configuration menu - View commit details
-
Copy full SHA for 4647d59 - Browse repository at this point
Copy the full SHA 4647d59View commit details
Commits on Jan 25, 2023
-
Allow for worker types to be merged and given a name.
In the Synapse-worker docker image, multiple worker types are defined but only allowed to be that one single definition. This makes it so that any two(or more) worker types can be merged into a single worker. For example: 1. If you wish to have one worker with both client_reader and federation_sender functions enabled, you can set SYNAPSE_WORKER_TYPES='client_reader+federation_sender' 2. If you wish to have all stream_writers(excepting event_persister) you can set SYNAPSE_WORKER_TYPES='account_data+presence+receipts+to_device+typing' Multiple types can be combined, but some error checking to dis-allow multiples of worker types that shouldn't be enabled more than once has been added. For example: SYNAPSE_WORKER_TYPES='background_worker+event_persister, background_worker+event_persister' will not work as background_worker is only allowed to have a single worker for the entire deployment. Giving a worker or a combination of workers a custom name is as simple as adding the name then an equal sign in front of the given worker type(s). For example: SYNAPSE_WORKER_TYPES='alice=federation_reader' or SYNAPSE_WORKER_TYPES='bob=federation_inbound+federation_sender' or SYNAPSE_WORKER_TYPES='charlie=event_persister:2, derek=media_repository + pusher + user_dir + appservice + event_creator'
Configuration menu - View commit details
-
Copy full SHA for e29ef6f - Browse repository at this point
Copy the full SHA e29ef6fView commit details -
Install examples in start_for_complement.sh.
Instead of adding stream writer workers individually, utilize new functionality to make one worker with all included. Also utilitize new shorthand for event_persister's.
Configuration menu - View commit details
-
Copy full SHA for 8538cb6 - Browse repository at this point
Copy the full SHA 8538cb6View commit details -
Configuration menu - View commit details
-
Copy full SHA for 02178ce - Browse repository at this point
Copy the full SHA 02178ceView commit details
Commits on Jan 26, 2023
-
Configuration menu - View commit details
-
Copy full SHA for bd6b73e - Browse repository at this point
Copy the full SHA bd6b73eView commit details -
Configuration menu - View commit details
-
Copy full SHA for c858d45 - Browse repository at this point
Copy the full SHA c858d45View commit details -
Configuration menu - View commit details
-
Copy full SHA for e00989f - Browse repository at this point
Copy the full SHA e00989fView commit details
Commits on Feb 5, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 497f01d - Browse repository at this point
Copy the full SHA 497f01dView commit details
Commits on Feb 14, 2023
-
Configuration menu - View commit details
-
Copy full SHA for beb2531 - Browse repository at this point
Copy the full SHA beb2531View commit details
Commits on Feb 19, 2023
-
1. Turn 'placeholder_name' into a magic constant. 2. Fix casting on a worker_type_list from 'list' to 'List[str]'. 3. In 'insert_worker_name_for_shared_config', use new WORKER_PLACEHOLDER_NAME constant and reduce complexity by using value from items() directly. 4. In 'is_sharding_allowed_for_worker_type', simplify the return value. 5. Adjust 'split_string_and_strip' to allow for using the 'maxsplit' kwarg on 'strip' and update comment. 6. Remove function 'increment_counter' and use 'defaultdict' instead to reduce complexity. 7. Remove function 'is_name_allowed_for_worker', as this can be accomplished with an 'if' statement and only checked for a worker base name in a dict. Adjust 'generate_worker_files' to accommadate. 8. Rename a few dict's to remove misnomer's and adjust comments for extra clarity. 9. In 'generate_worker_files': a. Use 'ValueError' exception handling to catch additional errors when parsing an integer and simplify counting of appended worker_type's. b. Add additional error cases to check(single and double quotes, spaces, and last character being an integer) for potential file name/nginx issues. c. Adjust various variable names to improve readability. 10. Update various comments throughout.
Configuration menu - View commit details
-
Copy full SHA for 8d1b37a - Browse repository at this point
Copy the full SHA 8d1b37aView commit details -
Configuration menu - View commit details
-
Copy full SHA for f964a1a - Browse repository at this point
Copy the full SHA f964a1aView commit details -
Configuration menu - View commit details
-
Copy full SHA for aba5585 - Browse repository at this point
Copy the full SHA aba5585View commit details
Commits on Feb 24, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 3a7df06 - Browse repository at this point
Copy the full SHA 3a7df06View commit details
Commits on Mar 3, 2023
-
1. Factor processing of all worker types from the environment out and away from generating config files. 2. Fix up early templating of 'worker_config' to remove boilerplate. 3. Wrangle nginx upstream processing to accommodate odd combinations for overlapping workers. e.g. 'user_dir, user_dir+presence' should be able to handle the(in this case single) endpoint over either worker. I think this was promised in a previous commit, consider it delivered. 4. Update a bunch of comments, and adjust some pre-existing to fit inside the line length count of 88(aka, make the green squiggles go away) 5. Move processing of unusable characters for worker names to a later position so it will check the name produced if one was not requested as well.
Configuration menu - View commit details
-
Copy full SHA for 47a965f - Browse repository at this point
Copy the full SHA 47a965fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 5dc3318 - Browse repository at this point
Copy the full SHA 5dc3318View commit details
Commits on Mar 5, 2023
-
Configuration menu - View commit details
-
Copy full SHA for dd9773c - Browse repository at this point
Copy the full SHA dd9773cView commit details -
Disambiguate a confusing mess of 'worker_roles', 'workers_roles', 'wo…
…rker_type', and similar and that should have been 'worker_types' when in use by sets and a couple other places.
Configuration menu - View commit details
-
Copy full SHA for 4442c0a - Browse repository at this point
Copy the full SHA 4442c0aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 98da503 - Browse repository at this point
Copy the full SHA 98da503View commit details
Commits on Mar 8, 2023
-
Update comments and a few other texty bits:
1. Update comments and edit(or create) docstrings and a single type hint that was incomplete(split_and_strip_string()). 2. Rename parse_worker_types_from_env to parse_worker_types as it's not coming from the environment.
Configuration menu - View commit details
-
Copy full SHA for b8741d7 - Browse repository at this point
Copy the full SHA b8741d7View commit details -
Update how worker_type specific options are merged into shared_config…
…, and update comments to match.
Configuration menu - View commit details
-
Copy full SHA for 204f61e - Browse repository at this point
Copy the full SHA 204f61eView commit details -
Remove condition based on number of ports
...so that even with a single port, a worker gets it's own upstream. Nginx says this can lead to performance improvements if some features are enabled. Revisit this later.
Configuration menu - View commit details
-
Copy full SHA for 7c32bdb - Browse repository at this point
Copy the full SHA 7c32bdbView commit details -
Configuration menu - View commit details
-
Copy full SHA for 0cf5e80 - Browse repository at this point
Copy the full SHA 0cf5e80View commit details
Commits on Mar 9, 2023
-
Try and apply review comments about sanitizing names.
1. Update a few comments for clarity. 2. Apply a replace() to remove whitespace in a requested worker name and use an underscore instead. 3. Stop-error if requesting a worker name, but forgetting to actually put one in. 4. If a worker name isn't requested, just use the set of worker types, sorted and concatenated with a hyphen. This avoids having to sanitize the worker name in this case. 5. Add regex to avoid nasty name surprises.
Configuration menu - View commit details
-
Copy full SHA for e2fecab - Browse repository at this point
Copy the full SHA e2fecabView commit details -
Configuration menu - View commit details
-
Copy full SHA for 97a0af0 - Browse repository at this point
Copy the full SHA 97a0af0View commit details -
Fix log message and the condition it fires on to reflect it only happ…
…ens if more than one worker type for a given worker is requested.
Configuration menu - View commit details
-
Copy full SHA for 9f3f9ab - Browse repository at this point
Copy the full SHA 9f3f9abView commit details -
Configuration menu - View commit details
-
Copy full SHA for 65de783 - Browse repository at this point
Copy the full SHA 65de783View commit details -
Configuration menu - View commit details
-
Copy full SHA for aeefc5d - Browse repository at this point
Copy the full SHA aeefc5dView commit details -
Configuration menu - View commit details
-
Copy full SHA for b5a66b4 - Browse repository at this point
Copy the full SHA b5a66b4View commit details
Commits on Mar 10, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 54c23c9 - Browse repository at this point
Copy the full SHA 54c23c9View commit details -
Configuration menu - View commit details
-
Copy full SHA for 32c04bb - Browse repository at this point
Copy the full SHA 32c04bbView commit details -
Tweak regex to not allow final character to be a number, and adjust w…
…hitespace on error message.
Configuration menu - View commit details
-
Copy full SHA for 78a43c8 - Browse repository at this point
Copy the full SHA 78a43c8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8ff80d7 - Browse repository at this point
Copy the full SHA 8ff80d7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6b90ccb - Browse repository at this point
Copy the full SHA 6b90ccbView commit details
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.