Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Partial progress on ClusterState #1514

Merged
merged 2 commits into from
May 21, 2024
Merged

Partial progress on ClusterState #1514

merged 2 commits into from
May 21, 2024

Conversation

AhmedSoliman
Copy link
Contributor

@AhmedSoliman AhmedSoliman commented May 14, 2024

Partial progress on ClusterState

ClusterState is a new data structure maintained by ClusterController. ClusterState is established by a scatter-gather request (GetProcessorsState) to all workers. For every worker, partition processor manager is the entity responsible for responding to GetProcessorsState requests, it keeps taps on running partition processors via a set of tokio watches.

This PR is a snapshot in time of progress done on this component, this is far from complete but it's best to merge the current progress until we circle back to it in a week or two.


Stack created with Sapling. Best reviewed with ReviewStack.

Copy link

github-actions bot commented May 14, 2024

Test Results

 99 files  ±0   99 suites  ±0   8m 27s ⏱️ -13s
 84 tests ±0   82 ✅ ±0  2 💤 ±0  0 ❌ ±0 
215 runs   - 1  209 ✅  - 1  6 💤 ±0  0 ❌ ±0 

Results for commit 55d7f03. ± Comparison against base commit ab3c57e.

This pull request removes 1 and adds 1 tests. Note that renamed tests count towards both.
dev.restate.e2e.JavaWorkflowAPITest ‑ Workflow cannot be submitted more than once
dev.restate.e2e.runtime.IngressTest ‑ Idempotent send then attach/getOutput

♻️ This comment has been updated with latest results.

@AhmedSoliman AhmedSoliman force-pushed the pr1514 branch 2 times, most recently from a53c19b to f31fcd9 Compare May 14, 2024 15:43
@AhmedSoliman AhmedSoliman changed the title Cluster State Partial progress on ClusterState May 14, 2024
@AhmedSoliman AhmedSoliman marked this pull request as ready for review May 14, 2024 15:47
Copy link
Contributor

@tillrohrmann tillrohrmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool stuff :-) Thanks for kicking off the work on the ClusterState @AhmedSoliman. +1 for merging.

crates/worker/src/partition_processor_manager.rs Outdated Show resolved Hide resolved
This PR includes:
- Attachment of PPM to cluster controller now use Networking
- PPM observes partition processor status through a buffered watch mechanism
- PPM can now send control messages (unused at this PR) to processor for future use.
- PPM collects state information from running processors. This will be used in a follow PR to response to controller requests about partitions.


Cluster controller grpc service is kept for external tooling integration (CLI, etc.)
ClusterState is a new data structure maintained by ClusterController. ClusterState is established by a scatter-gather request (GetProcessorsState) to all workers. For every worker, partition processor manager is the entity responsible for responding to GetProcessorsState requests, it keeps taps on running partition processors via a set of tokio watches.

This PR is a snapshot in time of progress done on this component, this is far from complete but it's best to merge the current progress until we circle back to it in a week or two.
@AhmedSoliman AhmedSoliman merged commit 9f28b58 into main May 21, 2024
10 checks passed
@AhmedSoliman AhmedSoliman deleted the pr1514 branch May 21, 2024 09:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants