-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Grpc tcp connection limiter #3
base: master
Are you sure you want to change the base?
Commits on Oct 29, 2023
-
Configuration menu - View commit details
-
Copy full SHA for be149b2 - Browse repository at this point
Copy the full SHA be149b2View commit details -
Add TCP connection limiter service to utils library
Implemented a new LimiterService within the utils library to control the maximum number of TCP connections for a service. This feature was created to prevent services, such as gRPC, JSON RPC, REST, etc., from exceeding their maximum allowed active TCP connections. This enhancement will avoid server overload and ensure more stable and reliable service performance. Other additions include appropriate dependencies in the Cargo.toml file and necessary modifications to the Cargo.lock file to support this implementation.
Configuration menu - View commit details
-
Copy full SHA for 33947b2 - Browse repository at this point
Copy the full SHA 33947b2View commit details -
Implement TCP connection limit on gRPC server
Added optional TCP connection limit functionality to the gRPC server Adaptor, ConnectionHandler, and GrpcService classes. This change allows limiting the number of active TCP connections to the gRPC server. If set, the server will refuse new connections once the limit is reached. This is intended to prevent server overload from too many concurrent connections. Implemented by using a TCP limiter method from the kaspa_utils library and integrating it with the gRPC server setup. This feature is optional and can be enabled by passing the limiter object during server creation. If not passed, the server will accept an unlimited number of connections.
Configuration menu - View commit details
-
Copy full SHA for bffb7ac - Browse repository at this point
Copy the full SHA bffb7acView commit details -
Implement TCP connection limit on gRPC server
Introduced an optional TCP connection limit to the gRPC server in Adaptor, ConnectionHandler, and GrpcService. This change allows the server to manage the number of active TCP connections, enhancing its ability to prevent server overload due to excess concurrent connections. Implemented via the TCP limiter method from the kaspa_utils library and integrated into the gRPC server setup. This functionality can be enabled during server creation via passing a limiter object. If not provided, the server will operate with an unlimited number of connections.
Configuration menu - View commit details
-
Copy full SHA for b38b4f1 - Browse repository at this point
Copy the full SHA b38b4f1View commit details -
Refactor import statements and add http dependency
The import statements in service.rs and layer.rs files in the tcp_limiter module were refactored for better code readability. Additional '\http' dependency was added in the Cargo.toml file which was reflected in the Cargo.lock file. Removing 'http' from 'tonic' in service.rs was necessary as it is now directly added as a dependency.
Configuration menu - View commit details
-
Copy full SHA for 1a78c11 - Browse repository at this point
Copy the full SHA 1a78c11View commit details -
Fix service shutdown in p2p and gRPC connection handlers
Replaced the serve method with serve_with_shutdown in both p2p and gRPC connection handlers. This change resolves the previous issue of lacking server shutdown functionality. The termination_receiver.map(drop) is now properly incorporated, enabling a more controlled server termination.
Configuration menu - View commit details
-
Copy full SHA for 14a23db - Browse repository at this point
Copy the full SHA 14a23dbView commit details -
Update crate dependencies for tonic, tonic-build, and related modules
This commit updates our crate dependencies by changing the tonic and tonic-build packages to reference the workspace instead of specific external versions. This will allow us to keep our internal versions in sync and avoid potential incompatibility issues. We have also added required updates in utils, rpc, protocol and grabbed the latest versions of pin-project and pin-project-internal. This optimizes the dependencies of the project and reduces the build size and time. Changes present in rpc/grpc/server, rpc/grpc/core, protocol/p2p, utils, rpc/grpc/client Cargo.toml, Cargo.lock and in Cargo.toml files.
Configuration menu - View commit details
-
Copy full SHA for 8211e8e - Browse repository at this point
Copy the full SHA 8211e8eView commit details -
Remove gRPC-specific limiter, introduce shared TCP limiter
The commit removes the gRPC-specific connection limiter and introduces a shared TCP connection limiter. The previous implementation was specific to gRPC only, but we want a solution that works across different services like gRPC, JSON RPC, REST, etc. As a result, the 'grpc' module has been deleted, and a new 'tcp_wrapper' module has been created. The new TCP limiter is shared and tracks the number of active TCP connections against a maximum limit.
Configuration menu - View commit details
-
Copy full SHA for d9f7139 - Browse repository at this point
Copy the full SHA d9f7139View commit details -
"Refactor connection limiter to be protocol agnostic
The TCP connection limiter has been made protocol agnostic, allowing it to work with multiple services like gRPC, JSON RPC, REST, etc. Previously, the limiter was limited to gRPC only. This change involved removing the gRPC-specific 'LimitLayer' and replacing it with a shared 'Wrapper' from the 'tcp_limiter' module. This refactoring makes it more reusable and simplifies the management of active TCP connections."
Configuration menu - View commit details
-
Copy full SHA for 61ccfe7 - Browse repository at this point
Copy the full SHA 61ccfe7View commit details -
"Update TCP Connection Limiter and Remove Redundant Tests
Updated the TCP connection limiter to use a loop for sync and made it protocol agnostic. Previously, the limiter was tightly coupled with gRPC, this change allows it to work with any protocol and makes it more reusable. Also, Removed redundant connection tests in client_server, as they were excessive and not related to the primary server functionality. These changes simplify management of active connections and improve test efficiency."
Configuration menu - View commit details
-
Copy full SHA for 5115685 - Browse repository at this point
Copy the full SHA 5115685View commit details -
"Add TCP connections limit in Kaspa
Integrate TCP limit option in daemon and rpc/wrpc modules to control the number of incoming connections. The limit option can be set during the launch of these services. By default, the value is None, meaning the number of connections are unlimited. This is essential for system resources optimization and prevention of any potential DoS attacks."
Configuration menu - View commit details
-
Copy full SHA for a4ec24c - Browse repository at this point
Copy the full SHA a4ec24cView commit details -
"Add option for maximum TCP connections
Added a new argument 'max_tcp_connections' to allow user to set a limit on number of TCP connections for kaspad. This helps in optimization of system resources and also acts as a preventive measure against potential DoS attacks. In absence of argument, defaults to 'None', indicating unlimited number of connections."
Configuration menu - View commit details
-
Copy full SHA for 9d65d56 - Browse repository at this point
Copy the full SHA 9d65d56View commit details -
"Exclude tcp_limiter and some dependencies for wasm32 target
Applied conditional compilation to the tcp_limiter module to exclude it when targeting wasm32. TCP connection limiting is not applicable in the context of WebAssembly, thereby removing unnecessary dependencies and reducing compilation time. Also reordered dependencies in utils/Cargo.toml under a specific target architecture to maintain clean code organization."
Configuration menu - View commit details
-
Copy full SHA for 359a218 - Browse repository at this point
Copy the full SHA 359a218View commit details -
"Refactor TCP limit calculation in kaspad/src/main.rs
Moved the TCP limit calculation from main.rs to daemon.rs to improve code coherence. Now, error management related to TCP limit exceeding is also done within daemon.rs. This change enhances code organization and readability by keeping all TCP-related logic in one place."
Configuration menu - View commit details
-
Copy full SHA for 652aa72 - Browse repository at this point
Copy the full SHA 652aa72View commit details -
"Add todo comments for future TCP limit implementation
Inserted comments in adaptor.rs and connection_handler.rs as reminders to implement TCP limit in connect_peer and connect_with_retry functions. These additions highlight the need to consider TCP limits when connecting to a peer and retrying connections respectively."
Configuration menu - View commit details
-
Copy full SHA for 0a42679 - Browse repository at this point
Copy the full SHA 0a42679View commit details -
"Limit TCP connections in integration tests
In tests, implemented new parameters for max_tcp_connections in both daemon.rs and daemon_integration_tests.rs. This changes the total file descriptor limit calculation by dividing it by two. This is critical in improving the robustness of our tests by ensuring they work under hardened network conditions."
Configuration menu - View commit details
-
Copy full SHA for eccbebf - Browse repository at this point
Copy the full SHA eccbebfView commit details
Commits on Oct 30, 2023
-
Merge branch 'master' into grpc-tcp-connection-limiter
# Conflicts: # Cargo.toml # protocol/p2p/Cargo.toml # rpc/grpc/client/Cargo.toml # rpc/grpc/core/Cargo.toml # rpc/grpc/server/Cargo.toml # rpc/grpc/server/src/adaptor.rs # rpc/grpc/server/src/connection_handler.rs # rpc/grpc/server/src/service.rs # rpc/grpc/server/src/tests/client_server.rs # testing/integration/src/daemon_integration_tests.rs
Configuration menu - View commit details
-
Copy full SHA for 6c04495 - Browse repository at this point
Copy the full SHA 6c04495View commit details -
Adjust file descriptor limit in integration tests
Changed the hardcoded file descriptors limit to a dynamic computation based on system's limitations in integration tests. This change makes the test environment more reliable because the file descriptor limit is adjusted according to the system's configuration and prevents potential issues with resource exhaustion. Also, set the maximum TCP connections for devnet in daemon cleaning test.
Configuration menu - View commit details
-
Copy full SHA for f0ac8a4 - Browse repository at this point
Copy the full SHA f0ac8a4View commit details