feat: cluster mode support for stress testing

[skynet2]

Cluster support implemented using Redis.
All available stress testing nodes registers under stress:nodes hset.

Starting stress test:

curl --location 'http://127.0.0.1:2775/run' \
--header 'Content-Type: application/json' \
--data '{
    "api_url" : "https://xx.com/vcs",
    "token_client_id" : "user_vcs_xxx",
    "token_client_secret" : "gxxxx",
    "user_count" : 5,
    "concurrent_requests" : 4,
    "issuer_profile_id" : "xxxx",
    "issuer_profile_version" : "latest",
    "verifier_profile_version" : "latest",
    "verifier_profile_id" : "xxxx",
    "verifier_presentation_id" : "xxxx",
    "credential_type" : "xxxxx",
    "disable_revoke_test" : false,
    "disable_vp_test" : false,
    "claim_data" : {
        "type" : ["xxxx"],
        "country" : "data:image/jpeg;base64"
    },
    "detailed" : false
}'

Example response:

{
    "id": "f6cbc007-e7a3-48ed-b325-a9c5697b5b0e",
    "nodes": {
        "127.0.0.1": "stress:result:30c49629-5ae7-46a5-9a0d-c6c7bee75750",
        "localhost": "stress:result:4a6fc2d1-56e3-48d1-9c74-980b89b44553"
    },
    "total_rate": 4
}

Once we start the stress test, the master node will read "concurrent_requests" and divide it to the count of available nodes.
For example:
Node count: 4
concurrent_requests : 200

Each node will execute 200 / 4 = 50 concurrent requests.

curl --location 'http://127.0.0.1:2775/run/f6cbc007-e7a3-48ed-b325-a9c5697b5b0e'

Is used to receive results.
Where f6cbc007-e7a3-48ed-b325-a9c5697b5b0e is the stress test id, which we can find in response of /run API.

Example response:

{
    "total_rate": 4,
    "total_nodes": 2,
    "total_requests": 5,
    "total_errors": 0,
    "grouped_errors": {},
    "combined_metrics": {
        "__network_latency": {
            "avg": "605ms",
            "max": "624ms",
            "min": "598ms"
        },
        "_vcs_ci_flow_duration": {
            "avg": "1.633s",
            "max": "1.689s",
            "min": "1.6s"
        },
        "_vcs_vp_flow_duration": {
            "avg": "1.505s",
            "max": "1.583s",
            "min": "1.446s"
        },
        "vci_create_wallet": {
            "avg": "15ms",
            "max": "19ms",
            "min": "14ms"
        },
        "vci_get_access_token": {
            "avg": "604ms",
            "max": "614ms",
            "min": "593ms"
        },
        "vci_get_credential": {
            "avg": "277ms",
            "max": "324ms",
            "min": "251ms"
        },
        "vci_get_issuer_credentials_oidc_config": {
            "avg": "149ms",
            "max": "157ms",
            "min": "144ms"
        },
        "vci_get_issuer_oidc_config": {
            "avg": "601ms",
            "max": "605ms",
            "min": "596ms"
        },
        "vp_create_authorized_response": {
            "avg": "0s",
            "max": "1ms",
            "min": "0s"
        },
        "vp_fetch_request_object": {
            "avg": "643ms",
            "max": "754ms",
            "min": "609ms"
        },
        "vp_query_credential_from_wallet": {
            "avg": "2ms",
            "max": "3ms",
            "min": "2ms"
        },
        "vp_send_authorized_response": {
            "avg": "862ms",
            "max": "901ms",
            "min": "829ms"
        },
        "vp_verify_authorization_request": {
            "avg": "671ms",
            "max": "790ms",
            "min": "617ms"
        }
    },
    "per_runner_info": {
        "127.0.0.1": {
            "raw_metrics": [
                {
                    "Name": "__network_latency",
                    "Avg": 605000000,
                    "Max": 624000000,
                    "Min": 598000000
                },
                {
                    "Name": "_vcs_ci_flow_duration",
                    "Avg": 1633000000,
                    "Max": 1689000000,
                    "Min": 1600000000
                },
                {
                    "Name": "_vcs_vp_flow_duration",
                    "Avg": 1505000000,
                    "Max": 1583000000,
                    "Min": 1446000000
                },
                {
                    "Name": "vci_create_wallet",
                    "Avg": 15000000,
                    "Max": 19000000,
                    "Min": 14000000
                },
                {
                    "Name": "vci_get_access_token",
                    "Avg": 604000000,
                    "Max": 614000000,
                    "Min": 593000000
                },
                {
                    "Name": "vci_get_credential",
                    "Avg": 277000000,
                    "Max": 324000000,
                    "Min": 251000000
                },
                {
                    "Name": "vci_get_issuer_credentials_oidc_config",
                    "Avg": 149000000,
                    "Max": 157000000,
                    "Min": 144000000
                },
                {
                    "Name": "vci_get_issuer_oidc_config",
                    "Avg": 601000000,
                    "Max": 605000000,
                    "Min": 596000000
                },
                {
                    "Name": "vp_create_authorized_response",
                    "Avg": 0,
                    "Max": 1000000,
                    "Min": 0
                },
                {
                    "Name": "vp_fetch_request_object",
                    "Avg": 643000000,
                    "Max": 754000000,
                    "Min": 609000000
                },
                {
                    "Name": "vp_query_credential_from_wallet",
                    "Avg": 2000000,
                    "Max": 3000000,
                    "Min": 2000000
                },
                {
                    "Name": "vp_send_authorized_response",
                    "Avg": 862000000,
                    "Max": 901000000,
                    "Min": 829000000
                },
                {
                    "Name": "vp_verify_authorization_request",
                    "Avg": 671000000,
                    "Max": 790000000,
                    "Min": 617000000
                }
            ],
            "error_count": 0,
            "request_count": 5,
            "rate": 2,
            "state": "complete",
            "result_error": null
        },
        "localhost": {
            "raw_metrics": null,
            "error_count": 0,
            "request_count": 0,
            "rate": 0,
            "state": "running",
            "result_error": null
        }
    },
    "is_finished": false
}

Most important fields here :
is_finished - true (stress test finished), false (stress test is still running)
combined_metrics - is a map of combined performance metrics from all nodes for specific stress test

[codecov]

Codecov Report

Patch and project coverage have no change.

Comparison is base (e107432) 87.47% compared to head (8976651) 87.47%.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1320   +/-   ##
=======================================
  Coverage   87.47%   87.47%           
=======================================
  Files         152      152           
  Lines        8648     8648           
=======================================
  Hits         7565     7565           
  Misses        714      714           
  Partials      369      369           

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.