From d55adbf149cfe038f0169cdb6411bbdb4afc3f7a Mon Sep 17 00:00:00 2001 From: Mikel Blanchard Date: Fri, 23 Feb 2024 22:59:41 -0800 Subject: [PATCH 1/3] Stress test README updates. --- .../OpenTelemetry.Tests.Stress.Logs/README.md | 20 ++- .../README.md | 36 ++++- .../README.md | 20 ++- test/OpenTelemetry.Tests.Stress/README.md | 136 ++++++++++++------ 4 files changed, 162 insertions(+), 50 deletions(-) diff --git a/test/OpenTelemetry.Tests.Stress.Logs/README.md b/test/OpenTelemetry.Tests.Stress.Logs/README.md index 20c4b87db6e..b2e50ece933 100644 --- a/test/OpenTelemetry.Tests.Stress.Logs/README.md +++ b/test/OpenTelemetry.Tests.Stress.Logs/README.md @@ -10,5 +10,23 @@ based on the [OpenTelemetry.Tests.Stress](../OpenTelemetry.Tests.Stress/README.m Open a console, run the following command from the current folder: ```sh -dotnet run --framework net6.0 --configuration Release +dotnet run --framework net8.0 --configuration Release +``` + +To see command line options available, run the following command from the +current folder: + +```sh +dotnet run --framework net8.0 --configuration Release -- --help +``` + +The help output includes settings and their explanations: + +```text + -c, --concurrency The concurrency (maximum degree of parallelism) for the stress test. Default value: Environment.ProcessorCount. + + -p, --internal_port The Prometheus http listener port where Prometheus will be exposed for retrieving internal metrics while the stress test is running. Set to '0' to + disable. Default value: 9464. + + -d, --duration The duration for the stress test to run in seconds. If set to '0' or a negative value the stress test will run until canceled. Default value: 0. ``` diff --git a/test/OpenTelemetry.Tests.Stress.Metrics/README.md b/test/OpenTelemetry.Tests.Stress.Metrics/README.md index 26c6ac1b1fe..3201c7c5900 100644 --- a/test/OpenTelemetry.Tests.Stress.Metrics/README.md +++ b/test/OpenTelemetry.Tests.Stress.Metrics/README.md @@ -5,11 +5,6 @@ This stress test is specifically for Metrics SDK, and is based on the * [Running the stress test](#running-the-stress-test) -> [!NOTE] -> To run the stress tests for Histogram, comment out the `Run` method -for `Counter` and uncomment everything related to `Histogram` in the -[Program.cs](../OpenTelemetry.Tests.Stress.Metrics/Program.cs). - ## Running the stress test Open a console, run the following command from the current folder: @@ -17,3 +12,34 @@ Open a console, run the following command from the current folder: ```sh dotnet run --framework net8.0 --configuration Release ``` + +To see command line options available, run the following command from the +current folder: + +```sh +dotnet run --framework net8.0 --configuration Release -- --help +``` + +The help output includes settings and their explanations: + +```text + -t, --type The metrics stress test type to run. Valid values: [Histogram, Counter]. Default value: Histogram. + + -m, --metrics_port The Prometheus http listener port where Prometheus will be exposed for retrieving test metrics while the stress test is running. Set to '0' to disable. + Default value: 9185. + + -v, --view Whether or not a view should be configured to filter tags for the stress test. Default value: False. + + -o, --otlp Whether or not an OTLP exporter should be added for the stress test. Default value: False. + + -i, --interval The OTLP exporter export interval in milliseconds. Default value: 5000. + + -e, --exemplars Whether or not to enable exemplars for the stress test. Default value: False. + + -c, --concurrency The concurrency (maximum degree of parallelism) for the stress test. Default value: Environment.ProcessorCount. + + -p, --internal_port The Prometheus http listener port where Prometheus will be exposed for retrieving internal metrics while the stress test is running. Set to '0' to + disable. Default value: 9464. + + -d, --duration The duration for the stress test to run in seconds. If set to '0' or a negative value the stress test will run until canceled. Default value: 0. +``` diff --git a/test/OpenTelemetry.Tests.Stress.Traces/README.md b/test/OpenTelemetry.Tests.Stress.Traces/README.md index 54998de1e76..005614d3361 100644 --- a/test/OpenTelemetry.Tests.Stress.Traces/README.md +++ b/test/OpenTelemetry.Tests.Stress.Traces/README.md @@ -10,5 +10,23 @@ based on the [OpenTelemetry.Tests.Stress](../OpenTelemetry.Tests.Stress/README.m Open a console, run the following command from the current folder: ```sh -dotnet run --framework net6.0 --configuration Release +dotnet run --framework net8.0 --configuration Release +``` + +To see command line options available, run the following command from the +current folder: + +```sh +dotnet run --framework net8.0 --configuration Release -- --help +``` + +The help output includes settings and their explanations: + +```text + -c, --concurrency The concurrency (maximum degree of parallelism) for the stress test. Default value: Environment.ProcessorCount. + + -p, --internal_port The Prometheus http listener port where Prometheus will be exposed for retrieving internal metrics while the stress test is running. Set to '0' to + disable. Default value: 9464. + + -d, --duration The duration for the stress test to run in seconds. If set to '0' or a negative value the stress test will run until canceled. Default value: 0. ``` diff --git a/test/OpenTelemetry.Tests.Stress/README.md b/test/OpenTelemetry.Tests.Stress/README.md index 1f953b1def1..ef97c2075d8 100644 --- a/test/OpenTelemetry.Tests.Stress/README.md +++ b/test/OpenTelemetry.Tests.Stress/README.md @@ -18,26 +18,45 @@ Open a console, run the following command from the current folder: ```sh -dotnet run --framework net6.0 --configuration Release +dotnet run --framework net8.0 --configuration Release +``` + +To see command line options available, run the following command from the +current folder: + +```sh +dotnet run --framework net8.0 --configuration Release -- --help +``` + +The help output includes settings and their explanations: + +```text + -c, --concurrency The concurrency (maximum degree of parallelism) for the stress test. Default value: Environment.ProcessorCount. + + -p, --internal_port The Prometheus http listener port where Prometheus will be exposed for retrieving internal metrics while the stress test is running. Set to '0' to + disable. Default value: 9464. + + -d, --duration The duration for the stress test to run in seconds. If set to '0' or a negative value the stress test will run until canceled. Default value: 0. ``` Once the application started, you will see the performance number updates from -the console window title. +the console window title and the console window itself. + +While a test is running... -Use the `SPACE` key to toggle the console output, which is off by default. +* Use the `SPACE` key to toggle the console output, which is on by default. -Use the `ENTER` key to print the latest performance statistics. +* Use the `ENTER` key to print the latest performance statistics. -Use the `ESC` key to exit the stress test. +* Use the `ESC` key to exit the stress test. + +Example output while a test is running: ```text -Running (concurrency = 1), press to stop... -2021-09-28T18:47:17.6807622Z Loops: 17,549,732,467, Loops/Second: 738,682,519, CPU Cycles/Loop: 3 -2021-09-28T18:47:17.8846348Z Loops: 17,699,532,304, Loops/Second: 731,866,438, CPU Cycles/Loop: 3 -2021-09-28T18:47:18.0914577Z Loops: 17,850,498,225, Loops/Second: 730,931,752, CPU Cycles/Loop: 3 -2021-09-28T18:47:18.2992864Z Loops: 18,000,133,808, Loops/Second: 724,029,883, CPU Cycles/Loop: 3 -2021-09-28T18:47:18.5052989Z Loops: 18,150,598,194, Loops/Second: 733,026,161, CPU Cycles/Loop: 3 -2021-09-28T18:47:18.7116733Z Loops: 18,299,461,007, Loops/Second: 724,950,210, CPU Cycles/Loop: 3 +Options: {"Concurrency":20,"PrometheusInternalMetricsPort":9464,"DurationSeconds":0} +Run OpenTelemetry.Tests.Stress.exe --help to see available options. +Running (concurrency = 20, internalPrometheusEndpoint = http://localhost:9464/metrics/), press to stop, press to toggle statistics in the console... +Loops: 17,384,826,748, Loops/Second: 2,375,222,037, CPU Cycles/Loop: 24, RunningTime (Seconds): 7``` ``` The stress test metrics are exposed via @@ -73,59 +92,88 @@ process_runtime_dotnet_gc_allocations_size_bytes 5485192 1658950184752 ## Writing your own stress test -> [!WARNING] -> These instructions are out of date and should NOT be followed. They will be - updated soon. - Create a simple console application with the following code: ```csharp -using System.Runtime.CompilerServices; +using OpenTelemetry.Tests.Stress; -public partial class Program +public static class Program { - public static void Main() + public static int Main(string[] args) { - Stress(concurrency: 10, prometheusPort: 9464); + return StressTestFactory.RunSynchronously(args); } - [MethodImpl(MethodImplOptions.AggressiveInlining)] - protected static void Run() + private sealed class MyStressTest : StressTest { - // add your logic here + public MyStressTest(StressTestOptions options) + : base(options) + { + } + + protected override void RunWorkItemInParallel() + { + } } } ``` -Add the Skeleton.cs file to your `*.csproj` file: +Add the following project reference to the project: ```xml - - - + ``` -Add the following packages to the project: +Now you are ready to run your own stress test. Add test logic in the +`RunWorkItemInParallel` method to measure performance. -```shell -dotnet add package OpenTelemetry.Exporter.Prometheus --prerelease -dotnet add package OpenTelemetry.Instrumentation.Runtime --prerelease -``` +To define custom options create an options class which derives from +`StressTestOptions`: -Now you are ready to run your own stress test. +```csharp +using CommandLine; +using OpenTelemetry.Tests.Stress; + +public static class Program +{ + public static int Main(string[] args) + { + return StressTestFactory.RunSynchronously(args); + } + + private sealed class MyStressTest : StressTest + { + public MyStressTest(MyStressTestOptions options) + : base(options) + { + } + + protected override void RunWorkItemInParallel() + { + // Use this.Options here to access options supplied + // on the command line. + } + } + + private sealed class MyStressTestOptions : StressTestOptions + { + [Option('r', "rate", HelpText = "Add help text here for the rate option. Default value: 0.", Required = false)] + public int Rate { get; set; } = 0; + } +} +``` Some useful notes: -* You can specify the concurrency using `Stress(concurrency: {concurrency - number})`, the default value is the number of CPU cores. Keep in mind that - concurrency level does not equal to the number of threads. -* You can specify a local PrometheusExporter listening port using - `Stress(prometheusPort: {port number})`, the default value is `0`, which will - turn off the PrometheusExporter. -* You want to put `[MethodImpl(MethodImplOptions.AggressiveInlining)]` on - `Run()`, this helps to reduce extra flushes on the CPU instruction cache. * You might want to run the stress test under `Release` mode rather than `Debug` - mode. + mode. `Debug` builds typically are not optimized and contain extra code which + will change the performance of the code under test. +* You can specify the concurrency using `-c` or `--concurrency` command line + argument, the default value if not specified is the number of CPU cores. Keep + in mind that concurrency level does not equal to the number of threads. +* You can use the duration `-d` or `--duration` command line argument to run the + stress test for a specific time period. This is useful when comparing changes + across multiple runs. ## Understanding the results @@ -134,4 +182,6 @@ Some useful notes: sliding window of few hundreds of milliseconds. * `CPU Cycles/Loop` represents the average CPU cycles for each `Run()` invocation, based on a small sliding window of few hundreds of milliseconds. -* `Runaway Time` represents the runaway time (seconds) since the test started. +* `Total Running Time` represents the running time (seconds) since the test started. +* `GC Total Allocated Bytes` (not available on .NET Framework) shows the total + amount of memory allocated while the test was running. From 8156fa9b372e6f1915a7550719ab448e33a1dc90 Mon Sep 17 00:00:00 2001 From: Mikel Blanchard Date: Mon, 26 Feb 2024 09:35:22 -0800 Subject: [PATCH 2/3] Cleanup. --- test/OpenTelemetry.Tests.Stress/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/OpenTelemetry.Tests.Stress/README.md b/test/OpenTelemetry.Tests.Stress/README.md index ef97c2075d8..0aa12febb17 100644 --- a/test/OpenTelemetry.Tests.Stress/README.md +++ b/test/OpenTelemetry.Tests.Stress/README.md @@ -56,7 +56,7 @@ Example output while a test is running: Options: {"Concurrency":20,"PrometheusInternalMetricsPort":9464,"DurationSeconds":0} Run OpenTelemetry.Tests.Stress.exe --help to see available options. Running (concurrency = 20, internalPrometheusEndpoint = http://localhost:9464/metrics/), press to stop, press to toggle statistics in the console... -Loops: 17,384,826,748, Loops/Second: 2,375,222,037, CPU Cycles/Loop: 24, RunningTime (Seconds): 7``` +Loops: 17,384,826,748, Loops/Second: 2,375,222,037, CPU Cycles/Loop: 24, RunningTime (Seconds): 7 ``` The stress test metrics are exposed via From b22813d821d6f11d614f2182e8a868988ee23690 Mon Sep 17 00:00:00 2001 From: Mikel Blanchard Date: Mon, 26 Feb 2024 09:40:19 -0800 Subject: [PATCH 3/3] Code review. --- test/OpenTelemetry.Tests.Stress/README.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/test/OpenTelemetry.Tests.Stress/README.md b/test/OpenTelemetry.Tests.Stress/README.md index 0aa12febb17..0aa032669fb 100644 --- a/test/OpenTelemetry.Tests.Stress/README.md +++ b/test/OpenTelemetry.Tests.Stress/README.md @@ -165,9 +165,12 @@ public static class Program Some useful notes: -* You might want to run the stress test under `Release` mode rather than `Debug` - mode. `Debug` builds typically are not optimized and contain extra code which - will change the performance of the code under test. +* It is generally best practice to run the stress test for code compiled in + `Release` configuration rather than `Debug` configuration. `Debug` builds + typically are not optimized and contain extra code which will change the + performance characteristics of the logic under test. The stress test will + write a warning message to the console when starting if compiled with `Debug` + configuration. * You can specify the concurrency using `-c` or `--concurrency` command line argument, the default value if not specified is the number of CPU cores. Keep in mind that concurrency level does not equal to the number of threads.