Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add benchmarks framework #11

Merged
merged 6 commits into from
Aug 9, 2019
Merged

Conversation

baronfel
Copy link
Contributor

@baronfel baronfel commented Aug 8, 2019

Minimal implementation of #7 that hooks up benchmarks into the build and provides a sample benchmark that compares serialization speed of an array of varying length containing multiple copies of a single test record instance.

The benchmark is done for Newtonsoft and System.Text.Json.

On my machine right now I get the following results:

// * Summary *

BenchmarkDotNet=v0.11.5, OS=macOS Mojave 10.14.5 (18F132) [Darwin 18.6.0]
Intel Core i7-7820HQ CPU 2.90GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=3.0.100-preview7-012821
  [Host]   : .NET Core 3.0.0-preview7-27912-14 (CoreCLR 4.700.19.32702, CoreFX 4.700.19.36209), 64bit RyuJIT DEBUG
  ShortRun : .NET Core 3.0.0-preview7-27912-14 (CoreCLR 4.700.19.32702, CoreFX 4.700.19.36209), 64bit RyuJIT

Job=ShortRun  Runtime=Core  IterationCount=3  
LaunchCount=1  WarmupCount=3  

|         Method | ArrayLength |         Mean |       Error |    StdDev |    Gen 0 |   Gen 1 |   Gen 2 | Allocated |
|--------------- |------------ |-------------:|------------:|----------:|---------:|--------:|--------:|----------:|
|     Newtonsoft |           1 |     2.309 us |   0.3416 us | 0.0187 us |   0.5341 |       - |       - |    2240 B |
| SystemTextJson |           1 |     3.093 us |   0.2150 us | 0.0118 us |   0.1640 |       - |       - |     688 B |
|     Newtonsoft |          10 |    17.766 us |   6.4282 us | 0.3524 us |   1.9836 |       - |       - |    8328 B |
| SystemTextJson |          10 |    25.582 us |   7.5592 us | 0.4143 us |   1.1902 |       - |       - |    5024 B |
|     Newtonsoft |         100 |   171.889 us |  96.7675 us | 5.3042 us |  19.7754 |  0.2441 |       - |   83048 B |
| SystemTextJson |         100 |   253.261 us |  19.8938 us | 1.0904 us |  11.2305 |       - |       - |   48400 B |
|     Newtonsoft |        1000 | 1,784.456 us |  86.2631 us | 4.7284 us | 121.0938 | 58.5938 | 58.5938 |  684768 B |
| SystemTextJson |        1000 | 2,598.975 us | 113.6843 us | 6.2314 us | 117.1875 | 58.5938 | 58.5938 |  495709 B |

@baronfel
Copy link
Contributor Author

baronfel commented Aug 8, 2019

I added benchmarks for pure classes as opposed to records (using the same sample data and the same number of items in the array):

classes:

BenchmarkDotNet=v0.11.5, OS=macOS Mojave 10.14.5 (18F132) [Darwin 18.6.0]
Intel Core i7-7820HQ CPU 2.90GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=3.0.100-preview7-012821
  [Host]     : .NET Core 3.0.0-preview7-27912-14 (CoreCLR 4.700.19.32702, CoreFX 4.700.19.36209), 64bit RyuJIT DEBUG
  DefaultJob : .NET Core 3.0.0-preview7-27912-14 (CoreCLR 4.700.19.32702, CoreFX 4.700.19.36209), 64bit RyuJIT


|         Method | ArrayLength |       Mean |     Error |    StdDev |
|--------------- |------------ |-----------:|----------:|----------:|
|     Newtonsoft |           1 |   2.421 us | 0.0292 us | 0.0259 us |
| SystemTextJson |           1 |   1.925 us | 0.0064 us | 0.0053 us |
|     Newtonsoft |          10 |  18.433 us | 0.2803 us | 0.2622 us |
| SystemTextJson |          10 |  14.533 us | 0.1299 us | 0.1085 us |
|     Newtonsoft |         100 | 176.174 us | 0.7278 us | 0.6808 us |
| SystemTextJson |         100 | 138.210 us | 0.5615 us | 0.5252 us |

records:

// * Summary *

BenchmarkDotNet=v0.11.5, OS=macOS Mojave 10.14.5 (18F132) [Darwin 18.6.0]
Intel Core i7-7820HQ CPU 2.90GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=3.0.100-preview7-012821
  [Host]     : .NET Core 3.0.0-preview7-27912-14 (CoreCLR 4.700.19.32702, CoreFX 4.700.19.36209), 64bit RyuJIT DEBUG
  DefaultJob : .NET Core 3.0.0-preview7-27912-14 (CoreCLR 4.700.19.32702, CoreFX 4.700.19.36209), 64bit RyuJIT


|         Method | ArrayLength |       Mean |     Error |    StdDev |
|--------------- |------------ |-----------:|----------:|----------:|
|     Newtonsoft |           1 |   2.318 us | 0.0186 us | 0.0165 us |
| SystemTextJson |           1 |   3.007 us | 0.0226 us | 0.0201 us |
|     Newtonsoft |          10 |  18.011 us | 0.0581 us | 0.0515 us |
| SystemTextJson |          10 |  25.219 us | 0.0812 us | 0.0720 us |
|     Newtonsoft |         100 | 171.588 us | 1.6340 us | 1.4485 us |
| SystemTextJson |         100 | 250.176 us | 2.2393 us | 2.0946 us |

The uptake is that on master right now, you take a ~60% hit by using records.

@baronfel
Copy link
Contributor Author

baronfel commented Aug 8, 2019

I had to ZIP this SVG to attach it here.

flamegrapy.records.svg.zip

It shows a flamegraph of the results of stripping down the benchmark to just serializing a 1000-element array of that same record instance 1000 times.

A large portion of the gap is from the reflection-based APIs forced on us by the getUnionTagReader and getUnionRecordReader functions :(

@Tarmil
Copy link
Owner

Tarmil commented Aug 8, 2019

Looks like we have our work cut out for us...

For the PR itself, I think we should leave the Benchmark target out of the main FAKE pipeline and instead put a separate "Build" ==> "Benchmark". We're not here to benchmark AppVeyor's servers 😄

@baronfel
Copy link
Contributor Author

baronfel commented Aug 8, 2019

Fair enough :) I'll send those changes tomorrow!

@baronfel
Copy link
Contributor Author

baronfel commented Aug 9, 2019

Ok, I pushed up that removal. I mostly included it by default because the outputs are written in github-markdown format, and so might be nice to include on a commit-by-commit basis. No worries though :)

@Tarmil Tarmil merged commit 9c5ac92 into Tarmil:master Aug 9, 2019
@baronfel baronfel deleted the add-benchmarks-framework branch September 16, 2019 02:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants