new(ci,benchmarks): added more benchmarks and make use of them in perf CI #2023
Conversation
Signed-off-by: Federico Di Pierro <[email protected]>
poiana
added
release-note-none
dco-signoff: yes
kind/feature
|
/milestone 0.18.0 |
There was a problem hiding this comment.
The updated action now builds and runs benchmarks too. Then store the resulting json in its outputs.
| @@ -37,6 +37,7 @@ jobs: | |||
| ${{ steps.perf.outputs.perf_scap }} | |||
| ${{ steps.perf.outputs.heaptrack_tests }} | |||
| ${{ steps.perf.outputs.heaptrack_scap }} | |||
| ${{ steps.perf.outputs.gbench_json }} | |||
There was a problem hiding this comment.
Pages CI will now upload also master gbench json too.
| @@ -43,6 +43,25 @@ jobs: | |||
| run: | | |||
| sudo heaptrack_print heaptrack_scap.data.zst -d ${{ steps.perf.outputs.heaptrack_scap }} &> heaptrack_scap_diff.txt | |||
|
|
|||
| - name: Diff from master - gbench | |||
There was a problem hiding this comment.
The algorithm is as follows:
- load all benchmarks present in master json
- for each of them, take the master
real_timevalue - for each of them, take the PR's head
real_timevalue - if PR's head does not have the benchmark, skip it
- compute the diff between head and master
- compute the diff in percentage between head and master
There was a problem hiding this comment.
Have you ever considered to use the compare.py tool https:/google/benchmark/blob/main/docs/tools.md#comparepy?
There was a problem hiding this comment.
Neat! I'll give it a look and eventually implement it!
There was a problem hiding this comment.
Implemented, thanks!
| @@ -75,6 +94,10 @@ jobs: | |||
| echo "\`\`\`" >> ./pr/COMMENT | |||
| tail -n3 "heaptrack_scap_diff.txt" >> ./pr/COMMENT | |||
| echo "\`\`\`" >> ./pr/COMMENT | |||
| echo "# Benchmarks diff from master" >> ./pr/COMMENT | |||
There was a problem hiding this comment.
In the PERF comment, echo the full benchmark diffs.
| @@ -135,3 +158,14 @@ jobs: | |||
| if [ -s heaptrack_scap_diff_above_thresh.txt ]; then | |||
| exit 2 | |||
| fi | |||
|
|
|||
| # Check will fail if there any google benchmark is slowed more than 5% | |||
| - name: Check >= 5% slowdown on google benchmarks | |||
There was a problem hiding this comment.
For every benchmark in our diff, check if diff in percentage is above 5% and eventually trigger a failure.
Signed-off-by: Federico Di Pierro <[email protected]>
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #2023 +/- ##
=======================================
Coverage 74.20% 74.20%
=======================================
Files 253 253
Lines 30832 30832
Branches 5411 5393 -18
=======================================
Hits 22880 22880
+ Misses 7952 7941 -11
- Partials 0 11 +11
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
|
Note: perf CI will fail because master report has no EDIT: indeed:
|
… value. Instead, rely on the user running the benchmark to eventually provide `--benchmark_repetitions` CLI arg. Provide `--benchmark_repetitions` CLI arg for perf CI. Signed-off-by: Federico Di Pierro <[email protected]> Co-authored-by: Andrea Terzolo <[email protected]>
…x typo. Signed-off-by: Federico Di Pierro <[email protected]>
…chmarks results. Signed-off-by: Federico Di Pierro <[email protected]>
|
This is now ready. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: Andreagit97, FedeDP The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
What type of PR is this?
/kind feature
Any specific area of the project related to this PR?
/area CI
Does this PR require a change in the driver versions?
What this PR does / why we need it:
Adds some more benchmarks (around sinsp
concatenate_pathsandsplit_container_imageutils).Moreover, it adds support for benchmark results in our perf CI.
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Does this PR introduce a user-facing change?: