Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add runtime observation type to LLVM service. #307

Merged
merged 21 commits into from
Jul 19, 2021

Conversation

ChrisCummins
Copy link
Contributor

@ChrisCummins ChrisCummins commented Jun 29, 2021

This adds support to the LLVM environments for optimizing for runtime. To evaluate the runtime, the LLVM-IR module is compiled down to a binary and executed on the host machine multiple times. The wall time of each execution is then returned as a list. This is exposed through a new Runtime observation space.

To compute the program runtimes, use the new Runtime observation space:

>>> import compiler_gym
>>> env = compiler_gym.make("llvm-v0")
>>> env.reset()
>>> env.observation["Runtime"]
[0.123, 0.142, 0.153]

The Runtime observation space returns a list of runtimes in seconds, resulting from running the compiled binary multiple times. Set the number of runtimes to measure using:

# Use 10 runtime observations:
>>> env.send_param("llvm.set_runtimes_per_observation_count", "10")

This supports the cbench-v1 dataset partially and the csmith-v0 dataset fully. To check if a benchmark is compatible with the runtime observation space, use the IsRunnable observation:

>>> env.observation["IsRunnable"]
1

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jun 29, 2021
@codecov-commenter
Copy link

codecov-commenter commented Jun 29, 2021

Codecov Report

Merging #307 (4ae99e3) into development (164cd97) will increase coverage by 0.28%.
The diff coverage is 95.52%.

❗ Current head 4ae99e3 differs from pull request most recent head eb1a4b8. Consider uploading reports for the commit eb1a4b8 to get more accurate results
Impacted file tree graph

@@               Coverage Diff               @@
##           development     #307      +/-   ##
===============================================
+ Coverage        85.59%   85.88%   +0.28%     
===============================================
  Files               87       87              
  Lines             4700     4754      +54     
===============================================
+ Hits              4023     4083      +60     
+ Misses             677      671       -6     
Impacted Files Coverage Δ
compiler_gym/service/proto/__init__.py 100.00% <ø> (ø)
compiler_gym/views/observation_space_spec.py 86.17% <71.42%> (+1.90%) ⬆️
compiler_gym/wrappers/core.py 93.84% <97.05%> (+4.37%) ⬆️
compiler_gym/envs/compiler_env.py 86.70% <100.00%> (+0.25%) ⬆️
compiler_gym/envs/llvm/datasets/__init__.py 100.00% <100.00%> (ø)
compiler_gym/envs/llvm/datasets/cbench.py 78.76% <100.00%> (+2.12%) ⬆️
compiler_gym/envs/llvm/datasets/csmith.py 93.26% <100.00%> (+0.13%) ⬆️
compiler_gym/spaces/scalar.py 96.66% <100.00%> (+0.11%) ⬆️
compiler_gym/spaces/sequence.py 90.00% <100.00%> (+1.53%) ⬆️
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 164cd97...eb1a4b8. Read the comment docs.

@ChrisCummins ChrisCummins changed the title [WIP] Add runtime observation type to LLVM service. Add runtime observation type to LLVM service. Jul 14, 2021
@ChrisCummins ChrisCummins marked this pull request as ready for review July 19, 2021 14:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants