Skip to content

Commit

Permalink
Simplify the post-install test targets.
Browse files Browse the repository at this point in the history
Remove the implicit `make install` from the post-install test
targets. Users can run `make install install-test` if required.
  • Loading branch information
ChrisCummins committed Feb 25, 2021
1 parent 3528fdb commit 52e5f98
Show file tree
Hide file tree
Showing 3 changed files with 33 additions and 25 deletions.
7 changes: 7 additions & 0 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,13 @@ jobs:
${{ runner.os }}-install-${{ matrix.python }}-
if: matrix.os == 'macos-latest'

- name: Install
run: make install
env:
CC: clang
CXX: clang++
BAZEL_BUILD_OPTS: --config=ci

- name: Test
run: make install-test
env:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release_test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -33,4 +33,4 @@ jobs:
run: python -m pip install -r tests/requirements.txt

- name: Test
run: make pytest
run: make install-test
49 changes: 25 additions & 24 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -17,29 +17,34 @@ Testing
runs are minimal and fast. Use this as your go-to target for testing
modifications to the codebase.

make install-test
Build and install the python package (equivalent to 'make install'),
then run the full test suite against the installed package, and any
other tests that require the python package to be installed. This is
expensive and not typically required for local development.

make install-fuzz
Build and install the python package (equivalent to 'make install'),
then run the fuzz testing suite. Fuzz tests are tests that generate
their own inputs and run in a loop until an error has been found, or
until a minimum number of seconds have elapsed. This minimum time is
controlled using a FUZZ_SECONDS variable. The default is 300 seconds (5
minutes). Override this value at the command line, for example
`FUZZ_SECONDS=60 make install-fuzz` will run the fuzz tests for a
minimum of one minute.

make itest
Run the test suite continuously on change. This is equivalent to
manually running `make test` when a source file is modified. Note that
`make install-test` tests are not run. This requires bazel-watcher.
See: https:/bazelbuild/bazel-watcher#installation


Post-installation Tests
-----------------------

make install-test
Run the full test suite against an installed CompilerGym package. This
requires that the CompilerGym package has been installed (`make
install`). This is useful for checking the package contents but is
usually not needed for interactive development since `make test` runs
the same tests without having to install anything.

make install-fuzz
Run the fuzz testing suite against an installed CompilerGym package.
Fuzz tests are tests that generate their own inputs and run in a loop
until an error has been found, or until a minimum number of seconds have
elapsed. This minimum time is controlled using a FUZZ_SECONDS variable.
The default is 300 seconds (5 minutes). Override this value at the
command line, for example `FUZZ_SECONDS=60 make install-fuzz` will run
the fuzz tests for a minimum of one minute. This requires that the
CompilerGym package has been installed (`make install`).


Documentation
-------------

Expand Down Expand Up @@ -204,34 +209,30 @@ test:
itest:
$(IBAZEL) $(BAZEL_OPTS) test $(BAZEL_TEST_OPTS) //...

tests-datasets:
install-test-datasets:
cd .. && python -m compiler_gym.bin.datasets --env=llvm-v0 --download=cBench-v0 >/dev/null

pytest:
install-test: install-test-datasets
mkdir -p /tmp/compiler_gym/wheel_tests
rm -f /tmp/compiler_gym/wheel_tests/tests
ln -s $(ROOT)/tests /tmp/compiler_gym/wheel_tests
cd /tmp/compiler_gym/wheel_tests && pytest -n auto tests -k "not fuzz"

install-test: | install tests-datasets pytest

# The minimum number of seconds to run the fuzz tests in a loop for. Override
# this at the commandline, e.g. `FUZZ_SECONDS=1800 make fuzz`.
FUZZ_SECONDS ?= 300

fuzz:
install-fuzz: install-test-datasets
mkdir -p /tmp/compiler_gym/wheel_fuzz_tests
rm -f /tmp/compiler_gym/wheel_fuzz_tests/tests
ln -s $(ROOT)/tests /tmp/compiler_gym/wheel_fuzz_tests
cd /tmp/compiler_gym/wheel_fuzz_tests && pytest tests -p no:sugar -x -vv -k fuzz --seconds=$(FUZZ_SECONDS)

install-fuzz: | install tests-datasets fuzz

post-install-test:
$(MAKE) -C examples/makefile_integration clean
SEARCH_TIME=3 $(MAKE) -C examples/makefile_integration test

.PHONY: test install-test post-install-test
.PHONY: test post-install-test


################
Expand Down

0 comments on commit 52e5f98

Please sign in to comment.