XTest

Testing suite for XTask

Usage

Options

./paces.sh [options]

Where valid options are:

• -w<n>: Limit the number of used cores/threads to n
• -r<n>: Run each enabled test n times, averaging the results.
• -x<test>: Disable a test
• -X: Disable all enabled tests
• -o<test>: Enable a test
• -C<cc> and -c<cflags>: Change C compiler and options
• -s<stcflags>: Change STC compilation options
• -S: Disable STC (and thus use previously compiled Tcl for Swift tests)
• XTASK_DIR: Environment variable set to the path for the base XTask directory (default is $HOME/xtask)
• SWIFT_DIR: Environment variable set to the path for the Swift/T install directory (default is $HOME/swift-t-install)

Output

The output for a run of paces.sh often looks like the following:

Using runtimes: jigstack single jigstackxd openmp xtasklua cilk
Compiling C with ${CC} -O3
[single]	Counting for test fib... 1300483309 tasks
[single]	     Running test fib... 2 in 6.66s, 3.31/3.33/3.35
[openmp]	Counting for test fib... 6534925 tasks
[openmp]	     Running test fib... 2 in 1.65s, 0.48/0.83/1.17
[cilk]		Counting for test fib... 10573732 tasks
...

Each test/implementation pair outputs the number of tasks, along with a status line of the form:

... <number of runs> in <total time>s, <min run time>/<avg run time>/<max run time>

Benchmark data is written to out/<test>-<imp>.dat, for instance

Threads TPS CPU
1 19385656.743809 0.975358
2 31380342.717404 1.982466

The columns give the

1. Number of threads requested (not nessesarily used, as in the case of single or previous versions of cilk).
2. Average tasks per second (calculated as tasks/total time).
3. Number of cores used (calculated as cpu time/total time).

Tuning

For good data, the parameters in paces.sh need tuning for each testbed, as follows:

0. cd xtest
1. Run paces.sh -w3, disabling tests until the script runs to completion without error.
2. Delete the outputs (rm out/*)
3. For each enabled test, run paces.sh -X -o<test> and edit the test's parameters until the avg run time is above 1-2 seconds. NOTE: Due to overflow, single, openmp and cilk are not able to achieve this. It's ok, just get close.
4. Finally, run the suite from step 1 with paces.sh -r10, go get a sandwich, and grab the results from out/
5. Make pretty graphs with your graphing software of choice.
6. Profit!