Metrics Oct 2020

Evaluation platform:

CPU Architecture - IBM Power9 and Intel Xeon

GPU Architecture - NVIDIA V100 (Volta)

Metrics report:

Tool	Languages	TP	FP	TN	FN	Recall	Specificity	Precision	Accuracy	TSR	Adjusted F1
Archer	C/C++	63	1	80	17	0.7875	0.9877	0.9844	0.8882	0.9360	0.8190
Intel Inspector	C/C++	71	40	45	8	0.8987	0.5294	0.6396	0.7073	0.9535	0.7126
ROMP	C/C++	59	11	73	18	0.7590	0.8876	0.8630	0.8256	1	0.8077
ThreadSanitizer	C/C++	64	1	84	15	0.8101	0.9882	0.9846	0.9024	0.9545	0.8375
Coderrect	C/C++	72	2	85	9	0.8889	0.9770	0.9730	0.9345	0.9767	0.9074
Archer	Fortran	53	0	63	16	0.8154	1	1	0.9063	0.7711	0.6927
Intel Inspector	Fortran	63	9	65	16	0.7975	0.8784	0.8750	0.8366	0.9217	0.7691
ROMP	Fortran	62	12	61	18	0.7750	0.8356	0.8378	0.8039	0.9217	0.7465
ThreadSanitizer	Fortran	52	0	65	13	0.8000	1	1	0.9000	0.7831	0.6961
Coderrect	Fortran	66	11	64	15	0.8148	0.8533	0.8571	0.8333	0.9398	0.8354

Metrics formula:

precision (P) = TP/(TP + FP)

recall (R) = TP/(TP +FN)

accuracy (A) = (TP +TN)/(TP +FP +TN +FN)

specificity (S) = TN / ( TN + FP)

F1 score (F1) = 2 * (P * R) / (P + R)

test support rate (TSR) = (TP +FP +TN +FN)/(TEST NUMBER)

adjusted F1 (AF1) = TSR * F1

Tool & compiler version:

Tool	Version	Compiler
ThreadSanitizer	10.0.0	Clang/LLVM 10.0.0
TSan(GPU)	10.0.1	Clang/LLVM 10.0.1
Archer	release_60	Clang/LLVM 6.0
Intel Inspector	2020 (build 603904)	Intel Compiler 19.1.0.166
ROMP	20ac93c	GCC/gfortran 7.4.0
Coderrect	0.8.0	Clang/LLVM 9.0.0

Note that Coderrect does not require a specific compiler version, as long as the user's code can be built on their machine. However, for the static analysis Coderrect generates LLVM IR using Clang-9.0.0, which is packaged into the tool.