Add predicate capturing results of test runs

Signed-off-by: Aditya Sirish <[email protected]>

spec/predicates/test-results.md

@@ -0,0 +1,172 @@
+# Predicate type: Test Results
+
+Type URI: https://in-toto.io/attestation/test-result/v0.1
+
+Version: 0.1.0
+
+Authors:
+
+## Purpose
+
+This predicate type defines a generic schema to express the result of running
+tests in software supply chains. The schema may be extended to support different
+types of testing or specific test harnesses.
+
+## Use Cases
+
+Software development processes include several types of tests. This attestation
+can be used to express the results of running those tests. It can be used to
+verify:
+
+1. that all tests were in fact run, and
+2. that all required tests passed
+
+Therefore, each attestation corresponds to one invocation of a test suite, and
+may include the results of several individual tests.
+
+### Asserting Test Configurations Used
+
+The supply chain owner creates a policy that records the expected test
+configurations. During verification, the policy checks that the test attestation
+used the right configurations. A custom inspection may optionally parse the
+`testRun.link` field to verify attestation matches the test run.
+
+### Asserting Test Results
+
+In addition to the previous use case, the supply chain owner creates a policy
+verifying that test results passed. In the simplest case, the policy applies to
+all tests. Therefore, it asserts the contents of `result` and that
+`testRun.failedTests` is empty. In more nuanced cases, a subset of tests may
+matter. For example, the tested artifact may be an OS image that's to be
+deployed to three types of devices. As such, the test harness tests the new
+image on an instance of each device. When verifying the attestation prior to an
+image being installed on a device, it only matters that the tests passed on the
+corresponding test device and not necessarily the others.
+
+As before, a custom inspection may examine the `testRun.link` contents to verify
+the contents of the attestation.
+
+## Prerequisites
+
+Understanding of the
+[in-toto attestation specification](https:/in-toto/attestation).
+
+## Model
+
+This predicate type includes one compulsory field, `result`, that describes the
+result of the test run. The `testRun` object can be used to communicate a link
+to the test run and list tests that passed, failed, and passed with warnings. It
+also may optionally contain a reference to the configuration for the test run.
+The expected `subject` for this are the source artifacts themselves or some
+reference to them such as a Git commit.
+
+## Schema
+
+```json
+{
+ "_type": "https://in-toto.io/Statement/v1",
+ "subject": [{...}],
+ "predicateType": "https://in-toto.io/attestation/test-result/v0.1",
+ "predicate": {
+ "result": "pass|fail",
+ "testRun": {
+ "link": "<URL>",
+ "configuration": ["<ResourceDescriptor>", ...],
+ "passedTests": ["<TEST_NAME>", ...],
+ "warnedTests": ["<TEST_NAME>", ...],
+ "failedTests": ["<TEST_NAME>", ...]
+ }
+ }
+}
+```
+
+### Parsing Rules
+
+This predicate follows the in-toto attestation parsing rules. Summary:
+
+- Consumers MUST ignore unrecognized fields.
+- The `predicateType` URI includes the major version number and will always
+ change whenever there is a backwards incompatible change.
+- Minor version changes are always backwards compatible and “monotonic.” Such
+ changes do not update the `predicateType`.
+- Producers MAY add extension fields using field names that are unlikely to
+ collide with names used by other producers. Field names SHOULD avoid using
+ characters like `.` and `$`.
+- Fields marked _optional_ MAY be unset or null, and should be treated
+ equivalently. Both are equivalent to empty for object or array values.
+
+### Fields
+
+`result` _boolean_ , _required_
+
+Indicates the result of the test run. If true, it indicates _all_ tests passed
+in the corresponding run.
+
+`testRun` _object_, _optional_
+
+`testRun.link` _URI_, _optional_
+
+Contains a link to the test run. Useful to point to information that cannot be
+captured in the attestation.
+
+`testRun.configuration` _list of ResourceDescriptor_, _required_
+
+Reference to the configuration used for the test run.
+
+`testRun.passedTests` _list_, _optional_
+
+Each entry corresponds to a single test that passed.
+
+`testRun.warnedTests` _list_, _optional_
+
+Each entry corresponds to a single test that expressed a warning.
+
+`testRun.failedTests` _list_, _optional_
+
+Each entry corresponds to a single test that failed.
+
+## Example
+
+```json
+{
+ "_type": "https://in-toto.io/Statement/v1",
+ "subject": [
+ {
+ "digest": {
+ "gitCommit": "d20ace7968ba43c0219f62d71334c1095bab1602"
+ }
+ }
+ ],
+ "predicateType": "https://in-toto.io/attestation/test-result/v0.1",
+ "predicate": {
+ "result": "pass",
+ "testRun": {
+ "link": "https:/in-toto/in-toto/actions/runs/4425592351",
+ "configuration": [{
+ "name": ".github/workflows/ci.yml",
+ "downloadLocation": "https:/in-toto/in-toto/blob/d20ace7968ba43c0219f62d71334c1095bab1602/.github/workflows/ci.yml",
+ "digest": {
+ "gitBlob": "ebe4add40f63c3c98bc9b32ff1e736f04120b023"
+ }
+ }],
+ "passedTests": [
+ "build (3.7, ubuntu-latest, py)",
+ "build (3.7, macos-latest, py)",
+ "build (3.7, windows-latest, py)",
+ "build (3.8, ubuntu-latest, py)",
+ "build (3.8, macos-latest, py)",
+ "build (3.8, windows-latest, py)",
+ "build (3.9, ubuntu-latest, py)",
+ "build (3.9, macos-latest, py)",
+ "build (3.9, windows-latest, py)",
+ "build (3.10, ubuntu-latest, py)",
+ "build (3.10, macos-latest, py)",
+ "build (3.10, windows-latest, py)",
+ "build (3.x, ubuntu-latest, lint)"
+ ],
+ "warnedTests": [],
+ "failedTests": []
+ }
+ }
+}
+```