tests: tmt runs all tests in one vm

[cgwalters]

We are going to need to split up the tests. And to do that we are going to need to figure out fmf/tmt more.

Just to track things, I am testing out this:

From 891b23bcb44fa7e003588553b7fb243a33d2e0b4 Mon Sep 17 00:00:00 2001
From: Colin Walters <[email protected]>
Date: Fri, 26 Jul 2024 08:48:39 -0400
Subject: [PATCH] wip

Signed-off-by: Colin Walters <[email protected]>
---
 plans/integration-run.fmf                      | 11 +++--------
 tests/basic-readonly/main.fmf                  |  2 ++
 tests/basic-readonly/test.sh                   |  4 ++++
 tests/booted/{ => readonly}/001-test-status.nu |  0
 tests/booted/{ => readonly}/basic.py           |  0
 tests/image-pushpull-upgrade/main.fmf          |  2 ++
 tests/image-pushpull-upgrade/test.sh           |  2 ++
 7 files changed, 13 insertions(+), 8 deletions(-)
 create mode 100644 tests/basic-readonly/main.fmf
 create mode 100755 tests/basic-readonly/test.sh
 rename tests/booted/{ => readonly}/001-test-status.nu (100%)
 rename tests/booted/{ => readonly}/basic.py (100%)
 create mode 100644 tests/image-pushpull-upgrade/main.fmf
 create mode 100755 tests/image-pushpull-upgrade/test.sh

diff --git a/plans/integration-run.fmf b/plans/integration-run.fmf
index 1190fcd..132babe 100644
--- a/plans/integration-run.fmf
+++ b/plans/integration-run.fmf
@@ -6,13 +6,8 @@ provision:
   image: file://./target/testvm/disk.qcow2
   disk: 20
 summary: Execute booted tests
+discover:
+  how: fmf
 execute:
   how: tmt
-  # There's currently two dynamic test frameworks; python and nushell.
-  # python is well known and understood. nushell is less well known, but
-  # is quite nice for running subprocesses and the like while making
-  # it easy to parse JSON etc.
-  script: |
-    set -xeu
-    pytest tests/booted/*.py
-    ls tests/booted/*-test-*.nu |sort -n | while read t; do nu $t; done
+
diff --git a/tests/basic-readonly/main.fmf b/tests/basic-readonly/main.fmf
new file mode 100644
index 0000000..aa0b710
--- /dev/null
+++ b/tests/basic-readonly/main.fmf
@@ -0,0 +1,2 @@
+summary: Concise summary describing what the test does
+test: ./test.sh
diff --git a/tests/basic-readonly/test.sh b/tests/basic-readonly/test.sh
new file mode 100755
index 0000000..585e584
--- /dev/null
+++ b/tests/basic-readonly/test.sh
@@ -0,0 +1,4 @@
+#!/bin/sh -eux
+
+pytest tests/booted/readonly/*.py
+ls tests/booted/readonly/*-test-*.nu |sort -n | while read t; do nu $t; done
diff --git a/tests/booted/001-test-status.nu b/tests/booted/readonly/001-test-status.nu
similarity index 100%
rename from tests/booted/001-test-status.nu
rename to tests/booted/readonly/001-test-status.nu
diff --git a/tests/booted/basic.py b/tests/booted/readonly/basic.py
similarity index 100%
rename from tests/booted/basic.py
rename to tests/booted/readonly/basic.py
diff --git a/tests/image-pushpull-upgrade/main.fmf b/tests/image-pushpull-upgrade/main.fmf
new file mode 100644
index 0000000..aa0b710
--- /dev/null
+++ b/tests/image-pushpull-upgrade/main.fmf
@@ -0,0 +1,2 @@
+summary: Concise summary describing what the test does
+test: ./test.sh
diff --git a/tests/image-pushpull-upgrade/test.sh b/tests/image-pushpull-upgrade/test.sh
new file mode 100755
index 0000000..c8bfe44
--- /dev/null
+++ b/tests/image-pushpull-upgrade/test.sh
@@ -0,0 +1,2 @@
+#!/bin/sh -eux
+exec tests/booted/002-test-image-pushpull-upgrade.nu
-- 
2.44.0

[cgwalters]

OK teemtee/tmt#3037 is hard blocking my local testing...

[cgwalters]

I am so lost in tmt...trying to figure out: if I want to run each test in a distinct VM, does that require a distinct plan for each one? Is that what https://tmt.readthedocs.io/en/stable/spec/plans.html#isolate is trying to help?

[cgwalters]

I asked on a chat channel, the answer is:

for now, the answer is indeed yes, you need to spread tests across multiple plans, or have multiple guests in one plan, one for each discover phase with corresponding where
tmt does not know yet how to isolate tests or how to automatically turn a set of tests into multiple plans on its own