Implement tmt try for interactive sessions

Basic functionality for the new `tmt try` command to be used for test development in an interactive session. Supports running tests from the current working directory as well as quick experimenting with selected distro image and provision method. Simple user plan config is included as well.
teemtee · Dec 5, 2023 · 5b7f892 · 5b7f892
1 parent 45cafe6
commit 5b7f892
Show file tree

Hide file tree

Showing 49 changed files with 948 additions and 82 deletions.
diff --git a/docs/overview.rst b/docs/overview.rst
@@ -357,6 +357,10 @@ TMT_PLUGINS
  Path to a directory with additional plugins. Multiple paths
  separated with the ``:`` character can be provided as well.
 
+TMT_CONFIG_DIR
+ Path to an alternative directory with config files. By default
+ ``~/.config/tmt`` is used.
+
 TMT_WORKDIR_ROOT
  Path to root directory containing run workdirs. Defaults to
  ``/var/tmp/tmt``.

diff --git a/docs/releases.rst b/docs/releases.rst
@@ -7,6 +7,14 @@
 tmt-1.30
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
+The new :ref:`tmt try</stories/cli/try>` command provides an
+interactive session which allows to easily run tests and
+experiment with the provisioned guest. The functionality might
+still change. This is the very first proof of concept included in
+the release as a **tech preview** to gather early feedback and
+finalize the outlined design. Give it a :ref:`/stories/cli/try`
+and let us know what you think! :)
+
 The :ref:`/spec/core/contact` key has been moved from the
 :ref:`/spec/tests` specification to the :ref:`/spec/core`
 attributes so now it can be used with plans and stories as well.

diff --git a/pyproject.toml b/pyproject.toml
@@ -226,6 +226,7 @@ ignore = [
  "tmt/lint.py",
  "tmt/queue.py",
  "tmt/templates.py",
+ "tmt/trying.py",
  "tmt/utils.py"
 ]
 

diff --git a/stories/cli/run.fmf b/stories/cli/run.fmf
@@ -37,15 +37,15 @@ story: 'As a user I want to execute tests easily'
  For example use ``execute -h beakerlib`` to execute
  tests using the beakerlib method.
  example: |
- /plans/default:
+ /default/plan:
  summary:
  Default plan when fmf metadata found
  discover:
  how: fmf
  execute:
  how: shell
 
- /plans/default:
+ /default/plan:
  summary:
  Default plan when no fmf metadata around
  discover:

diff --git a/stories/cli/try.fmf b/stories/cli/try.fmf
@@ -0,0 +1,129 @@
+story:
+ As a tester or developer I want an easy and comfortable way to
+ run tests and experiment with various versions of the
+ operating system.
+
+link:
+ - implemented-by: /tmt/trying.py
+ - verified-by: /tests/try
+
+/test:
+ story:
+ As a test developer I want an easy way to run a single
+ test on a freshly provisioned guest.
+
+ description: |
+ One of the most common scenarios is to run a single test
+ in a given environment so that the test developer can
+
+ * verify the newly created test works as expected
+ * debug failures discovered during testing and fix them
+
+ .. versionadded:: 1.30
+
+ example:
+ - |
+ # Run test stored in the current working directory using
+ # the default provision method and image
+ cd tests/core/smoke
+ tmt try
+
+ - |
+ # By default we run in verbose level 3, choose a lower
+ # level if more concise output is enough for you
+ cd tests/core/smoke
+ tmt -v try
+ tmt -vv try
+
+ - |
+ # Run it against the specified distro image
+ cd tests/core/smoke
+ tmt try fedora-37
+
+ - |
+ # Run it using given provision method and image
+ cd tests/core/smoke
+ tmt try fedora:38@container
+
+ - |
+ # Use custom filter to select tests instead of the current
+ # working directory
+ cd tests/core/smoke
+ tmt try --test feature
+
+ - |
+ # Select plan from the repository instead of default
+ cd tests/core/smoke
+ tmt try --plan plan
+
+/guest:
+ story:
+ As a user I want to quickly experiment with a freshly
+ provisioned guest.
+
+ description: |
+ Sometimes it's handy to be able to quickly spawn a new
+ clean guest and start experimeting in the sand box. Run
+ the ``tmt try`` command in a directory without tests or
+ explicitly ask for interactive shell with ``--login``.
+
+ .. versionadded:: 1.30
+
+ example:
+ - |
+ # Provision default distro, log into it for experimenting
+ # (make sure there are no tests in the current directory)
+ tmt try
+
+ - |
+ # Provision desired distro, log into it for experimenting
+ # (make sure there are no tests in the current directory)
+ tmt try rhel-9
+
+ - |
+ # Explicitly ask for login only if there are tests around
+ tmt try --login
+ tmt try -l
+
+ - |
+ # Do nothing, just provision the guest and ask what to do
+ tmt try --ask
+ tmt try -a
+
+/config:
+ story:
+ As a tester I usually run tests using my preferred
+ provision method and I don't want to write it on the
+ command line again and again.
+
+ description: |
+ Define your custom user plan to be used by default when
+ using the ``tmt try`` command and store it under the
+ config directory ``~/.config/tmt``. All plans starting
+ with ``/user/plan`` will be used for experimenting.
+
+ Note that the config directory is a tmt :ref:`Tree<tree>`
+ so don't forget to initialize it. You can copy & paste the
+ following commands to get quickly started:
+
+ .. code-block:: shell
+
+ mkdir -p ~/.config/tmt
+ cd ~/.config/tmt
+ tmt init
+ tmt plan create --template full /user/plan
+
+ .. versionadded:: 1.30
+
+ example: |
+ # ~/.config/tmt/user/plan.fmf
+ discover:
+ how: fmf
+ provision:
+ how: container
+ image: "fedora:fresh"
+ prepare:
+ how: install
+ package: vim
+ execute:
+ how: tmt
diff --git a/tests/discover/distgit.sh b/tests/discover/distgit.sh
@@ -134,8 +134,8 @@ done
  ) > $MOCK_SOURCES_FILENAME
 
  WORKDIR=/var/tmp/tmt/XXX
- WORKDIR_SOURCE=$WORKDIR/plans/default/discover/default-0/source
- WORKDIR_TESTS=$WORKDIR/plans/default/discover/default-0/tests
+ WORKDIR_SOURCE=$WORKDIR/default/plan/discover/default-0/source
+ WORKDIR_TESTS=$WORKDIR/default/plan/discover/default-0/tests
 
  rlRun -s "tmt run --keep --id $WORKDIR --scratch plans --default \
  discover -vvv -ddd --how fmf --dist-git-source \
@@ -196,8 +196,8 @@ done
  echo "simple-1.tgz" > $MOCK_SOURCES_FILENAME
 
  WORKDIR=/var/tmp/tmt/XXX
- WORKDIR_SOURCE=$WORKDIR/plans/default/discover/default-0/source
- WORKDIR_TESTS=$WORKDIR/plans/default/discover/default-0/tests
+ WORKDIR_SOURCE=$WORKDIR/default/plan/discover/default-0/source
+ WORKDIR_TESTS=$WORKDIR/default/plan/discover/default-0/tests
 
  rlRun -s 'tmt run --id $WORKDIR --scratch plans --default \
  discover -vvv -ddd --how fmf --dist-git-source \
@@ -226,8 +226,8 @@ done
  echo "simple-1.tgz" > $MOCK_SOURCES_FILENAME
 
  WORKDIR=/var/tmp/tmt/XXX
- WORKDIR_SOURCE=$WORKDIR/plans/default/discover/default-0/source
- WORKDIR_TESTS=$WORKDIR/plans/default/discover/default-0/tests
+ WORKDIR_SOURCE=$WORKDIR/default/plan/discover/default-0/source
+ WORKDIR_TESTS=$WORKDIR/default/plan/discover/default-0/tests
 
  rlRun -s 'tmt run --id $WORKDIR --scratch plans --default \
  discover -v --how fmf --dist-git-source \
@@ -255,8 +255,8 @@ done
  echo "simple-1.tgz renamed_simple.tgz" > $MOCK_SOURCES_FILENAME
 
  WORKDIR=/var/tmp/tmt/XXX
- WORKDIR_SOURCE=$WORKDIR/plans/default/discover/default-0/source
- WORKDIR_TESTS=$WORKDIR/plans/default/discover/default-0/tests
+ WORKDIR_SOURCE=$WORKDIR/default/plan/discover/default-0/source
+ WORKDIR_TESTS=$WORKDIR/default/plan/discover/default-0/tests
 
  rlRun -s 'tmt run --id $WORKDIR --scratch plans --default \
  discover -v --how fmf --dist-git-source \
@@ -284,8 +284,8 @@ done
  echo "simple-1.tgz simple.tgz" > $MOCK_SOURCES_FILENAME
 
  WORKDIR=/var/tmp/tmt/XXX
- WORKDIR_SOURCE=$WORKDIR/plans/default/discover/default-0/source
- WORKDIR_TESTS=$WORKDIR/plans/default/discover/default-0/tests
+ WORKDIR_SOURCE=$WORKDIR/default/plan/discover/default-0/source
+ WORKDIR_TESTS=$WORKDIR/default/plan/discover/default-0/tests
 
  rlRun -s 'tmt run --id $WORKDIR --scratch plans --default \
  discover -v --how fmf --dist-git-source \
@@ -369,8 +369,8 @@ done
  rlRun "tmt init"
 
  WORKDIR=/var/tmp/tmt/XXX
- WORKDIR_SOURCE=$WORKDIR/plans/default/discover/default-0/source
- WORKDIR_TESTS=$WORKDIR/plans/default/discover/default-0/tests
+ WORKDIR_SOURCE=$WORKDIR/default/plan/discover/default-0/source
+ WORKDIR_TESTS=$WORKDIR/default/plan/discover/default-0/tests
 
  rlRun -s "tmt run --keep --id $WORKDIR --scratch plans --default \
  discover -vvv -ddd --how fmf --dist-git-source \

diff --git a/tests/discover/force.sh b/tests/discover/force.sh
@@ -10,7 +10,7 @@ rlJournalStart
  rlPhaseEnd
 
  plan="plans --default"
- test_yaml="$workdir/run/plans/default/discover/tests.yaml"
+ test_yaml="$workdir/run/default/plan/discover/tests.yaml"
 
  rlPhaseStartTest
  # Fresh run with no required packages

diff --git a/tests/discover/serial-number.sh b/tests/discover/serial-number.sh
@@ -54,7 +54,7 @@ rlJournalStart
  rlPhaseStartTest "Single '/' test"
  rlRun 'pushd serial-number-root-test'
  rlRun "tmt -vv run --scratch --id $workdir discover -h fmf"
- rlRun -s "yq -er '.[] | \"\\(.name) \\(.\"serial-number\")\"' $workdir/plans/default/discover/tests.yaml"
+ rlRun -s "yq -er '.[] | \"\\(.name) \\(.\"serial-number\")\"' $workdir/default/plan/discover/tests.yaml"
  rlAssertGrep "/ 1" $rlRun_LOG
  rlRun 'popd'
  rlPhaseEnd

diff --git a/tests/execute/filesubmit/test.sh b/tests/execute/filesubmit/test.sh
@@ -9,7 +9,7 @@ rlJournalStart
  rlPhaseEnd
 
  # would be set by TMT_TEST_DATA
- tmt_test_data="plans/default/execute/data/guest/default-0/default-1/data"
+ tmt_test_data="default/plan/execute/data/guest/default-0/default-1/data"
 
  rlPhaseStartTest
  rlRun "tmt run -vfi $tmp -a provision -h container"

diff --git a/tests/execute/result/basic.sh b/tests/execute/result/basic.sh
@@ -91,7 +91,7 @@ EOF
 
  rlPhaseStartTest "Verify fmf context lands in results"
  rlRun -s "tmt -c foo=bar run --id ${run} --scratch -a provision --how local test -n '/pass'"
- rlAssertEquals "Context is stored in result" "$(yq -r ".[] | .context | to_entries[] | \"\\(.key)=\\(.value[])\"" $run/plans/default/execute/results.yaml)" "foo=bar"
+ rlAssertEquals "Context is stored in result" "$(yq -r ".[] | .context | to_entries[] | \"\\(.key)=\\(.value[])\"" $run/default/plan/execute/results.yaml)" "foo=bar"
  rlPhaseEnd
 
  rlPhaseStartCleanup

diff --git a/tests/execute/result/custom.sh b/tests/execute/result/custom.sh
@@ -27,7 +27,7 @@ rlJournalStart
  rlAssertExists "$(sed -n 's/ *another_log: \(.\+\)/\1/p' $rlRun_LOG)"
  rlAssertExists "$(sed -n 's/ *slash_log: \(.\+\)/\1/p' $rlRun_LOG)"
 
- rlRun -s "yq -er '.[] | \"\\(.name) \\(.\"serial-number\") \\(.result) \\(.guest.name)\"' $run/plans/default/execute/results.yaml"
+ rlRun -s "yq -er '.[] | \"\\(.name) \\(.\"serial-number\") \\(.result) \\(.guest.name)\"' $run/default/plan/execute/results.yaml"
  rlAssertGrep "/test/custom-results/test/passing 1 pass default-0" $rlRun_LOG
  rlAssertGrep "/test/custom-results/test/failing 1 fail default-0" $rlRun_LOG
  rlAssertGrep "/test/custom-results 1 pass default-0" $rlRun_LOG
@@ -51,7 +51,7 @@ rlJournalStart
  testName="/test/missing-custom-results"
  rlPhaseStartTest "${testName}"
  rlRun -s "${tmt_command} ${testName} 2>&1 >/dev/null" 2 "Test does not provide 'results.yaml' file"
- rlAssertGrep "custom results file not found in '/tmp/.*/plans/default/execute/data/guest/default-0/test/missing-custom-results-1/data" $rlRun_LOG
+ rlAssertGrep "custom results file not found in '/tmp/.*/default/plan/execute/data/guest/default-0/test/missing-custom-results-1/data" $rlRun_LOG
  rlPhaseEnd
 
  testName="/test/empty-custom-results-file"

diff --git a/tests/full/test.sh b/tests/full/test.sh
@@ -139,7 +139,7 @@ rlJournalStart
 
 
  rlRun "su -l -c 'tmt run --id $CONNECT_RUN plans --default provision -h virtual' $USER"
- CONNECT_TO=$CONNECT_RUN/plans/default/provision/guests.yaml
+ CONNECT_TO=$CONNECT_RUN/default/plan/provision/guests.yaml
  rlAssertExists $CONNECT_TO
 
  # Create new plans/provision/connect.fmf

diff --git a/tests/login/when.sh b/tests/login/when.sh
@@ -11,15 +11,15 @@ rlJournalStart
  rlPhaseEnd
 
  rlPhaseStartTest "Skipped"
- rlRun "$tmt true login -w fail -c true 2>&1 >/dev/null | tee output"
- rlAssertGrep "Skipping interactive" "output"
- rlAssertNotGrep "Starting interactive" "output"
+ rlRun -s "$tmt true login -w fail -c true"
+ rlAssertGrep "Skipping interactive" $rlRun_LOG
+ rlAssertNotGrep "Starting interactive" $rlRun_LOG
  rlPhaseEnd
 
  rlPhaseStartTest "Enabled"
- rlRun "$tmt false login -w fail -c true 2>&1 >/dev/null | tee output" 1
- rlAssertNotGrep "Skipping interactive" "output"
- rlAssertGrep "Starting interactive" "output"
+ rlRun -s "$tmt false login -w fail -c true" 1
+ rlAssertNotGrep "Skipping interactive" $rlRun_LOG
+ rlAssertGrep "Starting interactive" $rlRun_LOG
  rlPhaseEnd
 
  rlPhaseStartCleanup

diff --git a/tests/plan/import/basic.sh b/tests/plan/import/basic.sh
@@ -68,7 +68,7 @@ rlJournalStart
  rlAssertGrep "/plans/minimal" $rlRun_LOG
  rlAssertGrep "/lint/tests" $rlRun_LOG
  rlAssertGrep "/lint/plans" $rlRun_LOG
- rlAssertNotGrep "/plans/default" $rlRun_LOG
+ rlAssertNotGrep "/default/plan" $rlRun_LOG
  # logging import plan details in verbose mode
  rlAssertGrep "import url: https:/teemtee/tmt" $rlRun_LOG
  rlAssertGrep "import ref: 1.16.0" $rlRun_LOG

diff --git a/tests/run/default/test.sh b/tests/run/default/test.sh
@@ -26,7 +26,7 @@ rlJournalStart
  rlPhaseStartTest "Explicit default"
  rlRun "tmt plan create -t mini plans/plan"
  rlRun "tmt run $options plan --default 2>&1 >/dev/null | tee output"
- rlAssertGrep "/plans/default" "output"
+ rlAssertGrep "/default/plan" "output"
  rlAssertNotgrep "/plans/plan" "output"
  rlPhaseEnd
 

diff --git a/tests/run/follow/data/main.fmf b/tests/run/follow/data/main.fmf
@@ -7,4 +7,4 @@
  how: tmt
 
 /test:
- test: for i in {00..05}; do echo step-$i; sleep 1; done
+ test: for i in {00..10}; do echo step-$i; sleep 1; done
diff --git a/tests/run/follow/test.sh b/tests/run/follow/test.sh
@@ -10,11 +10,11 @@ rlJournalStart
  rlPhaseStartTest
  rlRun "tmt run --id $run &>/dev/null &" 0 "Start a tmt run in background"
  rlRun "sleep 1" 0 "Ignore logging during the first second"
- rlRun -s "timeout -s INT 3 tmt run --id $run --follow" 124 "Follow logs for 3 seconds"
+ rlRun -s "timeout -s INT 5 tmt run --id $run --follow" 124 "Follow logs for 5 seconds"
  rlAssertGrep "step-01" $rlRun_LOG
  rlAssertGrep "step-02" $rlRun_LOG
- rlAssertNotGrep "step-04" $rlRun_LOG
- rlAssertNotGrep "step-05" $rlRun_LOG
+ rlAssertNotGrep "step-09" $rlRun_LOG
+ rlAssertNotGrep "step-10" $rlRun_LOG
  rlRun "wait $!" 0 "Wait until the run is finished"
  rlPhaseEnd
 

diff --git a/tests/try/config/.fmf/version b/tests/try/config/.fmf/version
@@ -0,0 +1 @@
+1
diff --git a/tests/try/config/user.fmf b/tests/try/config/user.fmf
@@ -0,0 +1,10 @@
+/plan:
+ discover:
+ how: fmf
+ provision:
+ how: local
+ prepare:
+ how: shell
+ script: echo custom-prepare
+ execute:
+ how: tmt
diff --git a/tests/try/data/.fmf/version b/tests/try/data/.fmf/version
@@ -0,0 +1 @@
+1
diff --git a/tests/try/data/keep.exp b/tests/try/data/keep.exp
@@ -0,0 +1,7 @@
+#!/usr/bin/expect -f
+# Select a plan and keep the run
+
+spawn tmt try -p plan
+expect "What do we do next?"
+send -- "k\r"
+expect eof