Skip to content

Commit

Permalink
tests: Add pytest and nushell based tests
Browse files Browse the repository at this point in the history
I've been trying to keep this project in "one" programming
language by writing even tests in Rust...but specifically
for our integration tests it's pretty painful not just to
compile them but have to deal with baking them into the base image.

The tmt framework is very GHA like in that it scrapes the
git source tree and copies it into the target environment, which
works really well with scripts.

Now, if you know me you know I am not a fan of dynamic programming
languages like bash and Python. I'm one of those folks that actually
tries to use Rust for things that feel like "scripts" i.e. they're
*mostly* about forking external processes (see the xtask/
crate which uses "xshell").

Some of our testing code is in Rust too. However...there's a giant
tension here because:

- Iteration speed is very important for tests and scripts
- The artifact being an architecture-dependent binary pushes us
  to inject it into container images; having the binary part
  of the bootc image under test conceptually forces us to reprovision
  for each test change, which is super expensive

Most other people when faced with the testing challenge would
just write shell scripts (or Python); that's definitely what tmt
expects people to do.

The podman project has a mix of a "bats" suite which is all
bash based, and a Go-based framework.

The thing is: bash is easy to mess up and has very little ability
to do static analysis. Go (and Python) are very verbose for forking external
processes.

I've been using https://www.nushell.sh/ for my interactive shell
for quite a while; I know just enough to get by day to day
(but honestly sometimes I still type "bash" and run a few things there
 that I know how to express in bash but not nu)

Anyways though, nushell has a lot of desirable properties for
tests (which are basically scripts):

- Architecture independent
- Running an external process requires zero ceremony; it's the
  default!
- But it *is* easy to e.g. scrape JSON from an external binary
  into a rich data structure
- A decently rich standard library

The downside is, it's a new language. And in the end, I'm
not going to say it's the only way to write tests...maybe we
do end up with some more bash. It wouldn't be the end of the world.
But...after playing with this, I definitely like the result.

OK, and after some debate we decided to add Python too, so this
demos a pytest test.

Signed-off-by: Colin Walters <[email protected]>
  • Loading branch information
cgwalters committed Jun 24, 2024
1 parent 4874db3 commit 9a758e3
Show file tree
Hide file tree
Showing 9 changed files with 81 additions and 8 deletions.
8 changes: 8 additions & 0 deletions .github/workflows/python.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
name: Python static analysis
on: [push, pull_request]
jobs:
ruff:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: chartboost/ruff-action@v1
1 change: 1 addition & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ test-tmt:
validate:
cargo fmt
cargo clippy
ruff check
.PHONY: validate

vendor:
Expand Down
22 changes: 20 additions & 2 deletions hack/provision-derived.sh
Original file line number Diff line number Diff line change
@@ -1,9 +1,26 @@
#!/bin/bash
set -xeu
case "$1" in
variant=$1
# I'm a big fan of nushell for interactive use, and I want to support
# using it in our test suite because it's better than bash. First,
# enable EPEL to get it.
. /usr/lib/os-release
if echo $ID_LIKE $ID | grep -q centos; then
dnf config-manager --set-enabled crb
dnf -y install epel-release epel-next-release
fi
# Ensure this is pre-created
mkdir -p -m 0700 /var/roothome
mkdir -p ~/.config/nushell
echo '$env.config = { show_banner: false, }' > ~/.config/nushell/config.nu
touch ~/.config/nushell/env.nu
dnf -y install nu
# And we also add pytest, to support running tests written in Python
dnf -y install python3-pytest
case "$variant" in
tmt)
# tmt wants rsync
dnf -y install cloud-init rsync && dnf clean all
dnf -y install cloud-init rsync
ln -s ../cloud-init.target /usr/lib/systemd/system/default.target.wants
# And tmt wants to write to /usr/local/bin
rm /usr/local -rf && ln -sr /var/usrlocal /usr/local && mkdir -p /var/usrlocal/bin
Expand All @@ -14,3 +31,4 @@ case "$1" in
echo "Unknown variant: $1" exit 1
;;
esac
dnf clean all && rm /var/log/* -rf
18 changes: 12 additions & 6 deletions plans/integration-run.fmf
Original file line number Diff line number Diff line change
@@ -1,12 +1,18 @@
# This tmt test just demonstrates local tmt usage.
# We'll hopefully expand it to do more interesting things in the
# future and unify with the other test plans.
# Run this via `make test-tmt` which will build a container,
# and a disk image from it.
provision:
how: virtual
# Generated by `cargo xtask `
# Generated by make test-tmt
image: file://./target/testvm/disk.qcow2
disk: 20
summary: Basic smoke test
summary: Execute booted tests
execute:
how: tmt
script: bootc status
# There's currently two dynamic test frameworks; python and nushell.
# python is well known and understood. nushell is less well known, but
# is quite nice for running subprocesses and the like while making
# it easy to parse JSON etc.
script: |
set -xeu
pytest tests/booted/*.py
ls tests/booted/*-test-*.nu |sort -n | while read t; do nu $t; done
1 change: 1 addition & 0 deletions tests/booted/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
__pycache__
8 changes: 8 additions & 0 deletions tests/booted/001-test-status.nu
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
use std assert
use tap.nu

tap begin "verify bootc status --json looks sane"

let st = bootc status --json | from json
assert equal $st.apiVersion org.containers.bootc/v1alpha1
tap ok
4 changes: 4 additions & 0 deletions tests/booted/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# Booted tests

These are intended to run via tmt; use e.g.
`make test-tmt`.
12 changes: 12 additions & 0 deletions tests/booted/basic.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# Tests which are read-only/nondestructive

import json
import subprocess

def run(*args):
subprocess.check_call(*args)

def test_bootc_status():
o = subprocess.check_output(["bootc", "status", "--json"])
st = json.loads(o)
assert st['apiVersion'] == 'org.containers.bootc/v1alpha1'
15 changes: 15 additions & 0 deletions tests/booted/tap.nu
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# A simple nushell "library" for the
# "Test anything protocol":
# https://testanything.org/tap-version-14-specification.html
export def begin [description] {
print "TAP version 14"
print $description
}

export def ok [] {
print "ok"
}

export def fail [] {
print "not ok"
}

0 comments on commit 9a758e3

Please sign in to comment.