How to test GUI apps?

[abitrolly]

There is a screen recorder called [byzanz] In Fedora. It doesn't work at least since F34.

✗ byzanz-record            
usage: byzanz-record [OPTIONS] filename
       byzanz-record --help
✗ byzanz-record test.gif
Given area is not inside desktop.

When new version of Fedora is released, Bugzilla sends me a reminder that the bug expired. Instead of manually testing the software after each release, I decided to try tmt, which is capable to automatically mark packages as broken for release if tests fail.

I created the dir layout with tmt init, then created test with tmt init --template mini, and then edited the plan install the package and run the same commands I did above.

✗ cat plans/basic.fmf 
summary: Basic smoke test for byzanz
prepare:
  - name: install byzanz
    how: install
    package: byzanz
execute:
  script:
    - byzanz-record
    - byzanz-record test.gif

tmt run correctly selected Fedora 36 for the test run.

✗ tmt run                   
/var/tmp/tmt/run-050

/plans/basic
    discover
        how: shell
        summary: 2 tests selected
    provision
        how: virtual
        image: fedora
        user: root
        memory: 2048 MB
        disk: 10 GB
        arch: x86_64
        progress: booting...
        distro: Fedora Linux 36 (Cloud Edition)
        summary: 1 guest provisioned
    prepare
        how: install
        name: install byzanz
        package: byzanz
    
        summary: 1 preparation applied
    execute
        how: tmt
        summary: 2 tests executed 
    report
        how: display
        summary: 2 tests failed
    finish
        guest: stopped
        guest: removed
        summary: 0 tasks completed

total: 2 tests failed

The test failed. However, the message in both cases is different than mine.

✗ tmt run --last report -vvv
/var/tmp/tmt/run-050

/plans/basic
    report
        how: display
        order: 50
            fail /script-00
                output.txt: /var/tmp/tmt/run-050/plans/basic/execute/data/script-00/output.txt
                content: 
                    Wrong option: Cannot open display: 
                    usage: byzanz-record [OPTIONS] filename
                           byzanz-record --help
            fail /script-01
                output.txt: /var/tmp/tmt/run-050/plans/basic/execute/data/script-01/output.txt
                content: 
                    Wrong option: Cannot open display: 
                    usage: byzanz-record [OPTIONS] filename
                           byzanz-record --help
        summary: 2 tests failed

total: 2 tests failed

I guess something is wrong with the testing setup. So how to test the GUI apps?

[lukaszachy]

Hi @abitrolly , tmt spawns Cloud Edition which has minimal set of packages installed and doesn't run any graphics.
In the old days of Xorg you could use start Xvfb (X server rendering to memory), set DISPLAY variable and run your app.

In Beakerlib are helper functions (you need to install xorg-x11-server-Xvfb as well as beakerlib).

I've tried

source /usr/share/beakerlib/beakerlib.sh
rlJournalStart 
rlPhaseStart
rlPhaseStartTest
rlVirtualXStart foo
export DISPLY=$(rlVirtualXGetDisplay foo)
byzanz-record --help
byzanz-record image

and it recorded blank screen (as nothing else is running) So I guess you need to spawn some other gui app, maybe whole desktop manager?
Also nowdays Wayland is a way to go, so byzanz working in Xvfb might have issues in Wayland.

Fedora QA might/should know much better how to test - https://fedoraproject.org/wiki/QA has mailing lists/irc channel contacts.

[abitrolly]

@lukaszachy why GUI tests are out of scope for Test Management Tool?

[lukaszachy]

Test framework is outside of Test Management Tool - why would tmt bundle/support pytest, cucumber etc?
There is no single way how to test GUI - it is up to tester to create their tests. tmt will help to run the testsuite according to the plan (which can contain setup but it is up to tester to provide such setup).

[abitrolly]

That becomes more and more confusing. I thought that tmt is the test framework, with a feature to execute other test frameworks if necessary. If Python is covered by pytest, and basic tests are processed by handling output from bash, I would expect a cookbook for GUI tests as well. Otherwise I don't see the reason to learn yet another framework.

[lukaszachy]

tmt is not a test framework. tmt is to solve 'test management' with 'install & setup setup system so tests can run there' and report results.

[thrix]

Yeah, I would also not call it a test framework, it actually is intended to wrap around other test frameworks. We use mainly Beakerlib for shell tests here, for which we have some more fine-grained support, but otherwise it should be happy to host any test framework.