tmt doesn't fail on invalid YAML keys

[abitrolly]

The following main.fmf definition is invalid, but it doesn't stop ./test.sh from executing and failing.

summary: Test `tmt --help` output
test: ./test.sh
requires: tmt

test.sh is standard from template.

#!/bin/sh -eux

tmp=$(mktemp)
tmt --help > "$tmp"
grep -C3 'Test Management Tool' "$tmp"
rm "$tmp"

And here is the output.

$ tmt run --all provision --how container
/var/tmp/tmt/run-028

/default/plan
    discover
        how: fmf
        directory: /data/s/gitlab-ai-gateway/xxx
                warn: /test01: - 'requires' does not match any of the regexes: '^extra-'
        summary: 1 test selected
    provision
        queued provision.provision task #1: default-0
        
        provision.provision task #1: default-0
        how: container
        multihost name: default-0
        arch: x86_64
        distro: Fedora Linux 40 (Container Image)
    
        summary: 1 guest provisioned
    prepare
        queued push task #1: push to default-0
        
        push task #1: push to default-0
    
        queued prepare task #1: requires on default-0
        
        prepare task #1: requires on default-0
        how: install
        summary: Install required packages
        name: requires
        where: default-0
        package: /usr/bin/flock
    
        queued pull task #1: pull from default-0
        
        pull task #1: pull from default-0
    
        summary: 1 preparation applied
    execute
        queued execute task #1: default-0 on default-0
        
        execute task #1: default-0 on default-0
        how: tmt
        progress:              
    
        summary: 1 test executed
    report
        how: display
        summary: 1 error
    finish
    
        container: stopped
        container: removed
        container: network removed
        summary: 0 tasks completed

total: 1 error

[LecrisUT]

I guess you are referring to the json schema validation. It is tricky because tmt does not yet expose json schemas for plugins which would make even the example plugin un-runnable. #2946 can solve that and maybe after that the json schema validation can be promoted to required.