Remove support for junit-xml, use Jinja2 instead

[seberm]

This PR removes support for the deprecated junit-xml library and uses the Jinja2 to generate JUnit XML instead.

It also prepares support for various junit flavors (--flavor) and adds the possibility to specify user-defined JUnit Jinja templates using the --template-path and --flavor=custom options. These options are mutually exclusive.

The current implementation requires that custom templates must be valid XML files. If we want to allow users use non-XML custom templates, we would probably go with something like template report plugin for results.

/plans/example<br>
    report
        how: junit
        warn: The 'custom' JUnit flavor is used, you are solely responsible for the validity of the XML schema.
        warn: The pretty print is always disabled for 'custom' JUnit flavor
        warn: The generated XML output is not valid XML file or it is not valid against the XSD schema.
        warn: <string>:1:1:FATAL:PARSER:ERR_DOCUMENT_EMPTY: Start tag expected, '<' not found
        warn: Start tag expected, '<' not found, line 1, column 1 (<string>, line 1)

plan failed

The exception was caused by 1 earlier exceptions

Cause number 1:

    The generated XML output is not a valid XML file.

The implementation of the polarion flavor and its support in the Polarion report plugin will be addressed in a separate PR. In this PR, the Polarion report plugin just calls the junit.make_junit_xml function in nearly the same way as before, aiming to keep the Polarion xunit.xml file unchanged.

TODOs:

• Probably divide the changes into multiple PRs (polarion and junit separately)
• Define XSD JUnit for default flavor and validate the XML output against this schema. Warn user if there are any problems.
• [ ] Possibility to specify template variables directly from cmdline as key-values (e.g. --var <key> <value>) for a custom flavor / user-defined template? - This could be a part of "template" report plugin in the future.
• The difference between junit-xml generated files and the new junit files is that the quotes " are not escaped to " inside of tag data (they are correctly escaped inside attributes). According to my knowledge, this should be OK and the XML is also valid.
• Add tests
• Add test which checks there is no schema/XML warning in the tmt logs when running non-custom flavor The generated XML output is not a valid XML file or it is not valid against the XSD schema.
• The duration should be probably converted to float and propagated to XML time="<val>" attributes as float, not an integer. If this gets changed, is it possible to enforce float attributes in XSD schema? - tmt works with seconds, let's just convert them to floats using a jinja filter.

Pull Request Checklist

• implement the feature
• write the documentation
• extend the test coverage
• update the specification
• adjust plugin docstring
• modify the json schema
• mention the version
• include a release note

[seberm]

Hello @happz , thanks for your early hints. It's still very early for a review. Many code parts will probably change, especially due to the Polarion report plugin.

[LecrisUT]

Is it possible to validate the junit.xml files created?

[seberm]

Is it possible to validate the junit.xml files created?

Currently, this functionality is not supported, and the XML files are not validated against any XSD schema, even when using junit-xml.

From python-junit-xml README.rst:

As there is no definitive Jenkins JUnit XSD that I could find, the XML documents created by this module support a schema based on Google searches and the Jenkins JUnit XML reader source code. File a bug if something doesn't work like you expect it to. For Bamboo situation is the same.

The junit-xml repository currently includes only Python unit tests, while TMT tests have very basic XML checks (polarion, default junit). However, there is no existing code that validates the entire XML file against an XSD schema.

I plan to create simple XSD schemas for each JUnit flavor supported by TMT and add tests validating the generated JUnit files against these schemas.

Additionally, it would be a good idea to have TMT dynamically validate the schema of the generated JUnit files. If a file does not conform to its respective schema, the report plugin should fail, unless e.g. the --force argument is used, which would allow it to continue despite the schema mismatch.

[happz]

I plan to create simple XSD schemas for each JUnit flavor supported by TMT and add tests validating the generated JUnit files against these schemas.

Additionally, it would be a good idea to have TMT dynamically validate the schema of the generated JUnit files. If a file does not conform to its respective schema, the report plugin should fail, unless e.g. the --force argument is used, which would allow it to continue despite the schema mismatch.

+1. Check out #3153, it does similar validation when consuming results.yaml, but it's not a blocker, only a warning sign, at least for now. That said, these files are not created by tmt - junit files would be. Failing this particular validation would mean the plugin is creating a broken output, which would be a bug. I'm not sure --force would be needed - if users wanted to overcome this bug temporarily, they should be able to modify either the schema or the template, and --force would sort of normalize the behavior in my eyes.

[thrix]

LGTM