Improving testing of SDK built-in templates

[vlada-shubina]

Improving template tests in dotnet/installer

Background

One of the most common problems with .NET templates shipped within .NET SDK are the issues which defining grouping metadata: identity, group identity, precedence.
This issue is commonly reproduced only when multiple .NET SDKs are installed.
Invalid definition results in the conflict that is shown only when the template is attempted to be run and the user cannot resolve it.

During delivery of .NET 5 test templates and common templates had this issue.
During delivery of .NET 6 ASP.NET templates and WPF templates had this issue. Issue with item templates in asp.net was discovered post preview 7.

The problem is amplified because the templates are delivered by 5 teams as of now, using different approaches for authoring, infrastructure and testing their templates.
dotnet/installer seems to be the only common point for them.

Proposal

Since the dotnet/installer is the first and last point where the template packages are inserted, create the tests that can catch the issues described above and disallow releasing the build until they are solved.
The tests will be written by template engine team, the maintenance of them may be negotiated between template engine team, dotnet/installer team or template authors.

Technical details

The tests are suggested as the separate job in existing pipeline. Running tests on a single platform is enough - should the issue occur, it is identically reproduced on all the platforms.

The tests will do the following actions:

Preparation steps:

• download and install currently supported SDK versions using dotnet-install scripts (at the moment 3.1, 6.0, 7.0). This can be done as part of the build or from C# code.
• unpack build install on top (artifacts\packages\<configuration>\Shipping\dotnet-sdk-<version>.zip)

Test action:

• run all the available templates for all available frameworks (in case template support framework parameter), without restore - dotnet new <template> --framework <framework> --no-restore.
• check if template instantiated successfully, based on output or/and exit code.

There are 2 ways how to ensure the test matrix:

• prepare test data programmatically

  • run dotnet new list to get all available templates
  • run dotnet new <template> --help to see if template supports --framework and its values
  • prepare test matrix based on discovered data
  • pros: low maintenance costs, however if list of help output changes the test might need adjustments
  • cons: the risk of bugs in logic, the risk of missing the cases, the risk that templates are not supporting all the frameworks they normally should

• prepare test data manually

  • the test matrix to be hardcoded in code
  • when adding new template or supported TFM, the matrix has to be manually adjusted accordingly
  • to verify that the matrix is up-to-date, the sanity test will be written failing if matrix doesn't contain all the results from dotnet new list (and dotnet new <template> --help)
  • pros: eliminated risk of option 1
  • cons: possibly higher maintenance costs comparing to option 1, if list of help output changes the santiy test might need adjustments

Planned number of test cases (maximum): 40 templates x 2 languages (average) x 4 TFMs = 320 cases.
Note: it is a maximum number of tests as not all the templates support all the frameworks.

Predicted test run time: 320 cases * 2 sec = 640 sec ~ 11 minutes
Note: doesn't include build, test preparation steps, test matrix preparation.

Other possible improvements:

• There is a plan to allow the user to remediate the conflict, however it doesn't resolve the root cause.
• Improved template validation is considered, however it doesn't guarantee that the authors will use it and delivery is not confirmed.
• Existing template tests may benefit from matrix generation / sanity tests to ensure that all the templates are being tested.

[YuliiaKovalova]

Sorry, it's a bit vague to me: are you talking about a separate pipeline or adding a new job to an existing one?
If it is about a job I would suggest making it optional.

[vlada-shubina]

Sorry, it's a bit vague to me: are you talking about a separate pipeline or adding a new job to an existing one? If it is about a job I would suggest making it optional.

vlada-shubina/installer#1 indicates that the suggestion is to add a new job to a pipeline.
I would prefer to have it mandatory as insertions of template packages are currently automatic and there will be noone to run an optional pipeline.

[YuliiaKovalova]

Hi @ladipro , @JanKrivanek ,

Could you review this idea paper?
Thank you!

[ladipro]

@YuliiaKovalova, looks great. No strong opinion on programmatic vs. manual test matrix generation. Probably better to start with automatic and add manual adjustments as issues are discovered.

Would it be possible to include concrete examples of the issues described in the Background section? Have there been cases where a template was successfully run (in terms of output and exit code) but the generated artifacts had a bug?

[vlada-shubina]

@YuliiaKovalova, looks great. No strong opinion on programmatic vs. manual test matrix generation. Probably better to start with automatic and add manual adjustments as issues are discovered.

Would it be possible to include concrete examples of the issues described in the Background section? Have there been cases where a template was successfully run (in terms of output and exit code) but the generated artifacts had a bug?

Concrete examples from .NET 7
dotnet/wpf#7204 (comment)
dotnet/aspnetcore#44479
dotnet/aspnetcore#44095
dotnet/test-templates#118

.NET 6:
#4296
#4299
dotnet/test-templates#118

[YuliiaKovalova]

Thank you Vlada.
I added this info in doc on the sharepoint.

[vlada-shubina]

due to planning reasons will be delivered in #6026 instead.