Authoring tools: templates & template packages validation

[vlada-shubina]

As a template author, I want a way to validate the templates, so that I can validate template before the publish.
Audience: template authors

Background

The template configuration requires manual editing in of JSON configuration which is source of authoring mistakes.
The configuration involves a lot of conditions, which are not validated in any way until the template is being generated.
When grouping the templates the configuration is manually, and requires careful combination of the fields across multiple JSON files. The grouping errors are shown only when the template is being run (not when the package containing it is installed)

Planned validation checks:

(note: though some of the issues are closed, their comments contain suggested validation check)

• Blacklist top level commands from allowed template names/shortnames
• Flag optional parameters without default value
• Flag parameters that are prefixes of others (this can create ambiguous situations when parsing expressions via tries)
• All forms applied to source name provide different results (warning, see Invalid sourceName transformation in SLN files #1920)
• All forms applied to symbol provide different results
• Simplify NuGet package delivery and validation for template authors (via MSBuild Targets) #3828
• Validating precedence in template groups #4268
• Generate JSON schema programatically as part of dotnet/templating and dotnet/sdk build #5561 (nth)
• Improve localization model validation #3254
• Disallow using forward slashes in symbol names #3127
• --package template parameter has changed to --param:package in .NET 5 #5198
• Improve documentation for sourceName. #5671 - the template doesn't define name parameter/symbol
• Implement validation of Template Choice Parameter values #4650

Justification

• decrease the number of issues when authoring 1st party templates and detecting them post PR/even releasing them
• improve easiness of authoring and quality of 3rd party templates; detect potential issues prior to testing
• improve UX for final users by not allowing to install invalid templates / warning about possible issues

Technical details

We plan to expose 2 ways:

• via MSBuild tasks, see: Simplify NuGet package delivery and validation for template authors (via MSBuild Targets) #3828
• via authoring tools CLI
  Both MSBuild task and CLI tool already exist.

As part of this task, the validation in template engine and dotnet new should be improved as well.

• when reading the template, the generator should ensure it has a minimum required set of information in order to proceed. The JSON parsing errors and basic errors and warnings should be exposed.
• when installing the template, the conflicts should be detected
• the host should be able to inject the checks for host.json file.
• the way to present the errors should be potentially implemented in host itself

Feedback from @sayedihashimi

• Templates shouldn’t have to be installed in order to be validated
  • More specifically be able to pass in a folder path or a path to .nupkg file
• The tool should accept multiple values for folder path/path to .nupkg
  • For authors that have developed several templates this would be helpful
• After running the tool it should return a non-zero exit code if the validation is not successful
  • This will enable users to script the validate step, and be able to fail the CI build if so
• Error and warning messages should follow the MSBuild error/warning format so that errors/warnings should be auto-detected if a user executes the tool via an MSBuild file.
  • The MSBuild team may know of any formal docs.
  • MSBuild book

[KathleenDollard]

How does this fit in with @sayedihashimi 's tool?

[KathleenDollard]

One of the things we should validate for is the use of parens around complex conditions as discussed in #2209

[sayedihashimi]

How does this fit in with @sayedihashimi 's tool?

I think the idea is to have some similar validation, but it's built into dotnet new instead of a stand alone tool.

The tool should accept multiple values for folder path/path to .nupkg

This is good, but in addition to this I think that when a folder path is specified, the tool should search the folder recursively for all templates. If a user is authoring a template pack (i.e. AWS/Uno/etc) they can pass the root folder to the validator and then it will discover all the templates that the folder contains.

[jzabroski]

I am cool with your thought process, @KathleenDollard . Seems worth the story points vs the other discussion.

[vlada-shubina]

The following was already done:

• Component-based template validation PoC  #5838 - validation infrastructure was implemented. Validation checks can be inserted via components (ITemplateValidator and ITemplateValidatorFactory implementations). Existing checks were reimplemented in this infrastructure. ITemplateValidator and ITemplateValidatorFactory are currently internal but ready to be made public once the feature is fully ready for release.
• validate CLI command #5839 - validation was added to authoring tools CLI (via validate command) with no additional checks at the moment. It is possible to add additional checks by implementing ITemplateValidator and ITemplateValidatorFactory interfaces, adding them as additional component when loading host.
• MSBuild ValidateTemplates task PoC #5840 - validation was added as MSBuild task (Validate) with no additional checks at the moment.

The following tasks remain:

• implementing additional validation checks (see planned checks list in issue description)
• making interfaces public to allow extensibility by tools/hosts
• implementing checks in hosts, in particular for host.json configuration
• updating documentation and authoring tools guidelines.