Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create pseudo-runners for situations where we can't actually run tests but want to produce a report #1475

Open
CharliePoole opened this issue Sep 9, 2024 · 0 comments

Comments

@CharliePoole
Copy link
Collaborator

BACKGROUND

When the runner or engine encounters a file name for which we can't run tests, we could obviously just throw an exception and terminate. That's acceptable if the invalid file is referenced on the console runner command-line. It's not OK in several other situations:

  1. When a runner with a continuous UI display (GUI or otherwise) needs to get the information about an invalid file and display it. It can't do that if it terminates.
  2. The invalid file may be nested inside a project containing many other files. We may want to run the tests in the valid files and just display an error for the invalid files.

CURRENT HANDLING

We currently use special drivers in the engine to handle this. They all derive from the abstract class NotRunnableFrameworkDriver. Separate subclasses exist for invalid assemblies and assemblies, which we are skipping because they don't contain any tests.

DRAWBACKS OF THE CURRENT APPROACH

  1. Since we use drivers, we need to create an agent process in many cases. This is costly, since these drivers don't actually execute any code in the tests but merely report back a result in the correct format.
  2. Issue Unable to run nunit-console with VSProjectLoader that has a WPF 8.0 project #1468 pointed out to me that this approach doesn't work in all situations. That issue relates to unmanaged executables, which cannot be loaded by any of our agents.

PLANNED APPROACH

I plan to create a set of NotRunnableTestRunners to replace the drivers. The UnmanagedExecutableTestRunner I created for #1468 will serve as a prototype. Each existing NotRunnableFrameworkDriver, will e sysetmatically replaced by a NotRunnableTestRunner.

These runners (or pseudo-runners) will not follow the ProcessModel and DomainUsage settings used by all runnable test files. They will execute immediately, in process. None of the drivers actually examine the files whose names are passed to them because that has already been done when they are activated. That will remain true for the runners.


Suggestions or Comments? @nunit

@CharliePoole CharliePoole added this to the 3.18.2 milestone Sep 9, 2024
@CharliePoole CharliePoole self-assigned this Sep 9, 2024
@CharliePoole CharliePoole modified the milestones: 3.18.2, 3.19.0 Sep 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant