Incorrect unit test paths in side panel when using both -s and -t parameters

[am1ter]

Type: Bug

Behaviour

Expected vs. Actual

Expected: During unit test discovery with parameters -s and -t, all tests are found. The path to every file is correct, and all tests are available to run from the side panel. The Go to test button works properly.

Actual: During unit test discovery, all tests are found and able to run, but the paths to all tests are incorrect. In every path, there is an additional folder that doesn't exist.

Steps to reproduce:

1. Create the project with the following file hierarchy:

├── app
│   ├── src
│   │   ├── __init__.py
│   │   └── main.py
│   └── tests
│       ├── __init__.py
│       └── unit
│           ├── __init__.py
│           └── test_main.py
└── docs

2. Add the following code to the files main.py and test_main.py.

File main.py:

class Main:
    pass

File main.py:

import unittest

from src.main import Main

class TestMain(unittest.TestCase):
    def test_main(self):
        Main()

3. Configure .vscode/settings.json

{
    "python.testing.unittestArgs": [
        "-v",
        "-s",
        "./app/tests",
        "-p",
        "test_*.py",
        "-t",
        "./app",
    ],
    "python.testing.unittestEnabled": true,
}

4. Look at the side panel. It will look like:

├── tests
│   ├── tests
│   │   ├── unit
│   │   │   ├── test_main.py

As you see there are 2 folders tests in the side panel hierarchy, but in fact, there is only one folder named test. If you try to run the test test_main.py, it will run okay. However, if you try to press the Go to the test button, you will get an error: The editor could not be opened because the file was not found.

5. Meanwhile, the console unit discovery command with exact the same settings works properly:

python -m unittest discover -v -s ./app/tests -p test_*.py -t ./app
Ran 1 test in 0.000s
OK

Diagnostic data

• Python version (& distribution if applicable, e.g. Anaconda): 3.11.2
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Venv
• Value of the python.languageServer setting: Default

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

2023-05-18 12:23:36.915 [info] > ./.venv/bin/python ~/.vscode-server/extensions/ms-python.python-2023.8.0/pythonFiles/testing_tools/unittest_discovery.py ./app/tests test_*.py ./app
2023-05-18 12:23:36.915 [info] cwd: .

User Settings

languageServer: "Pylance"

testing
• unittestArgs: [
        "-v",
        "-s",
        "./app/tests",
        "-p",
        "test_*.py",
        "-t",
        "./app",
    ]
• unittestEnabled: true

Extension version: 2023.8.0
VS Code version: Code 1.78.2 (b3e4e68a0bc097f0ae7907b217c1119af9e03435, 2023-05-10T14:39:26.248Z)
OS version: Windows_NT x64 10.0.22621
Modes:
Sandboxed: No
Remote OS version: Linux x64 5.15.90.1-microsoft-standard-WSL2
Remote OS version: Linux x64 5.15.90.1-microsoft-standard-WSL2

System Info

Item	Value
CPUs	AMD Ryzen 7 6800H with Radeon Graphics (16 x 3194)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: disabled_off direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: enabled
Load (avg)	undefined
Memory (System)	13.69GB (1.29GB free)
Process Argv	--crash-reporter-id 66d79bf3-db91-4df3-a9cb-2b8b3d209253
Screen Reader	no
VM	0%

Item	Value
Remote	WSL: Ubuntu
OS	Linux x64 5.15.90.1-microsoft-standard-WSL2
CPUs	AMD Ryzen 7 6800H with Radeon Graphics (16 x 3193)
Memory (System)	6.62GB (3.10GB free)
VM	0%

Item	Value
Remote	WSL: Ubuntu
OS	Linux x64 5.15.90.1-microsoft-standard-WSL2
CPUs	AMD Ryzen 7 6800H with Radeon Graphics (16 x 3193)
Memory (System)	6.62GB (3.10GB free)
VM	0%

A/B Experiments

vsliv368:30146709
vsreu685:30147344
python383:30185418
vspor879:30202332
vspor708:30202333
vspor363:30204092
vswsl492cf:30256860
vslsvsres303:30308271
vserr242cf:30382550
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
vscorecescf:30445987
vscod805cf:30301675
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
cmake_vspar411:30581797
vsaa593:30376534
pythonvs932:30410667
cppdebug:30492333
vsclangdf:30486550
c4g48928:30535728
dsvsc012:30540252
pynewext54:30695312
azure-dev_surveyone:30548225
vscccc:30610679
2e4cg342:30602488
pyind779:30671433
f6dab269:30613381
pythonsymbol12:30671437
showlangstatbar:30737416
azdwalk:30721579
pythonms35:30701012
pythonfmttext:30731395
pythoncmv:30739700
fixhidewlkth:30730051
hidesbindicator:30730055
pythongtdpath:30739704

[eleanorjboyd]

Hi! Yes, I am definitely able to replicate this. We are working on a rewrite of how testing works in vscode and I just tested and it is a problem in our future infrastructure too. Thank you for catching this!

The problem occurs because the arg -s ./app/tests/ which makes the root node of the test search tests. The path to the actual test is something like /tests/unit/test_actual.py when it is discovered by unittest. This path is split and for each folder a test node is created to be displayed [tests, unit, test_actual]. This created the double tests node since it is both the root and in the test path.

Another example is changing it to -s ./app/tests/unit/. When this is run, it layers even worse:

@karthiknadig which is the best way to display tests in the case that the test directory path is not the root of the workspace?

A. generate up to the root of the workspace so the test UI would be
workspace/app/tests/unit/test_actual.py
B. generate up to the directory path as displayed in the test args
unit/test_actual.py if -s ./app/tests/unit/

Thanks!

[karthiknadig]

A. is what i would prefer. Think a case where there are multiple test directories. (A) would display that better.

[andreamah]

Are there minimal repro steps to verify this?