Node 22.1+ crashes process when a lot of VM processes are created/destroyed in parallel workers

[sheremet-va]

Version

22.1.0

Platform

Darwin MacBook-Air.local 23.5.0 Darwin Kernel Version 23.5.0: Wed May  1 20:16:51 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T8103 arm64

Subsystem

No response

What steps will reproduce the bug?

• Clone https:/vitest-dev/vitest
• Run pnpm install --prefer-offline
• Run pnpm build
• Run cd test/core
• Run pnpm test:vmThreads --run
• See a crash - sometimes it's instant, sometimes it's after a few tests ran

How often does it reproduce? Is there a required condition?

Always

What is the expected behavior? Why is that the expected behavior?

No crash

What do you see instead?

Command failed with exit code 129. with no error.

Additional information

This is most likely an issue with Vitest itself, but I am having a hard time debugging it. The error started happening on 22.1.0, 22.0.0 works as before. The changelog didn't help me.

The issue seems to be with Worker Threads + VM as vmForks pool works fine or at least doesn't crash completely (it uses child_process under the hood instead, but the code for VM is the same) - looks like child_process just exists without crashing the main process, but it also doesn't print anything useful.

[RedYetiDev]

Do you happen to have a repro without any dependencies?

This is most likely an issue with Vitest itself, but I am having a hard time debugging it.

Have you tried reporting it to vitest?

[sheremet-va]

Do you happen to have a repro without any dependencies?

Unfortunately no. The behaviour seems very unreliable so at least some amount of dependencies is required. It seems like different tests just exit the process on different reruns without any errors printed to stdout/stderr.

Have you tried reporting it to vitest?

I am representing Vitest.

---

I have found NODE_DEBUG=* variable, I'll try to minimize the reproduction.

[joyeecheung]

FYI if it's getting exited deliberately by some user-land code via process.exit(), you can use --trace-exit to see what code is calling that (although for child processes, they need to inherit it i.e. the execArgv etc. should not be replaced with something else).

[RedYetiDev]

I haven't attempted to reproduce, but given the report, I've optimistically added these labels.

---

@sheremet-va I saw you pushed vitest-dev/vitest#6448, did this resolve the issue?

[sheremet-va]

@sheremet-va I saw you pushed vitest-dev/vitest#6448, did this resolve the issue?

No, it is a separate issue.

FYI if it's getting exited deliberately by some user-land code via process.exit(), you can use --trace-exit to see what code is calling that (although for child processes, they need to inherit it i.e. the execArgv etc. should not be replaced with something else).

I see that it hits some specific module and panics, I don't think it's the process.exit call. Haven't figured out the module itself yet, but it's loaded by vm.runInContext. I will investigate more tomorrow. Just child_process or worker_thread that loads modules normally are not affected.

[sheremet-va]

I wasn't able to create a small reproduction, but I've made it easier to debug: vitest-dev/vitest#6452 You can see in the console the test that failed and it creates a file with stdout logs.

I am still not sure what is happening and why Node crashes with a custom module runner based on VM context. And just a reminder - it started happening in Node 22.1, 22.0 works without an issue.