Repository navigation
src: fix hang when joining workers in process.exit() - #66171
manzoorwanijk wants to merge 2 commits into
Conversation
|
Welcome to Node.js, and thank you for your first contribution! Before review, please take a moment to read:
Please make sure every commit is signed off. For a first pull request, GitHub Actions require collaborator approval and Jenkins CI must be started by a collaborator or triager, so an initial wait is normal. |
|
I have a question: Will this fix be backported to v24 when/if it lands? |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #66171 +/- ##
==========================================
- Coverage 92.78% 90.42% -2.36%
==========================================
Files 422 791 +369
Lines 193692 276617 +82925
Branches 29881 53126 +23245
==========================================
+ Hits 179714 250135 +70421
- Misses 13650 16871 +3221
- Partials 328 9611 +9283
🚀 New features to boost your workflow:
|
…13 of 30 against 0 of 30 The 28 September list, item 15: find the mechanism, add no timeout. Reproduced, then read off the stuck process: its main thread waits in uv_thread_join under node::DefaultProcessExitHandler, and the one Node thread left sleeps on a condition variable inside V8 — nodejs/node#54918 and #64274, process.exit() joining platform workers while a V8 background job is parked waiting for a GC the exiting thread never runs. The upstream fix (nodejs/node#66171) is unmerged; this machine runs Node 24.12.0. Separated by treatment rather than argued: 30 alternating pairs under the same launch, the proof as shipped hung 13 times after printing its pass, and the same file ending by process.exitCode hung 0 times. The class is 289 process.exit( calls in 147 scripts plus the engine host entries; converting them is its own unit (a script relying on process.exit to cut an open handle would trade this hang for another), recorded with the shape it should take. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
8da904b to
8816d66
Compare
…ent compilers On Node 24 `process.exit` can deadlock joining V8's platform workers (a concurrent Sparkplug or Maglev compile waits for a GC the main thread never runs), so a run that got Ctrl-C or `sandcastle stop` could write its end and then never exit (nodejs/node#66171, still open). `exitOnSignal` no longer calls `process.exit`: it sets the exit code to 128 + n, emits `exit` once so every listener runs, drops its own listener and re-sends the signal with the default action back. The kernel ends the process, and the shell still sees 129, 130 or 143. `bin/sandcastle` passes `--no-maglev --no-concurrent-sparkplug` on both exec lines; a detached run inherits them through `process.execArgv`. Tests: the signals fixture runs with the launcher's flags, asserts death by the signal, the exit listeners once with 128 + n, and no `process.exit` call; the launcher, mod and notify tests follow the new exec line and the signal death. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…63788) On Node 24 and 26, process.exit() joins V8's platform workers without disposing the isolate. A concurrent Maglev or Sparkplug compile job parked waiting for a main-thread GC then never finishes, so CLI commands, the hook relay, or a stopping Gateway can sit at 0% CPU forever after printing their output (nodejs/node#64274; fix pending in nodejs/node#66171). Every OpenClaw executable entrypoint (CLI entry, Gateway/service index, native hook relay, macOS node worker) now turns off Maglev and concurrent Sparkplug at startup when it runs as main, matching Node 22's tiering and the existing Vitest policy. Library imports keep stock V8; explicit --maglev or --concurrent-sparkplug flags still win. Testbox A/B (4 vCPU, Node 24.19, e2e CLI child replay, 8-way): 6/1019 hung with defaults, 0/1019 with the policy; re-enabling only Maglev hung 2/1019 and only concurrent Sparkplug 12/944, so both flags are needed.
|
Can we please expedite this bug fix? |
…enclaw#163788) On Node 24 and 26, process.exit() joins V8's platform workers without disposing the isolate. A concurrent Maglev or Sparkplug compile job parked waiting for a main-thread GC then never finishes, so CLI commands, the hook relay, or a stopping Gateway can sit at 0% CPU forever after printing their output (nodejs/node#64274; fix pending in nodejs/node#66171). Every OpenClaw executable entrypoint (CLI entry, Gateway/service index, native hook relay, macOS node worker) now turns off Maglev and concurrent Sparkplug at startup when it runs as main, matching Node 22's tiering and the existing Vitest policy. Library imports keep stock V8; explicit --maglev or --concurrent-sparkplug flags still win. Testbox A/B (4 vCPU, Node 24.19, e2e CLI child replay, 8-way): 6/1019 hung with defaults, 0/1019 with the policy; re-enabling only Maglev hung 2/1019 and only concurrent Sparkplug 12/944, so both flags are needed.
…nd a time limit Tests that spawned the CLI ran tsx's cli.mjs or .bin/tsx without the launcher's --no-maglev --no-concurrent-sparkplug, so they could hit the Node 24 exit deadlock (nodejs/node#66171), and their spawnSync calls had no timeout, so one stuck exit hung pnpm test, every sandbox gate run and full-check.sh. test/cli-spawn.ts (runKit, runNode, startKit, startNode) reads the flags from bin/sandcastle, runs node with tsx's loader in one process as the launcher does, and kills a child that outlives its limit (60 s, SIGKILL) with the test failing and the command named. Every test that spawned a kit script now uses it, and test/cli-spawn.test.ts refuses a test file that names a tsx entry itself. pnpm test gets --test-timeout=300000, which fails an async hang by name; it cannot interrupt a synchronous spawnSync, which is what the helper's limit is for. Where the ticket left a choice: the helper also covers the Herdr plugin entry (script option) and the non-CLI fixtures that import src/, since the guard test would otherwise need an allowlist for them. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…enclaw#163788) On Node 24 and 26, process.exit() joins V8's platform workers without disposing the isolate. A concurrent Maglev or Sparkplug compile job parked waiting for a main-thread GC then never finishes, so CLI commands, the hook relay, or a stopping Gateway can sit at 0% CPU forever after printing their output (nodejs/node#64274; fix pending in nodejs/node#66171). Every OpenClaw executable entrypoint (CLI entry, Gateway/service index, native hook relay, macOS node worker) now turns off Maglev and concurrent Sparkplug at startup when it runs as main, matching Node 22's tiering and the existing Vitest policy. Library imports keep stock V8; explicit --maglev or --concurrent-sparkplug flags still win. Testbox A/B (4 vCPU, Node 24.19, e2e CLI child replay, 8-way): 6/1019 hung with defaults, 0/1019 with the policy; re-enabling only Maglev hung 2/1019 and only concurrent Sparkplug 12/944, so both flags are needed. (cherry picked from commit 1aa0aeb) Co-authored-by: RomneyDa <6581799+RomneyDa@users.noreply.github.com>
review wanted
|
Thank you |
|
Independent reproduction on macOS (Darwin 25.5), Apple Silicon, Node v26.5.0 (Homebrew):
|
V8 background jobs, such as concurrent Sparkplug and Maglev compiles, can run into the heap limit. They then post a GC request to the main thread and park in CollectionBarrier::AwaitCollectionBackground() until the main thread collects garbage or the isolate is torn down. process.exit() does neither. DefaultProcessExitHandlerInternal() calls DisposePlatform() while the isolate is still alive and not running JS, and NodePlatform::Shutdown() joins the worker threads. A worker that is parked at that point never wakes up, and the join hangs forever. Pass the exiting isolate down to NodePlatform::Shutdown(). While the workers finish their current tasks, the main thread checks every 10ms whether a foreground task was posted for that isolate. If so, it drops the queued tasks, which would never run on this path, and performs a GC so that the parked workers can finish. Normal exits dispose the isolate before the platform and are not affected. Fixes: nodejs#64274 Signed-off-by: Manzoor Wani <manzoorwani.jk@gmail.com>
7511be4 to
ed0841e
Compare
aduh95
left a comment
There was a problem hiding this comment.
The added test seems to be flaky on main (i.e. without the fix), maybe it's related to the new V8 version. Here's the result of tools/test.py -t 20 --repeat=99 test/parallel/test-process-exit-background-gc.js on my machine
# Without my suggestion
[02:11|% 100|+ 64|- 35]: Done
# With my suggestion
[03:05|% 100|+ 54|- 45]: Done
The test only hung on an unfixed build about a third of the time, because the window in which the main thread is out of JS depended on how fast the machine runs pbkdf2, and the stress allocation task had to run into the heap limit inside that window. Calibrate the iteration count from a short probe so that the window is about a second everywhere, and use up the old generation budget just before exiting so that the next background allocation fails and parks the task. Run in the test process instead of spawning a child, which also drops the child process overhead. On an unfixed build the test now hangs in 20 of 20 runs, and it passes 30 of 30 with the fix. Signed-off-by: Manzoor Wani <manzoorwani.jk@gmail.com>
|
@aduh95 Thanks, you are right. The window where the main thread is out of JS depended on how fast the machine runs pbkdf2, and the stress allocation task had to run into the heap limit inside that window, so whether it hung was luck. Reworked in e7f8e56, taking your suggestions: no child process, body at top level, flags via the On an unfixed build it now hangs in 20 of 20 runs here, and passes 30 of 30 with the fix. Could you re-run your |
process.exit()can hang forever when a V8 background job (for example a concurrent Sparkplug or Maglev compile) is parked waiting for the main thread to collect garbage. The exit handler joins the platform workers without disposing the isolate, and only isolate teardown or a main-thread GC wakes such a worker. This makes the main thread perform that GC while it waits for the workers to exit.Fixes #64274
The test uses
--stress-concurrent-allocationplus a threadpool job thatuv_library_shutdown()waits on, which parks a worker inCollectionBarrier::AwaitCollectionBackground()before the join. Without the fix it hangs in 20 of 20 local runs; with it, it passes 30 of 30.This hang stalls our WordPress core builds that end with
process.exit()on Node.js 24 (WordPress/wordpress-develop#13608).