Skip to content

fix(daemon): wait out Windows local teardown contention - #2599

Merged
DeusData merged 7 commits into
DeusData:mainfrom
pcristin:fix/windows-local-transition-cleanup
Oct 20, 2026
Merged

DeusData merged 7 commits into
DeusData:mainfrom
pcristin:fix/windows-local-transition-cleanup

Conversation

@pcristin

@pcristin pcristin commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #2598.

Windows local teardown reacquires the startup-v2 gate and validated legacy mutex before releasing its participant claim. A busy gate currently shares the 500 ms cleanup deadline with native I/O failures. This can make a completed one-shot CLI command exit unsuccessfully while another client is starting.

This adds a nonblocking release result that distinguishes contention from native failure. Local cleanup retains its protection and waits for contention under the existing 120 s startup backstop. Repeated native failures still get the 500 ms deadline. The existing boolean release API remains available.

Tests cover both startup-v2 and legacy mutex contention, retained retry authority after native release failure, and a real supervised Windows indexing operation whose startup gate is held for more than 500 ms after indexing completes.

Verification

  • Full production security audit: make -j3 -f Makefile.cbm security passed with LLVM 20 on Linux. The install stage was also rerun successfully without inherited CODEX_HOME, so it used only its temporary home.
  • Native Windows baseline: the test-only run fails section_local_cleanup_waits_for_startup_gate with exit code 1 after a successful index response. The same run also fails cold-storm startup with coordination.cleanup_timeout component=local_transition_cleanup. Its production code is unchanged from upstream.
  • Normal pre-commit hook passed with LLVM 20: lint-ci, diff-scoped clang-tidy, source security audit, and the complete ASan/UBSan C suite (9,276 passed, 9 skipped). The test-only baseline had 9,275 passed, 9 skipped. The nine UBSan diagnostics match exactly between baseline and fix.
  • Native Windows regression guards passed: completed work waits for the live startup owner and exits 0, and all six racing cold clients succeed. The test file is byte-for-byte identical to the failing baseline.
  • Native Windows C suite passed: 8,927 passed, 0 failed, 91 skipped across all 182 suites. daemon_ipc has 38 passed, 0 failed, 0 skipped, including startup-v2/legacy mutex contention and native-error retry coverage.
  • Local sanitizer runs use ASAN_OPTIONS=detect_stack_use_after_return=0:quarantine_size_mb=16:thread_local_quarantine_size_kb=64 on this 8 GiB Linux host with 4 GiB temporary swap. The full test suite remains enabled. The baseline reports nine existing UBSan diagnostics in callable-signature, documentation-link, and ObjectScript scanner code; these are outside this patch.

This is a separate daemon bug found while checking #2305. The PR covers daemon cleanup and its regression tests.

AI assistance: written and tested with Codex on behalf of Pcristin. Pcristin has certified the DCO sign-off.

Signed-off-by: Pcristin <xxxokzxxx@protonmail.com>
@pcristin
pcristin requested a review from DeusData as a code owner October 11, 2026 22:59
@github-actions

Copy link
Copy Markdown

Thanks for opening this — it has been seen, and it is queued.

This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence.

Current review status: working through a backlog. 0.9.1-rc.1 is out, so the release freeze that held reviews is over — but it left a large queue of open pull requests behind it, and we are reading through them oldest-first. The background is in discussion #1144.

What that means for this PR, concretely:

  • It will not be closed for inactivity. No stale bot touches pull requests here.
  • It may still sit a while before a human reads it. That is on us, not on you.
  • Older PRs are read first, so a recent one is not being skipped — it is behind a queue.

Things that will genuinely speed it up whenever review does happen:

  • Keep it rebased on main — the tree is moving quickly right now, and a conflicting branch cannot be reviewed as the diff you intended.
  • Get CI green, or say which failures you believe are pre-existing.
  • Keep the change to one claim. Bundled features and refactors get split before they get merged, which costs you a round trip.
  • Every commit needs a sign-off (git commit -s) — CI enforces DCO.

If this fixes a bug, a reproduction we can run is worth more than a description of the symptom.

Thanks for contributing, and sorry in advance for the wait.

@DeusData DeusData added bug Something isn't working windows Windows-specific issues stability/performance Server crashes, OOM, hangs, high CPU/memory priority/high Needs near-term maintainer attention; high-impact bug, regression, safety issue, or release blocker. labels Oct 19, 2026
@DeusData

Copy link
Copy Markdown
Owner

Thanks for the clear distinction between Windows teardown contention and native failure, and for linking the failing and passing regression runs. I have read the proposal and its validation notes; it remains pending maintainer review.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
@DeusData

Copy link
Copy Markdown
Owner

Thank you, @pcristin -- this is a careful, well-argued fix. Telling "another client is starting right now" apart from a genuine native cleanup failure is exactly the right line to draw, and reusing the existing 120 s startup backstop keeps local teardown consistent with how acquisition already waits out a busy gate, instead of introducing a new budget. Keeping the boolean release API as a thin wrapper also left every other caller unchanged. The test-only baseline run that goes red on unchanged production code, next to the green run on this branch with the same regression section, made the change easy to verify.

We pushed one commit on top of your branch so it can be merged against current main:

  1. Merged current main (28a6905) into fix/windows-local-transition-cleanup as a signed-off merge commit (af312fb). Several other pull requests were merged since your last update, and we re-run the required checks on an up-to-date head before merging. The merge was clean: none of those commits touch the five files in this PR, so the diff against main is exactly what you wrote. Your own commits are untouched.

On our side, the daemon IPC suite passes on the merged tree, and a one-shot index plus six racing cold clients all exit 0 without a cleanup timeout on macOS (your try-release only reports contention on Windows, so the POSIX path behaves as before). Once the checks are green on the new head, this is ready to go in.

Thanks again for tracking this down while working on #2305, and for the clear write-up.

@DeusData
DeusData merged commit 94f952d into DeusData:main Oct 20, 2026
35 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working priority/high Needs near-term maintainer attention; high-impact bug, regression, safety issue, or release blocker. stability/performance Server crashes, OOM, hangs, high CPU/memory windows Windows-specific issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Windows local teardown treats live startup contention as a cleanup failure

2 participants