Skip to content

fix(sessions): recover writer authority and verify publication joins - #4388

Merged
ScriptedAlchemy merged 34 commits into
masterfrom
fix/4372-writer-authority-epoch-reopen
Oct 23, 2026
Merged

ScriptedAlchemy merged 34 commits into
masterfrom
fix/4372-writer-authority-epoch-reopen

Conversation

@ScriptedAlchemy

@ScriptedAlchemy ScriptedAlchemy commented Oct 21, 2026 •

Copy link
Copy Markdown
Owner

Losing the daemon authority record could reopen a store with a writer epoch below its durable checkpoint, leaving observation admission retrying generic infrastructure failures. Read the exact shard/incarnation checkpoint before publication, retain its epoch floor, and retire the refused physical attachment on the blocking executor. Retry once when no live graph owner retains the binding; otherwise return typed WriterAuthorityBehindPersisted while preserving live leases. Ledger fences remain typed and rejected writes roll back.

Named Hermes inline transcript callbacks join their own pinned source frontier. Success requires an active temporal generation, an applied relation receipt, and its expected graph watermark. Pending resets, retired generations, and future frontiers cannot supply publication proof. Another session's failure cannot reject a proven publication; the requested session's failed or unproven projection remains unavailable. Source sweeps and unproven provider identities retain the aggregate publication contract.

Pending joins finish advancing bounded discovery pages before waiting for derived-summary admission or the next historical window. This fixes the complete-receipt/zero-backlog batch-join stall without changing deadlines or work budgets. Preserve the exact registry idle-completion assertions through rekey and retirement.

Validation on integration c990a339252ee6c3e132b0e43cc713f85d332e97:

  • Merged master ca677ea030, retaining all peer changes. The sole manual conflict was in the host fixture: retain master's stricter Pi condition, the PR's baseline assertion diagnostic, and canonical typed Kiro/Pi zero-finding validation. There was no production-code conflict.
  • The exact real host-admission journey passed uncached remotely: 1 passed, 0 failed, 0 ignored, 27.62 seconds (528fe953-8f70-49e0-9464-1dc40a86b7a6). The root Bazel formatting check also passed.
  • All 7 reset/activation tests passed uncached remotely, with zero failures or ignored cases and 416 unrelated tests filtered, in 4.79 seconds (cae2bfa8-51af-4e44-a432-3469b825631e). This includes waiting-join refusal for the superseded candidate and successful new joins after automatic reset recovery.
  • Both test runs forced Rustc/TestRunner remote, disabled local fallback, used priority 10 and minimal outputs, and confirmed nonzero libtest execution. Frozen pnpm install, whitespace, and the commit hook passed. The normal push passed a fresh ancestry guard; the retained checkout is clean. Full hosted CI on this updated ready-PR head must pass before merge.

Historical evidence:

Tested merge 0fd2e7d8c662a3ccb42f23252d279c84d38adb65 integrates master d7e9afee0eed9f2a199aed2c4cf9ec930c620033, preserving reset rediscovery alongside per-session publication proof and failure isolation. All 61 focused tests passed uncached remotely on this tree: scheduler 48 (c89517d4-a154-4829-87d5-0034f910dcf5), activation/reset 7 (ad811eaa-b12c-4c3d-982b-59eacff40954), and MCP live-transcript joins 6 (8531fe87-9743-45f9-b6b0-f2a8fe366360). Nonzero libtest counts were confirmed; Rustc/TestRunner were remote with local fallback disabled and priority 10. Formatting and whitespace passed.

Earlier integration also passed the real daemon writer-authority reopen and host fixtures; those runs preceded this master merge. Subsequent integration c43f845c83770f2c3125180df488cc62052cfd37 added only eight .github files for fleet routing and Tailscale network-environment preservation; Rust source is unchanged from the 61-test tree. The draft CI run 39853962118 skipped Linux tests, Clippy, feature gates, and the shipped CLI, so its green rollup is not full validation. This PR is now ready for review; actual Linux execution and aggregate, Clippy, shipped CLI, and applicable Windows coverage must pass before merge.

Previous head 182ac69041f702222e5442e1840b75617336dfe7 also preserves peer test commit e3a679a806: the reset journey gates historical work, verifies waiting named and aggregate joins refuse superseded publication, and verifies new joins observe the automatically rebuilt publication. Production source is unchanged. That strengthened reset journey has now passed on the newer integration above. Earlier CI run 39864396638 does not establish readiness of the updated head.

Fixes #4372.

Resource-refusal diagnostics at 33a9bde3418fec56d2987460e148fe6a1acb0e3f:

  • The preceding dd84675530 adds only synchronization to the profile wake-coalescing test. Fleet run 39898392822 passed 169 of 170 targets; its sole failure was the real Codex PostCompact journey receiving transcript_background_resource_unavailable. The specific resource was erased by the hook error mapper.
  • On dd84675530, the exact journey passed (1 test, 3.31s; invocation 3de4c5c8-c471-42da-87d1-4df409be4a51) and the full transcript-ingest target passed all 207 tests (59.44s; invocation cc545d8e-92ed-4188-8fe5-106fe1057325). The original refusal was not reproduced; its resource and CI pressure state remain unknown.
  • The mapper now preserves the typed provider/resource labels for background-resource refusals, while keeping the same reason code, retryability, and backpressure status. Other error details retain their existing redaction. This is a diagnostic correction, not a bypass or a claim that resource backpressure is resolved.
  • The new regression failed before. All 11 error-mapping tests, including redaction cases, and the Bazel formatting check passed after.
  • The real PostCompact daemon-effect journey passed on the final tree: 1 passed, 0 failed, 206 filtered, 3.26s (invocation). Rustc/TestRunner were remote, local fallback disabled, priority 10, uncached execution. Commitlint and whitespace checks passed; normal push preserved all peer commits. Fresh current-head hosted CI remains required before merge.

@changeset-bot

changeset-bot Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 33a9bde

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@ScriptedAlchemy
ScriptedAlchemy marked this pull request as ready for review October 21, 2026 00:03
devin-ai-integration[bot]

This comment was marked as resolved.

@ScriptedAlchemy ScriptedAlchemy left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushed 6b9c4e2d96cea73b6d553b4e1fbd347396805755 to address the two concrete recovery blockers.

Both persisted-checkpoint reads use the async exact-SQL path. A refused physical attachment is drained, closed, and joined on the existing blocking executor before retry; close failures propagate.

For graph-first stores, a live graph publication's immutable binding cannot be replaced safely while owners or operation leases retain it. Recovery now retains the persisted epoch floor and fails once with typed WriterAuthorityBehindPersisted, preserving existing leases. It does not perform the same doomed physical open twice. After canonical owner/client release and graph close, reopening allocates a shared binding above the checkpoint and can commit. That follows #4372's explicit “adopts the persisted epoch, or fails once with a typed error” contract.

Added a real GraphDbRegistry owner/client + SQLite-checkpoint regression covering the refusal, continued validity of old leases, release/close, and successful reopen/write. Existing reopen coverage also checks native-worker wait telemetry. Pinned rustfmt and diff checks pass; native tests remain for CI.

The PR description now names //crates/tracedecay-cli:core_cli_suite for the isolated daemon journey. The previous blocked command selected a binary target, so it was not evidence that this journey had run.

devin-ai-integration[bot]

This comment was marked as resolved.

@ScriptedAlchemy
ScriptedAlchemy enabled auto-merge (squash) October 22, 2026 00:45

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 new potential issues.

3 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

Devin Review

Comment thread crates/tracedecay-session-runtime/src/session_temporal_refresh_scheduler/wake.rs Outdated
@ScriptedAlchemy
ScriptedAlchemy marked this pull request as draft October 22, 2026 00:49
auto-merge was automatically disabled October 22, 2026 00:49

Pull request was converted to draft

@ScriptedAlchemy ScriptedAlchemy changed the title fix(store): converge reopened writers on persisted authority fix(sessions): recover writer authority and join live publication Oct 22, 2026
@ScriptedAlchemy ScriptedAlchemy changed the title fix(sessions): recover writer authority and join live publication fix(sessions): recover writer authority and verify publication joins Oct 22, 2026
@ScriptedAlchemy
ScriptedAlchemy marked this pull request as ready for review October 22, 2026 21:38
@ScriptedAlchemy
ScriptedAlchemy merged commit 715f9a2 into master Oct 23, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Daemon rejects every project write with a stale writer authority epoch after reopening a store with stuck projections

1 participant