Skip to content

fix(qoder): fail over on streaming first-byte timeout and watchdog stuck workers - #315

Merged
caigee-cmd merged 2 commits into
caigee-cmd:mainfrom
favoroo:fix/qoder-stream-failover-watchdog
Oct 19, 2026
Merged

caigee-cmd merged 2 commits into
caigee-cmd:mainfrom
favoroo:fix/qoder-stream-failover-watchdog

Conversation

@favoroo

@favoroo favoroo commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

Summary

Split out from #313 per review — this PR contains only the streaming-robustness changes, independent of the PAT bridge.

  • 30s TTFB failover (Go side): a streaming chat request whose upstream response head never arrives fails fast with ErrFirstByteTimeout, classified as unavailable + failover with a 10s cooldown, and the executor retries on the next eligible account. The deadline covers the response head only; once headers land the SSE body streams under the parent context's lifetime, so long thinking-model output is not cut off. A parent-context cancel (client disconnect) stays a plain cancel and is not counted as an upstream failure.
  • Worker fetch timeouts: the worker's fetch to the Qoder gateway previously had no AbortSignal, so a stalled upstream held the worker slot forever. It now aborts on a 30s first-byte deadline plus a 300s total-stream backstop (QODER_STREAM_TTFB_MS / QODER_STREAM_TOTAL_MS tunable), returning stable first_byte_timeout / stream_total_timeout errors.
  • Worker liveness watchdog: health probes previously ran only at boot and on manual console refresh, so a worker that was alive as a process but stuck on /health kept being routed to. The 30s maintenance tick now issues one /health round-trip per worker account; 3 consecutive failures (~90s grace) stop the worker and hand it to the existing watchAccount + beginAccountRecovery backoff path. In-process accounts (WorkBuddy / Trae / Devin) are skipped.
  • worker/NOTICE: documents the verified Global-region check-in limitation (daily check-in is reserved for the desktop client's own session token).

Tests

  • go build ./..., go test ./internal/providers/qoder/ ./internal/executor/ ./internal/runtime/ — all green
  • New: first_byte_timeout_test.go, probe_worker_liveness_test.go, executor classification tests

@caigee-cmd caigee-cmd left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for splitting this out of #313. CI is green and it merges cleanly with current main, but I found two blockers that need fixing before merge (confirmed with local tests on a merge tree):

  1. Every Qoder stream dies right after headers (internal/providers/qoder/worker.go doWithFirstByteTimeout). The request runs under ttfbCtx and defer cancelTTFB() cancels it when the function returns. Go's HTTP client ties the response body to the request context, so that cancel kills the body. A multi-chunk SSE read ends with 0 bytes + context canceled. FastHeadReturnsBody only does one buffered Read, so it misses this. Please only cancel on the error path (or stop the timer without cancelling and tie cancel to resp.Body.Close), and add a test that drains a multi-chunk stream to EOF.
  2. Watchdog stops workers but never restarts them (manager_maintenance.go probeWorkerLiveness → stopAccount). stopAccount deletes m.processes[id] before Stop(), then watchAccount sees processes[id] != process and returns without beginAccountRecovery. A stuck worker becomes a permanently dead account. The PR test only asserts proc.stopped. Please restore the recovery path and add a test that expects a restart.

Also please address in the same pass:
3. False-positive kills: probeHealthOnly treats health.Ready == false (missing hotContext) as failure, so workers that are still starting / rewarming / need login can be killed after ~90s. Only skip dead today.
4. Worker fetchWithTimeout (daemon.mjs): the 300s total cap also applies via prepareUpstream to non-stream and long thinking streams (contradicts changelog); mid-stream timeout just res.end() (silent truncate); headReached catch branch looks unreachable; e.code not forwarded; totalTimer not cleared on success.

Nits: Go firstByteTimeout isn't configurable; worker/NOTICE gained an unrelated Global check-in paragraph; test comment says 60s but code uses 30s.

Please rebase onto latest main after the fixes.

@favoroo

favoroo commented Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

@caigee-cmd 感谢复查,两个 blocker 和其余问题都已处理(commit 4679af2):

  1. 流在 headers 后立即死掉 —— 确认是 doWithFirstByteTimeout 里 defer cancelTTFB() 在函数返回时取消了请求 context,而 Go 的 HTTP client 把 response body 绑在该 context 上。现在只在错误路径 cancel;成功时停掉定时器、不取消,并把 cancel 绑到 resp.Body.Close()(cancelOnCloseBody)。同时修正了错误路径的判定顺序:先算出 timedOut 再 cancelTTFB(),避免 cancel 后 ttfbCtx.Err() 恒非空把真正的错误误判成超时。新增 TestDoWithFirstByteTimeoutStreamsBodyToEOF:短 TTFB + 多 chunk 间隔发送,io.Copy 读到 EOF 并逐字节比对,正是能抓到该 bug 的用例。

  2. watchdog 只停不重启 —— 根因如你所说:stopAccount 先删 m.processes[id] 再 Stop(),watchAccount 看到进程不匹配直接 return,永远不调 beginAccountRecovery。新增 restartStuckWorker:从进程表摘除(避免 watchAccount 重复处理)→ Stop() → 直接调 beginAccountRecovery。测试 TestProbeWorkerLivenessRestartsStuckWorker 现在断言 starter 被再次调用并拿到新的进程。

  3. 误杀 —— probeHealthOnly 现在把 200 + ok=true 视为存活,ready=false(启动中/rewarm/待登录)不再算失败;只有传输错误或非 2xx / ok=false 才计入失败。新增 TestProbeHealthOnlyTreatsWarmingWorkerAsAlive 与 ...AsDead。

  4. worker fetchWithTimeout —— 去掉了 300s 的整段 wall-clock 上限,改成中途停滞看门狗:每收到一个 chunk 就重置,只有连续 QODER_STREAM_TOTAL_MS 无数据才 abort,所以长而持续输出的思考流不再被截断(与 changelog 一致)。超时通过 controller.abort(err) 携带 .code,SSE pipe 现在转发 e.code(stream_total_timeout / first_byte_timeout)而不是一律 upstream_stream_interrupted。totalTimer 成功未清理的问题随重构消失(改为按 chunk 起停)。helper 抽到 worker/src/upstream.mjs 并加了 worker/test/upstream.test.mjs(首字节超时、持续流不被截断、中途静默被中断)。

  5. Nits —— firstByteTimeout 改为构造时从 QODER_STREAM_TTFB_MS 解析(与 worker 端同名,下限 5s);worker/NOTICE 里无关的 Global 段落已删除;测试注释里 60s 改为与实际一致。

go build ./...、go test ./...、(cd worker && npm test)(87 passing)全绿。已 rebase 到最新 main(含 #309/#311)。

…uck workers

A Qoder stream whose response head never arrives hung the client until it
gave up. Symptoms were requests with only an entry log line and no
"chat reasoning resolved" exit line, and a worker that stayed alive as a
process but stopped answering /health while still being routed to.

- providers: add ErrFirstByteTimeout and doWithFirstByteTimeout on the
  qoder streaming path (adapter + Direct). The 30s deadline covers the
  response head only; once headers land the SSE body streams under the
  parent context's lifetime, so long thinking-model output is not cut off.
  A parent cancel still surfaces unchanged so it does not read as an
  upstream failure worth cooling down.
- executor: classify ErrFirstByteTimeout as unavailable + failover with a
  10s cooldown, ahead of the context.Canceled check, so a stuck upstream
  sheds the account briefly and the next eligible one serves the turn.
- worker: wrap the upstream fetch in fetchWithTimeout with a 30s TTFB
  deadline and a 300s total-stream backstop, returning stable
  first_byte_timeout / stream_total_timeout errors. Previously the fetch
  had no AbortSignal, so a stalled upstream held the worker's in-flight
  slot forever. Both deadlines are tunable per deployment.
- runtime: add a worker liveness watchdog to the maintenance loop. Each
  30s tick issues one lightweight /health round-trip per worker account;
  three consecutive failures (~90s grace, enough to ride out a transient
  blip or a mid-rewarm) stop the worker and hand it to the existing
  watchAccount + beginAccountRecovery reboot path. In-process accounts are
  skipped by probeHealthOnly and unaffected.
- docs: record in worker/NOTICE that Global's campaign service reserves the
  claimable daily check-in for the desktop client's own session token, so
  Global accounts cannot be auto-checked-in by the worker.

Tests cover the new classification, prefix resolution, TTFB helper, and the
watchdog threshold/backoff path.
- doWithFirstByteTimeout: stop cancelling the request context once the head
  arrives; Go ties the response body to that context, so the old defer cancel
  killed every SSE stream right after headers. Cancel is now bound to Body.Close,
  and the error path decides first_byte_timeout before cancelling.
- Make firstByteTimeout configurable via QODER_STREAM_TTFB_MS (mirrors the
  worker knob) instead of a hard-coded package var; resolve once at construction.
- Watchdog: stopAccount deleted m.processes[id] before Stop(), so watchAccount
  saw a mismatch and never called beginAccountRecovery, leaving stuck workers
  permanently dead. restartStuckWorker now detaches, stops, and hands off to
  beginAccountRecovery directly.
- probeHealthOnly: treat ok=true/ready=false as alive (starting/rewarming/login)
  so warming workers are not rebooted every ~90s; only transport errors or
  non-2xx/ok=false count as failure.
- Worker daemon: replace the 300s wall-clock total cap with a mid-stream stall
  watchdog that resets on every chunk (long steady thinking streams are no longer
  truncated); forward the timeout .code so the Go side can classify a mid-stream
  stall. Extracted to worker/src/upstream.mjs for testing.
- Drop the unrelated Global check-in paragraph from worker/NOTICE.
- Tests: multi-chunk drain-to-EOF, watchdog restart, warming-worker liveness,
  and worker fetch TTFB/stall cases.
@favoroo
favoroo force-pushed the fix/qoder-stream-failover-watchdog branch from 4679af2 to 9b52b51 Compare October 19, 2026 07:08

@caigee-cmd caigee-cmd left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

上次提的问题都已修复,本地复现测试和 CI 均通过,感谢!几处过时注释(watchdog 仍写着 calls stopAccount)和 QODER_STREAM_TOTAL_MS 的命名可以之后再顺手改。

@caigee-cmd
caigee-cmd merged commit ffd2141 into caigee-cmd:main Oct 19, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants