Repository navigation
fix(qoder): fail over on streaming first-byte timeout and watchdog stuck workers - #315
Conversation
caigee-cmd
left a comment
There was a problem hiding this comment.
Thanks for splitting this out of #313. CI is green and it merges cleanly with current main, but I found two blockers that need fixing before merge (confirmed with local tests on a merge tree):
- Every Qoder stream dies right after headers (
internal/providers/qoder/worker.godoWithFirstByteTimeout). The request runs underttfbCtxanddefer cancelTTFB()cancels it when the function returns. Go's HTTP client ties the response body to the request context, so that cancel kills the body. A multi-chunk SSE read ends with 0 bytes +context canceled.FastHeadReturnsBodyonly does one bufferedRead, so it misses this. Please only cancel on the error path (or stop the timer without cancelling and tie cancel toresp.Body.Close), and add a test that drains a multi-chunk stream to EOF. - Watchdog stops workers but never restarts them (
manager_maintenance.goprobeWorkerLiveness→stopAccount).stopAccountdeletesm.processes[id]beforeStop(), thenwatchAccountseesprocesses[id] != processand returns withoutbeginAccountRecovery. A stuck worker becomes a permanently dead account. The PR test only assertsproc.stopped. Please restore the recovery path and add a test that expects a restart.
Also please address in the same pass:
3. False-positive kills: probeHealthOnly treats health.Ready == false (missing hotContext) as failure, so workers that are still starting / rewarming / need login can be killed after ~90s. Only skip dead today.
4. Worker fetchWithTimeout (daemon.mjs): the 300s total cap also applies via prepareUpstream to non-stream and long thinking streams (contradicts changelog); mid-stream timeout just res.end() (silent truncate); headReached catch branch looks unreachable; e.code not forwarded; totalTimer not cleared on success.
Nits: Go firstByteTimeout isn't configurable; worker/NOTICE gained an unrelated Global check-in paragraph; test comment says 60s but code uses 30s.
Please rebase onto latest main after the fixes.
|
@caigee-cmd 感谢复查,两个 blocker 和其余问题都已处理(commit 4679af2):
|
…uck workers A Qoder stream whose response head never arrives hung the client until it gave up. Symptoms were requests with only an entry log line and no "chat reasoning resolved" exit line, and a worker that stayed alive as a process but stopped answering /health while still being routed to. - providers: add ErrFirstByteTimeout and doWithFirstByteTimeout on the qoder streaming path (adapter + Direct). The 30s deadline covers the response head only; once headers land the SSE body streams under the parent context's lifetime, so long thinking-model output is not cut off. A parent cancel still surfaces unchanged so it does not read as an upstream failure worth cooling down. - executor: classify ErrFirstByteTimeout as unavailable + failover with a 10s cooldown, ahead of the context.Canceled check, so a stuck upstream sheds the account briefly and the next eligible one serves the turn. - worker: wrap the upstream fetch in fetchWithTimeout with a 30s TTFB deadline and a 300s total-stream backstop, returning stable first_byte_timeout / stream_total_timeout errors. Previously the fetch had no AbortSignal, so a stalled upstream held the worker's in-flight slot forever. Both deadlines are tunable per deployment. - runtime: add a worker liveness watchdog to the maintenance loop. Each 30s tick issues one lightweight /health round-trip per worker account; three consecutive failures (~90s grace, enough to ride out a transient blip or a mid-rewarm) stop the worker and hand it to the existing watchAccount + beginAccountRecovery reboot path. In-process accounts are skipped by probeHealthOnly and unaffected. - docs: record in worker/NOTICE that Global's campaign service reserves the claimable daily check-in for the desktop client's own session token, so Global accounts cannot be auto-checked-in by the worker. Tests cover the new classification, prefix resolution, TTFB helper, and the watchdog threshold/backoff path.
- doWithFirstByteTimeout: stop cancelling the request context once the head arrives; Go ties the response body to that context, so the old defer cancel killed every SSE stream right after headers. Cancel is now bound to Body.Close, and the error path decides first_byte_timeout before cancelling. - Make firstByteTimeout configurable via QODER_STREAM_TTFB_MS (mirrors the worker knob) instead of a hard-coded package var; resolve once at construction. - Watchdog: stopAccount deleted m.processes[id] before Stop(), so watchAccount saw a mismatch and never called beginAccountRecovery, leaving stuck workers permanently dead. restartStuckWorker now detaches, stops, and hands off to beginAccountRecovery directly. - probeHealthOnly: treat ok=true/ready=false as alive (starting/rewarming/login) so warming workers are not rebooted every ~90s; only transport errors or non-2xx/ok=false count as failure. - Worker daemon: replace the 300s wall-clock total cap with a mid-stream stall watchdog that resets on every chunk (long steady thinking streams are no longer truncated); forward the timeout .code so the Go side can classify a mid-stream stall. Extracted to worker/src/upstream.mjs for testing. - Drop the unrelated Global check-in paragraph from worker/NOTICE. - Tests: multi-chunk drain-to-EOF, watchdog restart, warming-worker liveness, and worker fetch TTFB/stall cases.
4679af2 to
9b52b51
Compare
caigee-cmd
left a comment
There was a problem hiding this comment.
上次提的问题都已修复,本地复现测试和 CI 均通过,感谢!几处过时注释(watchdog 仍写着 calls stopAccount)和 QODER_STREAM_TOTAL_MS 的命名可以之后再顺手改。
Summary
Split out from #313 per review — this PR contains only the streaming-robustness changes, independent of the PAT bridge.
ErrFirstByteTimeout, classified asunavailable+ failover with a 10s cooldown, and the executor retries on the next eligible account. The deadline covers the response head only; once headers land the SSE body streams under the parent context's lifetime, so long thinking-model output is not cut off. A parent-context cancel (client disconnect) stays a plain cancel and is not counted as an upstream failure.fetchto the Qoder gateway previously had noAbortSignal, so a stalled upstream held the worker slot forever. It now aborts on a 30s first-byte deadline plus a 300s total-stream backstop (QODER_STREAM_TTFB_MS/QODER_STREAM_TOTAL_MStunable), returning stablefirst_byte_timeout/stream_total_timeouterrors./healthkept being routed to. The 30s maintenance tick now issues one/healthround-trip per worker account; 3 consecutive failures (~90s grace) stop the worker and hand it to the existingwatchAccount+beginAccountRecoverybackoff path. In-process accounts (WorkBuddy / Trae / Devin) are skipped.worker/NOTICE: documents the verified Global-region check-in limitation (daily check-in is reserved for the desktop client's own session token).Tests
go build ./...,go test ./internal/providers/qoder/ ./internal/executor/ ./internal/runtime/— all greenfirst_byte_timeout_test.go,probe_worker_liveness_test.go, executor classification tests