Skip to content

Fix PTY attach output ordering - #476

Merged
chruffins merged 1 commit into
mainfrom
hypeship/fix-pty-attach-replay
Oct 24, 2026
Merged

chruffins merged 1 commit into
mainfrom
hypeship/fix-pty-attach-replay

Conversation

@chruffins

@chruffins chruffins commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • drain PTY output independently of WebSocket attachment and retain the latest 64 KiB in a fixed-size ring buffer
  • forward output on a separate goroutine so WebSocket backpressure cannot stop PTY draining; flush pending output before the exit event
  • add regression coverage for replay/exit ordering, ring-buffer reuse, and continued draining while the sink is blocked

Overlap with #463

Checked open PR #463 first. It changes process lifecycle and process.go, but its description explicitly excludes TTY sessions from output capture; it does not cover PTY attach replay or ordering. This PR changes the same file, so a merge conflict with #463 is possible. The PTY changes are otherwise separate from #463's non-TTY output lifecycle work.

Validation

  • cd server && go test -race ./cmd/api/api -run 'TestPTYOutput' -count=10
  • cd server && go test ./cmd/api/api -count=1
  • cd server && go vet ./cmd/api/api

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit f83e4da. Configure here.

Comment thread server/cmd/api/api/process.go Outdated
@chruffins
chruffins requested review from Sayan- and rgarcia and removed request for rgarcia October 19, 2026 19:03
@chruffins
chruffins marked this pull request as ready for review October 22, 2026 14:43

@Sayan- Sayan- left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • p2: attaching before the first output can leave the terminal blank. changed is initialized by the first write, but an earlier attachment can already be waiting on nil. A real WebSocket probe receives neither later output nor exit; attaching after initial output works.

  • p2: the two-second PTY drain deadline can wait without a bound. A shell exiting with a background child holding the terminal open leaves the reader polling a closed descriptor and the waiter blocked. Reproduced on this PR; the same process test completes before this change. Cleanup and scale-to-zero reenable cannot run while blocked.

  • p2: a retiring attachment can corrupt the next attachment's output position. The forwarder is outside the handler's wait group. A controlled WebSocket reconnect probe pauses the old callback after enqueueing, reconnects, then resumes it: the stale worker advances the new cursor and the next byte disappears. The control without that overlap passes.

@chruffins
chruffins requested a review from Sayan- October 23, 2026 13:59
@chruffins

Copy link
Copy Markdown
Contributor Author

@Sayan- thanks for the detailed review. Pushed 06c8292 with fixes for all three items.

1. Attaching before the first output leaves the terminal blank. write created the change channel only on the first write, so an attachment that arrived earlier waited on a nil channel and never woke. attach now creates the channel if needed, and write no longer replaces it. Regression test: TestPTYOutputDeliversWritesAfterAttachBeforeFirstOutput.

2. The drain deadline can block without bound. After ptyDrainWait, the waiter closed the PTY master while the reader was still polling the raw fd. A closed fd returns POLLNVAL, which the reader's loop ignores, so it spun forever and <-h.ptyDrained never returned. The waiter now closes the reader's existing stop channel and waits for the reader to exit before closing the master. The reader checks that channel on every 100ms poll, so the wait is bounded. Regression test: TestProcessTTYExitWithBackgroundChildHoldingPTYDoesNotBlock, which uses trap "" HUP; sleep 30 & exit 0 to keep the slave open. Without the HUP trap, the shell exit hangs up the terminal and the reader sees EIO, so the test doesn't reproduce.

3. A retiring attachment can corrupt the next attachment's cursor. The forwarder ran outside any wait group, so a stale forwarder could finish an in-flight sink call after the next attach and advance the new cursor. detach now waits for the forwarder to exit before returning. Regression test: TestPTYOutputDetachWaitsForInFlightSink.

Cursor-level fix for #3 is in pty_output.go; the drain change is in process.go.

Cursor Bugbot's finding on the earlier commit (writes forwarding into the sink while holding the lock) doesn't match the current code: write only copies into the ring buffer and never calls the sink. TestPTYOutputContinuesDrainingDuringSinkBackpressure covers that.

Verification: go test -race passes for the PTY and process tests (3 runs), and the full unit suite from AGENTS.md passes. gofmt and go vet are clean.

Remaining limit: if the PTY produces more output than the reader can drain in the 2s deadline, the excess is dropped at close. I left that as is.

@Sayan- Sayan- left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rechecked early PTY attachment, bounded drain completion, and reconnect isolation; the regression and WebSocket tests pass with the race detector.

Drain PTY output for the process lifetime into a replay buffer so a client
that attaches after output was written still receives it.

- Attach replays buffered output, then forwards live output; the exit
  event is sent only after the attached sink has flushed.
- Detach waits for the forwarder to leave any in-flight sink call so a
  stale forwarder cannot move the next attachment's cursor.
- The waiter stops the PTY reader through its stop channel after a
  bounded drain before closing the master. The reader polls the raw fd
  and spins on a closed one.
@chruffins
chruffins force-pushed the hypeship/fix-pty-attach-replay branch from 06c8292 to a079358 Compare October 24, 2026 00:10
@chruffins
chruffins merged commit 03cae8e into main Oct 24, 2026
10 of 11 checks passed
@chruffins
chruffins deleted the hypeship/fix-pty-attach-replay branch October 24, 2026 17:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants