Repository navigation
relay: the wire_parity release test fails intermittently with BrokenPipe #974
Description
crates/nostream-relay/tests/wire_parity.rs fails intermittently with a BrokenPipe on the release profile. Measured during the #865 review (job 011-relay-strategies), which does not touch either file:
cargo test -p nostream-relay --release --test wire_parity # at bca0f6f
# 5 failures in 10 runs
Why it matters
A pre-existing flake that fires half the time in release turns every release-profile run into a coin toss, which is how a real regression gets dismissed as "the flake again". It also makes cargo test --workspace --all-features --release unusable as a gate, which job 011 needed in order to check a float-underflow assertion in both profiles.
Not this job's scope
wire_parity.rs and transport.rs are not in the diff of factory/011-relay-strategies (base bca0f6f), and the failure reproduces at the base commit. Recording it so the next run owns it with its own evidence rather than inheriting a mystery.
What the fix needs
- The failing test's name and the exact assertion or panic site, captured from a run that fails.
- Whether it is a test-harness race (the pipe reader dropped before the writer finished) or a production path that closes early — that distinction decides whether this is a test fix or a relay fix.
- A reproduction that fails reliably enough to prove the fix bites, or a bounded loop that shows the rate drop.
Activity
chappie-daemon commentedon Oct 24, 2026
Reproduction rate from the factory triage: the flake is load-sensitive, not a fixed rate.
The original 5/10 was taken during a review round with builds running, so it was a loaded box too — treat "half the time" as "under load", not as a deterministic coin.
This narrows the hypothesis rather than widening it: a timing race between the client writing the 614 600-byte frame and the relay closing fits a load-dependent rate, and does not fit a logic error, which would fail at a stable rate.
Consequence for whoever fixes it: reproduction needs load, classification needs quiet. The discriminating test — does the 1009 close frame still arrive when the write fails — must run on an idle machine or its answer is noise. Reproduce by loading the box (concurrent builds are enough); classify with nothing else running.
chappie-daemon commentedon Oct 24, 2026
Measured at 011's head
24b679con an idle box (load 1.5, no compile running): 3 failures in 10 standalone release runs, atcrates/nostream-relay/tests/wire_parity.rs:496.That retires the earlier rate. The previous figures — 2/12 under compile load and 0/25 idle — were both taken while a leaked process was spinning four cores, so neither was evidence about the flake's real rate; the 0/25 in particular was luck rather than quiet.
The failing site is the
expecton the client send in theoversizedstep, so the panic happens before the close frame is compared: the test currently proves the write lost, and says nothing about what the server did. The classification that settles harness-vs-production is whether the close still arrives with 1009 after a failed send:The factory job carrying this is 011-relay-strategies; its manager runs that classification before round 2 opens and comments here with the outcome.
chappie-daemon commentedon Oct 24, 2026
Classified by measurement: a test-harness race, not a relay defect.
Run on an idle box (load 1.34), at
24b679c, release profile:.expecton the oversized-frame sendfinal_close1009 still assertedrustfmtThe failure is the client's
expectatwire_parity.rs:496,Io(BrokenPipe), and the test aborted there — so the close frame was never compared in a failing run. With the write error tolerated and the session's close assertion left in place, every run passed, including the 11 where the write lost.That is the answer to the question this issue asked: the relay does send the 1009 close; a 614 600-byte write into a socket the relay has already closed on the declared length is expected to fail, and the test was asserting the wrong thing.
Fixed on
factory/011-relay-strategiesin2a428c2: an IO error on that send is tolerated, any other send failure still fails, and the close comparison remains the assertion. Closes with #865's merge.chappie-daemon commentedon Oct 24, 2026
Fixed by #976, merged as
2b2287fintorewrite/mainat 2026-10-24T20:28:09Z. Closing by hand because the keyword does not fire into a non-default branch.What the flake was.
wire_parity.rs:496called.expecton the client's write of the one-frame-past-limit event. The relay refuses an oversized frame from the header, so it can close before the client has finished writing 614,600 bytes; the write then fails withIo(BrokenPipe)and the test aborted before comparing the close frame it exists to assert. The test was proving the write lost, not what the server did.Classification, measured on an idle box (load 1.5, nothing else running):
.expecton the sendrustfmtEvery tolerated run still compared the close frame and found 1009, so the relay was correct throughout: this was a harness race, not a production defect. The tolerance is limited to
Error::Io— any other send failure still fails, with the session id.One measurement note worth keeping: without
--nocapture, the first tolerance run reported "0 send failures" because the diagnosticeprintln!is captured on a passing test. That would have read as "the flake did not occur" on a run where it fired 11 times in 20.The earlier figures in this thread are retired: 2/12 under compile load and 0/25 idle were both taken while a leaked process was spinning four cores, so neither was a rate.