Skip to content

relay: inbound frames dispatch out of order, so a CLOSE can overtake its REQ #958

Open

Description

@chappie-daemon

Inbound frames are dispatched as independent tasks (crates/nostream-relay/src/relay.rs:304 JoinSet, :498 handlers.spawn(..)), so two frames read in order can run in either order. A REQ followed immediately by CLOSE for the same id can therefore have the CLOSE run first:

  1. CLOSE runs: abort_read(id) finds no entry, unsubscribe(id) removes nothing.
  2. REQ runs: it registers the subscription and begin_read(id) creates a read that nothing will abort.
  3. The read streams to EOSE on a subscription the client closed.

abort_read leaves no tombstone (connection.rs:301-310), so the later begin_read cannot learn that the id was closed.

Why it is not the capture's behaviour

The capture's adapter handles messages in arrival order on one thread: a handler's synchronous prefix — including registering the subscription — has run before the next message is looked at. Rust's JoinSet discards that ordering guarantee. So this is a parity deviation, not a performance choice.

Relationship to #951

#951 restores the abort for a CLOSE processed while the invite mint is in flight, and it does so correctly: begin_read is registered before the mint's only await. But its claim that "a CLOSE is not lost" holds only when the CLOSE is processed after the REQ has registered. Under this defect the CLOSE can be processed first and is lost again, by a different route. #951's build and ADR state that precondition; this issue owns removing it.

A CLOSE for an unknown id is indistinguishable from a CLOSE that overtook its REQ without an ordering token, so a tombstone cannot decide it on its own.

Options, none chosen here

  • Run the registration prefix synchronously in the read loop — restores the capture's guarantee for the state that matters, and leaves the awaits (mint, stored read) concurrent as the capture has them.
  • Serialize whole handlers — simplest to reason about, but a slow mint would block a subsequent CLOSE, which the capture permits.
  • Ordering token per connection — each handler records the arrival sequence and the state machine rejects a CLOSE older than the REQ it would abort; more machinery, exact.

Deciding check

A test that sends REQ then CLOSE back to back with the runtime's task ordering forced (a yield in the REQ handler before it registers), asserting no frame after the CLOSE. At base it streams; after a fix it does not.

Activity

  1. added
    bugSomething isn't working
    rewritenostream 4.0 Rust/LMDB rewrite
    rustRust implementation work
    factoryBuilt by the software factory
    on Oct 24, 2026
  2. chappie-daemon commented on Oct 24, 2026

    @chappie-daemon
    CollaboratorAuthor

    Fixed on factory/012-req-close-order — PR #978, awaiting the operator's merge.

    The mechanism reproduced exactly as filed: 30 of 30 multi-thread runs left the subscription registered and streaming after the CLOSE, and 0 of 20 on current_thread — which is why the suite was green. The fix runs each handler's ordering-sensitive prefix on the connection loop (handlers::begin), so the capture's guarantee is restored for the state that matters while the stored read and the invite mint stay concurrent.

    Recon also found a second symptom of the same defect that this issue did not name: a pipelined AUTH followed by a restricted REQ was refused auth-required in 10 of 10 multi-thread runs. It is fixed here too, because it is the same missing ordering and splitting it would have shipped a known identical failure one frame away.

    Two corrections to the issue's own text, from the recon: the parity claim was narrower than stated (the capture does not abort a stored read on CLOSE, so the parity-visible symptom is live dispatch to a closed id — the trailing EVENT/EOSE belongs to #957), and the proposed forcing (a yield in the REQ handler) does not exist, since nothing between the spawn and the registration can suspend. The oracle that does decide it anchors on a loop-side NOTICE and fails 5/5 at base.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingfactoryBuilt by the software factoryrewritenostream 4.0 Rust/LMDB rewriterustRust implementation work

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions