Skip to content

[Brook S4] Added Stream backends and stream_backends= on the client and Replayer. - #374

Open
moetemp wants to merge 1 commit into
moe/brook/review/s3-data-modelfrom
moe/brook/review/s4-backends
Open

moetemp wants to merge 1 commit into
moe/brook/review/s3-data-modelfrom
moe/brook/review/s4-backends

Conversation

@moetemp

@moetemp moetemp commented Oct 25, 2026 •

Copy link
Copy Markdown
Owner

What was changed

This PR adds StreamBackend, the base class of the Stream backends, and stream_backends= on the client and the Replayer. Each Stream handle stores its backend's name, so any process can route a handle it gets.

  • A Worker uses its client's backends, and applies them after its plugins, so a plugin that replaces the Workflow runner doesn't drop them.
  • With more than one backend, exactly one must have default=True, since a replayed create must select the same backend.
  • The generated Stream service and writer clients start their operations on the crate's runtime. Their _create_nexus_client gives a Nexus client that converts each model with the client's converters and codec, as Workflow code does, and calls the runtime in place of an endpoint.
  • The Nexus client runs the codec once for each request and once for each response, on all of their payloads, since a remote codec costs a round trip for each call.

Depends on S3 (#373), and on B1 (#370) through feature-streams. No changelog fragment, since S9 has the Streams entry.
Part of the Brook review series.

Why?

The S6 backends and the Workflow, Activity and client handles all get their calls from this registration.

Open questions.

  • StreamBackend subclasses SimplePlugin for its hooks, but plugins= refuses it. Is that OK?
  • The default backend name is native, though this SDK has no native backend. Is that the right default?
  • ClientConfig.stream_backends is Required, so a hand-built ClientConfig must now set it. Is that OK?

Checklist

  1. How was this tested:
    • poe lint, poe gen-docs and pytest tests/contrib/streams. Fork CI passes, except the arm jobs, since the fork has no arm runners.

@moetemp moetemp added the skip-changelog Changelog entry rides another PR label Oct 25, 2026
@moetemp
moetemp force-pushed the moe/brook/review/s4-backends branch from b146c64 to ce8bd6d Compare October 26, 2026 00:14
@moetemp moetemp changed the title [Brook S4] Added Stream backends and stream_backends= on the client, Worker and Replayer. [Brook S4] Added Stream backends and stream_backends= on the client and Replayer. Oct 26, 2026
@moetemp
moetemp force-pushed the moe/brook/review/s4-backends branch from ce8bd6d to e453c8e Compare October 26, 2026 02:08
@moetemp
moetemp force-pushed the moe/brook/review/s3-data-model branch 2 times, most recently from 9e969ce to 9b54b39 Compare October 26, 2026 08:44
@moetemp
moetemp force-pushed the moe/brook/review/s4-backends branch from e453c8e to 0d0ab14 Compare October 26, 2026 08:44
@moetemp
moetemp force-pushed the moe/brook/review/s3-data-model branch from 9b54b39 to fda6c46 Compare October 27, 2026 09:07
@moetemp
moetemp force-pushed the moe/brook/review/s4-backends branch 2 times, most recently from 9f62387 to 9afffa1 Compare October 27, 2026 09:17
@moetemp
moetemp force-pushed the moe/brook/review/s3-data-model branch 2 times, most recently from 5a0f9bf to 7a7e940 Compare October 28, 2026 01:33
@moetemp
moetemp force-pushed the moe/brook/review/s4-backends branch from 9afffa1 to 36ed7f5 Compare October 28, 2026 01:33
A backend registers a name that Stream handles store, so any process can
route a handle that it receives. A Worker uses its client's backends.
@moetemp
moetemp force-pushed the moe/brook/review/s4-backends branch from 36ed7f5 to c0c0a86 Compare October 28, 2026 07:54
@moetemp
moetemp force-pushed the moe/brook/review/s3-data-model branch from 7a7e940 to cbec633 Compare October 28, 2026 07:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-changelog Changelog entry rides another PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant