Repository navigation
Conversation
The Streams API moves into temporalio.common.streams, so the module becomes a package with the same contents.
The first-class API lives next to the converter it registers with. The paths of the generators and the generated imports follow.
A Stream's lifecycle is its owner's to manage, so a run end closes nothing. drain() still lets unawaited writes finish before the run ends.
Only the run-end close on Redis used them, along with workflow.Info.execution_expiration_time.
Backends go in stream_backends= of Client.connect, a Worker takes its client's, and the Replayer and WorkflowEnvironment take them too. temporalio.workflow, Client.stream and WorkflowHandle.stream replace StreamClient, and Redis moves to temporalio.contrib.stream.
A read is stream-level only, so Topic objects lose subscribe and pass to the Stream's subscribe as topics. The proto field names stay.
The crate reserves the auto_close fields, so the pb2 and the system bindings lose them. The generators write into the new package.
s23 drops auto_close from the Stores and the protos, and c8 drops the cache-off eviction reason that only the run-end close used.
A Local Activity close outlived a Workflow Task that failed after it, so the retry's create failed the run. The block end now leaves the Stream open in a cron run and on a failure that a retry run follows.
A ClientStream kept a failed or cancelled Describe forever, and the handle map kept a failed one with its handle. A found id now stays as text, and a failed call goes.
A Replayer serves one backend as a Worker does, a Worker configures its backends after its plugins, and a generator of backends stays. A backend in plugins= fails before the connect, and the per-client calls are private.
A Worker serves one backend, so the backend-move recipe can't run. A continued run reads its Streams' backends after @workflow.init.
Redis keeps its regular Activity close, and a backend that closes another way overrides _close_in_workflow. Workers that share a task queue all register the Redis backend, since it polls Activities there.
A Local Activity writes before its Workflow Task commits, so a close there stayed when the task failed, and the task's retry could not create the Stream again.
A run's cancel reaches the block as the error of an Activity, a Local Activity or a child, and no retry run follows a cancelled run.
The block end decided from the exception and the retry policy alone, so it left Streams open when no retry run followed.
A Store scans a limited number of entries per call, so a filtered read ended early after 10,000 entries of other Topics.
The open of a new attempt withdraws earlier Records, and a Stream that takes no write failed the attempt for good before user code ran.
The read path converts Record bodies later, so the failure kept RawValue details and an encoded message.
The decode of a dataclass inside a JSON value reads its fields, so the encoder's transfer type broke the round trip.
An entry left only on success, a rerun under a new epoch added another, and the count bound let up to 2 GiB stay.
The close of a Workflow's Stream is a regular Activity, so such a Worker alone left every close waiting. Another Worker on the queue may run it, so it is a warning.
This was referenced Oct 25, 2026
Owner
Author
|
Closed in favor of the review series. The net change is G1 to G3, B1 and S1 to S9, #367 to #379, cut fresh onto upstream |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR moves the Streams API into
temporalio.common, with the API of the SDK review and the fixes of an independent review.The SDK review asked for Streams as a first-class SDK API, with native as the default backend. It also asked for backends registered on the client, and an owner that closes each Stream. An earlier commit of this PR put the API in
temporalio.common.streams. It's now intemporalio.commonitself, with the same public names.What changed?
Modules.
temporalio/common.pyis now a package. The Stream code is in the privatetemporalio.common._streams.temporalio.commonloads the Stream names on first use, soimport temporalio.commonloads no Stream code.temporalio.contrib.stream(RedisStreamBackend,create(client, ...)andworkflow.*). The memory backend istemporalio.contrib.stream.testing.MemoryStreamBackend.temporalio.contrib.streamsis gone.Registration and entry points.
Client.connect(..., stream_backends=[...]). A Worker uses the backends of its client, andReplayerandWorkflowEnvironmenttakestream_backends=. With more than one backend, exactly one hasdefault=True. A backend inplugins=is refused.native. Brook has no native backend (Jetty adds it), so a create with no backend raisesStreamConfigError.workflow.create_stream,create_stream_nowaitandstream_handle,Client.stream, andWorkflowHandle.stream.StreamClientis gone.offsetandtopic_prefixes. Only a Stream subscribes, not a Topic.WorkflowTopic.publishis a plain method. It starts the write when called and returns itsasyncio.Future[int], so the write has its place in the Stream's order also with noawait.awaitgives the offset or raises as before. A failed write that no caller awaits or checks fails the run at the run end.*_nowaitdocstrings say that aStreamErrorfails the run and a config error fails the Workflow Task. The behavior is the same as before.Lifecycle.
auto_closeand noclose_with_run. On Redis, a run end closes nothing.CANCELED.Plugin.on_workflow_evicted,workflow.Info.execution_expiration_timeand the plugin shutdown wait of the Worker are removed.Other review fixes.
StreamExpiredError. A Record or batch over the Store limit raises the newStreamTooLargeError.next_offset. ACloseditem's failure is decoded with the reader's converter. A Record that fails to decode raisesStreamErrorwith its offset, andnext_offsetis the offset after it. The reader stores no offsets of client Records.async withon a client handle of a Workflow's Stream raisesStreamNotOwnerErrorbefore the block runs.Client.streamrefuses an empty id.Endis public. A run adds the Stream key header to its Activities only when it uses Streams.Subscription.aclose, the errors and the eviction match the code.Pins and CI.
ee6d1e51and Core c8f7826366, with regenerated protos and bindings.interceptors. The generated Stream Nexus clients are hidden from the API docs. Upstream's pydantic 2.14 annotation (fix: annotate pydantic TypeAdapter cache for pydantic 2.14 temporalio/sdk-python#2011) is cherry-picked fortest-latest-deps. Changelog fragments inadded/andchanged/describe the new API.Known limits.
no_remote_activities=Truelogs a warning, since another Worker must run the close.Why?
A close must not apply if its Workflow Task fails, so it runs as an Activity. The independent review of Brook and Jetty found that a Local Activity could still close early. It also found that a cancel closed a Stream as failed, that filtered reads could stall, and that an Activity retry failed on a closed Stream.
How did you test it?
Each review fix has a test that fails without it, apart from the docstring fixes. At the head,
pytest tests/contrib/streamsand the worker, replayer and testing suites pass on memory and Redis. New tests cover a publish with noawaitand a failed publish that no caller gets. A test checks thatimport temporalio.commonloads notemporalio.common._streamsin a fresh interpreter. The client, plugin, pydantic and converter suites pass. The upstreamtest_schedule_basicsfailed once on a shared dev server with schedules of other tests, and it passes alone.poe lint(ruff, pyright, mypy, basedpyright and pydocstyle) andpoe gen-docsare clean.On the fork CI, every job passes except
check-protosand the arm64 jobs.check-protoswaits on a nexgen release with temporalio/nexgen#260, which merged after v0.2.10. The arm64 jobs have no runner on the fork.The superproject branch
temporalio/internal-ai-prototypes2026/10/ai198-streamspins the head,415fb36a. sdk-python j9 and the harness use the newtemporalio.commonnames.