Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
95c6553
Turned temporalio.common into a package.
moetemp Oct 24, 2026
8e0d7ce
Moved the Streams package to temporalio.common.streams.
moetemp Oct 24, 2026
e55fb79
Removed auto_close, close_with_run and the run-end close on Redis.
moetemp Oct 24, 2026
0aa2dd8
Removed the Worker eviction hook and the plugin shutdown wait.
moetemp Oct 24, 2026
8aaff83
Registered Stream backends on the client, and added the entry points.
moetemp Oct 24, 2026
5eb5bce
Renamed the read options to offset and topic_prefixes.
moetemp Oct 24, 2026
4d67eec
Vendored the s23 Stream protos and WIT, and regenerated the bindings.
moetemp Oct 24, 2026
602b1b6
Pinned the streams crate to s23 and sdk-core to c8.
moetemp Oct 24, 2026
a66eab7
Closed a Workflow Stream with an Activity committed with its task.
moetemp Oct 24, 2026
c9323c1
Tried a failed first-run lookup of a Workflow's Stream again.
moetemp Oct 24, 2026
e7271cd
Tightened the Stream backend registration.
moetemp Oct 24, 2026
df4c0d9
Corrected the Workflow Stream docs on backends and on init.
moetemp Oct 24, 2026
74b759c
Put the Workflow close behind a backend method.
moetemp Oct 24, 2026
0a8927f
Noted in the Redis package docs that Workers poll the close Activity.
moetemp Oct 24, 2026
e771693
Decoded the Stream test vectors as UTF-8 on every OS.
moetemp Oct 25, 2026
b880a6d
Took a client config with no interceptors in the Stream backends.
moetemp Oct 25, 2026
3ef573b
Hid the generated Stream Nexus clients from the API docs.
moetemp Oct 25, 2026
c3b3798
fix: annotate pydantic TypeAdapter cache for pydantic 2.14 (#2011)
jmaeagle99 Oct 22, 2026
e033b82
Added a changelog fragment for Streams.
moetemp Oct 25, 2026
71ae539
Hid only the generated Stream clients from the API docs.
moetemp Oct 25, 2026
da1da37
Named the Stream entry points and other API changes in the changelog.
moetemp Oct 25, 2026
3fe0dfa
Refused a Local Activity's close of a Workflow's Stream.
moetemp Oct 25, 2026
85894c5
Closed a Workflow's Stream as canceled when a cancel ends its block.
moetemp Oct 25, 2026
4959553
Settled a deferred block-end close from the run's real outcome.
moetemp Oct 25, 2026
76e918c
Kept a no-wait read going while an empty page moves on.
moetemp Oct 25, 2026
8382d32
Let an Activity retry go on with a closed or expired Stream argument.
moetemp Oct 25, 2026
802b7a4
Decoded a Closed item's failure with the reader's converter.
moetemp Oct 25, 2026
3885277
Wrote a transfer type dataclass in a top-level list as its fields.
moetemp Oct 25, 2026
0d39c78
Bounded the pending Workflow appends by bytes and dropped refused ones.
moetemp Oct 25, 2026
a488efe
Warned when a Worker with Streams polls no Activities.
moetemp Oct 25, 2026
0c173b5
Raised StreamExpiredError at the create of an expired Stream.
moetemp Oct 25, 2026
2e47ec1
Raised a StreamError past a Record that fails to decode.
moetemp Oct 25, 2026
8358aef
Refused async with on a client handle of a Workflow's Stream.
moetemp Oct 25, 2026
0cd056c
Refused an empty Stream id at Client.stream.
moetemp Oct 25, 2026
1de5999
Kept no offsets of client Records in a read.
moetemp Oct 25, 2026
6c60f97
Exported End and dropped the unused RPC error mapping.
moetemp Oct 25, 2026
c0da527
Keyed a Workflow's Activities only when its run uses Streams.
moetemp Oct 25, 2026
d06cba4
Corrected the Retention, aclose and Stream error docstrings.
moetemp Oct 25, 2026
68bb7b7
Narrowed the item and failure types in the new Stream tests.
moetemp Oct 25, 2026
2ef3bcf
Added StreamTooLargeError for the reason stream_too_large.
moetemp Oct 25, 2026
2e2ffe5
Pinned crate s23 ee6d1e51 and Core c8 f7826366.
moetemp Oct 25, 2026
2ff42e1
Moved the Streams API into `temporalio.common`.
moetemp Oct 25, 2026
b4c5cee
Tightened the comments and docs on this branch.
moetemp Oct 25, 2026
8f71f3e
Dropped the figurative wording from three Stream error messages.
moetemp Oct 25, 2026
415fb36
Made WorkflowTopic.publish start the write and return its future.
moetemp Oct 25, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions changelog/added/babbling-brook.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Add experimental Streams to `temporalio.common`. A Stream is an ordered log of Records that Workflows, Activities and clients write, and that clients and Activities read. Register a backend with `stream_backends=` on `Client.connect`, `Replayer` or `WorkflowEnvironment`, then use `workflow.create_stream`, `Client.stream` or `WorkflowHandle.stream`.
Add a Redis Stream backend in `temporalio.contrib.stream`, and an in-memory one for tests in `temporalio.contrib.stream.testing`. The Redis backend needs a build with the bridge's `streams` feature, which the official builds have.
Add `activity.Info.workflow_task`, the Workflow Task attempt in which a Worker dispatched a Local Activity.
1 change: 1 addition & 0 deletions changelog/changed/babbling-brook.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Convert values of a `transfer_type_convertible` class inside JSON values too, such as list items and dataclass fields, not only at the top level of a payload. A dataclass with the decorator is still written as its fields inside a JSON value.
16 changes: 9 additions & 7 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -186,9 +186,9 @@ exclude = [
'temporalio/api',
'temporalio/bridge/proto',
'temporalio/nexus/system/workflow_service',
'temporalio/contrib/streams/proto',
'temporalio/contrib/streams/system',
'temporalio/contrib/streams/writer_system',
'temporalio/common/_streams/proto',
'temporalio/common/_streams/system',
'temporalio/common/_streams/writer_system',
]

[[tool.mypy.overrides]]
Expand All @@ -197,9 +197,9 @@ ignore_errors = true

[[tool.mypy.overrides]]
module = [
"temporalio.contrib.streams.proto.*",
"temporalio.contrib.streams.system.*",
"temporalio.contrib.streams.writer_system.*",
"temporalio.common._streams.proto.*",
"temporalio.common._streams.system.*",
"temporalio.common._streams.writer_system.*",
]
ignore_errors = true

Expand Down Expand Up @@ -239,6 +239,8 @@ privacy = [
"HIDDEN:temporalio.worker.workflow_instance",
"HIDDEN:temporalio.worker.workflow_sandbox.importer",
"HIDDEN:temporalio.worker.workflow_sandbox.in_sandbox",
"HIDDEN:temporalio.common._streams.system.client",
"HIDDEN:temporalio.common._streams.writer_system.client",
"HIDDEN:**.*_pb2*",
]
project-name = "Temporal Python"
Expand Down Expand Up @@ -272,7 +274,7 @@ exclude = [
"temporalio/api",
"temporalio/bridge/proto",
"temporalio/bridge/_visitor.py",
"temporalio/contrib/streams/proto",
"temporalio/common/_streams/proto",
"tests/worker/workflow_sandbox/testmodules/proto",
]

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -131,8 +131,7 @@ message StreamRecord {
CloseStatus close_status = 9;
}

// Idempotent for one id, one Retention and one auto_close. ALREADY_EXISTS for another
// Retention or another auto_close.
// Idempotent for one id and one Retention. ALREADY_EXISTS for another Retention.
message CreateStreamRequest {
string namespace = 1;
// Required from Workflow code.
Expand All @@ -141,11 +140,8 @@ message CreateStreamRequest {
google.protobuf.Duration retention = 3;
// Workflow only. A retry run's first create: the previous run, whose Records it withdraws.
string withdraw_run = 4;
// Close the Stream with no result when no Record was appended for this long, counted from
// the create. Empty never closes on its own. It is separate from Retention and the safety
// TTL. The Store applies it on the next access, so no call sees the Stream open after the
// period. UNIMPLEMENTED (detail stream_unsupported) on a Store that can't.
google.protobuf.Duration auto_close = 5;
reserved 5;
reserved "auto_close";
// The Workflow run that owns the Stream. Workflow kind, with its run_id and epoch.
Producer owner = 6;
// Close the Stream with the result or the failure of the owner's run chain when the chain
Expand Down Expand Up @@ -282,8 +278,8 @@ message DescribeStreamResponse {
// Zero when the Stream has no Record.
int64 last_offset = 2;
google.protobuf.Duration retention = 3;
// The auto_close of the create. Empty when the Stream has none.
google.protobuf.Duration auto_close = 4;
reserved 4;
reserved "auto_close";
}

// The details of a failed Stream call, in the gRPC status details and in the details of the
Expand Down
4 changes: 2 additions & 2 deletions scripts/_proto/temporal/sdk/streams/v1/extension.proto
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ package temporal.sdk.streams.v1;
import "google/protobuf/duration.proto";
import "temporal/api/streamservice/v1/stream_service.proto";

// The settings of the Redis Stream extension. Lang builds it from the plugin options.
// The settings of the Redis Stream extension. Lang builds it from the backend options.
message RedisStreamConfig {
// One URL, or the seed URLs of a cluster. Credentials and TLS come from the URL.
repeated string urls = 1;
Expand Down Expand Up @@ -96,7 +96,7 @@ message FreeWriterRequest {
message FreeWriterResponse {
}

// Method `CloseWriters`, asynchronous. Drops every open writer, for example when the plugin's
// Method `CloseWriters`, asynchronous. Drops every open writer, for example when the backend's
// client or Worker shuts down.
message CloseWritersRequest {
}
Expand Down
10 changes: 3 additions & 7 deletions scripts/_wit/stream-service.wit
Original file line number Diff line number Diff line change
Expand Up @@ -169,17 +169,15 @@ interface stream-service {
close-status: option<close-status>,
}

/// @nexus.doc "Create a Stream idempotently. ALREADY_EXISTS for another Retention or auto-close."
/// @nexus.doc "Create a Stream idempotently. ALREADY_EXISTS for another Retention."
/// @nexus.proto "temporal.api.streamservice.v1.CreateStreamRequest"
record create-stream-request {
/// @nexus.doc "Required from Workflow code. Empty elsewhere asks the service for an id."
/// @nexus.doc "Required on Redis and memory. On native, empty asks the server for an id."
stream-id: string,
/// @nexus.doc "The age after which a Record is dropped. Empty is the Store default."
retention: option<duration>,
/// @nexus.doc "Retry run only. The previous run, whose Records this withdraws."
withdraw-run: option<string>,
/// @nexus.doc "Close with no result after this long with no append. Empty never auto-closes."
auto-close: option<duration>,
/// @nexus.doc "The Workflow run that owns the Stream. The Workflow transport sets it."
owner: option<producer>,
/// @nexus.doc "Close with the owner run chain's result when it closes. Native only."
Expand Down Expand Up @@ -330,8 +328,6 @@ interface stream-service {
/// @nexus.offset "stream-offset"
last-offset: option<s64>,
retention: option<duration>,
/// @nexus.doc "The auto-close of the create. Empty when the Stream has none."
auto-close: option<duration>,
}

// Its stream-ref is its serialized form, so a handle passes through Workflow input, Activity
Expand Down Expand Up @@ -375,7 +371,7 @@ interface stream-service {
create-wait: option<duration>,
) -> read-records-response;

/// @nexus.doc "Describe the Stream: its status, newest offset, Retention and auto-close."
/// @nexus.doc "Describe the Stream: its status, newest offset and Retention."
/// @nexus.operation "describe-stream"
describe: func() -> describe-stream-response;
}
Expand Down
10 changes: 5 additions & 5 deletions scripts/gen_protos.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@

api_out_dir = base_dir / "temporalio" / "api"
sdk_out_dir = base_dir / "temporalio" / "bridge" / "proto"
streams_out_dir = base_dir / "temporalio" / "contrib" / "streams" / "proto"
streams_out_dir = base_dir / "temporalio" / "common" / "_streams" / "proto"

py_fixes = [
partial(re.compile(r"from temporal\.api\.").sub, r"from temporalio.api."),
Expand All @@ -53,11 +53,11 @@
),
partial(
re.compile(r"from temporal\.sdk\.streams\.").sub,
r"from temporalio.contrib.streams.proto.",
r"from temporalio.common._streams.proto.",
),
partial(
re.compile(r"'__module__' : 'temporal\.sdk\.streams\.").sub,
r"'__module__' : 'temporalio.contrib.streams.proto.",
r"'__module__' : 'temporalio.common._streams.proto.",
),
partial(
re.compile(r"'__module__' : 'temporal\.api\.").sub,
Expand All @@ -82,7 +82,7 @@
partial(re.compile(r"temporal\.sdk\.core\.").sub, r"temporalio.bridge.proto."),
partial(
re.compile(r"temporal\.sdk\.streams\.").sub,
r"temporalio.contrib.streams.proto.",
r"temporalio.common._streams.proto.",
),
]

Expand Down Expand Up @@ -231,7 +231,7 @@ def generate_protos(output_dir: Path):
for p in (output_dir / "temporal" / "sdk" / "core").iterdir():
shutil.rmtree(sdk_out_dir / p.name, ignore_errors=True)
p.replace(sdk_out_dir / p.name)
# The stream extension's own protos go with the contrib package that uses them
# Only the Streams package imports the stream extension protos
streams_out_dir.mkdir(parents=True, exist_ok=True)
for p in (output_dir / "temporal" / "sdk" / "streams").iterdir():
shutil.rmtree(streams_out_dir / p.name, ignore_errors=True)
Expand Down
23 changes: 11 additions & 12 deletions scripts/gen_stream_service.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,9 @@
"""Generates the Stream service bindings of temporalio.contrib.streams.
"""Generates the Stream service bindings of the Streams API in temporalio.common.

The WIT and the protos are copies of the temporalio-streams crate's, in scripts/_wit and
scripts/_proto. Run scripts/gen_protos.py first for the proto modules. No nexgen release has the
offset, long-poll and client features yet, so the script installs nexgen at
NEX_GEN_REVISION from moetemp/nexgen. NEX_GEN_STREAMS_BIN may name a build of that revision
instead.
offset, long-poll and client features, so the script installs nexgen at NEX_GEN_REVISION from
moetemp/nexgen. Set NEX_GEN_STREAMS_BIN to use a local build of that revision instead.
"""

import os
Expand All @@ -30,12 +29,12 @@
/ "v1"
/ "stream_service.proto"
)
output_dir = base_dir / "temporalio" / "contrib" / "streams" / "system"
# The writer is client-only, so it has a caller and its resource class, and no Workflow code.
writer_output_dir = base_dir / "temporalio" / "contrib" / "streams" / "writer_system"
output_dir = base_dir / "temporalio" / "common" / "_streams" / "system"
# Only clients call the writer service, so its bindings have no Workflow code.
writer_output_dir = base_dir / "temporalio" / "common" / "_streams" / "writer_system"

NEX_GEN_REPOSITORY = "https://github.com/moetemp/nexgen"
# The head of moe/brook/v3/n8-stream-resource. A branch moves, so the pin is the commit.
# A commit on the moe/brook/v3/n8-stream-resource branch.
NEX_GEN_REVISION = "204b566ee996939b165ccd1d18c8c432bfdcc0dd"
nex_gen_root = base_dir / ".nexgen-streams" / NEX_GEN_REVISION

Expand Down Expand Up @@ -81,14 +80,14 @@ def generate_stream_service() -> None:
raise RuntimeError(f"missing generator input: {path}")
with tempfile.TemporaryDirectory(dir=base_dir) as temp_dir:
descriptor_path = Path(temp_dir) / "stream_service.bin"
# The extension messages live in temporalio.contrib.streams.proto, so the descriptor
# names their file by that module path.
# The generated extension messages are in temporalio.common._streams.proto, so the
# descriptor must name their file by that module path.
extension_root = Path(temp_dir) / "extension"
extension_path = (
extension_root
/ "temporalio"
/ "contrib"
/ "streams"
/ "common"
/ "_streams"
/ "proto"
/ "v1"
/ "extension.proto"
Expand Down
4 changes: 2 additions & 2 deletions temporalio/activity.py
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ class WorkflowTaskAttempt:
"""ID of the ``WorkflowTaskStarted`` event of the run's last completed task. 0 if none."""

failed_tasks: Sequence[int] = ()
"""``WorkflowTaskStarted`` event IDs of the run's failed and timed out tasks, the last 100."""
"""``WorkflowTaskStarted`` event IDs of the run's last 100 failed and timed out tasks."""


@dataclass(frozen=True)
Expand Down Expand Up @@ -167,7 +167,7 @@ class Info:

workflow_task: WorkflowTaskAttempt | None = None
"""For a local activity, the Workflow Task in which the Worker dispatched this execution.
Retries that the Worker runs without a new task keep it. None for other activities.
Retries that the Worker runs in the same task have the same value. None for other activities.

.. warning::
This API is experimental.
Expand Down
Loading
Loading