/run_sse stream stays open when a ParallelAgent branch has no terminal event

Description

With google-adk==1.31.1 + LiteLlm in SSE streaming mode, one leaf of a
ParallelAgent can stop emitting events after tool responses have been
returned, without emitting a final event (partial=false, finishReason=STOP)
and without any exception appearing in the pod stdout or being yielded on the
SSE stream as an error event.

When that happens the parent ParallelAgent never completes, the enclosing
SequentialAgent never advances to the next step, and the /run_sse HTTP
stream remains open — no further data: event, no EOF. The downstream SSE
consumer eventually gives up at its own timeout layer.

We have observed this in two invocations in production traffic today; both
stall in the same leaf agent (sub_workout_recommend_agent) after it has
completed a tool-call round and started streaming its text response. Sibling
branches in the same ParallelAgent complete normally.

Environment

google-adk: 1.31.1
litellm: 1.83.12
Python: 3.13.11
Container OS: Debian GNU/Linux 13 (trixie)
Host kernel: Linux 6.8.0
Package location: /usr/local/lib/python3.13/site-packages
Runtime: Kubernetes pod, FastAPI app produced by
AdkWebServer.get_fast_api_app(), endpoint /run_sse
vLLM (server-side): 0.19.1.dev6+g6d4a8e6d2 (git dev build from commit 6d4a8e6d2, not a tagged release)
Upstream model served by vLLM: google/gemma-4-31B-it (max_model_len=26000), exposed as alias ait-edge-da
LiteLlm model string: hosted_vllm/ait-edge-da

LiteLlm additional args:

model = LiteLlm(model="hosted_vllm/ait-edge-da", api_base="<internal-url>")
model._additional_args["extra_body"] = {
    "chat_template_kwargs": {"enable_thinking": False}
}

Agent structure:
- SequentialAgent
  - LlmAgent (health status analysis, text only)
  - ParallelAgent
    - LlmAgent sub_workout_recommend_agent (3 tools)
    - LlmAgent sub_sleep_recommend_agent (1 tool)
  - LlmAgent (summary)
Frequency: intermittent — 4 occurrences in ~3h20m of production traffic today.

Observed behavior — two stalled invocations

Redacted full event dumps are attached. Values (text chunks, state-delta
fields, tool args, tool response content) are replaced with
[REDACTED_TEXT len=N] / [REDACTED]; structural fields (partial,
finishReason, author, timestamp, id, invocationId, functionCall.name,
functionResponse.name) are kept verbatim.

Invocation A (`e-beb6c3e3`) — attached: `redacted_e-beb6c3e3.jsonl` (765 events)

All four partial=false events:

time (KST)	author	role
16:07:34	`sub_health_analysis_agent`	text final STOP
16:07:35	`sub_sleep_recommend_agent`	tool-call round STOP
16:07:35	`sub_workout_recommend_agent`	tool-call round STOP
16:07:56.119	`sub_sleep_recommend_agent`	sleep text final STOP — last event of invocation

sub_workout_recommend_agent has no partial=false for its text response.
sub_summary_agent is never invoked (0 events).
No events are generated after 16:07:56.119. No EOF on the SSE stream.
Downstream SSE consumer gives up at 16:12:56
(= 16:07:56 + 300 s idle timeout).

Invocation B (`e-d9e02105`) — attached: `redacted_e-d9e02105.jsonl` (1359 events)

time (KST)	author	role
16:12:55	`sub_health_analysis_agent`	text final STOP
16:12:55	`sub_workout_recommend_agent`	tool-call round STOP (3 calls)
16:12:55	`sub_sleep_recommend_agent`	tool-call round STOP (1 call)
16:13:10	`sub_sleep_recommend_agent`	sleep text final STOP
16:13:20.318	`sub_workout_recommend_agent`	last event: `partial=true`

sub_workout_recommend_agent has no partial=false for its text response.
sub_summary_agent is never invoked (0 events).
No events after 16:13:20.318. Downstream consumer gives up at 16:18:20.

Common pattern

In both invocations:

All four tool responses return successfully.
sub_sleep_recommend_agent completes its text response normally.
sub_workout_recommend_agent does not emit a partial=false for its
text response. It either stops mid-chunk (B) or produces no text events
that terminate (A).
Because one branch of recommend_parallel_agent never completes, the
parent SequentialAgent cannot advance to sub_summary_agent.
/run_sse stays open without further events and without EOF. Our gateway
eventually emits its own application-level event after a 300 s idle:
```
{"chunkCount":N,"error":"upstream-timeout"}
```
This payload is from our gateway, not from ADK.

No exception was logged in pod stdout and no error event was yielded on the
SSE stream during the idle window.

Expected behavior

ADK should not leave a branch indistinguishable from an active run indefinitely.
When the upstream model stream ends, errors, or stalls beyond a configured
timeout, the branch should:

emit a final event (partial=false with finishReason) and let the parent
agent complete, or
propagate an exception / error / cancellation event so the parent agent and
the enclosing workflow can observe and react.

In the observations above there is no final event, no error, and no
cancellation — the branch remains open without an observable terminal signal, which is what makes the stall
undetectable to the parent workflow.

#5342 — LiteLlm streaming bypass for function-call argument delta
buffering. Different symptom; does not cover stopped text streaming after
tool responses.
#3665 — LiteLlm streaming finish_reason missing (closed). Different
symptom; the sibling branch in our observation terminates normally with
finishReason=STOP.

Additional context

Locally (different versions — litellm 1.83.7, Python 3.14.4) we have
not reproduced this. The issue may be version- or environment-sensitive,
but we have not verified which component is the cause.
Attached redacted traces:
- redacted_e-beb6c3e3.jsonl — stall A
- redacted_e-d9e02105.jsonl — stall B
- redacted_e-7b6bbdbb.jsonl — a successful invocation for comparison
- was_upstream_timeouts.log — downstream gateway timeouts
- summary_agent_check.txt — sub_summary_agent event counts per invocation
LiteLLM DEBUG traces for the same window can be provided on request;
they contain token-level deltas that likely need additional redaction
before public posting.

adk-stall-dumps.zip

I ran local probes against the ADK source to validate the mechanism.

What I confirmed:

If the LiteLlm async stream reaches EOF, ADK synthesizes a final partial=false STOP response even when finish_reason is missing.
If the upstream async iterator hangs after a partial text chunk, ADK does not emit a final/error event before the iterator resumes, ends, or raises.
In that state, the ParallelAgent child branch remains open, so the parent SequentialAgent does not advance to the next step.

So this no longer looks like ADK simply dropping a final event after EOF. It looks more like the upstream LiteLLM/vLLM stream, or the transport path feeding it, remains open without EOF/error after the last partial text chunk.

This is still a mechanism check, not a full production root-cause confirmation. The remaining useful evidence would be a raw upstream LiteLLM/vLLM stream trace or task stack captured while the Kubernetes run is stuck.

Follow-up with a production repro captured via a SIGUSR1 dump + a coroutine-chain walker that crosses async-generator boundaries.

Deepest observed wait for the stall task in today's repro:

parallel_agent.py:62   process_an_agent   (async for event in events_for_one_agent)
  -> base_agent.py:297     run_async
  -> llm_agent.py:487      _run_async_impl
  -> base_llm_flow.py:{379, 804, 881, 1239, 1261, 1322}
  -> lite_llm.py:2321      generate_content_async
  -> litellm/.../streaming_handler.py:2017   __anext__
  -> litellm/.../base_model_iterator.py:167  __anext__
  -> httpx/_models.py:{997, 1018, 1031, 1055}  aiter_raw/bytes/text/lines
  -> httpx/_client.py:176  __aiter__
  -> litellm/llms/custom_httpx/aiohttp_transport.py:86  __aiter__
  -> aiohttp/streams.py:{53, 456, 372}  (_wait)   <- chain end at FutureIter

At that moment the pod had no matching outbound TCP socket to the vLLM endpoint and the asyncio loop was idle in ep_poll. The aiohttp StreamReader future is pending and does not resolve.

Our app was not passing timeout into LiteLlm(...), so LiteLLM's aiohttp transport was building ClientTimeout(sock_read=None) — no read-idle escape hatch at the transport layer. We are mitigating on our side by passing timeout=httpx.Timeout(read=60), which the transport maps to ClientTimeout(sock_read=60).

The original observation in this issue stands independently of that fix: even after we set a transport-side timeout, a stuck upstream iterator leaves ParallelAgent and /run_sse pending with no visible error. An opt-in stream-idle timeout on LlmAgent that cancels the leaf and surfaces a terminal error event would cover this regardless of the upstream cause.

Happy to draft a PR adding an optional stream_idle_timeout on LlmAgent. Let me know if Runner or a plugin hook would be the preferred scope instead.

Update on the mitigation angle.

Setting DISABLE_AIOHTTP_TRANSPORT=True changes the failure from a silent stall to an immediate httpx.RemoteProtocolError (malformed chunk footer from the upstream). /run_sse closes fast instead of hanging for 300s.

That shifts but does not close this issue. The ParallelAgent leaf now exits via exception, SequentialAgent never reaches summary, and the generator closes with one error event. A transport-level exception is a better failure mode than a silent hang, but the leaf still has no proper terminal event.

An opt-in stream-idle timeout on LlmAgent that surfaces a well-formed terminal event would cover both the original silent-stall case and this fast-fail case consistently. Still on for the PR — LlmAgent vs Runner, whichever you prefer.

Root cause update from simultaneous packet captures.

Captured the same response on both sides at the same event:

H100 server egress (where vLLM runs, in a docker container)
NKS pod receive side (inside the pod's network namespace, via kubectl debug ephemeral container + tcpdump)

Across a 4-minute window with 6 malformed chunk footer events:

Same 24 HTTP responses on both sides, same body size (1,436,524 bytes total, 10 TCP flows)
4 flows: byte-identical
6 flows: body length identical but a segment-sized region (40–46 bytes) differs — in-place substitution, not insertion/deletion
The differing region starts at the same offset inside each affected TCP segment (byte 186) and extends to the segment's end
Each affected segment carries exactly one chunked HTTP frame (size line + data + footer, ~228–232 bytes)
The differing bytes on the receive side are random-looking; TCP and IP checksums pass on both sides, so something downstream is recomputing the checksum after substitution

So the malformed chunk footer bytes that h11 reports are real — the bytes actually arrive that way at the client. They are not produced by vLLM/uvicorn/LiteLLM/h11. The corruption is in the network path between the H100 host egress and the NKS pod (in our case a cloud network fabric on the pod side — we've opened a support inquiry with the provider).

This narrows the application-side scope of this issue. I'll keep it open because the original ask — an opt-in terminal/error event from LlmAgent when the upstream stream dies abruptly — is still useful regardless of why the stream died. Happy to close if you'd prefer this be a separate FR.

@dlwldnjs1009

Hello @dlwldnjs1009, Thank you for the detailed report.
Could you please provide a minimal reproducible code snippet? This will allow us to investigate the issue accurately.
We also noticed that PR #5465 is currently open to fix this. If you have a moment, would you mind testing it to see if it resolves the issue on your end?
Thank you!

@sanketpatil06

Thanks @sanketpatil06.

One important update from a follow-up 4-point capture on 2026-04-28: this no longer looks like an ADK/LiteLLM/vLLM root-cause bug for the
malformed chunk footer bytes.

We ran simultaneous packet captures at four points:

P1: H100 vLLM container netns
P2: H100 host NIC egress
P3: NKS worker node eth0
P4: NKS pod netns

Result on the affected responses:

P1 ↔ P2: byte-identical
P2 ↔ P3: byte mismatch (the corruption is introduced between these two capture points)
P3 ↔ P4: byte-identical

So the corrupted chunk-footer bytes are already present when the response reaches the NKS worker, and they are not produced by ADK,
LiteLLM, or vLLM. The root cause is now being investigated as a network/provider-path issue between the H100 host egress and the NKS
worker ingress.

That said, the ADK-side failure mode is still relevant but should be scoped differently: when an upstream stream stalls without a clean
terminal event, ParallelAgent / /run_sse should not hang indefinitely. PR #5465 appears to address that defensive failure-surfacing
part by turning a silent stalled branch into a TimeoutError when branch_idle_timeout_secs is configured.

On the minimal repro request: the smallest behavioral shape that triggers this is a BaseLlm subclass whose generate_content_async
yields one partial=True LlmResponse and then awaits an asyncio.Event() that never sets, wrapped as one ParallelAgent branch
alongside a normal-completing branch, all inside a SequentialAgent.

Given the latest packet-capture evidence, our production environment is no longer a good validator for whether PR #5465 fixes the
original network-triggered symptom. I do not expect PR #5465 to fix the corrupted bytes themselves. It still looks useful as a defensive
failure-surfacing improvement for stalled upstream streams, but I will leave validation of the isolated timeout behavior to the PR's
unit/integration tests unless you specifically need an external repro from my side.

fix(agents): add branch_idle_timeout_secs to ParallelAgent to detect …

fix: deliver parallel sub-agent failures to the caller

Navigation Menu

Repository navigation

/run_sse stream stays open when a ParallelAgent branch has no terminal event #5455

Description

Environment

Observed behavior — two stalled invocations

Invocation A (`e-beb6c3e3`) — attached: `redacted_e-beb6c3e3.jsonl` (765 events)

Invocation B (`e-d9e02105`) — attached: `redacted_e-d9e02105.jsonl` (1359 events)

Common pattern

Expected behavior

Related

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

/run_sse stream stays open when a ParallelAgent branch has no terminal event #5455

Description

Description

Environment

Observed behavior — two stalled invocations

Invocation A (e-beb6c3e3) — attached: redacted_e-beb6c3e3.jsonl (765 events)

Invocation B (e-d9e02105) — attached: redacted_e-d9e02105.jsonl (1359 events)

Common pattern

Expected behavior

Related

Additional context

Activity

dlwldnjs1009 commented on Apr 23, 2026

dlwldnjs1009 commented on Apr 24, 2026

dlwldnjs1009 commented on Apr 24, 2026

dlwldnjs1009 commented on Apr 24, 2026

sanketpatil06 commented on Apr 28, 2026

dlwldnjs1009 commented on Apr 28, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Invocation A (`e-beb6c3e3`) — attached: `redacted_e-beb6c3e3.jsonl` (765 events)

Invocation B (`e-d9e02105`) — attached: `redacted_e-d9e02105.jsonl` (1359 events)