OpenTelemetry context detach errors when BaseAgent wrapper exits inner agent's run_async() early

Describe the bug

When a BaseAgent subclass wraps another agent and exits early from iterating over self.main_agent.run_async(ctx) (via return or break inside async for), multiple Failed to detach context errors are logged by OpenTelemetry. This happens because base_agent.run_async() wraps the generator yield in tracer.start_as_current_span(), and when GeneratorExit propagates through the generator chain, the OTel context tokens cannot be detached — they were created in a different contextvars.Context.

google-adk version: 1.27.2
Python version: 3.13
opentelemetry-api version: 1.33.0

This appears to be the same root cause as #423 and #1670 (both closed), but with a distinct and reproducible trigger path.

To Reproduce

Create a BaseAgent subclass that wraps another agent and conditionally exits early:

from google.adk.agents import BaseAgent, LlmAgent
from google.adk.events.event import Event
from typing import AsyncGenerator
from typing_extensions import override

class WrapperAgent(BaseAgent):
    main_agent: BaseAgent

    @override
    async def _run_async_impl(self, ctx) -> AsyncGenerator[Event, None]:
        async for event in self.main_agent.run_async(ctx):
            if should_block(event):
                yield create_redirect_event(ctx)
                return  # <-- triggers GeneratorExit on inner generator chain
            yield event

When should_block() returns True and the wrapper does return inside the async for loop, the inner generator (main_agent.run_async(ctx)) is abandoned. Python sends GeneratorExit into the generator chain, which unwinds through base_agent.run_async():

base_agent.run_async()
  → tracer.start_as_current_span('invoke_agent ...')  # attaches OTel context token
    → Aclosing(self._run_async_impl(ctx))
      → async for event in agen:
        → yield event  # GeneratorExit thrown here
    # __exit__ tries to detach token → ValueError

Error output

ERROR:opentelemetry.context:Failed to detach context
Traceback (most recent call last):
  File ".../opentelemetry/trace/__init__.py", line 589, in use_span
    yield span
  File ".../opentelemetry/sdk/trace/__init__.py", line 1105, in start_as_current_span
    yield span
  File ".../google/adk/agents/base_agent.py", line 304, in run_async
    yield event
GeneratorExit

During handling of the above exception, another exception occurred:

  File ".../opentelemetry/context/__init__.py", line 155, in detach
    _RUNTIME_CONTEXT.detach(token)
  File ".../opentelemetry/context/contextvars_context.py", line 53, in detach
    self._current_context.reset(token)
ValueError: <Token ...> was created in a different Context

This repeats for each nesting level in the agent/tracing hierarchy (4 errors for a typical pipeline).

Root cause analysis

base_agent.run_async() at line 294 uses a synchronous context manager (tracer.start_as_current_span()) wrapping an async generator's yield point:

with tracer.start_as_current_span(f'invoke_agent {self.name}') as span:
    # ...
    async with Aclosing(self._run_async_impl(ctx)) as agen:
        async for event in agen:
            yield event  # <-- GeneratorExit enters here

The start_as_current_span context manager calls context.attach() on __enter__ (storing a contextvars.Token) and context.detach(token) on __exit__. When GeneratorExit is thrown into the generator, __exit__ fires, but the ContextVar.reset(token) fails because the token was created in a different contextvars.Context (the async context has shifted between generator suspension and cleanup).

Workaround

Application-level: Suppress the OTel context logger:

logging.getLogger("opentelemetry.context").setLevel(logging.CRITICAL)

Wrapper agent level: Use break instead of return inside the async for loop, and yield the redirect event after the loop exits. This ensures the inner generator's cleanup (including OTel span detach) completes before the wrapper yields:

@override
async def _run_async_impl(self, ctx) -> AsyncGenerator[Event, None]:
    redirect_event = None
    async for event in self.main_agent.run_async(ctx):
        if should_block(event):
            redirect_event = create_redirect_event(ctx)
            break  # inner generator closed here, in same async context
        yield event
    if redirect_event is not None:
        yield redirect_event

Note: The break workaround may or may not resolve the issue depending on whether Python's async for cleanup after break preserves the contextvars.Context. In our testing, it reduces but may not fully eliminate the errors.

Suggested fix

Consider wrapping the tracing in base_agent.run_async() with GeneratorExit-safe cleanup:

async def run_async(self, parent_context):
    with tracer.start_as_current_span(f'invoke_agent {self.name}') as span:
        ctx = self._create_invocation_context(parent_context)
        tracing.trace_agent_invocation(span, self, ctx)
        try:
            if event := await self._handle_before_agent_callback(ctx):
                yield event
            if ctx.end_invocation:
                return
            async with Aclosing(self._run_async_impl(ctx)) as agen:
                async for event in agen:
                    yield event
            if ctx.end_invocation:
                return
            if event := await self._handle_after_agent_callback(ctx):
                yield event
        except GeneratorExit:
            # Suppress OTel context detach errors during generator cleanup
            span.end()
            return

Or use an async-generator-safe tracing wrapper that doesn't rely on contextvars.Token for cleanup.

Related issues

Bug: Failed to detach context #423 (closed) — same error, different trigger (parallel tool downloads)
ERROR:opentelemetry.context:Failed to detach context #1670 (closed) — same error, SSE streaming context
Context Detachment Errors with Human-in-the-Loop Tool Confirmation #3607 (closed) — same error, Human-in-the-Loop tool confirmation
Support for Disabling or Extending Internal OpenTelemetry Tracing in Google ADK (Python) #2792 — request to disable/extend internal OTel tracing

The suggested fix doesn't work:

from opentelemetry.sdk.trace import TracerProvider
import asyncio
from opentelemetry.trace import get_current_span

tracer_provider = TracerProvider()
tracer = tracer_provider.get_tracer('my_tracer')


async def run_async():
    with tracer.start_as_current_span(f'invoke_agent') as span:
        try:
            yield 1
            yield 2
        except GeneratorExit:
            raise Exception("This doesn't actually get triggered when the loop breaks")
            span.end()
            return


async def main():
    async for value in run_async():
        print(value)
        break

    print(get_current_span())


asyncio.run(main())

The exception is still logged, and also the span remains active:

Failed to detach context
Traceback (most recent call last):
  File "/Users/alex/work/logfire/.venv/lib/python3.12/site-packages/opentelemetry/trace/__init__.py", line 589, in use_span
    yield span
  File "/Users/alex/work/logfire/.venv/lib/python3.12/site-packages/opentelemetry/sdk/trace/__init__.py", line 1105, in start_as_current_span
    yield span
  File "/Users/alex/Library/Application Support/JetBrains/PyCharm2025.3/scratches/scratch_2774.py", line 10, in run_async
    yield 1
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/alex/work/logfire/.venv/lib/python3.12/site-packages/opentelemetry/context/__init__.py", line 155, in detach
    _RUNTIME_CONTEXT.detach(token)
  File "/Users/alex/work/logfire/.venv/lib/python3.12/site-packages/opentelemetry/context/contextvars_context.py", line 53, in detach
    self._current_context.reset(token)
ValueError: <Token var=<ContextVar name='current_context' default={} at 0x10d912070> at 0x10dc7c180> was created in a different Context
1
_Span(name="invoke_agent", context=SpanContext(trace_id=0x8500018b8cdc046f8cd55c3405cf026f, span_id=0x6529f990576b69fe, trace_flags=0x01, trace_state=[], is_remote=False))

See https://logfire.pydantic.dev/docs/reference/advanced/generators/

The fact is you can't wrap generators in spans like this. This means the async for event in runner.run_async interface itself has to change into one where the user first opens a context manager, so that they can reliably signal when they're done.

The root cause is that start_as_current_span spans the yield point inside run_async, so the OTel context token gets created in one contextvars.Context — but when the generator is abandoned early, the cleanup fires in a different one, which is what triggers the error.

One workaround worth trying: in base_agent.py, replace the with tracer.start_as_current_span(...) block with explicit tracer.start_span() + otel_context.attach().
Then in the finally clause, briefly push the opentelemetry.context logger level above CRITICAL before calling otel_context.detach(token), restore it afterwards, and call span.end().
That should suppress the ValueError from surfacing without leaking the span.

Could you give this a shot on your side and see if it consistently clears the errors in your pipeline?
If it holds up across both the early-exit and normal-completion paths, that'd be a solid foundation for a PR — just want to make sure it actually works in your setup before taking it further.

No, the whole point is that the detach fails, hence the log. If you have to suppress the log, the error is still happening. The span still leaks.

from opentelemetry.sdk.trace import TracerProvider
import asyncio
from opentelemetry.trace import get_current_span, set_span_in_context
from opentelemetry import context

tracer_provider = TracerProvider()
tracer = tracer_provider.get_tracer('my_tracer')


async def run_async():
    with tracer.start_span(f'invoke_agent') as span:
        token = context.attach(set_span_in_context(span))
        try:
            yield 1
            yield 2
        finally:
            print("Detaching context")
            context.detach(token)


async def main():
    async for value in run_async():
        print(value)
        break
    print(get_current_span())


asyncio.run(main())

Failed to detach context
Traceback (most recent call last):
...
ValueError: <Token var=<ContextVar name='current_context' default={} at 0x1163da6b0> at 0x11676ff80> was created in a different Context
1
_Span(name="invoke_agent", context=SpanContext(trace_id=0xac52ae30130dcade960b8a26d26fe95d, span_id=0x3d0461f9ae583b45, trace_flags=0x01, trace_state=[], is_remote=False))
Detaching context

Note print(get_current_span()) printing _Span(name="invoke_agent", showing the leakage. Detaching context only gets printed after because the finally isn't triggered by the loop ending, it's triggered by the generator being garbage collected. That happens at an unreliable time in a different context, hence the error log.

If you do this:

async def main():
    g = run_async()
    async for value in g:
        print(value)
        break
    await g.aclose()
    print(get_current_span())

then it works correctly:

1
Detaching context
NonRecordingSpan(SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False))

Note the change in order of the lines as well as the NonRecordingSpan. And then you can just go back to start_as_current_span without needing any special stuff in run_async.

fix(telemetry): prevent OTel context detach errors on generator cance…

test(telemetry): add tests for OTel context-detach safety on generato…

You're right — suppressing the log was just hiding the symptom while the span continued to leak underneath, which is the real problem.
And your point about the finally running at GC time in the wrong async context is exactly why my suggestion didn't hold up.

The explicit aclose() approach you showed is the correct fix — it forces the generator cleanup to happen in the right context, so the detach succeeds and the span closes cleanly.

Looks like mmphego has already filed PR #4919 with a fix — will take a look to make sure it covers both the early-exit path and normal completion correctly.

@jawoszek

@jawoszek Please have a look into it once.

Our approach to this was to use aclosing for every async generator - see #1670.

All snippets here are missing the aclosing, so I'm assume that's the missing piece. Taking the repro snipped from issue description, I think the correctly working code should look like:

import contextlib
from google.adk.agents import BaseAgent, LlmAgent
from google.adk.events.event import Event
from typing import AsyncGenerator
from typing_extensions import override

class WrapperAgent(BaseAgent):
    main_agent: BaseAgent

    @override
    async def _run_async_impl(self, ctx) -> AsyncGenerator[Event, None]:
        async with contextlib.aclosing(self.main_agent.run_async(ctx)) as agen:
            async for event in agen:
                if should_block(event):
                    yield create_redirect_event(ctx)
                    return  # <-- triggers GeneratorExit on inner generator chain
                yield event

Could you check if that works?

This issue has been automatically marked as stale because it has not had recent activity for 7 days after a maintainer requested clarification. It will be closed if no further activity occurs within 7 days.

This has been automatically closed because it has been marked as stale for over 7 days.

Still reproducing in google-adk==2.2.0 with streaming + async generators

Environment:

google-adk==2.2.0
Python 3.12
PostgreSQL session service (DatabaseSessionService with asyncpg)
Gemini 3 Flash (streaming mode)

When using async streaming agents with run_async(), OpenTelemetry raises:

ERROR:opentelemetry.context:Failed to detach context
ValueError: <Token var=<ContextVar name='current_context' ...> at 0x...>
was created in a different Context
This causes ADK to retry the request up to 5 times, adding ~10–11 seconds
of latency per invocation.

Root cause (as understood):
OTel span tokens are created in one asyncio.Task context but detach()
is called during GeneratorExit cleanup in a different context — violating
PEP 567 / contextvars design.

Workarounds we've tried:

OTEL_SDK_DISABLED=true — eliminates the error but disables all tracing.
Monkey-patching opentelemetry.context.detach at startup:

import opentelemetry.context as _otel_ctx
_orig_detach = _otel_ctx.detach
def _safe_detach(token):
    try:
        _orig_detach(token)
    except ValueError:
        pass
_otel_ctx.detach = _safe_detach
This suppresses the crash but the underlying context mismatch still occurs.
Or admin can fixing this error

@hungpq-ai

HI @hungpq-ai

Thanks for reporting your findings.

Could you please prepare a minimal reproducible example?

I'm most interested in if your call to run_async() wraps the async generator in contextlib.aclosing. If not, this is likely to fix the issue. If it doesn't fix your issue we may have missed this wrapping in the ADK runtime. OTel API unfortunately relies on contextvars, which doesn't play very well with async generators, hence the need for contextlib.aclosing.

@RKest

@RKest, my environment is very similar to that of @hungpq-ai and I believe I'm experiencing the same issue with the instrumentation here: https://github.com/Arize-ai/openinference/blob/789d41974c08a9a13147977f28ef4142a07e2106/python/instrumentation/openinference-instrumentation-google-adk/src/openinference/instrumentation/google_adk/__init__.py#L54

This looks to me like a regression in the 2.0 release. Prior to then the call to run_async is wrapped in aclosing: https://github.com/google/adk-python/blob/v1.31.1/src/google/adk/runners.py#L605. With the new node execution path it is not:

adk-python/src/google/adk/agents/base_agent.py

Line 322 in 54c5fd2

async for event in self.run_async(

Navigation Menu

Repository navigation

OpenTelemetry context detach errors when BaseAgent wrapper exits inner agent's run_async() early #4894

Describe the bug

To Reproduce

Error output

Root cause analysis

Workaround

Suggested fix

Related issues

Still reproducing in google-adk==2.2.0 with streaming + async generators

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

OpenTelemetry context detach errors when BaseAgent wrapper exits inner agent's run_async() early #4894

Description

Describe the bug

To Reproduce

Error output

Root cause analysis

Workaround

Suggested fix

Related issues

Activity

alexmojaki commented on Mar 19, 2026

surajksharma07 commented on Mar 20, 2026

alexmojaki commented on Mar 20, 2026

surajksharma07 commented on Mar 23, 2026

surajksharma07 commented on Mar 23, 2026

jawoszek commented on Apr 17, 2026

adk-bot commented on Apr 25, 2026

adk-bot commented on May 2, 2026

hungpq-ai commented on Jun 16, 2026

Still reproducing in google-adk==2.2.0 with streaming + async generators

RKest commented on Jun 16, 2026

rmoneys commented on Jun 23, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions