"Failed to detach context" (Token created in a different Context) logged as ERROR when caller stops runner.run_async() early

Description

When a caller stops iterating Runner.run_async() early (a break/return as soon as the final response arrives — a very common pattern), an ERROR-level log is emitted on every invocation:

ERROR opentelemetry.context Failed to detach context
Traceback (most recent call last):
  File ".../opentelemetry/trace/__init__.py", line 589, in use_span
    yield span
  File ".../opentelemetry/trace/__init__.py", line 508, in start_as_current_span
    yield span
  File ".../google/adk/runners.py", line 572, in _run_node_async
    yield event
GeneratorExit

During handling of the above exception, another exception occurred:
  File ".../opentelemetry/context/__init__.py", line 143, in detach
    _RUNTIME_CONTEXT.detach(token)
  File ".../opentelemetry/context/contextvars_context.py", line 42, in detach
    self._current_context.reset(token)
ValueError: <Token ...> was created in a different Context

Functionally the invocation completes correctly (OpenTelemetry swallows the exception inside detach() and only logs it), so this is cosmetic — but it is logged at ERROR level on every early-terminated run, which creates significant log noise and false-positive error-rate alerts (e.g. Datadog).

Environment

google-adk: observed in production on 2.3.0; the relevant structure is unchanged through v2.6.1 (verified by source inspection)
Python 3.13 (also reproduces on 3.9–3.12)
FastAPI + uvicorn (--workers 2), run under ddtrace-run (Datadog APM); logs shipped to Datadog
OTEL_SDK_DISABLED=true is set
opentelemetry-api / -sdk: current 1.x (its context.detach() wraps the failure in try/except and calls logger.exception("Failed to detach context"))

Note: setting OTEL_SDK_DISABLED=true does not suppress this — that flag disables the SDK (export/sampling), but context.attach()/detach() live in the context API and keep running. Running under ddtrace-run (which proxies the OTel context) makes the cross-context mismatch surface reliably.

Root cause

runners._run_node_async is an async generator whose event-yielding loop is wrapped in an OpenTelemetry current-span context manager:

# runners.py -> telemetry/_instrumentation.record_invocation (schema v1)
with tracer.start_as_current_span("invocation"):
    ...
    async for event in agen:
        yield event

start_as_current_span does attach() on entry (creating a contextvars Token) and detach(token) on exit. In an async generator the attach() happens in the execution context where the generator is resumed to produce a value. When the caller stops early, the generator is not finalized in that same context — it is closed later by GC / loop.shutdown_asyncgens() / a different task, which raises GeneratorExit (or CancelledError) at the yield. The context manager's __exit__ then calls detach(token) in a different context than the one where the token was created, which is exactly the situation contextvars.Token forbids → ValueError: <Token ...> was created in a different Context.

This is the well-known "start_as_current_span around an async generator" anti-pattern; it only manifests on early close, never on full consumption.

Minimal reproduction (no ADK / API key required)

Reproduces the exact OTel interaction with only opentelemetry-sdk:

import asyncio, contextlib, logging
logging.basicConfig(level=logging.ERROR)
from opentelemetry import trace, context as context_api
from opentelemetry.sdk.trace import TracerProvider
trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer("adk-repro")

@contextlib.contextmanager
def record_invocation_current():            # current ADK structure
    with tracer.start_as_current_span("invocation"):
        yield

async def event_queue():
    for i in range(10):
        await asyncio.sleep(0)
        yield i

async def run_node():                       # ~ runners._run_node_async
    with record_invocation_current():
        async for ev in event_queue():
            yield ev

async def caller_early_stop():
    async for ev in run_node():
        if ev == 2:
            return                          # early stop -> generator closed later
                                            # in a different context

asyncio.run(caller_early_stop())            # -> ERROR "Failed to detach context"

Scenario	Current structure	With proposed fix
Caller stops early (`break`/`return`)	❌ `Failed to detach context` ERROR	✅ clean
Caller consumes to completion	✅ clean	✅ clean (unchanged)

Proposed fix (minimal — schema v1 `invocation` span)

Manage the span/context explicitly and only detach() when the generator body completed normally; skip detach() on early close (the token belongs to a different context and the generator's context is discarded anyway). The span is always ended, so trace completeness is preserved.

--- a/src/google/adk/telemetry/_instrumentation.py
+++ b/src/google/adk/telemetry/_instrumentation.py
@@ def record_invocation(...):
     if resolve_schema_version() < SCHEMA_VERSION_SEMCONV_ALIGNED:
-      with tracing.tracer.start_as_current_span("invocation"):
-        yield
-      return
+      # NOTE: this context manager wraps an async generator (runners.
+      # _run_node_async). If the caller stops iterating early, the generator is
+      # finalized (GeneratorExit / CancelledError) in a different execution
+      # context than the one where the span was attached, so an automatic
+      # detach() would raise "Token was created in a different Context" (OTel
+      # swallows it but logs an ERROR). Detach only on normal completion; always
+      # end the span.
+      span = tracing.tracer.start_span("invocation")
+      token = context_api.attach(trace.set_span_in_context(span))
+      completed = False
+      try:
+        yield
+        completed = True
+      finally:
+        if completed:
+          context_api.detach(token)
+        span.end()
+      return

(trace and context_api are already imported in _instrumentation.py.)

The same start_as_current_span-around-async-generator pattern also exists on the schema v2 path (node_tracing._use_invoke_workflow_span) and could get the same treatment, but this issue/PR intentionally scopes to the default (schema v1) invocation span that users hit today.

Workaround (for users, until fixed)

Consume the generator to completion instead of returning early — capture the final response and let the loop end naturally:

final = None
async for event in runner.run_async(...):
    if event.is_final_response():
        final = event          # don't break/return here
# loop ends naturally -> attach/detach occur in the same context, no error

Related (same root cause, different layers)

ADK Dev UI - Graph error #1170 (secondary "Failed to detach context" note during GeneratorExit)
Multi-agent setting - final agent crashing often with this message #1028, ContextVar Token Error with exit_loop Tool #949, opentelemetry ValueError: "Token was created in a different Context" with ParallelAgent and LlmAgent #860, Crash while exiting loop agent in adk web #501 (context-token detach in Loop/Parallel/AgentTool paths)

Happy to send a PR with the fix + a unit test if this direction looks good.

fix: avoid OTel "Failed to detach context" error on early run_async c…

@GanziMan

Hello @GanziMan ,
Thank you for your contribution. We will review it and reach out to you if we need any additional information.
Since you mentioned that you tested this with version 2.3.0, could you please try testing it with version 2.6.2 as well and share your observations?

@llalitkumarrr

Hello @GanziMan , Thank you for your contribution. We will review it and reach out to you if we need any additional information. Since you mentioned that you tested this with version 2.3.0, could you please try testing it with version 2.6.2 as well and share your observations?

@llalitkumarrr
Thanks for the quick response! I tested on google-adk==2.6.2 (Python 3.13, opentelemetry-api==1.42.1), and the issue still reproduces.

I isolated it to a minimal repro (no LLM/model needed) that drives the actual record_invocation instrumentation and closes the async generator early — this mirrors our production usage, where we iterate runner.run_async(...) and return as soon as we get the final response:

import asyncio, logging
logging.basicConfig(level=logging.INFO)

from google.adk.telemetry._instrumentation import record_invocation
from google.adk.telemetry._schema_version import resolve_schema_version
print("schema version =", resolve_schema_version())  # 1 (LEGACY, default)

async def gen():                              # mirrors _run_node_async
    with record_invocation(None, "conv-1"):   # actual adk 2.6.2 instrumentation
        for i in range(5):
            yield i

async def main():
    async for _ in gen():
        break                                 # consumer early-stop == our early return

asyncio.run(main())

Output:

ERROR:opentelemetry.context:Failed to detach context
Traceback (most recent call last):
  ...
  File ".../opentelemetry/trace/__init__.py", line 443, in start_as_current_span
    yield span
  File ".../google/adk/telemetry/_instrumentation.py", line 72, in record_invocation
    yield
  ...
ValueError: <Token ...> was created in a different Context

It reproduces on both telemetry schema versions — default LEGACY (v1) and ADK_TELEMETRY_SCHEMA_VERSION_OPT_IN=2 (SEMCONV v2).

Root cause: record_invocation (and _run_node_async in 2.3.0) wraps the async generator's yield inside tracer.start_as_current_span(...), which is a sync context manager doing contextvars attach/detach. When the consumer closes the generator early (GeneratorExit / cancellation), the span's __exit__ calls context.detach(token) in a different contextvars.Context than where attach happened, violating the Token reset rule → ValueError.

Suggested direction: avoid wrapping an async-generator yield in a sync start_as_current_span context manager, since the context token can't be safely detached across the generator's suspend/close boundary. Handling GeneratorExit/cancellation explicitly, or guarding the detach against the different-context case, should resolve it.

Happy to help test a fix.

@GanziMan

Hello @GanziMan,

Thank you for the detailed report. There appears to be some conflict in your contribution, could you please take a look at it and fix it?

fix: avoid OTel "Failed to detach context" error on early run_async c…

@llalitkumarrr

Hello @GanziMan,

Thank you for the detailed report. There appears to be some conflict in your contribution, could you please take a look at it and fix it?

@llalitkumarrr @rohityan
Thanks! I've resolved the conflict — the branch is now rebased onto the latest main and the PR shows as mergeable again.

For transparency on what the conflict was: it was only in tests/unittests/telemetry/test_instrumentation.py, where main had meanwhile added new record_invocation tests. I kept all of those upstream tests as-is and simply re-appended my two regression tests after them
(plus merged the imports). The actual source change in _instrumentation.py had no conflict and is unchanged.

After the rebase I ran the full file and all tests pass (the upstream record_invocation span-emit tests + my two new ones), confirming the fix doesn't alter the existing span behavior — the invocation span is still started, made current, and ended exactly as before; only detach() is now skipped on early generator close.

Navigation Menu

Repository navigation

"Failed to detach context" (Token created in a different Context) logged as ERROR when caller stops runner.run_async() early #6559

Description

Environment

Root cause

Minimal reproduction (no ADK / API key required)

Proposed fix (minimal — schema v1 `invocation` span)

Workaround (for users, until fixed)

Related (same root cause, different layers)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

"Failed to detach context" (Token created in a different Context) logged as ERROR when caller stops runner.run_async() early #6559

Description

Description

Environment

Root cause

Minimal reproduction (no ADK / API key required)

Proposed fix (minimal — schema v1 invocation span)

Workaround (for users, until fixed)

Related (same root cause, different layers)

Activity

llalitkumarrr commented on Aug 5, 2026

GanziMan commented on Aug 5, 2026

llalitkumarrr commented on Aug 7, 2026

GanziMan commented on Aug 7, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Proposed fix (minimal — schema v1 `invocation` span)