Skip to content

Exposed FastAPI server is not instrumented automatically using get_fast_api_app #4767

Closed

Description

@conorhodder

🔴 Required Information

Please ensure all items in this section are completed to allow for efficient
triaging. Requests without complete information may be rejected / deprioritized.
If an item is not applicable to you - please mark it as N/A

Describe the Bug:
When the ADK API server (get_fast_api_app) receives inbound HTTP requests containing W3C traceparent headers, the headers are ignored. The server starts new trace roots instead of joining the caller's trace. This breaks distributed trace correlation for any service calling the ADK agent over HTTP.

The root cause is that AdkWebServer.get_fast_api_app() creates a TracerProvider and configures W3C TraceContextTextMapPropagator, but never applies FastAPIInstrumentor to the FastAPI app. Without the ASGI instrumentation middleware, the traceparent header on inbound requests is never extracted, so each request starts a new trace.

Steps to Reproduce:

  1. Deploy an ADK agent with get_fast_api_app(agents_dir=..., web=False)
  2. Set OTEL_EXPORTER_OTLP_ENDPOINT to a Jaeger/collector endpoint and OTEL_SERVICE_NAME to identify the agent
  3. From a calling service (e.g. Next.js with @opentelemetry/sdk-node), send a request to the agent's /run_sse endpoint — the caller's OTel SDK will automatically attach a traceparent header
  4. Observe in Jaeger that the caller and agent traces are separate trace roots with different trace IDs, rather than a single correlated trace

Expected Behavior:
The ADK server should extract the traceparent header from inbound HTTP requests and create child spans under the caller's trace, producing a single end-to-end distributed trace (e.g. caller → agent → LLM).

Observed Behavior:
The agent creates a new trace root for every request. The caller's trace ends at the outbound fetch span, and the agent's spans appear as a disconnected trace in Jaeger. The traceparent header is present on the wire but never consumed.

Workaround: manually applying FastAPIInstrumentor().instrument_app(app) after get_fast_api_app() returns fixes the issue, since the TracerProvider and TraceContextTextMapPropagator are already correctly configured by the ADK.

Environment Details:

  • ADK Library Version: google-adk 1.25.0
  • Desktop OS: Linux 6.10.14-linuxkit (Docker)
  • Python Version: 3.14.3

Model Information:

  • Are you using LiteLLM: No
  • Which model is being used: gemini-2.5-flash

🟡 Optional Information

Providing this information greatly speeds up the resolution process.

Regression:
N/A — the ADK has never applied FastAPIInstrumentor to the app returned by get_fast_api_app(). The _setup_instrumentation_lib_if_installed() function only instruments GoogleGenAiSdkInstrumentor, not FastAPI.

Logs:
Verified by inspecting the ADK source:

# In AdkWebServer.get_fast_api_app():
#   _setup_telemetry() → configures TracerProvider + W3C propagator ✓
#   _setup_instrumentation_lib_if_installed() → only instruments GoogleGenAiSdkInstrumentor ✗
#   FastAPIInstrumentor is never called on the app ✗

# Propagator IS correctly configured:
>>> from opentelemetry import propagate
>>> p = propagate.get_global_textmap()
>>> [type(x).__name__ for x in p._propagators]
['TraceContextTextMapPropagator', 'W3CBaggagePropagator']

# But without FastAPI middleware, the propagator is never invoked on inbound requests.

Minimal Reproduction Code:
Please provide a code snippet or a link to a Gist/repo that isolates the issue.

# server.py — ADK agent server
from google.adk.cli.fast_api import get_fast_api_app

app = get_fast_api_app(agents_dir="./agents", web=False)
# No FastAPIInstrumentor applied → traceparent ignored

# caller.py — service calling the agent
import httpx
from opentelemetry import trace

tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("call-agent"):
    # OTel httpx instrumentation injects traceparent header automatically
    resp = httpx.post("http://agent:8000/run_sse", json={...})
    # Agent receives traceparent but creates a new trace root

How often has this issue occurred?:

  • Always (100%)

Suggested Fix
In AdkWebServer.get_fast_api_app(), after the app is created, add:

try:
    from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
    FastAPIInstrumentor().instrument_app(app)
except ImportError:
    pass

This is consistent with how _setup_instrumentation_lib_if_installed() already handles GoogleGenAiSdkInstrumentor with a try/except ImportError pattern.

Activity

  1. added
    tracing[Component] This issue is related to OpenTelemetry tracing
    on Mar 9, 2026
  2. added theissue type on Mar 9, 2026
  3. self-assigned this
    on Mar 9, 2026
  4. saschabuehrle commented on Mar 9, 2026

    @saschabuehrle

    Thanks for the detailed repro — this explains the broken parent/child linkage clearly.

    I’d support default instrumentation in get_fast_api_app() (best-effort, non-breaking), plus a short docs note clarifying the tracing contract:

    • ADK should extract inbound traceparent when available,
    • continue trace through agent execution,
    • and preserve propagation on outbound calls.

    Having that explicitly documented helps teams debug correlation issues quickly and aligns expectations for production observability.

  5. klateefa commented on Mar 11, 2026

    @klateefa

    @conorhodder Thanks for reporting this. The issue happens because the FastAPI app created by get_fast_api_app() is not connected to OpenTelemetry. So when a request comes with trace headers, the agent does not read them and starts a new trace.
    A simple fix is to instrument the FastAPI app when OpenTelemetry is available, so the agent can continue the same trace.

    try:
        from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
        FastAPIInstrumentor.instrument_app(app)
    except ImportError:
        pass
    

    This keeps everything working as before and enables proper end-to-end tracing.
    Thank you

  6. added 3 commits that reference this issue on Mar 24, 2026
    8f964b5
    8d9c478
    6e32259
  7. added
    request clarification[Status] The maintainer need clarification or more information from the author
    on Apr 17, 2026
  8. added a commit that references this issue on Apr 24, 2026
    ef47fe4
  9. added 2 commits that reference this issue on May 9, 2026
    72d3f8d
    f0ffa9a
  10. surajksharma07 commented on May 27, 2026

    @surajksharma07
    Collaborator

    @conorhodder — just following up here. The fix hasn't landed in a released version yet — FastAPIInstrumentor is still not auto-applied in get_fast_api_app(), so the manual workaround from the Mar 11 comment still applies: call FastAPIInstrumentor().instrument_app(app) right after get_fast_api_app() returns.

    Community PR #4972 from @STHITAPRAJNAS is actively in review and should bring the proper upstream fix once merged.

    Since we haven't heard back, going ahead and closing this out — feel free to reopen once PR #4972 lands or if the workaround isn't covering your case!

  11. added a commit that references this issue on Jun 7, 2026
    da42210
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

request clarification[Status] The maintainer need clarification or more information from the authortracing[Component] This issue is related to OpenTelemetry tracing

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions