Repository navigation
OpenTelemetry integration creates span ID mismatch between LLM_REQUEST and LLM_RESPONSE/LLM_ERROR (BigQuery Analytics Plugin) #4851
Description
When OpenTelemetry tracing is enabled in the ADK, the span_id no longer matches between LLM_REQUEST and its corresponding LLM_RESPONSE (and LLM_ERROR) events.
This happens because before_model_callback runs before the framework's internal call_llm span begins. As a result, the BigQueryAgentAnalyticsPlugin grabs the parent span ID (e.g. from the execute_tool span) to log the LLM_REQUEST. Then, the model executes inside the child call_llm span. The on_model_error_callback and after_model_callback then execute within that child span.
Because of this, LLM_REQUEST.span_id == LLM_RESPONSE.parent_span_id.
This can break any downstream data warehouses or analysis tools that rightfully expect an atomic request/response LLM pair to share the identical span_id. Notably, without OpenTelemetry, span_id matches perfectly between all three events
When querying raw event tables with OTel enabled:
TOOL_STARTING to TOOL_COMPLETED still match perfectly on span_id.
AGENT_STARTING to AGENT_COMPLETED still match perfectly on span_id.
The issue is completely isolated to the LLM flow callbacks.
Activity
fix: ensure LLM callbacks share the same OTel span context (google#4851)
surajksharma07 commentedon Mar 17, 2026
@evekhm the mismatch comes down to timing. before_model_callback fires just before the call_llm span opens in _call_llm_async, which means LLM_REQUEST ends up picking up the parent span context while LLM_RESPONSE captures the child span.
That's what's causing the pairing to break.
One thing worth trying: move _handle_before_model_callback inside the with tracer.start_as_current_span('call_llm') block so all three callbacks are working within the same span context.
Could you test that on your end and see if it fixes the pairing for you?
If it holds up after some thorough testing, it'd be great to carry that forward into #4854!
brucearctor commentedon Mar 18, 2026
@surajksharma07 Spot on with the diagnosis! PR #4854 implements exactly that — but found one additional subtlety worth noting:
Just moving the
before_model_callbackinside thecall_llmspan isn't enough on its own. Theafter_model_callbackruns inside an async for loop that consumes yields from the error handler, which creates a childgenerate_contentspan viause_inference_span. Because async generators preserve their span context across yields, the after model callback ends up seeing the child span instead ofcall_llm.The fix in #4854 addresses both sides:
v0.0.4 (Bug Fixes & Observability Hardening)
surajksharma07 commentedon Mar 26, 2026
Good catch on the trace.use_span detail, @brucearctor — the async generator span context persisting across yields is easy to miss, and wrapping after_model_callback to re-activate the call_llm span makes the fix properly complete on both sides.
@evekhm given your v0.0.4 commit it looks like you may have already verified this on your end — could you confirm whether the two-part approach in PR #4854 fully resolves the LLM_REQUEST/LLM_RESPONSE pairing for you?
Once we have that confirmation, we'll get it through review with @jawoszek and move toward merge.
Team is looking more into it.
surajksharma07 commentedon Mar 26, 2026
@jawoszek Please have a look into it once.
fix(telemetry): ensure all model callbacks observe the same call_llm …
fix: pass call_llm span to error callbacks and use real TracerProvide…
haiyuan-eng-google commentedon Mar 30, 2026
Fixed in #5041
Summary
before_model_callbackinside thecall_llmspan so plugins see a valid span (previously ran before the span started)after_model_callbackwithtrace.use_span(span, end_on_exit=False)to rebind to thecall_llmspan (previously could see a child inference span)call_llm_spaninto_run_and_handle_errorand wrapson_model_error_callbackthe same waycall_llmspan IDChanges
src/google/adk/flows/llm_flows/base_llm_flow.py:_call_llm_async(): Movedbefore_model_callback, label setup, and__get_llm()inside thewith tracer.start_as_current_span('call_llm') as span:blockafter_model_callbackcall sites wrapped withtrace.use_span(span, end_on_exit=False)_run_and_handle_error(): New optionalcall_llm_spanparameter; error callbacks run underuse_span(call_llm_span)when providedcall_llm_spantests/unittests/flows/llm_flows/test_llm_callback_span_consistency.py(new):test_before_and_after_callbacks_share_same_span— success pathtest_callbacks_same_trace_id— same tracetest_before_and_error_callbacks_share_same_span— error pathtest_short_circuit_before_callback_sees_valid_span— short-circuit pathtest_all_three_callbacks_share_span_on_error— error recovery path (all 3 callbacks)Design decisions
end_on_exit=False: Thecall_llmspan is already managed by thewith tracer.start_as_current_span(...)context manager.use_spanis only used to re-activate the span context, not to end it.call_llm_spandefaults toNone; when absent, behavior is unchanged.BigQueryAgentAnalyticsPlugin. That should be a separate follow-up PR after this fix is proven in production.Test plan
llm_flowstests pass (0 regressions)