feat: project finish_reason from LlmResponse to attributes in BigQueryAgentAnalyticsPlugin LLM_RESPONSE events

Summary

BigQueryAgentAnalyticsPlugin does not project the Vertex AI finish_reason field to the attributes column of LLM_RESPONSE events written to BigQuery. This makes it impossible to classify model failure modes (MAX_TOKENS, SAFETY, MALFORMED_FUNCTION_CALL, RECITATION, etc.) via SQL queries against the analytics table — operators must instead parse unstructured Cloud Logging output to recover this signal.

Adding the field would be a small change (~5 lines) with significant observability value for any project using the plugin.

Current behavior (verified empirically 2026-05-08)

SELECT
  JSON_VALUE(attributes,'$.finishReason')                AS camel,
  JSON_VALUE(attributes,'$.finish_reason')               AS snake,
  JSON_VALUE(attributes,'$.usage_metadata.finish_reason') AS nested,
  COUNT(*) AS row_count
FROM `<project>.<dataset>.events`
WHERE event_type = 'LLM_RESPONSE'
  AND timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)
GROUP BY camel, snake, nested
ORDER BY row_count DESC;

Result against a 7-day window in production:

camel	snake	nested	row_count
NULL	NULL	NULL	309

finish_reason is not present in any path (camelCase, snake_case, or nested under usage_metadata). Source inspection confirms why: in google/adk/plugins/bigquery_agent_analytics_plugin.py, _EVENT_VIEW_DEFS[\"LLM_RESPONSE\"] (around lines 1813–1846 in 1.32.0) extracts only response, usage_*_tokens, cached_content_token_count, context_cache_hit_rate, total_ms, ttft_ms, model_version, usage_metadata, cache_metadata. There are zero references to finish_reason / finishReason anywhere in the 3500+ lines of the plugin file. The EventData dataclass and after_model_callback likewise do not capture it, even though the LlmResponse parameter the callback receives includes it from Vertex.

Use case / motivation

Failure mode breakdown via SQL: today, distinguishing "this LLM call ended with MAX_TOKENS" from "this LLM call ended with MALFORMED_FUNCTION_CALL" via the analytics table is impossible. The information lives only in unstructured stderr logs, which are not joinable to the structured event stream.
Cache hit ratio segmented by finish_reason: the existing cached_content_token_count projection enables computing cache hit ratio (we use this heavily). But understanding the quality of the cached responses (did the cached prefix lead to clean STOPs or to MALFORMED retries?) requires the finish_reason dimension.
Model migration health checks: when migrating between model families (e.g., Gemini 2.5 → 3.x), SRE-grade observability needs to compare MALFORMED rate, SAFETY rate, etc., across revisions side by side. Without finish_reason in BQ, the comparison must be reconstructed from logs — slow, brittle, and not amenable to alert policies.
Alerting on regression: a Cloud Monitoring alert like "MALFORMED_FUNCTION_CALL rate > 5% sustained over 1h" is straightforward to write against the BQ events table if finish_reason is projected. Without it, the same alert requires a log-based metric pipeline (more moving parts, more cost).

Proposed solution

Add finish_reason (and optionally finish_message) to the LLM_RESPONSE view definition. The field is already present on LlmResponse from Vertex; the plugin only needs to read and project it. Roughly (against 1.32 source structure):

# In _EVENT_VIEW_DEFS[\"LLM_RESPONSE\"]:
\"finish_reason\": lambda llm_response: (
    llm_response.finish_reason.name
    if llm_response.finish_reason is not None
    else None
),
\"finish_message\": lambda llm_response: llm_response.finish_message,

Backwards-compatible: existing consumers reading the documented fields are unaffected; new consumers can opt in via JSON_VALUE(attributes,'$.finish_reason').

If a different snake_case vs camelCase convention is preferred to match other ADK projections, happy to adjust — the empirical probe checked both spellings to be safe.

Alternatives considered

after_model_callback writing to state_delta: feasible as a downstream workaround, but every consumer of the plugin needs to re-implement it, and it bloats the event stream with one extra row per LLM call.
Custom plugin subclass overriding LLM_RESPONSE EventData: surgical but creates maintenance burden when the upstream plugin schema evolves.
Cloud Logging sink → BigQuery via Logs Router: works but introduces a second pipeline (sink config, log-based filters, parsing) for data that is already flowing through the analytics plugin one level up.

All three alternatives are strictly more code and more complexity than projecting the field in the plugin where the data already lives.

Environment

google-adk 1.32.0
Python 3.12
Vertex AI backend (`vertexai=True`)
Plugin: BigQueryAgentAnalyticsPlugin (default config, single dataset, single events table)
Models tested: gemini-2.5-flash

Happy to send a PR if the team agrees this is in scope. Thanks for the great library.

@roanny

Hi @roanny , Thanks for providing the detailed description. Please go ahead and raise a PR; our team will be happy to review it.

@roanny

@roanny - Are you working on this ? If not, I'd be happy to collaborate and work on this

@darshil3011

@darshil3011 Thanks for the offer — I'm actively working on it now and will open a PR shortly with the finish_reason (and finish_message) projection plus a unit test. Happy to tag you for review when it's up.

feat(plugins): project LlmResponse finish_reason to BigQuery analytics

@roanny

Sure, I can review @roanny

@roanny

Hi @roanny, thanks for raising a PR. Our team will review it and let you know if anything further is needed.
We appreciate your contribution.

@shobsi

@shobsi Could you please help with this?

I'm working on this issue for now, would keep you posted

@roanny

Thanks for the detailed report and the empirical probe — confirmed this is still the case at HEAD: the plugin doesn't project finish_reason anywhere, while LlmResponse.finish_reason is available in after_model_callback.

Status update on the fix:

@roanny's PR feat(plugins): project LlmResponse finish_reason to BigQuery analytics #5704 has the right approach (EventData field → attributes["finish_reason"] via .name → JSON_VALUE view column), but it's now conflicting — the plugin file has been substantially restructured since May. @roanny, are you still able to rebase? If not, I'll carry the change forward with attribution.
One adjustment to the original proposal: LlmResponse has no finish_message field — the candidate's finish message is folded into LlmResponse.error_message at response-creation time. So the plan is to project finish_reason (enum name, e.g. STOP, MAX_TOKENS, MALFORMED_FUNCTION_CALL) and, separately, route llm_response.error_message into the existing error_message column for non-STOP outcomes.
Field will be snake_case ($.finish_reason), matching the other enrichment attributes. Streaming note: only the final chunk of a streamed response carries finish_reason, so partial-chunk rows will simply omit the key — final rows are the ones to filter on.

Your alerting/migration use cases (MALFORMED rate, SAFETY rate over model revisions) will work directly via JSON_VALUE(attributes,'$.finish_reason') once this lands, and there's also a dedicated view column planned so you won't need the JSON path.

Implementation anchors for whoever carries this forward (verified against current HEAD, 1026d2da):

EventData dataclass: bigquery_agent_analytics_plugin.py:3811-3840 — add the finish_reason field here.
Extraction point: after_model_callback, EventData construction at :6797-6805.
Attribute assembly: _build_attributes at :5764-5784.
Typed view: _EVENT_VIEW_DEFS["LLM_RESPONSE"] at :3610-3653 — add JSON_VALUE(attributes, '$.finish_reason') AS finish_reason.
Test anchors: test_after_model_callback_text_response (tests/unittests/plugins/test_bigquery_agent_analytics_plugin.py:1815), the LLM_RESPONSE view-column test (:6568), and the streaming lifecycle tests (:2965-3038) for the partial-vs-final-chunk case.

Rebase scope for #5704: the plugin file has roughly doubled since May (~3.5k → ~7k lines) and the attribute/view sections moved, so the hunks won't apply cleanly — but the design maps 1:1 onto the anchors above.

One extra nuance worth deciding in the PR: a response that returns content and a non-STOP finish_reason (e.g. MAX_TOKENS with partial text) currently lands as LLM_RESPONSE with the default status="OK". Routing error_message for non-STOP outcomes (as proposed above) covers the message, but I'd keep status orthogonal — consumers filter on finish_reason directly rather than overloading the error predicate. Flagging so it's a conscious choice, not an accident.

@roanny

Follow-up after checking the current schema, view, and streaming paths in more detail:

One scope correction to my previous note: I recommend keeping #5704 strictly focused on finish_reason. Although non-STOP candidate text can reach LlmResponse.error_message, copying that text into the existing row-level error_message field would change both status and privacy semantics. That deserves a separate design/review and should not block this additive fix.

Merge checklist:

Add optional finish_reason to EventData.
Populate it from llm_response.finish_reason using one stable string representation (the enum name/value, not str(enum)).
Add attributes.finish_reason only when present.
Expose finish_reason in the generated BigQuery view; no physical event-table schema migration is needed because the event body is JSON.
Cover STOP, MAX_TOKENS, SAFETY/MALFORMED-style outcomes, missing values, streaming partial events, and the final response.
Keep response/error text out of this change and verify it remains covered by the payload/privacy filters.

#5704 still has merge conflicts. I’ll wait for @roanny’s rebase response; if they cannot continue, the patch can be ported onto current main with attribution.

Implementation follow-up: the current-main port is now in the mono draft PR caohy1988#12 (commit 8994a025).

It adds the optional EventData.finish_reason, records the stable enum .name on final responses, omits it on partial/missing values, and exposes a typed finish_reason column in v_llm_response. Coverage includes STOP, MAX_TOKENS, SAFETY, MALFORMED_FUNCTION_CALL, and missing values.

One deliberate update to my last scope note: the draft also routes llm_response.error_message into the existing row-level error_message column. It goes through the plugin's existing sanitizer/privacy pipeline, and tests assert that sensitive text is redacted. status remains orthogonal and stays OK for a returned LLM response, so this does not reclassify non-STOP responses as callback failures. Legacy response-like objects without either new attribute remain supported.

The complete plugin unit file passes: 477 tests.

feat(plugins): Expose BigQuery LLM termination details

@roanny

Correction / final status: this is fixed at upstream HEAD. The change merged to main on 2026-08-10 in 04b8b72 (feat(plugins): add BigQuery Agent Analytics delivery and termination observability), landing via the internal review + Copybara flow. My earlier note describing it as still pending on a fork draft was out of date — apologies for the confusion.

What shipped (re-verified line-by-line at current HEAD):

EventData.finish_reason records the stable enum .name (with a plain-string fallback for backends that surface finish_reason as a string, e.g. LiteLLM), and is recorded only on final responses — partial streaming chunks omit it, so progressive SSE cannot double count.
Projected as attributes.finish_reason and as a typed finish_reason column in the v_llm_response view, so both JSON_VALUE(attributes,'$.finish_reason') and the view column support the failure-mode breakdown, cache-quality segmentation, migration-health, and alerting queries from the issue description.
llm_response.error_message is routed into the existing error_message column through the plugin's sanitizer pipeline; status stays OK for returned responses, so non-STOP outcomes are not reclassified as failures.
Coverage includes STOP / MAX_TOKENS / SAFETY / MALFORMED_FUNCTION_CALL, missing values, string finish reasons, and the partial-vs-final streaming split.

Availability: the merge postdates v2.6.3, so it ships in the next release; until then it's available from main.

Closing as fixed. Thanks @roanny for the unusually precise report and empirical probe — the shipped design follows your proposal, adjusted only where LlmResponse differs from the Vertex candidate shape (no finish_message field; that text arrives via error_message).

Fixed at HEAD by 04b8b72; ships in the next release after v2.6.3.

Navigation Menu

Repository navigation

feat: project `finish_reason` from LlmResponse to attributes in BigQueryAgentAnalyticsPlugin LLM_RESPONSE events #5644

Summary

Current behavior (verified empirically 2026-05-08)

Use case / motivation

Proposed solution

Alternatives considered

Environment

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

feat: project finish_reason from LlmResponse to attributes in BigQueryAgentAnalyticsPlugin LLM_RESPONSE events #5644

Description

Summary

Current behavior (verified empirically 2026-05-08)

Use case / motivation

Proposed solution

Alternatives considered

Environment

Activity

sanketpatil06 commented on May 11, 2026

darshil3011 commented on May 13, 2026

roanny commented on May 14, 2026

darshil3011 commented on May 14, 2026

sanketpatil06 commented on May 15, 2026

xuanyang15 commented on May 18, 2026

haiyuan-eng-google commented on Aug 5, 2026

caohy1988 commented on Aug 5, 2026

caohy1988 commented on Aug 5, 2026

caohy1988 commented on Aug 5, 2026

caohy1988 commented on Aug 5, 2026

caohy1988 commented on Aug 11, 2026

caohy1988 commented on Aug 11, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

feat: project `finish_reason` from LlmResponse to attributes in BigQueryAgentAnalyticsPlugin LLM_RESPONSE events #5644