Repository navigation
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
Projects `finish_reason` from `LlmResponse` to the `attributes` JSON column of `LLM_RESPONSE` events written to BigQuery by `BigQueryAgentAnalyticsPlugin`, and exposes it as a typed column in the generated per-event SQL view. Without this, classifying model failure modes (MAX_TOKENS, SAFETY, MALFORMED_FUNCTION_CALL, RECITATION, etc.) via SQL against the analytics table is impossible — the signal lives only in unstructured Cloud Logging output that is not joinable to the structured event stream. Changes: - `EventData` gets a `finish_reason: Optional[str]` field. - `after_model_callback` reads `llm_response.finish_reason.name` via a defensive getattr (same back-compat pattern as `cache_metadata`, covering legacy response shapes). - `_enrich_attributes` writes `attrs["finish_reason"]` only when set, keeping existing consumers and partial streaming events unaffected. - `_EVENT_VIEW_DEFS["LLM_RESPONSE"]` projects `JSON_VALUE(attributes, '$.finish_reason')` as `finish_reason`. - Schema-field description for `attributes` mentions the new key with example values. Tests: - `test_after_model_callback_projects_finish_reason` — MAX_TOKENS round-trips end-to-end into the BQ row's attributes. - `test_after_model_callback_omits_finish_reason_when_absent` — key is omitted when `LlmResponse.finish_reason` is None. - `TestEnrichAttributes::test_finish_reason_included_when_set` and `..._omitted_when_none` — direct `_enrich_attributes` coverage. - Existing 229 plugin tests still pass; 5 skipped (pre-existing). Fixes google#5644
5063422 to
2198289
Compare
|
@googlebot I signed it! |
|
Hi @roanny , Thank you for your contribution! We appreciate you taking the time to submit this pull request. Your PR has been received by the team and is currently under review. We will provide feedback as soon as we have an update to share. |
Project finish_reason into LLM response attributes and typed views, and route model response diagnostics through the existing sanitized error_message column without changing status semantics. The finish_reason projection follows the approach proposed by @roanny in google#5704. Refs google#5644
Project finish_reason into LLM response attributes and typed views, and route model response diagnostics through the existing sanitized error_message column without changing status semantics. The finish_reason projection follows the approach proposed by @roanny in google#5704. Refs google#5644
|
shipped in 04b8b72, v2.7.0: EventData.finish_reason, the attributes write, reading the enum name off LlmResponse in after_model_callback, and the JSON_VALUE column on the LLM_RESPONSE view. that's the approach from this pr, and the commit should have credited it. head additionally suppresses the key on streaming partials and tolerates a plain string value. #5644 is closed. thanks for the pr and for the issue that drove it. |
Summary
Projects
finish_reasonfromLlmResponseto theattributesJSON column ofLLM_RESPONSEevents written to BigQuery byBigQueryAgentAnalyticsPlugin, and exposes it in the generated per-event-type SQL view.Without this projection, classifying model failure modes (MAX_TOKENS, SAFETY, MALFORMED_FUNCTION_CALL, RECITATION, etc.) via SQL against the analytics table is impossible — the signal lives only in unstructured Cloud Logging output that is not joinable to the structured event stream.
Fixes #5644.
Changes
EventDatagets afinish_reason: Optional[str]field.after_model_callbackreadsllm_response.finish_reason.nameand passes it through toEventData._enrich_attributeswritesattrs["finish_reason"]when present (omitted entirely whenNone, so the JSON shape stays backwards-compatible for existing consumers)._EVENT_VIEW_DEFS["LLM_RESPONSE"]projectsJSON_VALUE(attributes, '$.finish_reason') AS finish_reasonso it shows up as a typed column on the per-event-type view.attributescolumn mentions the new key with example values.Backwards compatibility
The new key is only added to the JSON when
LlmResponse.finish_reasonis notNone, so events emitted before this change and events for streaming partials (wherefinish_reasonis unset) remain unchanged. SQL consumers reading the existing documented fields are unaffected. New consumers can opt in viaJSON_VALUE(attributes, '$.finish_reason')or via thefinish_reasoncolumn exposed by the generated view.Test plan
New unit tests (all passing locally;
229 passed, 5 skippedfor the full plugin suite):test_after_model_callback_projects_finish_reason— assertsMAX_TOKENSround-trips end-to-end into the BQ row'sattributes.test_after_model_callback_omits_finish_reason_when_absent— asserts the key is omitted whenLlmResponse.finish_reasonisNone.TestEnrichAttributes::test_finish_reason_included_when_set— direct_enrich_attributescoverage.TestEnrichAttributes::test_finish_reason_omitted_when_none— direct_enrich_attributescoverage of the None branch.To run locally:
cc @sanketpatil06 @darshil3011