Repository navigation
fix(evaluation): populate developer_instructions when invocation_even… - #5595
cthurston-clgx wants to merge 3 commits into
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
|
Response from ADK Triaging Agent Hello @cthurston-clgx, thank you for your contribution! Before we can merge this pull request, you'll need to sign the Contributor License Agreement (CLA). It seems that the CLA check has failed. Please visit https://cla.developers.google.com/ to sign the agreement. This is a necessary step for us to be able to accept your contribution. Thanks! |
…ts is empty (google#5593) The rubric_based_final_response_quality_v1 evaluator fails to pass developer_instructions to the judge when the agent makes zero tool calls (empty invocation_events list). This occurs because the agent name is resolved exclusively from invocation_events[0].author, with no fallback for the zero-event case. This is critical for evaluating out-of-scope rejection behavior where an agent correctly declines a request without calling any tools. The judge receives an empty <developer_instructions> block and cannot validate rubrics that reference the system prompt. Fix: When invocation_events is empty, fall back to the first agent name in app_details.agent_details to resolve developer_instructions. This mirrors how hallucinations_v1.py handles the same scenario. Fixes google#5593
67ab346 to
ecdc7fd
Compare
rohityan
added
the
request clarification
|
Hi @cthurston-clgx , Thank you for your contribution! We appreciate you taking the time to submit this pull request. Please fix pre-commit check. |
|
Thank you @cthurston-clgx for your contribution! 🎉 Your changes have been successfully imported and merged via Copybara in commit 69c9090. Closing this PR as the changes are now in the main branch. |
4 participants
The rubric_based_final_response_quality_v1 evaluator fails to pass developer_instructions to the judge when the agent makes zero tool calls (empty invocation_events list). This occurs because the agent name is resolved exclusively from invocation_events[0].author, with no fallback for the zero-event case.
This is critical for evaluating out-of-scope rejection behavior where an agent correctly declines a request without calling any tools. The judge receives an empty <developer_instructions> block and cannot validate rubrics that reference the system prompt.
Fix: When invocation_events is empty, fall back to the first agent name in app_details.agent_details to resolve developer_instructions. This mirrors how hallucinations_v1.py handles the same scenario.
Fixes #5593