Repository navigation
AgentEvaluator crashes with ValidationError when evaluating conversation_scenario eval cases #5214
Description
Bug
AgentEvaluator.evaluate() raises a pydantic ValidationError when processing eval cases that use conversation_scenario (LLM-backed user simulation) instead of explicit conversation arrays.
Error
pydantic_core._pydantic_core.ValidationError: 1 validation error for _EvalMetricResultWithInvocation
expected_invocation
Input should be a valid dictionary or instance of Invocation [type=model_type, input_value=None, input_type=NoneType]
at agent_evaluator.py:639
Root Cause
Type mismatch between the public and private invocation result models:
-
EvalMetricResultPerInvocationineval_metrics.py:323correctly declares:expected_invocation: Optional[Invocation] = Field(default=None, ...)
-
_EvalMetricResultWithInvocationinagent_evaluator.py:93incorrectly declares:expected_invocation: Invocation # required, non-optional
local_eval_service.py:285-287 intentionally sets expected_invocation=None when eval_case.conversation is None (i.e., when using conversation_scenario):
EvalMetricResultPerInvocation(
actual_invocation=actual,
expected_invocation=eval_case.conversation[idx]
if eval_case.conversation
else None, # <-- None for conversation_scenario cases
)This None flows through _get_eval_metric_results_with_invocation() at line 636 and into _EvalMetricResultWithInvocation() at line 639, which rejects it.
Fix
Two changes needed in agent_evaluator.py:
1. Make the field optional (line 93):
expected_invocation: Optional[Invocation] = None2. Guard the downstream attribute accesses (lines 439-449 in _print_details):
"prompt": AgentEvaluator._convert_content_to_text(
per_invocation_result.expected_invocation.user_content
if per_invocation_result.expected_invocation else None
),
"expected_response": AgentEvaluator._convert_content_to_text(
per_invocation_result.expected_invocation.final_response
if per_invocation_result.expected_invocation else None
),
...
"expected_tool_calls": AgentEvaluator._convert_tool_calls_to_text(
per_invocation_result.expected_invocation.intermediate_data
if per_invocation_result.expected_invocation else None
),Both _convert_content_to_text and _convert_tool_calls_to_text already accept Optional parameters and handle None gracefully.
Reproduction
# evalset with conversation_scenario (no explicit conversation array)
{
"eval_set_id": "test",
"eval_cases": [{
"eval_id": "scenario_1",
"conversation_scenario": {
"starting_prompt": "Hello",
"conversation_plan": "Ask the agent a question and accept the answer."
},
"session_input": {"app_name": "my_agent", "user_id": "user1", "state": {}}
}]
}# test_eval.py
import pytest
from google.adk.evaluation import AgentEvaluator
@pytest.mark.asyncio
async def test_scenario():
await AgentEvaluator.evaluate(
"my_agent",
"path/to/evalset.json",
num_runs=1,
)Crashes during post-processing in _get_eval_metric_results_with_invocation after all metrics have been computed.
Environment
google-adk[eval]>=1.28.0(also confirmed onmainbranch — same code)- Python 3.11
Activity
fix: make _EvalMetricResultWithInvocation.expected_invocation Optional
fix(eval): make _EvalMetricResultWithInvocation.expected_invocation o…
fix(eval): make _EvalMetricResultWithInvocation.expected_invocation o…
surajksharma07 commentedon Apr 9, 2026
@ASRagab _EvalMetricResultWithInvocation.expected_invocation (line 93) needs to be Optional[Invocation] = None, and _print_details (lines 439–449) also needs None guards on all three expected_invocation attribute accesses.
The good news is that both _convert_content_to_text and _convert_tool_calls_to_text already handle None cleanly, so those should be straightforward one-liners.
Could you check whether PR #5215 covers both spots and run it through a few scenario-based eval cases before we push it forward?
Just want to make sure it holds up end-to-end before merging.
Team is looking more into it.
ASRagab commentedon Apr 9, 2026
#5215 (comment)
surajksharma07 commentedon Apr 10, 2026
That's exactly the kind of evidence we needed @ASRagab — before/after output with all three guard paths verified and a regression check is thorough testing done right!
We currently have two PRs open for the same fix (#5215 and #5221), so the team will review both and consolidate before merging.
Really appreciate you going the extra mile on the testing — we'll get this closed out soon!
Team is looking more into it.
surajksharma07 commentedon Apr 10, 2026
@ankursharmas Please have a look into it once.
fix(eval): make _EvalMetricResultWithInvocation.expected_invocation o…
fix: make `_EvalMetricResultWithInvocation.expected_invocation` `Opti…
fix: make `_EvalMetricResultWithInvocation.expected_invocation` `Opti…