Repository navigation
LocalEvalService._evaluate_single_inference_result crashes when inferences is None #6071
Description
Description
LocalEvalService._evaluate_single_inference_result() accesses inference_result.inferences without a None check. When an inference fails (timeout, API error, etc.), inferences can be None, causing an unhandled crash.
Reproduction
When running GEPA optimization with MCP tool agents, if an MCP session drops during inference, the inference result may have inferences=None. The method then crashes at:
# local_eval_service.py, line ~41-42
if eval_case.conversation_scenario is None and len(
inference_result.inferences # <-- crashes if NoneImpact
A single failed inference case crashes the entire GEPA optimization run instead of gracefully skipping the case with a NOT_EVALUATED status.
Suggested Fix
Add a None guard before accessing inference_result.inferences:
if inference_result.inferences is None:
return inference_result, EvalCaseResult(
eval_id=inference_result.eval_case_id,
eval_set_id=inference_result.eval_set_id,
final_eval_status=EvalStatus.NOT_EVALUATED,
overall_eval_metric_results=[],
eval_metric_result_per_invocation=[],
session_id="skipped",
)Environment
- google-adk 2.2.0
- Python 3.11/3.12
Activity
sanketpatil06 commentedon Jun 11, 2026
Hi @tottenjordan, thanks for reporting this.
The PR (#5878) that addresses this issue is already open and under review. Could you please test it and let us know if it resolves the issue on your end? Thanks.
tottenjordan commentedon Jun 11, 2026
Tested PR #5878 against our production codebase (107 tests, all pass). The fix works correctly.
Repro: When
InferenceResult.inferencesisNone(e.g., agent auth failure, timeout, or null response),_evaluate_single_inference_resultcallslen(inference_result.inferences)which throwsTypeError: object of type 'NoneType' has no len().With PR #5878: The null check returns a proper
EvalCaseResultwithfinal_eval_status=FAILEDand empty metric results. No crash, evaluation continues for remaining cases.Compatibility: The fix is backwards-compatible with our existing workaround (a higher-level null guard in our optimizer). Once this PR merges, we can remove our workaround.
fix(evaluation): handle None inferences in LocalEvalService
fix(evaluation): handle None inferences in LocalEvalService
ftnext commentedon Aug 2, 2026
This appears to have been fixed by #5878, so I think this issue can be closed.
GWeale commentedon Aug 6, 2026
thanks for flagging this, fixed in v2.3.0, local_eval_service now returns a FAILED result when inferences is None instead of crashing, so you can drop the workaround.