Repository navigation
GEPA optimizer crashes with KeyError on failed eval cases #6004
Description
🔴 Required Information
Describe the Bug
When using GEPA optimization (AgentOptimizer.optimize()) with evaluation sets that include failed cases (e.g., due to inference failures, user simulator errors, or API timeouts), the GEPA adapter crashes with a KeyError in gepa_root_agent_prompt_optimizer.py:150.
Failed eval cases don't populate score entries in the result.scores dictionary, but the GEPA adapter assumes all batch examples have corresponding scores. This causes the optimization loop to crash mid-evaluation.
This is Part 2 of a three-part error cascade:
- Bug 1 (PR fix(eval): handle failed inference results without invocations #5878): Failed inferences leave
inference_result.inferences = None→ TypeError when iterating - Bug 2 (This Issue): Failed cases missing from
result.scoresdict → KeyError when accessing scores - Bug 3: Failed cases have
score = None→ TypeError when rounding (will be mitigated by Bug 1 fix)
See related issues: #5876, #5115, #5403, and PR #5878.
Steps to Reproduce
- Create an eval set with some cases that will trigger inference failures
- E.g., use
conversation_scenario(user simulation) with edge cases that cause "LLM returned only thinking tokens" - Or configure cases that timeout or fail authentication
- E.g., use
- Run
AgentOptimizer.optimize()with this eval set - Expected: Optimization completes with "X PASSED, Y FAILED" and continues
- Observed: After initial eval if Bug 1 is fixed, crashes in iteration 2 with:
KeyError: '<example_id>' File ".../gepa_root_agent_prompt_optimizer.py", line 150, in evaluate score = result.scores[example_id] # ← CRASH
Expected Behavior
GEPA optimization should gracefully handle failed eval cases by:
- Assigning a default score (e.g., 0.0) to failed cases
- Continuing the optimization loop with successful cases
- Logging which cases failed and why
Observed Behavior
Traceback (most recent call last):
...
File ".../google/adk/optimization/gepa_root_agent_prompt_optimizer.py", line 150, in evaluate
score = result.scores[example_id]
~~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: '151d97ad'
Optimization terminates prematurely, blocking the ability to use GEPA with any eval sets containing transient failures.
Environment Details
- ADK Library Version: 1.31.1+
- Desktop OS: macOS/Linux
- Python Version: 3.14.0
Model Information
- Are you using LiteLLM: Yes/No
- Which model is being used: (e.g., gemini-2.5-pro)
🟡 Optional Information
Root Cause Analysis
File: google/adk/optimization/gepa_root_agent_prompt_optimizer.py:150
Current Code:
for example_id in batch:
score = result.scores[example_id] # ← CRASH if example_id not in result.scores
scores.append(score)
eval_data = result.data.get(example_id, {}) if result.data else {}
outputs.append(eval_data)
trajectories.append(eval_data)The Problem:
- Failed eval cases are excluded from
result.scores - But they're still in the
batchlist being iterated - Direct dictionary access
[example_id]raises KeyError
Note: This manifests AFTER Bug 1 (PR #5878) is fixed, because:
- Bug 1 fix → failed inferences don't crash during iteration
- Evaluation completes with some cases marked as FAILED
- Failed cases don't populate
result.scores - This code assumes all cases in batch have scores → KeyError
Suggested Fix
Use .get() with a conservative default of 0.0 for failed cases:
for example_id in batch:
score = result.scores.get(example_id, 0.0) # Default to 0.0 for failed cases
scores.append(score)
eval_data = result.data.get(example_id, {}) if result.data else {}
outputs.append(eval_data)
trajectories.append(eval_data)Rationale:
- Graceful degradation: Optimization continues with successful cases
- Conservative default: 0.0 treats failure as worst performance (penalizes failing prompts)
- Semantic correctness: Failed inference IS a failure and should contribute negatively to score
- Minimal change: One
.get()call with default value - No breaking changes: Behavior identical when all cases succeed
Relationship to Other Issues
- PR fix(eval): handle failed inference results without invocations #5878: Fixes Bug 1 (inference_result.inferences = None) by returning early with failed EvalCaseResult
- Issue LocalEvalService crashes with len(None) when inference_result.inferences is None #5876: Reports Bug 1; PR fix(eval): handle failed inference results without invocations #5878 is the pending fix
- Issue [ADDING MORE DETAIL] TypeError: round() argument must be a number, not NoneType in local_eval_sampler.py during optimizer GEPA loop #5115, TypeError in LocalEvalSampler when metric evaluation fails #5403: Report Bug 3 (round(None)) but likely not merged
Fix order:
- PR fix(eval): handle failed inference results without invocations #5878 merges (Bug 1) → failed cases no longer crash during iteration
- This PR (Bug 2) → failed cases don't crash during score aggregation
- Bug 3 becomes a safety net (defensive None check before rounding)
How Often This Issue Occurs
Always (100%) when:
- GEPA optimization is run with eval sets containing failed cases
- AND any of those failed cases are sampled during the optimization loop
- AND the initial baseline eval completes (Bug 1 doesn't prevent baseline eval from completing with some fixes)
In production environments with transient failures (rate limits, timeouts, network errors), this is a regular occurrence.
Local Workaround
Monkeypatch the optimizer to handle missing scores:
# Monkey-patch before calling optimizer
original_evaluate = GepaRootAgentPromptOptimizer.evaluate
def patched_evaluate(self, batch):
result = original_evaluate(self, batch)
# Ensure all batch examples have scores
for example_id in batch:
if example_id not in result.scores:
result.scores[example_id] = 0.0
return result
GepaRootAgentPromptOptimizer.evaluate = patched_evaluateTesting
The fix should be validated with:
- Unit test covering batch evaluation with missing score entry
- Integration test: GEPA optimization with mixed passing/failing eval cases
- Regression test: Verify scoring identical when all cases pass
Additional Context
This bug is part of a broader resilience issue in GEPA optimization. The three cascading bugs prevent optimization from completing when ANY eval case fails, even transiently. Together they block production usage of GEPA with real-world eval sets.
The fixes are minimal (1-2 lines each) and maintain backward compatibility while enabling graceful degradation.
Activity
adk-bot commentedon Jun 7, 2026
GitHub Issue #6004 Analysis: GEPA optimizer crashes with KeyError on failed eval cases
Executive Summary
KeyErrorwhen attempting to aggregate scores in gepa_root_agent_prompt_optimizer.py.Detailed Analysis
1. Root Cause Analysis ("What is broken?")
The failure resides within the
evaluatemethod of the dynamically constructed adapter class _AgentGEPAAdapter.When evaluating a batch of instances:
sample_and_score()asynchronously and waits for the UnstructuredSamplingResult.result.scoresdictionary.example_idin the requestedbatchat gepa_root_agent_prompt_optimizer.py#L149-L155:example_idis excluded fromresult.scores, direct dictionary subscription throws aKeyError: '<example_id>', abruptly terminating the entire optimization loop.2. Existing Pull Requests ("Is there a linked PR that fixes this issue?")
fix(eval): handle failed inference results without invocations(Open)TypeError: object of type 'NoneType' has no len()during evaluation.KeyErrorin score collection.3. Recommendation
.get()dictionary retrieval with a conservative penalty fallback (e.g.,0.0score) avoids the KeyError and penalizes prompts that cause failures:Summary of Work
surajksharma07 commentedon Jun 9, 2026
@thoang3 the inconsistency is clear: line 153 in the same block already uses .get() for result.data, but line 150 uses bare subscript for result.scores, so failed cases that never populate the scores dict will always crash there.
A workaround worth trying locally: swap result.scores[example_id] for result.scores.get(example_id, 0.0) at gepa_root_agent_prompt_optimizer.py:150 — the 0.0 default penalizes failed cases conservatively while letting the optimization loop complete.
Could you verify this resolves the crash on your eval set and confirm the scoring output is unchanged when all cases pass?
fix(optimization): Handle failed inference cases gracefully in GEPA p…
surajksharma07 commentedon Aug 26, 2026
@thoang3 closing this out for inactivity but flagging it's not actually resolved. result.scores[example_id] on line 150 is still a bare subscript on current main. Your fix in #6005 has the right approach but has gone stale, unreviewed and now showing merge conflicts against main so it'll need a rebase before anyone can pick it back up. Ping me or reopen if you get a chance to rebase it or if a maintainer wants to take the fix from here directly.
fix: handle failed inference cases gracefully in GEPA pipeline