Repository navigation
EvalBaseModel extra='forbid' rejects custom fields in session_input and eval cases #5906
Description
Description
EvalBaseModel in src/google/adk/evaluation/common.py uses extra="forbid" in its Pydantic model_config. This causes all eval-related models (EvalCase, SessionInput, IntermediateData, Invocation, ConversationScenario, etc.) to reject any fields not explicitly defined in the model.
This blocks common use cases like adding custom metadata to session_input for downstream analysis (e.g., category, tier, complexity tags on eval cases).
Reproduction
from google.adk.evaluation.eval_case import SessionInput
# This raises ValidationError because 'category' is not a defined field
session_input = SessionInput(
app_name="my_app",
user_id="eval_user",
category="search", # rejected
tier="low", # rejected
)pydantic.ValidationError: 2 validation errors for SessionInput
category
Extra inputs are not permitted [type=extra_forbidden]
tier
Extra inputs are not permitted [type=extra_forbidden]
Expected Behavior
session_input and other eval models should accept additional fields. The ADK's own EvalConfig already uses extra="allow" — eval case models should follow the same pattern, or at minimum SessionInput should allow extra fields since it's the natural place for user-defined metadata.
Workaround
Currently we patch all EvalBaseModel subclasses at runtime:
from google.adk.evaluation import eval_case, eval_set
for mod in (eval_case, eval_set):
for name in dir(mod):
cls = getattr(mod, name)
if isinstance(cls, type) and hasattr(cls, "model_config"):
if cls.model_config.get("extra") == "forbid":
cls.model_config["extra"] = "ignore"
cls.__pydantic_complete__ = False
for name in dir(mod):
cls = getattr(mod, name)
if isinstance(cls, type) and hasattr(cls, "model_rebuild"):
cls.model_rebuild(force=True)Affected Classes (as of 2.1.0)
All 11 classes inheriting from EvalBaseModel:
eval_case.AppDetailseval_case.ConversationScenarioeval_case.EvalBaseModeleval_case.EvalCaseeval_case.IntermediateDataeval_case.Invocationeval_case.InvocationEventeval_case.InvocationEventseval_case.Rubriceval_case.SessionInputeval_set.EvalCase
Suggested Fix
Change extra="forbid" to extra="ignore" (or extra="allow") in EvalBaseModel.model_config:
class EvalBaseModel(pydantic.BaseModel):
model_config = pydantic.ConfigDict(
alias_generator=alias_generators.to_camel,
populate_by_name=True,
extra="ignore", # was "forbid"
arbitrary_types_allowed=True,
)Environment
- google-adk: 1.34.0 (also verified on main/2.1.0)
- Python: 3.12
Activity
surajksharma07 commentedon Jun 1, 2026
@tottenjordan this makes sense — extra="forbid" on EvalBaseModel is stricter than it needs to be for user-facing eval models, and the inconsistency with BaseCriterion using extra="allow" is a good catch.
As a quick workaround before #5922 lands, try patching each affected class at import time by setting cls.model_config["extra"] = "ignore" and calling cls.model_rebuild(force=True).
Worth noting that each subclass holds its own copy of model_config, so the patch needs to be applied per class rather than just on EvalBaseModel.
Could you give it a thorough run against your eval pipeline and confirm it holds up before we move forward with the PR review?
tottenjordan commentedon Jun 10, 2026
Confirming — the per-class workaround works reliably. We've been running it across 20+ GEPA optimization pipeline runs (both local and in Vertex AI Pipelines) with ADK 2.2.0 and it holds up without issues.
Our implementation matches what you suggested — patch each class individually and rebuild:
Verified on ADK 2.2.0 with eval cases containing custom metadata fields (
tier,category,description,tags). Looking forward to #5922 landing so we can remove the workaround.surajksharma07 commentedon Jun 12, 2026
@tottenjordan thanks for running it through 20+ GEPA pipeline runs — that's solid validation that the per-class patch holds up in both local and Vertex AI Pipelines!
One nuance worth knowing for when #5922 lands: your patch sets extra="ignore", which stops the ValidationError but silently drops the custom fields at construction — they never live on the model instance.
The PR goes with extra="allow", so tier/category/tags will actually be stored and round-trip through serialized eval results, which should remove any need to carry that metadata out-of-band in your GEPA runs.
Once it merges, you can drop the patch and get strictly better behaviour.
#5922 is rebased onto current main and is with the team for review (reviewer assigned), so it's progressing.
Also noting #6070 tracks the same root cause — we'll consolidate on this issue and #5922 to keep things in one place!
fix(eval): preserve custom eval metadata
fix(eval): preserve custom eval metadata