Skip to content

Consistency improvements between adk eval and AgentEvaluator #4410

Closed
Feature
Closed
Consistency improvements between adk eval and AgentEvaluator#4410
Feature
Assignees
haranrksasha-gitg
Labels
eval[Component] This issue is related to evaluationneeds review[Status] The PR/issue is awaiting review from the maintainer

Description

@ftnext
ftnext
opened on Feb 8, 2026

Please make sure you read the contribution guide and file the issues in the right place.
Contribution guide.

🔴 Required Information

Please ensure all items in this section are completed to allow for efficient
triaging. Requests without complete information may be rejected / deprioritized.
If an item is not applicable to you - please mark it as N/A

Is your feature request related to a specific problem?

Yes. In v1.24.1, evaluation behavior differs between adk eval and AgentEvaluator.evaluate,
which makes it harder to run and compare evals consistently across CLI and API/test workflows.

Concrete gaps I hit:

  • adk eval has no --num_runs, so I need external loops to repeat runs.
  • AgentEvaluator.evaluate does not provide built-in result persistence, so historical comparison is less convenient than CLI-based runs.
  • In adk eval, --config_file_path is optional, but when omitted it falls back to in-code default criteria and does not auto-discover per-test test_config.json next to each eval/test file.
  • adk eval resolves only agent_module.agent.root_agent and does not look for get_agent_async.

Describe the Solution You'd Like

I would like feature parity/consistency between both entry points:

  1. Add --num_runs to adk eval (default 1) and aggregate results per eval case across runs.
  2. Add optional result persistence support to AgentEvaluator.evaluate via an EvalSetResultsManager (local by default).
  3. In adk eval, when --config_file_path is omitted, auto-discover test_config.json adjacent to each input eval/test file, then fall back to default criteria if no file is found.
  4. Align agent resolution behavior between adk eval and AgentEvaluator: support both root_agent and get_agent_async consistently

Impact on your work

This impacts reproducibility and operational efficiency in day-to-day evals.

  • Repeated runs are needed to reduce nondeterminism, but currently require custom scripting for CLI.
  • Comparing historical outcomes is easier on one path than the other.
  • The same dataset/config setup behaves differently depending on whether I use CLI or programmatic evaluation.
  • Agent modules that are valid in one evaluation path can fail in the other, forcing workflow-specific agent definitions.

Willingness to contribute

Yes

🟡 Recommended Information

Describe Alternatives You've Considered

  • Keeping current differences and documenting them: this still leaves manual repetition/persistence work and split workflows.
  • Wrapper scripts outside ADK: works short-term, but duplicates logic and reduces maintainability.

Proposed API / Implementation

Pseudo-proposal:

# CLI
adk eval <agent_dir> <eval_set...> --num_runs=3 [--config_file_path=...]

# Internals (conceptual)
for i in range(num_runs):
  inference_results += run_inference(...)

aggregated = aggregate_eval_case_results(inference_results)
save_if_configured(aggregated)
# AgentEvaluator
await AgentEvaluator.evaluate(
    agent_module=...,
    eval_dataset_file_path_or_dir=...,
    num_runs=3,
    eval_set_results_manager=LocalEvalSetResultsManager(...),  # optional
)
# Config resolution for adk eval (conceptual)
if config_file_path:
  eval_config = load(config_file_path)
else:
  eval_config = discover_test_config_near_input_or_default(...)
# Conceptual shared loader used by both entry points.
agent_module = _get_agent_module(...)
if hasattr(agent_module.agent, "root_agent"):
  agent = agent_module.agent.root_agent
elif hasattr(agent_module.agent, "get_agent_async"):
  agent = await agent_module.agent.get_agent_async()
else:
  raise ValueError("Expected `root_agent` or `get_agent_async` in agent module.")

Additional Context

Goal: make eval behavior consistent regardless of entry point (adk eval vs AgentEvaluator.evaluate) so that the same eval assets produce comparable outcomes with less custom glue code.

Activity

  1. adk-bot
    added
    eval[Component] This issue is related to evaluation
    on Feb 8, 2026
  2. adk-bot
    added the
    Feature
    issue type on Feb 8, 2026

  3. rohityan commented on Feb 10, 2026

    @rohityan
    rohityan
    on Feb 10, 2026
    Collaborator

    Hi @ftnext ,

    1. The _get_eval_results_by_eval_id method explicitly uses inference_requests = [...] * num_runs to repeat inference requests, so the framework itself manages the repetition to address non-determinism.
    2. The EvalSetResultsManager interface, specifies how evaluation results are stored and retrieved. Implementations like GcsEvalSetResultsManager and LocalEvalSetResultsManager handle the actual storage, either in Google Cloud Storage or locally on disk, respectively. This enables the persistence and reuse of evaluation results across different ADK components
    3. The AgentEvaluator.evaluate static method is designed to automatically discover test_config.json files when the eval_dataset_file_path_or_dir argument points to a directory or a file.
  4. rohityan
    self-assigned this
    on Feb 10, 2026

  5. ftnext commented on Feb 11, 2026

    @ftnext
    ftnext
    on Feb 11, 2026
    ContributorAuthor

    @ryanaiagent Thanks for the detailed references and for clarifying the current implementation.

    I think we may be talking past each other slightly, so let me clarify the core request in this issue:
    I’m not claiming these capabilities are missing in general; I’m pointing out behavior differences between adk eval and AgentEvaluator.evaluate that make workflows inconsistent.

    Specifically:

    1. num_runs
      AgentEvaluator supports it (agreed).
      My request is that adk eval also supports num_runs so repeated runs do not require external scripting. (PR Support adk eval --num_runs N as AgentEvaluator #4411)
    2. Result persistence
      I understand EvalSetResultsManager exists and is used by some components (agreed).
      My request is that AgentEvaluator.evaluate can optionally persist results through this manager, to match CLI-style history workflows. (PR Add optional eval set result persistence to AgentEvaluator #4414)
    3. Config discovery
      I agree AgentEvaluator.evaluate auto-discovers test_config.json.
      My request is that adk eval should offer equivalent discovery behavior when --config_file_path is omitted, instead of only falling back to in-code defaults. (PR Auto-discover test_config.json only for single-file adk eval input #4412)
    4. Agent resolution
      adk eval currently resolves agent_module.agent.root_agent directly, while AgentEvaluator can also use get_agent_async as a fallback.
      My request is to align this resolution behavior across both entry points. (PR fix(eval): support get_agent_async in adk eval #4413)

    So the goal is entry-point consistency (adk eval vs AgentEvaluator), not to dispute the existence of the underlying primitives.

  6. rohityan
    added
    needs review[Status] The PR/issue is awaiting review from the maintainer
    on Feb 13, 2026

  7. rohityan commented on Feb 13, 2026

    @rohityan
    rohityan
    on Feb 13, 2026
    Collaborator

    Hi @sasha-gitg , can you PTAL .

  8. rohityan
    assigned
    sasha-gitg
    and unassigned
    rohityan
    on May 12, 2026
  9. haranrk
    self-assigned this
    on Jun 15, 2026
  10. copybara-service
    added a commit that references this issue on Jul 22, 2026

    fix(eval): support `get_agent_async` in `adk eval`

    ebaef9f
  11. copybara-service
    added 4 commits that reference this issue on Aug 7, 2026

    feat(cli): Support `adk eval --num_runs N` as AgentEvaluator

    39c8cfa

    feat(cli): Support `adk eval --num_runs N` as AgentEvaluator

    c9d5ed6

    feat(cli): auto-discover test_config.json for single eval file in adk…

    0477e57

    feat(evaluation): add optional eval set result persistence to AgentEv…

    76027dd
  12. FrigaZzz
    added 3 commits that reference this issue on Aug 11, 2026

    fix(eval): support `get_agent_async` in `adk eval`

    c69563d

    feat(cli): auto-discover test_config.json for single eval file in adk…

    aa6e12a

    feat(evaluation): add optional eval set result persistence to AgentEv…

    ef55584

  13. DeanChensj commented on Oct 14, 2026

    @DeanChensj
    DeanChensj
    on Oct 14, 2026
    Collaborator

    Hi @ftnext, thank you for reporting this issue!

    This has been resolved by PR #4412 and merged into the main branch in commit 0477e57.

    We are closing this issue as resolved. If you find that the issue persists on the latest ADK release or if this was closed in error, please feel free to reopen it or leave a comment! Thank you again for your contribution.

  14. DeanChensj
    closed this as completedon Oct 14, 2026

  15. github-actions commented on Oct 14, 2026

    @github-actions
    github-actionsBot
    on Oct 14, 2026 – with GitHub Actions

    Are you satisfied with the resolution of your issue?
    Yes
    No

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

eval[Component] This issue is related to evaluationneeds review[Status] The PR/issue is awaiting review from the maintainer

Type

Feature

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions