Skip to content

Feature: Support custom metrics in AgentEvaluator eval flow #4343

Closed

Description

@ftnext

** Please make sure you read the contribution guide and file the issues in the right place. **
Contribution guide.

🔴 Required Information

Please ensure all items in this section are completed to allow for efficient
triaging. Requests without complete information may be rejected / deprioritized.
If an item is not applicable to you - please mark it as N/A

Is your feature request related to a specific problem?

Thanks to supporting custom metrics in v1.23.0.
When running evals via AgentEvaluator.evaluate in tests, custom metrics defined in test_config.json are not registered, even though adk eval (CLI) supports custom metrics.
This makes it hard to use the same eval configs in pytest and adk eval.

For example
https://github.com/ftnext/agent-practice/tree/1835d834b32796bba36a05aa85c32eb4427ad831/adk-evaluation/tests

% PYTHONPATH=. adk eval home_automation_agent tests/fixtures/home_automation_agent/simple_test.test.json --config_file_path tests/fixtures/home_automation_agent/test_config.json

Eval Run Summary
b305bd06-38c5-4796-b9c7-d9c7454338b9:
  Tests passed: 1
  Tests failed: 0

But

@pytest.mark.asyncio
async def test_with_single_test_file():
    await AgentEvaluator.evaluate(
        agent_module="home_automation_agent",
        eval_dataset_file_path_or_dir="tests/fixtures/home_automation_agent/simple_test.test.json",
        num_runs=1,
    )
  File "/.../adk-evaluation/.venv/lib/python3.13/site-packages/google/adk/evaluation/metric_evaluator_registry.py", line 64, in get_evaluator
    raise NotFoundError(f"{eval_metric.metric_name} not found in registry.")
google.adk.errors.not_found_error.NotFoundError: practice_tool_trajectory_metric not found in registry.

Describe the Solution You'd Like

Make AgentEvaluator.evaluate_eval_set register custom metrics from EvalConfig.custom_metrics, matching the behavior in adk eval.
The registration should use a local metric registry (no global side effects) and a shared helper for default MetricInfo construction.

Impact on your work

I want to reuse the same eval configurations across CLI and pytest without duplicating logic or writing custom registry code in tests.
This affects our local eval workflow and CI coverage.

Willingness to contribute

Are you interested in implementing this feature yourself or submitting a PR?
Yes


🟡 Recommended Information

Describe Alternatives You've Considered

Manually registering custom metrics in each test, but this is repetitive and diverges from adk eval behavior.
example: https://github.com/ftnext/agent-practice/blob/3e845dd8327bdf28716528d4f50cebce91231542/adk-evaluation/tests/test_home_automation_agent.py#L41-L203

Proposed API / Implementation

Use a local MetricEvaluatorRegistry in AgentEvaluator.evaluate_eval_set and register custom_metrics with _CustomMetricEvaluator, similar to the CLI.
Share a common helper for default MetricInfo creation in the evaluation layer.

Additional Context

Keeping CLI and AgentEvaluator behavior consistent reduces confusion and makes it easier to adopt custom metrics in tests.

Custom metrics are very promising as they empower individual developers to handle evaluation issues themselves.

I hope that custom metrics support will continue to improve with ADK version upgrades.

Activity

  1. added
    eval[Component] This issue is related to evaluation
    on Feb 1, 2026
  2. added theissue type on Feb 1, 2026
  3. adk-bot commented on Feb 8, 2026

    @adk-bot
    Collaborator

    Notification: The author has updated the issue description. Maintainers, please review.

  4. klateefa commented on Feb 11, 2026

    @klateefa

    @ftnext Thank you for bringing this to our attention. The request is under review, and I’ll keep you updated if any additional information is required. We appreciate your patience.
    Thanks.

  5. added
    needs review[Status] The PR/issue is awaiting review from the maintainer
    on Feb 11, 2026
  6. ftnext commented on Feb 11, 2026

    @ftnext
    ContributorAuthor

    @klateefa Thank you for reviewing this request!

    I've been actively working on adopting ADK's evaluation features in production and in Japanese-language environments.
    In my experience, the current built-in metrics alone are not sufficient to cover the evaluation needs we encounter in these real-world scenarios — for example, see issue #4122.

    Custom metrics are a critical capability that would bridge this gap, allowing teams like ours to define evaluation criteria tailored to their specific use cases.
    I'm very excited about the potential of this feature.

    Even if my PR (#4344) is not the right approach, I would greatly appreciate prioritizing custom metric support in AgentEvaluator to make ADK more adoptable for diverse production environments. Thank you!

  7. self-assigned this
    on Jul 28, 2026
  8. added a commit that references this issue on Aug 18, 2026
    babb11c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

eval[Component] This issue is related to evaluationneeds review[Status] The PR/issue is awaiting review from the maintainer

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions