Running eval through pytest is not saving evalset_result.json file

** Please make sure you read the contribution guide and file the issues in the right place. **
Contribution guide.

Describe the bug
When I run the evals either through adk web or adk eval command, I see the evalset_result.json file getting stored under .adk/eval_history folder under the agent folder. But which I run the eval through pytest, the result.json is not getting stored

To Reproduce
Steps to reproduce the behavior:

Install '...'
Run '....'
Open '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.
The evalset_result.json should get stored just like what happens when we run the evals through adk web or through adk eval command

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

OS: Windows
Python version(python -V): 3.12.10
ADK version(pip show google-adk): 1.11.0

Model Information:
For example, which model is being used.
gemini-2.5-pro

Additional context
Add any other context about the problem here.

This is not supported. This will be a a feature request.

Noted this issue; commenting to receive updates when it’s fixed

implementation example
#2652 (comment)

@CloudIQTechnologies

@CloudIQTechnologies Thanks for the update. The request is currently being reviewed, and I will let you know if anything further is needed. Thank you for your understanding.

#4414 resolves this issue like:

@pytest.mark.asyncio
async def test_with_single_test_file_saves_eval_set_result(
    tmp_path,
):
  """Persists eval set results with derived app_name when app_name is omitted."""
  eval_set_results_manager = LocalEvalSetResultsManager(
      agents_dir=str(tmp_path)
  )
  await AgentEvaluator.evaluate(
      agent_module="tests.integration.fixture.home_automation_agent",
      eval_dataset_file_path_or_dir=(
          "tests/integration/fixture/home_automation_agent/simple_test.test.json"
      ),
      num_runs=2,
      eval_set_results_manager=eval_set_results_manager,
  )

This issue has been resolved in newer versions of the library (PR #4414 introduced LocalEvalSetResultsManager to persist results when running via pytest).

@DeanChensj

@DeanChensj Thanks for the update. I may be missing something, but PR #4414 still appears to be open and not merged yet.

Since #4414 is the PR that adds LocalEvalSetResultsManager support to AgentEvaluator for pytest-based eval result persistence, I don’t think the fix is available on main yet. I also don’t see the new eval_set_results_manager argument or the related persistence logic in AgentEvaluator on the current main branch.

Could you clarify why this issue was closed before #4414 was merged or otherwise released?

@ftnext

Thanks @ftnext for the context, reopening.

feat(evaluation): add optional eval set result persistence to AgentEv…

Navigation Menu

Repository navigation

Running eval through pytest is not saving evalset_result.json file #2602

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Running eval through pytest is not saving evalset_result.json file #2602

Description

Activity

hangfei commented on Nov 4, 2025

sivakumar-d3v commented on Nov 24, 2025

ftnext commented on Jan 14, 2026

klateefa commented on Feb 11, 2026

ftnext commented on Feb 11, 2026

DeanChensj commented on Jun 24, 2026

ftnext commented on Jun 24, 2026

DeanChensj commented on Jun 25, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions