I want to mock the Agent tool when using pytest to run eval in CI/CD

** Please make sure you read the contribution guide and file the issues in the right place. **
Contribution guide.

Is your feature request related to a problem? Please describe.
I am trying to implement regression tests using pytest by loading evalset.json and a rubric, following the instructions in the document below:
10. CI/CD with Pytest (pytest)

However, the agent I am developing uses tools that call APIs returning real-time data from an external system. Because the tool outputs are dynamic, there is a risk that the evaluation results (pass/fail) may change depending on when the test is executed.

To address this, following the traditional unit testing approach, I would like to mock the tool implementations during eval execution so that they return fixed JSON responses for testing purposes.

Is there currently any built-in support in ADK for mocking tools in this way?

Describe the solution you'd like
When the evaluation is executed, the tool is given the expected JSON to be returned, and the tool is mocked so that the specified JSON is returned as a fixed value.

Describe alternatives you've considered
I have already implemented an approach where the agent checks an environment variable and switches the exported tool accordingly.

However, this approach is quite custom, and it requires modifying production code solely for the purpose of mocking, which I would prefer to avoid.

Below is a simplified example of the current implementation.

call_api_tool.py

# create CallApiTool And MockCallApi Tool by BaseTool
if os.getenv("IS_EVAL") == "1":
  call_api_tool = MockCallApiTool()
else:
  call_api_tool = CallApiTool()

agent.py

root_agent = Agent(
    name="my_api_agent",
    description="Agent that calls an external API",
    tools=[call_api_tool],
)

Ideally, I would like to mock or override tool implementations during eval execution without introducing environment-based branching logic into the agent or tool source code.

Additional context
None

Notification: The author has updated the issue description. Maintainers, please review.

@dev-fjk

@dev-fjk Thank you for submitting this feature request. It’s currently under review by our team, and I’ll keep you posted on any updates. We appreciate your input!

Is there any update on this? Would also be valuable for us

We would also like to see this feature. Additionally, it would be highly useful to be able to mock tool results for tools that aren't necessarily in the expected execution path too (ensuring that if the agent calls an unexpected tool, it doesn't hit the live API).

For now, we are using a workaround by mocking the underlying client method. Here is a simplified and generic version of what worked for us:

from unittest.mock import AsyncMock, MagicMock

import pytest
from google.adk.evaluation.agent_evaluator import AgentEvaluator


@pytest.mark.asyncio
async def test_my_agent_workflow(mocker):
    """Test that the agent invokes the mock tool and evaluates successfully."""

    # 1. Patch the external service/client method
    mock_client_method = mocker.patch(
        "my_module.tools.MyServiceClient.get_data_method",
        new_callable=AsyncMock,
    )
    
    # 2. Return a generic mock object
    mock_response = MagicMock()
    mock_response.status = "SUCCESS"
    mock_response.data = "dummy_value"
    mock_client_method.return_value = mock_response

    # 3. Run the evaluator
    await AgentEvaluator.evaluate(
        agent_module="my_module.my_agent_directory.agent",
        agent_name="my_agent_name",
        eval_dataset_file_path_or_dir="path/to/eval_dataset.json",
    )

While this works, scaling this approach across a large test suite where tool results vary is quite tedious. A built-in way to mock these results natively would be a much appreciated addition.

I found an edge case in the existing environment simulation support: InjectionConfig(injected_response={}) is rejected, while specifying both an error and {} is accepted. The engine also treats an empty injected response as missing.

Would a focused fix for those presence checks be welcome? I can include regression tests and a Runner test confirming that {} is returned without executing the real tool. The scope would be limited to empty JSON responses in the current simulation API.

For the fixed-JSON part there is a built-in piece now: the experimental environment simulation in google.adk.tools.environment_simulation (I read it at v2.12.0). An InjectionConfig with injected_response (optionally narrowed with match_args) returns that dict instead of running the tool, and EnvironmentSimulationFactory.create_callback() turns the config into a before_tool_callback. That means it can be attached from the test file, with no IS_EVAL switch in the agent code.

One thing to know before relying on it in CI, for the "an unexpected tool must not hit the live API" case raised above. The callback returns None for a tool that is not in the config (engine L79-L80), and for a call that matches no injection when no mock strategy is set (L120-L129, logged as "Returning no-op."). A before_tool_callback that returns None lets ADK run the real tool (LlmAgent.before_tool_callback: "Return None to continue"). So an unconfigured call still reaches the live API. #7547 fixes the empty-{} version of this; the two paths above are separate from it.

Wrapping it so that case fails closed:

import pytest
from google.adk.evaluation.agent_evaluator import AgentEvaluator
from google.adk.tools.environment_simulation import EnvironmentSimulationFactory
from google.adk.tools.environment_simulation.environment_simulation_config import (
    EnvironmentSimulationConfig, InjectionConfig, ToolSimulationConfig)
from my_module.my_agent_directory import agent as agent_module

simulate = EnvironmentSimulationFactory.create_callback(EnvironmentSimulationConfig(
    tool_simulation_configs=[ToolSimulationConfig(
        tool_name="call_api",  # your tool's name
        injection_configs=[InjectionConfig(injected_response={"status": "SUCCESS", "data": "fixed"})],
    )]))
unmocked = []

async def fixture_or_refuse(tool, args, tool_context):
    result = await simulate(tool, args, tool_context)
    if result is None:  # None means ADK would run the real tool
        unmocked.append((tool.name, args))
        return {"error": f"no fixture for {tool.name}"}
    return result

@pytest.mark.asyncio
async def test_eval_with_fixed_tools(monkeypatch):
    monkeypatch.setattr(agent_module.root_agent, "before_tool_callback", fixture_or_refuse)
    await AgentEvaluator.evaluate(
        agent_module="my_module.my_agent_directory.agent",
        eval_dataset_file_path_or_dir="path/to/eval_dataset.json",
    )
    assert not unmocked, unmocked

Patching root_agent in the test works because AgentEvaluator loads the agent with importlib.import_module (agent_evaluator.py L650), which hands back the module object the test already imported. Sub-agents with their own tools need the same callback on each LlmAgent; an agent that already has a before_tool_callback should get a list with this one first.

I read this from the v2.12.0 source and have not run it against a model, so please check the experimental names against the version you have installed.

— LEEVAR

Navigation Menu

Repository navigation

I want to mock the Agent tool when using pytest to run eval in CI/CD #4193

call_api_tool.py

agent.py

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

I want to mock the Agent tool when using pytest to run eval in CI/CD #4193

Description

call_api_tool.py

agent.py

Activity

adk-bot commented on Jan 26, 2026

klateefa commented on Feb 11, 2026

Strandor commented on Mar 3, 2026

adrisanjuan commented on Jun 3, 2026

Samecho commented on Oct 15, 2026

forevercrab321-svg commented on Oct 21, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions