Repository navigation
MCP Toolset cleanup race condition when running parallel evaluations #4155
Description
Description
When running agent eval set with multiple eval cases and MCPToolset, the evaluation fails most of the time with CancelledError and warnings about cancel scope violations. This fails in version 1.22.1 also.
This happens from the UI, adk eval or pytest. The root cause is a race condition where multiple parallel Runner instances share the same MCPToolset (via a shared root_agent), and each Runner independently calls close() on exit.
As mentioned this fails randomly, since depending on your agent/tools used and eval set.
I think this also relates to #3161
Bug Behavior
The evaluation fails intermittently with errors like:
asyncio.exceptions.CancelledError: Cancelled via cancel scope <id> by <Task ...>
And warnings:
WARNING - Toolset MCPToolset cleanup cancelled: Cancelled via cancel scope...
Warning: Error during MCP session cleanup for stdio_session: Attempted to exit cancel scope in a different task than it was entered in
Steps to Reproduce
- Create an agent that uses
MCPToolset, you can either use stdio or streamable-http - Create a eval set with at least two eval cases, if you only have one is not going to parellallise anything so won't fail.
- Run evaluation with
adk evalorpytestusingAgentEvaluator.evaluate(), or from the UI selecting both eval cases - The test fails randomly, depending on the test most times will fail.
Inspired on the example shared in PR #3161 in the following comment
And following https://google.github.io/adk-docs/evaluate/#recommendations-on-criteria approach.
Files needed:
test_agent/__init__.py- test_agent/agent.py
- test_agent/eval/test_eval.py
- mcp_server.py
- and your eval_sets
Example agent (agent.py):
from google.adk.agents import Agent
from google.adk.tools.mcp_tool.mcp_toolset import MCPToolset, StdioServerParameters
root_agent = Agent(
name="test_agent",
model="gemini-2.5-flash",
instruction="You are a calculator assistant.",
tools=[
MCPToolset(
connection_params=StdioServerParameters(
command="python",
args=["mcp_server.py"],
),
)
],
)from fastmcp import FastMCP
# Initialize the MCP server
mcp = FastMCP("Addition Server")
# Define a tool to add two numbers
@mcp.tool()
def add_two_numbers(a: int, b: int) -> int:
"""
Adds two numbers together.
"""
return a + b
# Run the MCP server
if __name__ == "__main__":
mcp.run()You can create a evalset and two eval cases using the adk web UI.
Then, either run both eval cases in the UI or create the followin test:
Example test (test_agent/eval/test_eval.py):
@pytest.mark.asyncio
async def test_basic_mcp_connection():
await AgentEvaluator.evaluate(
"test_agent",
"test_agent/eval/eval_data/basic_test.evalset.json",
num_runs=1,
)Root Cause Analysis
The issue is in the evaluation flow:
-
LocalEvalService.perform_inference()(local_eval_service.py#L175-L189) runs multiple inferences in parallel (defaultparallelism=4) using the sameroot_agent -
EvaluationGenerator._generate_inferences_from_root_agent()(evaluation_generator.py#L236-L243) creates a new Runner for each parallel inference, but passes the sharedroot_agent -
Each Runner's
close()(runners.py#L1489-L1493) callstoolset.close()on all toolsets from the shared agent -
Race condition: When Runner 1 finishes first, it closes the shared
MCPToolset. Runners 2, 3, 4 are still running and either:- Try to use the now-closed MCP connection
- Also try to close the already-closed toolset
- Violate anyio's
CancelScopetask context rules
Proposed Solution
I'm not very familiar with the code base yet, but I think my approach would be to update the LocalEvalService and MCPToolset, make McpToolset.close() idempotent and manage lifecycle explicitly in LocalEvalService since that would have minimum impact.
I know the project is very active and not sure if there is any related progress on this issue, but happy to open a PR to fix it if that's helpful.
Environment
- Python: 3.13.0
- OS: macOS (Darwin)
- ADK version: 1.22.1
- Related packages: anyio, fastmcp
Additional Context
- For few tools and eval cases, the bug sometimes tests pass if the parallel tasks happen to complete in a "safe" order
- Setting
parallelism=1in evaluation config works around the issue but defeats the purpose of parallel evaluation - The issue affects both
adk evalCLI andAgentEvaluator.evaluate()in pytest
Activity
challenger71498 commentedon Jan 16, 2026
Hi @alexhermida, I also encountered a similar issue you have experienced when I tried to run agent as a tool in parallel.
It seems like the issue happens if:
Both my case and yours match the case above.
For my case:
AgentToolcreates a runner to execute agent as a tool.AgentTools which share its agent triggers the issue.For your case:
I created an issue for this, take a look at #4045 if you are interested in.
challenger71498 commentedon Jan 16, 2026
For the case with running
AgentTools in parallel, I created a PR #4046 to resolve the issue (only my case though).At the PR,
AgentToolManager) to count runner reference to an agent when usingAgentTool.AgentToolinitializes the runner, it registers the runner to the ref counter.AgentToolderegisters the runner from the ref counter.You may also look into the PR if you are interested in. :)
challenger71498 commentedon Jan 16, 2026
And just one more thing..
The PR above only addresses
AgentToolcase though, not your case. It does not fix the original issue: concurrency issue when executing runner in parallel with a same agent. This is on purpose since I thought only my case would be a practical issue at that time. (Well, now we do have another actual issue: your case 👍)I tried to resolve the original issue, but found out that this is rather an architectural issue and seems like the issue cannot be resolved by simple PRs. I think we need to discuss the design of resource management of the toolset in runner with maintainers.
From what I have researched about the architecture about the ADK is that:
Agent,RunnerandToolsethave different lifecycles.Agentis rather a blueprint; it does not have a lifecycle(Basically it is just a pydantic model).Runneris a stateful object. An agent is injected to the runner.Toolsetis a group of tools, and it may have state, depends on its implementation. It is injected to theAgent.So if
Toolsetis stateful, should runner cleanup the toolset when the runner is closing?IMO this is NO. Even if a
Runnerand aToolsetmay have states and should be cleanup when their life is done, it should be handled seperately. Since aToolsetis injected to theAgent, a code scope where injects theToolsetalso has resposibility to cleanup the toolset.I have not researched further to determine where should we put this
Toolsetmanagement though, before we go I think we need to have enough conversation to whether should we consider this approach or try something else.Please let me know if there is a correction needed.
Your opinions are greatly appreciated :)
Thank you.
alexhermida commentedon Jan 16, 2026
Hi @challenger71498! Thank you for all your detailed comments. After a quick look to your PR, as you said, I think that doesn't directly address the problem with the original MCPToolset but looks progress and might made the fix needed for this simpler, since you're already implementing
cleanup_toolsetsin theRunner.close()To be honest, I've already encountered this issue just running multiple agents with tools throught the Runner, but in that case I have more control and I workarounded it. I've opened this issue focused on fixing the out-of-the-box evals.
I agree with you on the more fundamental arquitectural decision here. I think your change in the
Runner.close()looks good short term but that way assumes that owns the toolsets and closes them, but toolsets are being injected into theAgent, not theRunner, so should be the agent responsible of closing the tools.Personally, would be great to see this issues fixed soon but also having more medium/long term view on the approach. Happy to contribute, at the same time I know that this kind of changes might need several reviews 😄
challenger71498 commentedon Jan 16, 2026
Thank you for your reply, @alexhermida!
Glad to hear that you have found a workaround. Just out of curiosity, I wonder how you workarounded the issue. Was it the same way I've done, or something else?
Totally agree with your view. The PR could fix the issue temporarily, but the root cause should be resolved as you mentioned.
alexhermida commentedon Jan 16, 2026
It's kind of simple approach, in the agent I have a factory function that returns the agent and the toolset, like
In my runners module, I use asyncio.gather to run all tasks concurrently so I create a new agent instance, that ensures each parallel task has its own independent agent and MCP connections.
surajksharma07 commentedon Jan 29, 2026
The root cause is in
runners.py:1528-1532— each parallel Runner independently calls_cleanup_toolsets()on the shared agent's toolsets, so when Runner 1 finishes it tears down the MCPToolset connection while Runners 2-4 are still using it.A ref-counting guard on
Runner.__aenter__/__aexit__that only callstoolset.close()when the last Runner exits would fix this.As a workaround, you can use an agent factory that returns a fresh
Agent+MCPToolsetinstance per eval case instead of sharing one agent across parallel inferences — @challenger71498's comment above describes the same pattern.alexhermida commentedon Jan 29, 2026
Thanks @surajksharma07
Yes, that's what we did 😄
Great to see this moving forward, thanks again.
surajksharma07 commentedon Feb 8, 2026
Still unresolved as of v1.24.1 —
runners.py_cleanup_toolsets()has no ref-counting or idempotent-close guard, andlocal_eval_service.py:174still shares the same agent instance across parallel inferences.Commit
cce430dafixed cancel-scope task issues but not this race condition.The agent-factory workaround (fresh Agent+MCPToolset per eval case) remains the recommended approach until this is addressed architecturally.
surajksharma07 commentedon Feb 8, 2026
@seanzhou1023 Please have a look into it once.
surajksharma07 commentedon Apr 28, 2026
The agent-factory workaround (fresh Agent + MCPToolset per eval case) you confirmed in January remains the recommended path until a ref-counting fix lands.
Closing this out since there's been no follow-up for a few months — feel free to reopen or link a PR here if you'd like to contribute the fix, happy to review it!