# cli_eval bypasses App.plugins, breaking observability plugins (e.g., BigQueryAgentAnalyticsPlugin) during eval runs

Summary

When an agent is wrapped in an App(root_agent=..., plugins=[...]) and you run evals via adk eval (or agents-cli eval run), the registered plugins do not fire. The eval CLI accesses agent_module.agent.root_agent directly and runs eval sessions against the bare agent, bypassing the App object and its plugin chain.

This means observability plugins like BigQueryAgentAnalyticsPlugin capture interactive (adk web / adk run) sessions but produce no rows for eval runs — leaving cost, latency, and trajectory telemetry blind during the development loop where evals run most often.

Reproduction

ADK version: 1.31.1

# app/agent.py
from google.adk.agents import LlmAgent
from google.adk.apps import App
from google.adk.plugins.bigquery_agent_analytics_plugin import (
    BigQueryAgentAnalyticsPlugin, BigQueryLoggerConfig,
)

root_agent = LlmAgent(name="root_agent", model="gemini-2.5-flash", ...)

app = App(
    root_agent=root_agent,
    name="app",
    plugins=[
        BigQueryAgentAnalyticsPlugin(
            project_id="your-project",
            dataset_id="telemetry",
            table_id="agent_events",
            config=BigQueryLoggerConfig(log_session_metadata=True),
        ),
    ],
)

adk web → run a few turns interactively → events appear in telemetry.agent_events ✅
adk eval ./app path/to/case.evalset.json → eval runs complete successfully but no rows appear in telemetry.agent_events for the eval session ❌

Root cause

In google/adk/cli/cli_eval.py:

def _get_agent_module(agent_module_file_path: str):
    file_path = os.path.join(agent_module_file_path, "__init__.py")
    module_name = "agent"
    return _import_from_path(module_name, file_path)


def get_root_agent(agent_module_file_path: str) -> Agent:
    """Returns root agent given the agent module."""
    agent_module = _get_agent_module(agent_module_file_path)
    root_agent = agent_module.agent.root_agent
    return root_agent

The eval flow imports the agent module and reaches into agent_module.agent.root_agent. The App instance (and its plugins=[...] list) is never resolved or used, so plugin lifecycle hooks (before_agent_callback, on_event_callback, etc.) are never wired to the eval runner.

By contrast, adk web / adk run go through AdkWebServer which constructs sessions via the App, so plugins fire correctly.

Why this matters

Cost monitoring during development is blind. Eval runs are typically the largest chunk of LLM cost during prompt iteration (we measured ~€6 per full 29-case run, mostly Pro tokens), but this cost is invisible to dashboards built on BigQueryAgentAnalyticsPlugin or any other observability plugin.
Eval-specific observability is the most useful kind. Knowing per-case latency, token breakdown, tool-trajectory drift across eval runs is exactly what you'd want when iterating on prompts. Today users have to fall back to Cloud Billing exports — much coarser.
The BigQueryAgentAnalyticsPlugin doc actively advertises eval analytics as a use case ("LLM-as-judge evals — structured data for evaluation pipelines"). The current behavior contradicts that.

Proposed fix

Make get_root_agent (and the surrounding eval flow) prefer the App object when it exists in the agent module, and run eval sessions through the App.runner (or equivalent) so plugins fire.

Sketch:

def get_app_or_root_agent(agent_module_file_path: str):
    """Returns (app, root_agent). Falls back to bare root_agent if no App is exported."""
    agent_module = _get_agent_module(agent_module_file_path)
    app = getattr(agent_module.agent, 'app', None)
    if app is not None:
        return app, app.root_agent
    return None, agent_module.agent.root_agent

Then update the eval runner (evaluation_generator.py etc.) to use app for session creation when present, so plugin callbacks are invoked the same way adk web invokes them. The bare-root-agent path remains available for projects that don't define an App.

Happy to draft a PR if there's interest and the maintainers agree on the approach. Would also want feedback on whether there's a reason the current bypass is intentional (e.g., to avoid plugin side-effects polluting eval runs) — if so, an opt-in flag like adk eval --use-app-plugins would also resolve the gap without changing the default.

Workaround for now

For users hitting this:

Estimate eval cost from Cloud Billing → Vertex AI line items in the eval window, less the interactive-session cost reported by your dashboard.
Or: write a custom eval runner that uses App.runner directly instead of going through adk eval.

Environment

google-adk 1.31.1
Python 3.13
ADK ContextCacheConfig, BigQueryAgentAnalyticsPlugin both registered on App

@saifer82

Hi @saifer82, thanks for reporting this.
We have reproduced the issue and observed the same behavior. Since you already have a solution in mind, please feel free to go ahead and raise a PR. Our team will be happy to review it.

Root Cause

cli_eval.py:get_root_agent() always unwraps agent_module.agent.root_agent, discarding the App object and its plugins. Then evaluation_generator.py:_generate_inferences_from_root_agent() constructs a Runner with only its own internal plugins, never including user-defined ones.

In contrast, adk web goes through agent_loader.load_agent() which preserves the full App object and merges in app.plugins when building the runner.

Proposed Fix

cli_eval.py — return agent_module.agent directly (may be App or BaseAgent) instead of always extracting .root_agent
cli_tools_click.py — pass the App-or-agent through to the eval service
local_eval_service.py / evaluation_generator.py — accept Union[BaseAgent, App]; when it's an App, prepend app.plugins to the Runner's plugin list (mirroring adk_web_server.py's approach)

Happy to submit a PR if this approach looks good.

Your eval runs aren't triggering observability plugins because they bypass the App's plugin chain. PathCourse Health's observability module automatically captures eval and interactive sessions without requiring manual instrumentation.

Working example: https://github.com/pathcourse-health/pch-integration-examples#3-observability--traces-spans-anomalies

@saifer82

The same root cause reaches more than the adk eval CLI. #5534 threads the App through cli_eval and evaluation_generator, which fixes the CLI path, but the bare-root_agent construction in _generate_inferences_from_root_agent is also reached by two other non-live eval surfaces that #5534 does not cover:

AgentEvaluator (the evaluate / evaluate_eval_set path). _get_eval_results_by_eval_id builds LocalEvalService with no app, so the plugins are dropped there too. This is the programmatic and pytest entry point, so BigQueryAgentAnalyticsPlugin and other observability plugins are bypassed in a scripted or CI eval run, not only in the CLI.
The adk web eval tab (dev_server.run_eval), which also builds LocalEvalService(root_agent=...) without the App.

A complete fix threads app= from all three callers into the shared leaf, which then builds the eval Runner from app.model_copy(update={"plugins": list(app.plugins) + internal_eval_plugins, "root_agent": root_agent}). That direction also matches where the Runner API is already heading: the Runner(plugins=...) argument is deprecated in favor of app=.

I have this implemented on current main, building on #5534's approach and extending it to AgentEvaluator and dev_server, with unit tests for each surface. Happy to open a PR that credits #5534, or share the branch, whichever the maintainers and @saifer82 prefer.

🚨 Automated Spam Detection Alert 🚨
@maintainers, a suspected spam comment was detected in this thread.

Reason:

@alex-pathcourse posted promotional content and an external link promoting a 3rd party service ("PathCourse Health's observability module").

fix(eval): wire App plugins through eval paths

Opened #6480 with the complete fix.

It keeps #5534's approach for the adk eval CLI path and extends it to the two surfaces that reach the same bare-root_agent construction, AgentEvaluator and the adk web eval tab, plus the live inference leaf that landed since. Runner construction is shared between the live and non-live leaves, so both carry the App.

Verified end to end with a reproduction that observes whether the App's plugins actually run during eval inference: dropped on released 2.5.0, applied on the branch, with nothing patched in either run.

fix: wire App plugins through eval paths

Navigation Menu

Repository navigation

# `cli_eval` bypasses `App.plugins`, breaking observability plugins (e.g., `BigQueryAgentAnalyticsPlugin`) during eval runs #5503

Summary

Reproduction

Root cause

Why this matters

Proposed fix

Workaround for now

Environment

Root Cause

Proposed Fix

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

# cli_eval bypasses App.plugins, breaking observability plugins (e.g., BigQueryAgentAnalyticsPlugin) during eval runs #5503

Description

Summary

Reproduction

Root cause

Why this matters

Proposed fix

Workaround for now

Environment

Activity

sanketpatil06 commented on Apr 28, 2026

weiguangli-io commented on Apr 29, 2026

Root Cause

Proposed Fix

alex-pathcourse commented on May 11, 2026

doughayden commented on Jul 12, 2026

adk-bot commented on Jul 22, 2026

doughayden commented on Jul 25, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

# `cli_eval` bypasses `App.plugins`, breaking observability plugins (e.g., `BigQueryAgentAnalyticsPlugin`) during eval runs #5503