Skip to content

Agent Engine: _init_session replays all events via append_event causing 429 RESOURCE_EXHAUSTED on long sessions #5714

Closed

Description

@JoeyC1990

🔴 Required Information

Is your feature request related to a specific problem?

Yes. When an ADK agent is deployed to Vertex AI Agent Engine and invoked mid-conversation (e.g. via @agent in Gemini Enterprise after building context with the standard Gemini agent), the _init_session() method in vertexai/agent_engines/templates/adk.py (line ~810) replays every prior session event by calling session_service.append_event() individually:

# adk.py line ~810
await session_service.append_event(session, Event(**event))

For sessions with many events (common when users build context in the standard agent before handing off to a custom ADK agent), this triggers:

google.genai.errors.ClientError: 429 RESOURCE_EXHAUSTED
Quota exceeded for quota metric 'Session Event Append Requests'
and limit 'Session Event Append Requests per minute per region'
of service 'aiplatform.googleapis.com'

This is a self-perpetuating failure: each retry attempt consumes more quota, making recovery impossible until the user starts a fresh session.

Steps to reproduce:

  1. Deploy an ADK agent to Vertex AI Agent Engine with A2A enabled
  2. In Gemini Enterprise, have a multi-turn conversation with the standard Gemini agent (10+ turns with file attachments, images, etc.)
  3. @mention the custom ADK agent in the same conversation
  4. Observe the 429 error in Reasoning Engine stderr logs — the agent never reaches user code

Describe the Solution You'd Like

Any of the following (in order of preference):

  1. Batched event replay with rate-limiting: _init_session should batch append_event calls or use a bulk append API rather than individual sequential calls
  2. Incremental session resumption: Only replay events that haven't already been persisted, rather than replaying the entire history on every invocation
  3. Session event windowing: Allow configuring a maximum number of recent events to replay (similar to the GetSessionConfig.num_recent_events pattern from Allow limiting num. of Session events fetched when calling Runner.run_async #3562), discarding or summarising older context
  4. Expose session configuration on Agent Engine: Allow developers to configure session replay behaviour (e.g. max_replay_events, replay_strategy) when deploying to Agent Engine

Impact on your work

Critical. We have deployed multi-brand A2A agents (PowerPoint generation agents) to Agent Engine for enterprise users. The intended workflow is:

  1. Users discuss their presentation needs with the standard Gemini agent in Gemini Enterprise (building context, sharing images, iterating on ideas)
  2. Users then @mention our custom ADK agent to generate the actual presentation

This workflow is completely broken for any non-trivial conversation because the session replay crashes the agent before any user code executes. Users are forced to start a brand new conversation and re-provide all context, which defeats the purpose of the A2A cross-agent handoff pattern.

This is blocking production usage for our organisation today.

Willingness to contribute

Yes — happy to test and provide feedback on proposed solutions. Would consider submitting a PR if pointed to the right code path (the replay logic appears to be in the vertexai SDK rather than adk-python directly).


🟡 Recommended Information

Describe Alternatives You've Considered

  1. Starting a new conversation — Works but defeats the purpose of building context with the standard agent first. Poor UX.
  2. Trimming llm_request.contents in before_model_callback — Only prevents future growth; cannot fix the initial _init_session crash since it runs before any callbacks.
  3. Requesting a quota increase — May raise the ceiling but doesn't address the O(n) replay design. Sessions will still break at some threshold.
  4. "Context brief" pattern — Ask the standard agent to summarise context, then paste into a new conversation. Functional workaround but requires user discipline and breaks the seamless handoff experience.

Proposed API / Implementation

Option A — Add a max_replay_events config to Agent Engine deployment:

# In agent_engine deployment config
session_config = SessionConfig(
    max_replay_events=50,  # Only replay last 50 events
    replay_strategy="recent"  # or "summarise", "checkpoint"
)

Option B — Use bulk append in _init_session:

# Instead of:
for event in events:
    await session_service.append_event(session, Event(**event))

# Use a batch API:
await session_service.append_events_batch(session, [Event(**e) for e in events])

Additional Context

Environment:

  • OS: macOS
  • Python: 3.12
  • ADK version: Latest (deployed via Agent Engine)
  • Deployment: Vertex AI Agent Engine (Reasoning Engine) in europe-west1
  • Model: Gemini 3.1 Pro Preview / Gemini 3 Flash
  • A2A: Enabled

Related Issues:

Activity

  1. mahasarabesh commented on May 16, 2026

    @mahasarabesh
    Contributor

    Hi! I took a look at this issue and submitted a fix in #5720.

    Root cause: The external vertexai SDK's _init_session() replays all prior session events by calling �ppend_event() one at a time, which hits the Vertex AI API rate limit on sessions with many events.

    Since the root cause is in the external SDK, my PR adds defensive handling on the �dk-python side:

    1. Retry with exponential backoff on append_event() in VertexAiSessionService — catches 429 RESOURCE_EXHAUSTED and retries up to 5 times with jitter (1s initial, 2x backoff, 30s cap). Non-429 errors are raised immediately.

    2. ** append_events_batch()** on BaseSessionService (default sequential) with a VertexAiSessionService override that uses asyncio.Semaphore(5) for bounded concurrency — available for callers that need to replay many events at once.

    All existing tests pass and I've added 5 new tests covering retry behavior and batch append.

    Would love to hear opinions on this approach — particularly:

    • Are the retry parameters (5 attempts, 1s initial, 2x backoff) reasonable for the Vertex AI rate limits you've encountered?
    • Is append_events_batch() useful to include, or would you prefer the PR scoped to just the retry logic?

    Happy to adjust based on feedback!

  2. added
    agent engine[Component] This issue is related to Vertex AI Agent Engine
    on May 18, 2026
  3. surajksharma07 commented on May 22, 2026

    @surajksharma07
    Collaborator

    @JoeyC1990 the A2A handoff triggering a full sequential replay in _init_session is a real production gap, especially when sessions carry a lot of history from the standard Gemini agent side.

    One thing worth trying as a workaround: adding exponential-backoff retry logic around the append_event API call in VertexAiSessionService (5 attempts, ~1s initial delay, 2× backoff, capped at ~30s).
    That should absorb the per-minute burst and let the replay complete without hitting the quota ceiling.

    Could you test that against your europe-west1 setup and let us know whether it consistently gets past the threshold for your typical session sizes?
    PR #5720 from @mahasarabesh already has a concrete implementation of this approach — if your testing validates it in production, that would be great supporting evidence to help move that review forward!

  4. surajksharma07 commented on Jul 8, 2026

    @surajksharma07
    Collaborator

    Closing this out since @JoeyC1990 hasn't been back to confirm the workaround — but worth being clear that the underlying bug is still live and PR #5720 is still sitting unreviewed, so this isn't a "resolved" close.

    @mahasarabesh's retry/backoff + batch-append approach looks like the right shape for the in-repo mitigation — flagging for a maintainer to take a review pass on #5720 specifically so it doesn't stall just because this issue is no longer open.

    Feel free to reopen if you hit this again or want to pick the thread back up!

  5. added a commit that references this issue on Sep 3, 2026
    cc5985a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

agent engine[Component] This issue is related to Vertex AI Agent Enginerequest clarification[Status] The maintainer need clarification or more information from the author

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions