Skip to content

Orphaned function_call permanently and silently disables context compaction (once triggered)Β #7236

Closed

Description

@erickleal-azos

πŸ”΄ Required Information

Describe the Bug:
Once the first context compaction happens in a session, if there is an
orphaned function_call (a tool call whose function_response never
arrives β€” e.g. the process handling the tool died, such as a Kubernetes pod
restart while the agent was waiting on a tool), every subsequent
compaction is silently and permanently blocked
, even as the prompt keeps
growing past the configured token_threshold. There is no log and no error.

Root cause: google.adk.apps.compaction._longest_self_contained_prefix()
walks the compaction window keeping a set of open obligations keyed by call
id (function_call opens an id, the matching function_response closes
it), and only records a cut point where that set is completely empty. Once
the first compaction has happened, an orphaned call becomes the first
candidate event of every subsequent round, so the set can never empty again
β€” safe_length freezes at 0 forever, and _events_to_compact_for_token_threshold
(and the sliding-window path, which shares the same helper) returns [].

This affects both compaction strategies since both call the same helper.

Steps to Reproduce:

  1. Install: pip install google-adk==2.9.2
  2. Persist a session (e.g. via DatabaseSessionService) that already had a
    successful 1st compaction, followed by one orphaned function_call
    (no matching function_response anywhere in the session), followed by
    several normal, balanced turns.
  3. Reload the session and call the internal
    google.adk.apps.compaction._events_to_compact_for_token_threshold(...)
    (or exercise the sliding-window path) on it.
  4. Observe the returned list is empty β€” no compaction β€” even though there
    are dozens of eligible events after the orphan.

A minimal, deterministic, runnable reproduction (no LLM call / API key
needed β€” the bug lives entirely in the pure event-selection logic) is here:
https://github.com/erickleal-azos/simulate-adk-fc-orphan

git clone https://github.com/erickleal-azos/simulate-adk-fc-orphan
cd simulate-adk-fc-orphan
poetry install
poetry run python repro_orphan_function_call.py

Expected Behavior:
An orphaned function_call from an already-finished invocation, with no
response anywhere in the session and not a long-running/pending
auth/tool-confirmation call, should eventually be treated as unanswerable
so that compaction can resume over the healthy events that follow it
(optionally with a warning log, since today the stall is silent).

Observed Behavior:
_events_to_compact_for_token_threshold (and the sliding-window variant)
returns [] forever after the orphan, so the session's prompt never
compacts again and grows without bound. Example from the linked repro:

Scenario A -- NO orphaned function_call (baseline)
Baseline: events_to_compact = 36 event(s)
  -> 2nd compaction would proceed normally.

Scenario B -- orphaned function_call right after the 1st compaction
Buggy: events_to_compact = 0 event(s)
  -> BLOCKED: no 2nd compaction, even with 42 eligible events in the session.

Scenario B (healed) -- removing only the orphaned call event
Healed: events_to_compact = 36 event(s)
  -> 2nd compaction would proceed normally.

Environment Details:

  • ADK Library Version (pip show google-adk): 2.9.2 (verified byte-identical on main, so also present on latest)
  • Desktop OS: Linux (also observed in a GKE/Kubernetes)
  • Python Version (python -V): 3.11

Model Information:

  • Are you using LiteLLM: No
  • Which model is being used: N/A β€” the bug is in pure event-selection logic (google.adk.apps.compaction), no LLM call is involved in triggering or reproducing it.

🟑 Optional Information

Regression:
Not a regression from a specific prior version as far as I can tell β€” the
guard this interacts with was added in PR #5832 (fixing #5602, "Compaction
breaks LongRunningFunctionTool resume") and is deliberate/tested for the
first stall. What wasn't accounted for is that the stall is unbounded:
once the window's start point is pinned at last_compacted_end_timestamp,
an orphaned call there blocks that window from ever balancing again.

Logs:
None β€” this is precisely the problem: the block is completely silent, no
log or error is emitted, which made the original incident hard to diagnose.

Additional Context:
This was found in a real staging incident: a Kubernetes pod running the
tool-execution side was restarted while an agent was waiting on a tool call.
The session came back with a function_call that would never be answered,
and from that point on it never compacted again.

I have a proposed fix with a 7-scenario validation matrix (including the
two existing upstream tests that encode the conservative window,
test_sliding_window_excludes_pending_function_call_events and
test_sliding_window_plain_orphaned_function_call_dropped_from_contents,
replayed to confirm no regression) here:
https://github.com/erickleal-azos/simulate-adk-fc-orphan#the-proposed-solution

Summary of the approach: keep the strict rule exactly as-is (if it yields a
non-empty prefix, return it untouched β€” behavior for the normal/healthy
case does not change). Only when it yields nothing (the stall) do we ask a
second question β€” is each open obligation still fulfillable? An id keeps
blocking if: (1) a matching function_response exists anywhere in the
session (not just the window, to avoid splitting a pair across the window
boundary when the response lives past the window's end β€” this is what the
original guard protects), (2) it is in Event.long_running_tool_ids or a
pending auth/tool-confirmation request, or (3) it was opened by the newest
invocation in the session (may still be in flight). Everything else is a
normal tool call from an already-finished invocation with no response
anywhere β€” provably dead, since a normal tool always emits a response event
(including on error), so its absence means the process died holding the
obligation. Event.long_running_tool_ids is persisted in its own DB column
and survives the kind of restart that causes this, which the fix relies on.

I'm open to opening a PR against src/google/adk/apps/compaction.py with
this fix, if the maintainers agree the approach makes sense β€” just let me
know and I'll follow up with tests in tests/unittests/apps/test_compaction.py.

Minimal Reproduction Code:
See https://github.com/erickleal-azos/simulate-adk-fc-orphan/blob/main/repro_orphan_function_call.py
for the full runnable script. Core of the buggy helper it exercises
(google/adk/apps/compaction.py):

open_ids: set[str] = set()
safe_length = 0
for index, event in enumerate(events):
  open_ids -= _event_function_response_ids(event)
  open_ids |= _event_function_call_ids(event)
  if event.actions:
    open_ids |= set(event.actions.requested_tool_confirmations)
    open_ids |= set(event.actions.requested_auth_configs)
  if not open_ids:
    safe_length = index + 1
return events[:safe_length]

How often has this issue occurred?:

  • Rare (<1%) β€” requires the process handling a tool call to die or be
    killed mid-call (pod restart, OOM kill, crash) after at least one
    compaction has already happened in the session. In practice an
    uncommon window, but when it does line up the block is permanent and
    silent.

Activity

  1. chelsealong commented on Sep 22, 2026

    @chelsealong
    Contributor

    Picking this one up now β€” opening a PR shortly. Flagging it here so nobody duplicates the work; if someone is already on it, say so and I will drop mine.

  2. erickleal-azos commented on Sep 22, 2026

    @erickleal-azos
    Author

    @chelsealong
    I was hoping to send a PR if one of the maintainers agreed with the solution

  3. added
    core[Component] This issue is related to the core interface and implementation
    on Sep 23, 2026
  4. surajksharma07 commented on Sep 23, 2026

    @surajksharma07
    Collaborator

    @erickleal-azos The fix in #7237 lines up with your proposed approach and holds up against our own repro. Could you verify it against your actual staging scenario (the pod-restart case) before it goes in? Once you've confirmed it's solid on your end go ahead and let the PR proceed to review rather than raising a separate one.

  5. erickleal-azos commented on Sep 23, 2026

    @erickleal-azos
    Author

    @surajksharma07 I patched #7237 into my repro repo (applied it directly on top of google-adk==2.9.2 in a clean venv) and re-ran the exact scenario from the staging incident. After the patch: compaction resumes normally, So yeah, it holds up.

    The approach is basically identical to what I was going to propose: strict pass first, and only fall back to discounting call ids that are provably dead (no response anywhere in the session, not long-running/pending, not from the newest invocation) if the strict pass comes back empty.

    One thing I noticed while comparing: the fix doesn't add any logging when it decides to compact past a dead call. Given the original issue was partly "this fails completely silently," might be worth a logger.warning there so a future incident like this one leaves a trace (but that's a nice-to-have, not a blocker on my end).

    Go ahead and let #7237 proceed to review, no need for me to open a separate one.

    Not gonna pretend I'm not bummed someone beat me to it while I was waiting on your read on the approach, I'd have liked to land the actual fix myself. Lesson learned though, next time I'll just open the issue and the PR together instead of waiting around.
    But honestly the fix is right, so let's just get it merged.

  6. surajksharma07 commented on Sep 25, 2026

    @surajksharma07
    Collaborator

    Fair point on the timing @erickleal-azos. The root cause, the repro repo and the "provably dead" rules all came from you and #7237 is built on them. Opening the PR together with the issue next time is definitely the way to go.

    Agree on the logging. compaction.py already has a module logger so @chelsealong could you add a logger.warning in _longest_self_contained_prefix when the fallback pass kicks in with the dead call ids and the session/invocation? That way the next incident like this leaves a trace.

    Also merged #7237 onto current main (applies cleanly on top of the recent compaction threshold change). test_compaction.py passes (58/58) and the guard cases still hold: an orphan from the newest invocation, a long-running call and a call whose response sits past the window all still block. Only the dead orphan gets compacted.

    Once the log line is in @DeanChensj this should be good for review. Until then the issue is still present in 2.9.2 and main.

  7. added a commit that references this issue on Oct 2, 2026
    73ad447
  8. added 2 commits that reference this issue on Oct 2, 2026
    e697990
    3f34675
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

core[Component] This issue is related to the core interface and implementation

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions