Persistent crash loop caused by missing tool_result in conversation history after interrupted execution

Describe the bug
The application enters a persistent crash loop when resuming a session where the previous execution was interrupted immediately after a tool_use event but before the corresponding tool_result could be saved.

This interruption typically occurs in two scenarios:

Server-side interruption: Server restarts, OOM crashes, or deployment cycles occurring exactly during tool execution.
Client-side interruption: A user refreshes the browser or closes the connection while the agent is processing a tool call. The server receives a cancellation signal and aborts the task likely without persisting the failure/cancellation state to the database, leaving the history incomplete.

When the session resumes, ADK (via LiteLLM) attempts to send this "corrupted" history (a tool_use in the last assistant message without a following tool_result) to strict LLM providers like Anthropic or OpenAI. These providers reject the request with a BadRequestError (400), rendering the session permanently unrecoverable ("bricked") without manual database intervention.

To Reproduce
I have created a standalone reproduction script using litellm to simulate the API rejection that occurs inside ADK.

Prerequisites:

Set ANTHROPIC_API_KEY or OPENAI_API_KEY.
Install litellm.

Reproduction Script:

import asyncio
import os
from litellm import acompletion, BadRequestError

async def test_model(model_name: str, api_key_env: str):
    print(f"\n📡 Testing Model: {model_name} ...")
    if not os.getenv(api_key_env):
        print(f"⚠️  SKIPPING: {api_key_env} not found.")
        return

    # Broken history: assistant called a tool, but no tool_result follows.
    broken_messages = [
        {"role": "user", "content": "What is the weather in Seoul?"},
        {
            "role": "assistant",
            "content": "I will check the weather.",
            "tool_calls": [
                {
                    "id": "tool_u_reproduce_123", 
                    "type": "function",
                    "function": {
                        "name": "get_weather",
                        "arguments": "{\"location\": \"Seoul\"}"
                    }
                }
            ]
        },
        # --- MISSING TOOL RESULT DUE TO REFRESH/RESTART ---
        # The next message is a new user turn, violating the strict tool_use -> tool_result sequence.
        {"role": "user", "content": "Wait, tell me about Tokyo instead."}
    ]

    tools = [{
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather",
            "parameters": {
                "type": "object",
                "properties": {"location": {"type": "string"}},
                "required": ["location"]
            }
        }
    }]

    try:
        await acompletion(model=model_name, messages=broken_messages, tools=tools)
        print(f"❌ {model_name}: PASSED (Resilient).")
    except BadRequestError as e:
        print(f"✅ {model_name}: FAILED (Vulnerable). Crash Reproduced!\n   Error: {e}")

async def run_all_tests():
    for model, key_env in [("anthropic/claude-3-5-sonnet-20240620", "ANTHROPIC_API_KEY"), ("openai/gpt-4o", "OPENAI_API_KEY")]:
        await test_model(model, key_env)

if __name__ == "__main__":
    asyncio.run(run_all_tests())

Expected behavior
The ADK framework should be resilient to history corruption caused by interruptions. When loading or preparing conversation history for LLM execution:

Validation: Detect any assistant message containing tool_calls that is NOT immediately followed by the required tool result messages.
Auto-Healing: Automatically inject a placeholder tool_result (e.g., {"role": "tool", "content": "Error: Execution interrupted (server restart or page refresh).", "tool_call_id": "..."}) into the message list to satisfy API constraints.
Recovery: This allows the LLM to understand that the previous action failed/interrupted and enables the session to continue seamlessly instead of crashing.

Desktop:

OS: macOS / Linux
Python version: 3.12
ADK version: 1.21.0

Model Information:

LLM Provider: Anthropic (Claude 3.5 Sonnet), OpenAI (GPT-4o)
Interface: LiteLLM

@donggyun112

@donggyun112 This is a critical issue - when interruptions leave orphaned tool_use events in history, ADK has no validation to detect or heal them before sending to LiteLLM, causing 400 errors.

The fix needs to be in flows/llm_flows/contents.py in _get_contents() (after line 445) - scan for tool_calls without matching tool_results and inject synthetic error responses (see example in your expected behavior).

As a temporary workaround, wrap session loading with validation logic that checks event sequences and adds placeholder tool_results for incomplete pairs before resuming - this prevents the session from bricking and allows graceful recovery.

Your reproduction script clearly demonstrates the issue with both Anthropic and OpenAI providers.

@DeanChensj

@DeanChensj Please have a look into it.

@donggyun112

@donggyun112 This is a critical issue - when interruptions leave orphaned tool_use events in history, ADK has no validation to detect or heal them before sending to LiteLLM, causing 400 errors.

The fix needs to be in flows/llm_flows/contents.py in _get_contents() (after line 445) - scan for tool_calls without matching tool_results and inject synthetic error responses (see example in your expected behavior).

As a temporary workaround, wrap session loading with validation logic that checks event sequences and adds placeholder tool_results for incomplete pairs before resuming - this prevents the session from bricking and allows graceful recovery.

Your reproduction script clearly demonstrates the issue with both Anthropic and OpenAI providers.

Thanks for the quick response!
Before implementation, I’d like to discuss one design concern: Silent healing vs. Explicit error.

• Auto-healing concern: While auto-injecting a placeholder tool_result fixes the crash, developers might not realize why their tool didn't execute as expected.

• Proposal: Should we also emit a warning log or a callback event when this auto-healing occurs? This would maintain stability while ensuring transparency for debugging.

I’ve already traced the code path to contents.py:_get_contents() and would be happy to contribute the PR if you’re aligned with this approach. What do you think?

I believe this should follow the existing tool failure pattern. By injecting a tool_result with an error message like 'Error: execution interrupted', the LLM can handle the situation naturally, just like any other tool failure. This maintains consistency and prevents the session from being bricked while allowing the model to decide the next step.

@donggyun112

@donggyun112 Your approach is exactly right - auto-healing should follow ADK's existing tool failure pattern, and adding logging/warning is crucial for debugging transparency.

Here's the recommended implementation:

Auto-healing: Inject function_response with error content (like existing tool failures in ADK):

# In contents.py after line 445
logger.warning(
    f"Auto-healing orphaned function_call (id={call_id}, name={call_name}). "
    "This indicates execution was interrupted before tool completion."
)
# Inject error response matching ADK's tool failure pattern

Consistency: This matches how ADK handles other tool failures - the LLM sees an error response and can naturally recover (retry, apologize, or take alternative action).
Observability: The warning log gives developers clear visibility into when/why healing occurs, essential for debugging production issues.

We would love to see your PR!

Feel free to reference the tool error handling in flows/llm_flows/functions.py for the response format.

If you need any clarification on the implementation details, let us know.

Thanks for contributing!

fix: auto-heal orphaned function_calls to prevent crash loop

@surajksharma07

@surajksharma07 FYI - As we discussed, I've submitted this PR to address the orphaned call issues.

fix: auto-heal orphaned function_calls to prevent crash loop

@donggyun112

Thanks for the PR @donggyun112!

Team is reviewing #4056 - the implementation looks solid with proper auto-healing logic, warning logs, and comprehensive test coverage.
The approach of injecting synthetic error responses with 'user' as author is correct for multi-agent scenarios.

Team will prioritize reviewing and merging this since it fixes a critical session-bricking bug.
Once merged, it will be included in the future ADK releases.
In the meantime, users experiencing this issue can apply the PR as a patch.

Great work on the thorough implementation and tests!

fix: Heal missing tool results before LiteLLM requests

@surajksharma07

@surajksharma07 I need to understand what happened here.

Before writing any code, I discussed the approach with you — silent healing vs explicit error, and whether to emit warning logs. You said my approach was "exactly right" and encouraged me to submit a PR.

I followed your guidance exactly:

Location: contents.py near line 445, as you specified

Approach: The one we discussed and you approved

Tests: 7 comprehensive tests

Instead, a different implementation was merged without any notice:

Different location: lite_llm.py instead of contents.py

Test coverage: Only 1 test vs my 7 tests

Process: I wasn't informed about the parallel internal work

Since we collaborated on the design and my PR addresses more edge cases with comprehensive tests, I’d like to understand the reasoning behind this decision. What should I do with PR #4056?

I tested the merged implementation (commit 6b7386b) and found bugs.

The current fix processes messages sequentially — if a tool response isn't immediately after its call, it is incorrectly marked as orphaned. However, ADK officially supports async/out-of-order responses, which is why _rearrange_events_for_async_function_responses_in_history() exists.

My implementation handles this correctly by building a complete function_call_id -> response mapping first, then performing the match. The merged implementation breaks async responses.

Test Case	Merged Implementation	This PR (#4056)
Async responses	FAIL	PASS
Out-of-order responses	FAIL	PASS

Given these bugs, should I submit a follow-up fix PR to lite_llm.py, or would you prefer to revert and use the approach we originally discussed here?

@donggyun112

@donggyun112 I apologize for the miscommunication - we should have notified you about the parallel internal work before merging.
Your concerns about the different implementation location and approach are completely valid, and I understand the frustration.

I'd like to keep PR #4056 open for the team to review.
Your implementation in contents.py with 14 comprehensive tests provides a different approach (Event-level vs Message-level handling) that may offer additional robustness.
The team will evaluate whether both approaches have merit or if we should adopt your more thorough test coverage.

Regarding your concern about the merged implementation breaking async/out-of-order responses: the team will review this specific edge case you've identified.
If you can provide a concrete test case that demonstrates the failure scenario (where async responses are incorrectly marked as orphaned), that would be extremely helpful for the evaluation.

Thank you for the detailed analysis and your thorough testing approach - your contribution and the comprehensive test suite in #4056 are valuable to the project.
We appreciate your collaboration on making ADK more robust.

@surajksharma07

@surajksharma07 Thank you for the thoughtful response and for keeping the PR open for review.

As requested, I’ve prepared a test script that demonstrates the failure scenarios with the current merged implementation: https://gist.github.com/donggyun112/e0f01d2eace4dbef14267b30b55ca55f

The script covers several critical edge cases where async responses can be incorrectly handled:

Async tool responses arriving late in the history.
Parallel tool calls with out-of-order responses.
Conversations continuing while a tool is still running.

Regarding the architectural concern: ADK’s existing async response handling is centralized in _rearrange_events_for_async_function_responses_in_history() within contents.py. The recently merged implementation in lite_llm.py creates a separate code path that doesn't leverage this logic.

This fragmentation means:

If LiteLLM is called directly (bypassing the contents layer), async responses won't be handled correctly.

It introduces maintenance overhead by having similar logic in two different places.

My PR extends the existing function instead of adding a parallel implementation, ensuring that the handling logic remains robust and unified in one place.

@donggyun112

@donggyun112 The merged fix in lite_llm.py uses sequential processing that removes tool_call_ids immediately upon seeing a tool response, which indeed breaks when responses arrive out-of-order (as ADK officially supports via _rearrange_events_for_async_function_responses_in_history).

Your PR #4056's approach of building a complete function_call_id→response mapping first is architecturally correct and aligns with ADK's existing async handling in contents.py.

The concern about fragmentation is also valid - having healing logic in two different layers (contents vs lite_llm) creates maintenance overhead.

The team will review your test script demonstrating the async response failures.
Your 14 comprehensive tests provide valuable coverage that the merged implementation lacks.
We'll evaluate whether to adopt your approach in contents.py or enhance the lite_llm.py fix to handle async responses correctly.

Thanks for the thorough edge case analysis and maintaining PR #4056 open for evaluation.
Your contribution significantly improves the robustness discussion around this critical bug fix.

Thank you for the detailed review and for recognizing the concerns. I’ll keep PR #4056 available for the team’s evaluation. Let me know if you need anything else

fix: auto-heal orphaned function_calls to prevent crash loop

fix: Heal missing tool results before LiteLLM requests

fix(flows): prune orphaned function calls on interrupted turns

Navigation Menu

Repository navigation

Persistent crash loop caused by missing tool_result in conversation history after interrupted execution #3971

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Persistent crash loop caused by missing tool_result in conversation history after interrupted execution #3971

Description

Activity

surajksharma07 commented on Dec 30, 2025

surajksharma07 commented on Dec 30, 2025

donggyun112 commented on Dec 30, 2025

donggyun112 commented on Dec 30, 2025

surajksharma07 commented on Dec 30, 2025

donggyun112 commented on Jan 2, 2026

surajksharma07 commented on Jan 3, 2026

donggyun112 commented on Jan 6, 2026

donggyun112 commented on Jan 6, 2026

surajksharma07 commented on Jan 6, 2026

donggyun112 commented on Jan 6, 2026

surajksharma07 commented on Jan 7, 2026

donggyun112 commented on Jan 7, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions