DatabaseSessionService.append_event reload-and-continue in 1.24+ corrupts session under concurrent run_async

Summary

ADK 1.24 changed DatabaseSessionService.append_event() from raising ValueError on stale sessions to silently reloading session state and continuing with the write. This breaks session integrity when two run_async() calls target the same session concurrently (e.g., cross-server recovery via run_async(invocation_id=X)), causing interleaved events that permanently corrupt the LLM's conversation history.

Reproduction

Server A starts run_async() on session S — begins writing events
Server B calls run_async(invocation_id=X) to resume the same session S (cross-server recovery)
Both servers write events concurrently to the same session via append_event()
ADK ≤1.23: Server B's append_event raises ValueError("stale session") → Server B stops → session trajectory stays clean
ADK ≥1.24: Server B's append_event reloads session state and continues → both servers write interleaved events → session trajectory is corrupted

Impact

The interleaved event trajectory is loaded as conversation history on subsequent run_async() calls. The LLM sees duplicate/contradictory tool calls and responses, producing unpredictable behavior for all future turns. This is permanent data corruption — not just a display issue.

Example corrupted trajectory:

event 1: [Server A] model_response: "I'll call tool X"
event 2: [Server A] tool_call: do_something("arg1")
event 3: [Server B] model_response: "I'll call tool X"          ← duplicate from resumed turn
event 4: [Server A] tool_response: {result...}
event 5: [Server B] tool_call: do_something("arg1")             ← duplicate
event 6: [Server B] tool_response: {result...}

Root Cause

In database_session_service.py, the stale session check changed from reject to reload:

ADK 1.23 (rejects concurrent writes):

if storage_session.update_timestamp_tz > session.last_update_time:
    raise ValueError("... Please check if it is a stale session.")

ADK 1.24+ (current behavior — reloads and continues):

if storage_session.get_update_timestamp(is_sqlite) > session.last_update_time:
    # Reload the session from storage
    session.state = _merge_state(app_state, user_state, session_state)
    session.events = [e.to_event() for e in storage_events]
    # ... then writes the event anyway

This was introduced in commit 1063fa53 (closing #1733) to fix false-positive stale session errors caused by timestamp precision loss. #3717 reports the same underlying issue. The asyncio.Lock (_with_session_lock) added in the same release (f5084746) only serializes writes within a single process — it provides no cross-server protection.

The deeper issue: timestamps are not sequence numbers

The original #3717 bug was caused by using float timestamps for optimistic concurrency control. The conversion chain time.time() → datetime.fromtimestamp() → DB DATETIME/TIMESTAMP column → read back → .timestamp() → float introduces precision loss at each step. On MySQL (DATETIME(fsp=6)) and PostgreSQL (TIMESTAMP), a single writer's own previous write can appear "newer" than the stored session.last_update_time due to microsecond rounding differences, triggering a false-positive stale detection.

Spanner provides externally consistent COMMIT_TIMESTAMP, but ADK doesn't use it — it sets update_time from the application clock via datetime.fromtimestamp(event.timestamp).

The reload-and-continue change in 1.24 addressed the symptom rather than the root cause.

Suggested Fix

Replace the timestamp-based OCC with a monotonically increasing integer sequence number per session:

# Per-session integer, incremented on every append_event commit
storage_session.event_sequence += 1

# OCC check: reject if another writer advanced the sequence
if storage_session.event_sequence != session.expected_next_sequence:
    raise ValueError("Another writer modified this session")

This eliminates both problems:

No precision loss: integers are exact, no float↔datetime conversion issues (fixes Title: DatabaseSessionService.append_event raises stale session error with LongRunningFunctionTool ADK Version #3717 root cause)
Cross-server rejection: a foreign writer's sequence increment is reliably detected (fixes this issue)
Self-write safe: a single writer's own sequence always matches (no false positives)

Environment

ADK versions affected: ≥1.24.0 (tested 1.24.0, 1.25.0, 1.26.0)
Last working version: 1.23.0
Database: Spanner via sqlalchemy-spanner-async
Deployment: Cloud Run (multiple instances)

The last_update_time provided in the session object '2025-07-01 18:59:42' is earlier than the update_time in the storage_session '2025-07-01 18:59:56'. Please check if it is a stale session #1733 — the issue whose fix (1063fa53) introduced this regression
Title: DatabaseSessionService.append_event raises stale session error with LongRunningFunctionTool ADK Version #3717 — same underlying stale session error, reported with LongRunningFunctionTool

@pztang9

Thanks for reporting this critical issue, @pztang9!

My research confirms your analysis: the change in DatabaseSessionService.append_event() introduced in ADK 1.24 (commit 1063fa53) to "reload-and-continue" instead of raising a ValueError on stale sessions is indeed leading to permanent data corruption in concurrent run_async() scenarios.

Root Cause:
The core problem is the reliance on float timestamps for optimistic concurrency control (OCC), which suffers from precision loss across database operations. The ADK 1.24 change was a symptomatic fix for false-positive stale session errors (as highlighted in #3717) but completely bypasses true concurrency protection, especially in cross-server deployments. The asyncio.Lock (_with_session_lock) provides only in-process serialization and offers no protection against concurrent writes from different server instances.

Suggested Solution:
I strongly agree with your proposed fix: replace the timestamp-based OCC with a monotonically increasing integer sequence number per session. This provides robust cross-server concurrency control and eliminates precision loss issues.

Here's how this could be implemented within src/google/adk/sessions/database_session_service.py in append_event method:

Add event_sequence to StorageSession: Introduce a new integer column, event_sequence, initialized to 0.
OCC Check: Before applying changes, verify storage_session.event_sequence matches an expected_next_sequence (from the in-memory session). If they don't match, raise a ValueError or ConcurrentModificationError.
Increment Sequence: If the check passes, increment storage_session.event_sequence by 1 before committing the transaction.
Remove Timestamp Stale Check: The if storage_session.get_update_timestamp(is_sqlite) > session.last_update_time: block can be removed as the sequence number handles concurrency.

This approach will restore fail-fast behavior for true concurrent modifications, preventing data corruption, while avoiding false positives related to timestamp precision.

# Conceptual change in DatabaseSessionService.append_event

# ... (existing code to load storage_session, etc.) ...

# Assuming StorageSession has 'event_sequence' and Session has 'expected_next_sequence'
if storage_session.event_sequence != session.expected_next_sequence:
    raise ValueError('''Another writer modified this session concurrently.''')

# ... (existing state merge logic) ...

storage_session.event_sequence += 1 # Increment for the current write

# The update_time logic can remain for auditing, but not for OCC.
# ... (existing update_time logic) ...

sql_session.add(schema.StorageEvent.from_event(session, event))
await sql_session.commit()

# Update the in-memory session's expected_next_sequence
session.expected_next_sequence = storage_session.event_sequence

This ensures data integrity by explicitly rejecting concurrent modifications rather than silently corrupting the session history.

Fix google#4751: Implement optimistic concurrency control with event …

Fix google#4751: Address code review feedback on optimistic concurren…

@pztang9

@pztang9 the reload-and-continue change in 1.24 (commit 1063fa53) does remove cross-server concurrency protection entirely, and the asyncio.Lock in _with_session_lock only serializes within a single process.

As an interim workaround, pinning back to 1.23.0 should restore the fail-fast behavior and stop the session corruption — could you confirm whether that holds in your Spanner/Cloud Run setup?

If it does, PR #4752's sequence-number OCC approach looks like the right long-term fix, but it'd be good to validate the workaround is stable on your end before we move forward with that.

@surajksharma07

@surajksharma07 I pinned my app to 1.23.0 to retain the cross-server concurrency protection. It works for me as far as I am concerned. However this is not without nuances.

Spanner as session store does not work out of the box for beyond version 1.18 because DatabaseSessionService switch from sync to async, therefore requiring an async SQLAlchemy dialet. The upstream sqlalchemy-spanner package has no async driver (open since 2022, p3 priority, no ETA). On my side we attempted to provide an async adapter, this though reveals some other issues.

We are also trying Postgres, but the set up in 1.23.0 has a bug where the append_event() query be written as SELECT ... FOR UPDATE to ensure serialization with this read-modify-write operation. The impelmentation in 1.23.0 lacks the FOR UPDATE part. I monkey patched it to inject the FOR UPDATE clause, and so far it looks like it is working.

Given the above experience, I also filed issue #4772 and #4771 in hope we can get to a state of a correct working spanner session store.

@pztang9

Thanks for the detailed follow-up, @pztang9 — glad 1.23.0 restores cross-server protection for your Spanner setup.

The async dialect gap (no upstream async driver for sqlalchemy-spanner) and the missing SELECT ... FOR UPDATE in Postgres append_event() are both valid independent gaps — the latter is a real serialization hole that exists regardless of the OCC check, since without row-level locking the read-modify-write can still race between concurrent processes even with PR #4752's
sequence check applied.

Your filed issues #4771 and #4772 are the right place to track the broader Spanner/Postgres correctness work; we'll keep those alongside #4752 as we move toward a proper fix.

@GWeale

@GWeale Please have a look into it once.

fix: Reject appends to stale sessions in DatabaseSessionService

@GWeale

Unfortunately, the fix b8e7647 doesn't help much. Precision lost is still in place. For example, I have the following false positive error due to 6 digit millis truncation to 3 digit millis in MSSQL:

session._storage_update_marker = {str} '2026-04-21T14:52:54.488582'
storage_session.get_update_marker() = {str} '2026-04-21T14:52:54.490000'

cc @GWeale

fix: Reject appends to stale sessions in DatabaseSessionService

Navigation Menu

Repository navigation

DatabaseSessionService.append_event reload-and-continue in 1.24+ corrupts session under concurrent run_async #4751

Summary

Reproduction

Impact

Root Cause

The deeper issue: timestamps are not sequence numbers

Suggested Fix

Environment

Related

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

DatabaseSessionService.append_event reload-and-continue in 1.24+ corrupts session under concurrent run_async #4751

Description

Summary

Reproduction

Impact

Root Cause

The deeper issue: timestamps are not sequence numbers

Suggested Fix

Environment

Related

Activity

9chait9 commented on Mar 7, 2026

surajksharma07 commented on Mar 11, 2026

pztang9 commented on Mar 11, 2026

surajksharma07 commented on Mar 11, 2026

surajksharma07 commented on Mar 11, 2026

EugeneYushin commented on Apr 21, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions