Skip to content

fix(models) initialize Anthropic client off event loop - #6861

Open
sylvesterkaczmarek wants to merge 1 commit into
google:mainfrom
sylvesterkaczmarek:fix/anthropic-client-init
Open

fix(models) initialize Anthropic client off event loop#6861
sylvesterkaczmarek wants to merge 1 commit into
google:mainfrom
sylvesterkaczmarek:fix/anthropic-client-init

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown

Link to Issue or Description of Change

Problem

The Anthropic client is a synchronous cached property. Constructing it for the first request on Vertex Agent Engine can block the serving event loop while credentials and transport state are initialised, leaving the deployment unresponsive.

Solution

  • Initialise the Anthropic client with asyncio.to_thread.
  • Protect first construction with an async lock so concurrent first requests create one client.
  • Retain the cached fast path after initialisation.
  • Use the helper for both streaming and non-streaming generation.
  • Preserve preconfigured-client and validation behaviour.

Testing Plan

  • Added a concurrency regression test.
  • Relevant tests pass.

pytest tests/unittests/models/test_anthropic_llm.py -q

137 passed in 1.92s

The regression test verifies that concurrent calls construct the client exactly once and that construction runs on a different thread from the event loop.

All applicable pre-commit hooks passed.

Manual E2E

Not run on Vertex Agent Engine because that requires a deployed project and credentials. The unit test exercises the isolated first-construction path that caused the serving-loop wedge.

Checklist

  • I have read the CONTRIBUTING document.
  • I have self-reviewed this change.
  • I have added a regression test.
  • Relevant tests pass locally.
  • I have deployed this change to Vertex Agent Engine.
  • This change has no downstream dependencies.

Additional context

The implementation follows the issue reporter's confirmed workaround of moving first client construction off the serving event-loop thread, while making it automatic and concurrency-safe.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants