Skip to content

0.7.0: time out document reads and opens - #11

Merged
davidackerman merged 3 commits into
mainfrom
document-timeouts
Oct 23, 2026
Merged

davidackerman merged 3 commits into
mainfrom
document-timeouts

Conversation

@davidackerman

Copy link
Copy Markdown
Member

0.7.0: store opens time out. No placement change and no unit table change.

The bug (cellmap-flow's finding). Retry bounded data reads only. Suppose an http server accepts the connection and never answers. Then wa.open and wa.metadata waited on the metadata fetch forever. Reproduced locally: still waiting after 40 s.

The fix.

  • Every metadata document read or written, and every store open, now waits under Retry as data reads do. Each is treated as an operation of no bytes, so by default an attempt waits 5 s times its number, for up to ten attempts.
  • tensorstore's own request retries still run inside each attempt.
  • The policy is the calling function's retries. wa.metadata, wa.detect and wa.open_tensorstore gain retries=; wa.open, wa.open_pyramid and wa.create already took it.
  • With a 0.3 s, two-attempt policy, all three raise TimeoutError within seconds. The error names the URL and the attempts.
  • Two things are left alone. Listings, because their size is unknown. The open that creates an array, because a second attempt would find it already made.
  • This lets cellmap-flow drop the 5 s requests probe in front of its picker's dtype poll.

Also in this PR: spec open question 5 is resolved, with no funlib_attrs. No live consumer reads funlib attributes from cellmap-analyze or cellmap-flow outputs:

  • cellmap-analyze 0.6.0 writes OME only.
  • cellmap-flow is the only reader of its own funlib keys.
  • dacapo isn't fed these outputs.
  • cellmap-data and fibsem-tools read OME only.

Open question 11 now points at an OME-shaped array attribute. The spec also notes that cellmap-data reads an OME translation as voxel 0's corner, which puts it half a voxel off.

186 tests pass locally (1 opt-in stress test skipped), and ruff passes.

https://claude.ai/code/session_0141wgF8mkGbjGfBFKfKhp64

A server that accepts a connection and never answers held wa.open and wa.metadata
forever: Retry bounded data reads only (cellmap-flow's finding). Every document read or
written and every store open now waits under the calling function's retries, as an
operation of no bytes, with tensorstore's request retries inside each attempt.
wa.metadata, wa.detect and wa.open_tensorstore take retries= as wa.open already did.
Listings and the creating open are left alone.

Claude-Session: https://claude.ai/code/session_0141wgF8mkGbjGfBFKfKhp64
No live consumer reads funlib attributes from cellmap-analyze or cellmap-flow outputs:
cellmap-analyze 0.6.0 writes OME only, cellmap-flow reads back its own, dacapo is fed
none of them, cellmap-data and fibsem-tools read OME only. Notes cellmap-data's
translation-as-corner reading.

Claude-Session: https://claude.ai/code/session_0141wgF8mkGbjGfBFKfKhp64
@davidackerman
davidackerman marked this pull request as ready for review October 23, 2026 18:43
@davidackerman
davidackerman merged commit f2620bc into main Oct 23, 2026
4 checks passed
@davidackerman
davidackerman deleted the document-timeouts branch October 23, 2026 18:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant