[filimonov]

The background CAS mount lease renewal used to give up when it got no definitive answer from the object store inside its budget. It then tripped the fence and started a remount under a fresh epoch. A short object-store outage (tens of seconds) cost about 40 s without writes, restarted merges and counted a CASMountLeaseLost, although nothing had touched the slot and the holder still had its lease on its own clock. Now the renewal retries until the store gives a definitive answer. While the lease cannot cover a write, writes pause and resume under the same epoch; the mount is no longer lost.

The branch is rebased on antalya-26.6-20261008, which already contains #2509 and #2513. It is not stacked any more. Read it commit by commit.

Rollout

• Date a GC mount-token sighting after the read that returned it #2509 (GC dates a sighting after the read) must run on the GC servers before this change reaches the holders, because a live holder now keeps one slot token for the whole outage. It is merged and in the base.
• Wait for the lease a CAS mount holder declares before fencing or reclaiming its slot #2516 (observers wait for the lease the holder declares) is still open. Until it is merged and rolled out, every pool member must use the same cas_mount_lease_ttl_ms.

How it works

• Every cas_mount_renew_period_ms the renewal thread builds one tuple and keeps it for the whole series of attempts: key, body bytes, the expected ETag of the last confirmed body, write_attempt_id, and start, taken before the first request.
• The conditional PUT runs on the lease plane under Retry::untilDefinitive, which has no deadline. After an unclear attempt, the next one waits a uniform draw in [0.8, 1.2] of 1000 ms minus what the attempt took (Retry::pauseAfterAttempt). There is no growing backoff, so the first good second after an outage ends the series.
• Any store refusal (credentials, AccessDenied, MalformedXML, …) waits one renewal period, then a new engine call sends the same tuple. A refused read waits the same period. A clean 412 whose read failed repeats only the read. A refusal after an unclear attempt is settled inside the call.
• Four things end the series:
  • A commit. DriverLease::finish publishes the deadline start + lease under driver_mutex, before the driver leaves the call, so a remount cannot arm a new epoch in between. The store is unconditional; it is monotonic because a renewal starts only after the previous one finished, and the renewer cannot be replaced during a call.
  • A slot that moved: another owner, our own body gc_fenced, a deleted slot, or our token over other bytes. This goes to the existing loss handling.
  • A local failure.
  • A stop or a remount request. Every pause on the lease plane wakes on both.
• Stop semantics.
  • A stop between engine calls, after a refusal or after a read that found the slot unchanged, is NotAttempted: nothing of ours is in flight, and the farewell still hands the slot back.
  • A stop during a call that sent an attempt, or while the reads after a refused precondition repeat, is terminal with classification cancelled: the request may still land, or the slot may have moved. No farewell runs.
  • A remount request (park) always follows a fence trip and stays a loss.
• The pause.
  • Writes stop being admitted once the lease has less than 2 × envelope + cas_lease_safety_margin_ms left. This rule is unchanged.
  • From the same moment, computed by the same predicate, system.cas_mounts shows lifecycle = 'not_live' and lifecycle_reason = 'lease_expired', with lifecycle_since = when writes paused.
  • At the next renewal attempt the server logs one WARNING per pause. It names the writer_epoch and the last renewal error: CAS mount lease of '<srid>' under writer_epoch <E> has too little time left for a write … Last renewal error: ….
  • A confirmed renewal re-admits writes under the same epoch only if it leaves more than that reservation. A renewal confirmed later than that leaves writes paused, and the next one starts at once.
  • A refused write's error says writes are paused and resume once the renewal is confirmed.
  • A clean shutdown during a pause skips the farewell, which has no lease room left, logs one INFO, and leaves the slot for GC to observe.
• The bounded renewals stay bounded by the last confirmed lease minus the safety margin:
  • the startup re-anchor;
  • the remount re-anchor;
  • the background renewal of a pool opened for decommission (bounded_background_renewal).
• Late commit. A renewal confirmed after an outage longer than the lease stores an expires_at_ms already in the past. GC does not fence the slot for it, because GC fences only a token it watched unchanged for its whole threshold.

Settings (no new ones)

• cas_mount_lease_ttl_ms: the length of the right to write from each confirmed renewal.
• cas_mount_renew_period_ms: the interval between renewal starts, and the wait after a refusal.
• cas_attempt_timeout_ms: the budget of one attempt, hence the admission envelope.
• cas_lease_safety_margin_ms: unchanged.
• The 1000 ms spacing is a constant.

Risks / notes

• Nothing changes on the store or in the observer protocol. The remount machinery stays as the reaction to a definitive loss.
• CASMountRenewalDeadlineExceeded now counts only bounded renewals: the startup and remount re-anchors, and a decommission's background renewal. An outage of a normal mount no longer moves it or CASMountLeaseLost; alert on system.cas_mounts.lifecycle_reason = 'lease_expired' instead.
• During a pause, background merges restart about once a second and fail at commit (the pool stays Live while admission is closed). Tracked as backlog task CAS-256.
• The S39 soak card still expects a remount after the long fault and needs CAS-344.
• Docs: mounts-and-leases.md, debugging.md, monitoring.md, troubleshooting.md, metrics_and_events.md, cas_mounts.md.

Testing

• Unit, 30 new or rewritten tests on two shared fixtures:
  • CASRetrySpacing.* and CASRequestsSpacing.*: no deadline, spaced attempts, the standard backoff unchanged.
  • CASHeartbeatUntilDefinitive.*, one tuple past the lease and the refusal pause: RetriesOneTuplePastTheLease, ARefusedRenewalIsSentAgainAfterTheRefusalPause, AfterACleanPreconditionRefusalOnlyTheReadRepeats.
  • CASHeartbeatUntilDefinitive.*, every way a series ends after a refused precondition: AReadThatFindsAForeignHolderEndsTheSeries, AReadThatFindsOurSlotFencedEndsTheSeries, AReadThatFindsTheSlotDeletedEndsTheSeries, OurTokenOverOtherBytesEndsTheSeries.
  • CASHeartbeatUntilDefinitive.*, stops and liveness: AStopBetweenCallsEndsTheSeriesAsCancelled, AStopDuringTheReadsEndsTheSeriesWithoutAFarewell, ALostLivenessDuringTheReadsEndsTheSeriesAsALoss.
  • CASHeartbeatUntilDefinitive.ALateCommitRestartsTheGcObservation: a late commit against GC.
  • CASMountRuntime.*: a long outage is a pause under the same epoch; an old epoch's renewal does not move the new deadline; a late commit reports a true reason; a remount request ends a series.
  • CASMountRuntime.*, stop and park: they end the refusal wait at once, and the farewell survives a stop.
  • CASMountRuntime.*, the rest: one refusal waits one period; the decommission renewal ends inside its lease; the pause is logged once with the last error; a clean shutdown during a pause leaves the slot.
  • CASLifecycleSnapshot.*: the row is not_live from the admission edge.
  • CASHeartbeat.CancellationAfterSendIsTerminalAndForbidsRelease now also asserts classification cancelled.
  • Each new behaviour was shown red by reverting it on the final tree.
• Integration: test_long_mount_put_outage_pauses_under_the_same_epoch; test_cas_mount_renewal_retry 6 passed.
• CAS* gate: 2670 passed (release), 2674 passed (ASan, no error).
• TLA+ model run models/mount-lease-series-v2 @1e4f3222015, all batteries green. Pause safety ResumedEpochNeverWritesAfterGcActed holds in CaRenewalDriver.

Closes: #2431
Related: #2509
Related: #2513
Related: #2516

Changelog category (leave one):

• Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

A CAS mount lease renewal retries until the object store gives a definitive answer instead of giving up at a deadline. While the lease is too short to admit a write, writes pause and resume under the same epoch once a renewal is confirmed, instead of losing the mount and remounting.

Documentation entry for user-facing changes

• Documentation is written (mandatory for new features)

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Unit tests
• Performance tests
• Aarch64 tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• CAS (content-addressed storage; Antalya only)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• OAuth (5m)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [54d1258]

[k-morozov]

LGTM. My comments are mostly related to refactoring and don't affect the resolution of the issue.

[k-morozov]

Suggest renewUntilResolved.

[k-morozov]

untilLeaseSafe use deadline. Maybe could specify this info?

[k-morozov]

Suggest to use particular policy instead bool flag. Like policies in CasOperation::writeLoop.

[k-morozov]

I am wondering that we could forget to process new value in std::variants.

[k-morozov]

Maybe we could update this place because it depends on the specific policy.

[k-morozov]

It is not related with current PR.
I suggest rename open_requests_ to `lease_requests' becasee in the next it is confusing.

[k-morozov]

Suggest change name. For example reissue_interval_ms?

[k-morozov]

Different cas requests are being used here inside the methods, though it’s not immediately obvious. I can’t suggest any simple refactoring for this. Need to think.

[filimonov]

CI triage for 54d1258. No red is caused by this PR.

• Build (amd_release), Build (arm_release): Docker Hub pull of the builder image failed (TLS handshake timeout, connection reset by peer). Infra.
• Integration tests (amd_asan_ubsan, db disk, old analyzer, 2/8): UBSan in utils/self-extracting-executable/decompressor.cpp:209 while unpacking the binary, before any test ran. Infra.
• iceberg_1, parquet_aws_s3: could not bring up docker-compose cluster. Infra.
• Unit tests (asan_ubsan): only IcebergMetadataGenerator.ModifyColumnAfterItselfWithoutTypeIsNoop, the UBSan failure seen on every PR of this base. All CAS* tests pass.
• cas_selects: 1 scenario, against 8 on siblings (CAS: refuse new nested server_root_id, delete S3 staging on abort, actionable member refusals #2566). clickhouse1 hit 8894 Cannot assign requested address (errno 99, client port exhaustion) to the store, and one part-manifest PUT ran out of its retry budget, giving one client Code: 210. The lease-loss form is gone: 0 mount lease not held; renewal paused writes twice and recovered after 12 attempts / 10.7 s and 9 attempts / 7.9 s.
• cas_lightweight_delete_4: a lost lightweight DELETE in the base, not a flake. Mutation 23 registered 7 ms before mutation 22 (prepareMutationEntry writes the file outside the lock), the part was mutated straight to version 23 (Part 5_6_6_0_21 doesn't change up to mutation version 23), and mutation 22 was never applied: 50000 rows left. Fixed upstream in Fix a concurrent ALTER UPDATE being skipped and reported done in MergeTree ClickHouse/ClickHouse#124167, not in antalya-26.6 yet.