Skip to content

Handle context cancellation while waiting for ghost table migration - #1758

Open
ggilder wants to merge 1 commit into
github:masterfrom
ggilder:handle-ctx-cancellation-ghost-table-migrated
Open

Handle context cancellation while waiting for ghost table migration#1758
ggilder wants to merge 1 commit into
github:masterfrom
ggilder:handle-ctx-cancellation-ghost-table-migrated

Conversation

@ggilder

@ggilder ggilder commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Description

Migrate() blocks on an unbuffered receive from ghostTableMigrated while waiting for the ghost table to be created:

mgtr.migrationContext.Log.Infof("Waiting for ghost table to be migrated. Current lag is %+v", initialLag)
<-mgtr.ghostTableMigrated

The only sender is onChangelogStateEvent(), which publishes via base.SendWithContext(). If the migration aborts during this window, abort() cancels the migration context, so SendWithContext() takes its ctx.Done() branch and returns without ever sending. Nothing else writes to the channel, so Migrate() blocks forever:

  • Migrate() never returns, so its deferred teardown() never runs
  • finishedMigrating is therefore never set
  • the status and throttler tickers exit on finishedMigrating rather than on the context, so they keep looping

The result is a process that has already decided to abort but stays alive indefinitely, logging a frozen status line, until it is killed externally. This is the same type of deadlock, and the same fix, as #1677 applied to consumeRowCopyComplete()ghostTableMigrated is its remaining sibling.

Change

Extract the wait into waitForGhostTableMigrated() and select on the migration context alongside the channel, returning checkAbort() so the original abort error is surfaced rather than a bare context error. The extraction mirrors consumeRowCopyComplete() and makes the behaviour testable without a database.

TestAbort_DuringGhostTableWait follows the existing TestAbort_* convention and blocks until its timeout (i.e. fails) without this change; TestWaitForGhostTableMigrated covers the normal path.

Possibly related

Not claiming these are fixed by this change — their root causes aren't established, and this only addresses hangs caused by an abort cancelling the context during the wait — but they are reports of hanging at this exact point, so noting them for reference: #884, #380. #1735 concerns the same ghostTableMigrated channel in a different scenario.

In case this PR introduced Go code changes:

  • contributed code is using same conventions as original code
  • script/cibuild returns with no formatting errors, build errors or unit test errors.

On script/cibuild: formatting, build, and all unit tests pass. The three testcontainers-based suites (TestApplier, TestMigrator, TestEventsStreamer) cannot run in my environment — no rootless Docker provider — and fail identically on unmodified master, so they are unaffected by this change and left to CI.

Migrate() blocks on an unbuffered receive from ghostTableMigrated while
waiting for the ghost table to be created. The only sender is
onChangelogStateEvent(), which publishes via base.SendWithContext().

If the migration aborts during this window, abort() cancels the
migration context, so SendWithContext() takes its ctx.Done() branch and
returns without ever sending. Nothing else writes to the channel, so
Migrate() blocks forever: it never returns, its deferred teardown()
never runs, finishedMigrating is never set, and the status and throttler
tickers -- which exit on finishedMigrating rather than on the context --
keep looping. The process stays alive indefinitely, logging a frozen
status line, until it is killed externally.

Extract the wait into waitForGhostTableMigrated() and select on the
migration context alongside the channel, returning checkAbort() so the
original abort error is surfaced rather than a bare context error. The
extraction mirrors consumeRowCopyComplete() and makes the behaviour
testable without a database.

This is the same deadlock, and the same fix, as github#1677 applied to
consumeRowCopyComplete; ghostTableMigrated is its remaining sibling.

TestAbort_DuringGhostTableWait follows the existing TestAbort_* pattern
and fails (blocking until its timeout) without this change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings August 20, 2026 20:06

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Prevents migration shutdown deadlocks when cancellation occurs while awaiting ghost-table migration.

Changes:

  • Adds a context-aware ghost-table wait that preserves the abort error.
  • Adds regression tests for cancellation and successful signaling.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
go/logic/migrator.go Replaces the blocking receive with a cancellation-aware helper.
go/logic/migrator_test.go Tests abort and normal completion paths.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@ggilder

ggilder commented Aug 20, 2026

Copy link
Copy Markdown
Contributor Author

Passing CI on my fork ggilder#8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants