Handle context cancellation while waiting for ghost table migration - #1758
Open
ggilder wants to merge 1 commit into
Open
Handle context cancellation while waiting for ghost table migration#1758ggilder wants to merge 1 commit into
ggilder wants to merge 1 commit into
Conversation
Migrate() blocks on an unbuffered receive from ghostTableMigrated while waiting for the ghost table to be created. The only sender is onChangelogStateEvent(), which publishes via base.SendWithContext(). If the migration aborts during this window, abort() cancels the migration context, so SendWithContext() takes its ctx.Done() branch and returns without ever sending. Nothing else writes to the channel, so Migrate() blocks forever: it never returns, its deferred teardown() never runs, finishedMigrating is never set, and the status and throttler tickers -- which exit on finishedMigrating rather than on the context -- keep looping. The process stays alive indefinitely, logging a frozen status line, until it is killed externally. Extract the wait into waitForGhostTableMigrated() and select on the migration context alongside the channel, returning checkAbort() so the original abort error is surfaced rather than a bare context error. The extraction mirrors consumeRowCopyComplete() and makes the behaviour testable without a database. This is the same deadlock, and the same fix, as github#1677 applied to consumeRowCopyComplete; ghostTableMigrated is its remaining sibling. TestAbort_DuringGhostTableWait follows the existing TestAbort_* pattern and fails (blocking until its timeout) without this change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Prevents migration shutdown deadlocks when cancellation occurs while awaiting ghost-table migration.
Changes:
- Adds a context-aware ghost-table wait that preserves the abort error.
- Adds regression tests for cancellation and successful signaling.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
go/logic/migrator.go |
Replaces the blocking receive with a cancellation-aware helper. |
go/logic/migrator_test.go |
Tests abort and normal completion paths. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Contributor
Author
|
Passing CI on my fork ggilder#8 |
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Migrate()blocks on an unbuffered receive fromghostTableMigratedwhile waiting for the ghost table to be created:The only sender is
onChangelogStateEvent(), which publishes viabase.SendWithContext(). If the migration aborts during this window,abort()cancels the migration context, soSendWithContext()takes itsctx.Done()branch and returns without ever sending. Nothing else writes to the channel, soMigrate()blocks forever:Migrate()never returns, so its deferredteardown()never runsfinishedMigratingis therefore never setfinishedMigratingrather than on the context, so they keep loopingThe result is a process that has already decided to abort but stays alive indefinitely, logging a frozen status line, until it is killed externally. This is the same type of deadlock, and the same fix, as #1677 applied to
consumeRowCopyComplete()—ghostTableMigratedis its remaining sibling.Change
Extract the wait into
waitForGhostTableMigrated()and select on the migration context alongside the channel, returningcheckAbort()so the original abort error is surfaced rather than a bare context error. The extraction mirrorsconsumeRowCopyComplete()and makes the behaviour testable without a database.TestAbort_DuringGhostTableWaitfollows the existingTestAbort_*convention and blocks until its timeout (i.e. fails) without this change;TestWaitForGhostTableMigratedcovers the normal path.Possibly related
Not claiming these are fixed by this change — their root causes aren't established, and this only addresses hangs caused by an abort cancelling the context during the wait — but they are reports of hanging at this exact point, so noting them for reference: #884, #380. #1735 concerns the same
ghostTableMigratedchannel in a different scenario.script/cibuildreturns with no formatting errors, build errors or unit test errors.On
script/cibuild: formatting, build, and all unit tests pass. The three testcontainers-based suites (TestApplier,TestMigrator,TestEventsStreamer) cannot run in my environment — no rootless Docker provider — and fail identically on unmodifiedmaster, so they are unaffected by this change and left to CI.