Repository navigation
Wait for the port-forward instead of sleeping in the smoke test - #861
Merged
Merged
Conversation
The smoke test intermittently fails on CI immediately after printing "Check the admin URL to make sure it works": make: *** [Makefile:227: smoke-test] Error 1 The run log shows the cause. The backgrounded 'kubectl port-forward' logged "Forwarding from 127.0.0.1:18983" at 23:05:40.029, which is 540ms *after* the curl that follows the fixed 'sleep 2' had already been refused at 23:05:39.489. With 'set -e' a refused connection aborts the whole smoke test. Replace the two 'sleep 2' calls that follow a port-forward with a helper that polls the port until it answers, for up to 60 seconds. The 'sleep 15' before the prometheus exporter query is deliberately left alone: it waits for the exporter to scrape rather than just for the forward to come up, and has a much larger margin. This flake is not related to pinning the actions. The same smoke test passed on another branch on the same ubuntu-26.04 runner image, and the pinned SHAs are the commits the floating major tags already pointed at.
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The readiness helper must enforce the documented 60-second timeout before approval.
1 open issue
What changed in this PR
This pull request replaces fixed delays after Solr kubectl port-forward commands with readiness polling to reduce intermittent smoke-test failures.
Changes:
- Adds a reusable helper to poll forwarded ports.
- Replaces both Solr-related
sleep 2calls with port-readiness checks. - Keeps the Prometheus exporter scrape delay unchanged.
| File | Summary |
|---|---|
hack/release/smoke_test/test_cluster.sh |
Adds and applies port-forward readiness polling. The helper may wait substantially longer than the documented 60-second limit because it counts attempts rather than elapsed time; it should track a wall-clock deadline and bound probes by the remaining time. |
Review effort: Lite
Counting attempts let a hanging connection stretch the 60s limit to 180s, since each probe could burn its own --max-time plus the sleep.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The smoke test intermittently fails right after
Check the admin URL to make sure it works. The run log shows why: the backgroundedkubectl port-forwardloggedForwarding from 127.0.0.1:18983540ms after thecurlfollowing the fixedsleep 2had already been refused. Underset -ea refused connection aborts the whole smoke test.Replaces the two
sleep 2calls that follow a port-forward with a helper that polls the port until it answers, up to 60s. Thesleep 15before the prometheus exporter query is left alone, since it waits for the exporter to scrape rather than for the forward.Seen on run 40035568922 (PR #856); the same test passed on another branch on the same runner image, so it is a flake rather than a regression.
🤖 AI assisted