Re-download truncated e2e-bin archives once in CI

[Mikola Lysenko (mikolalysenko)]

Problem

actions/download-artifact v4 (pinned v4.3.0) can finish successfully and leave a truncated file. In run 38059192473 attempt 1 (PR #1345), job e2e (ubuntu-latest, e2e_vex_build, hatch:: --ignored, 1.18.1) logged Starting download of artifact and nothing after it. The next step then failed:

target/e2e-archive/e2e-bin.tar.zst : Read error (39) : premature end

Attempt 2 on the same SHA passed. It's the only time this happened in the 3-day window I checked (step-level scan of 107 failed CI runs since 2026-10-08). The exposure is large, though: every merge group downloads the ~94 MB e2e-bin archive once per e2e, e2e-full and cargo-vex-matrix leg. In the merge queue a single hit evicts the entry and costs a full rebuild.

Root cause

A transient short read in the artifact download. The action doesn't check that the file is complete, so the truncation only shows up later, when zstd -d runs in "Unpack the e2e binaries". The upload side already has a retry wrapper (.github/actions/upload-artifact). The download side had nothing.

Fix

• New composite action .github/actions/download-artifact, built like the upload wrapper. It downloads once (continue-on-error) and runs a caller-supplied verify bash check. If the download or the check fails, it clears path, waits 10 s, downloads again and checks again. Neither of those last two steps continues on error, so a second bad download fails the job.
• The two e2e-bin consumers in ci.yml (the &e2e-steps and &cargo-vex-steps anchors, which cover e2e, e2e-windows/macos/full and every cargo-vex-matrix job) now use the wrapper with verify: zstd -q -t target/e2e-archive/e2e-bin.tar.zst.
• The pattern: e2e-bin-${{ matrix.os }}* contract that test_ci_scheduling.py checks is unchanged.

No test is removed or moved. Required-check names and ci-ok are unchanged.

Proof

• New tests in scripts/tests/test_ci_e2e_archive.py:
  • Each consumer's real verify command passes a good archive and fails a half-truncated one and a missing one.
  • The wrapper's step order and retry gating are pinned.
• python3 -B -m unittest discover -s scripts/tests: 327 tests OK, with zstd installed, so the archive tests actually run.
• zizmor --offline: the new action has no findings. In ci.yml the only new notes are help-level self-repository, the same note the existing ./.github/actions/upload-artifact uses already get.
• actionlint: no new findings. Its alias syntax-check errors were already there on main.

Review findings not fixed here

From `/code-review high`. Fixed in cd71fbf: the second check is gated on the second download succeeding, `verify` is required, and the test pins each step's exact `if:` gate (checked by mutating one gate).

• Two matching artifacts extracting into one directory (`-retry-N` names from the upload wrapper): not changed. This is how things already worked: the upload wrapper only publishes a retry name after attempt 1 failed. I found no failure with that signature in the window, and the one incident left no second artifact.
• docker-base-image and the compatibility workflows' downloads: left for a follow-up. They have no hits in the window. This PR changes only the e2e-bin consumers, the ones on the merge-queue path with dozens of downloads per run. Any consumer can switch to the wrapper by passing a `verify`.
• Bump to download-artifact v8 instead: v8 makes `digest-mismatch` default to `error`. That turns a bad download into a failed step but doesn't retry it, so the wrapper would still be needed. A major bump across 20 call sites is a separate change.
• `zstd -t` decompresses the archive a second time: kept. It costs well under a second on a ~94 MB archive and keeps the check separate from Unpack, so a retry is possible.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KdaGJqi1VLtSWMnHY4hP6g

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

bugbot run

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

bugbot run

---

Generated by Claude Code

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit cd71fbf. Configure here.}

[Mikola Lysenko (mikolalysenko)]

Ready for review at cd71fbf.

• CI: 36/36 non-skipped checks green on the head; no main-wide failures.
• Bugbot: reviewed cd71fbf, no findings.
• Mergeable: no conflicts with main; no CHANGELOG.md changes.

Slack announcement pending (no Slack send tool in this run; the next run retries).

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

Final review brief

What it does. Adds a composite action .github/actions/download-artifact that downloads, runs a caller-supplied verify check, and on failure clears the path and downloads/verifies once more (the second attempt fails hard). The two e2e-bin consumers in ci.yml use it with verify: zstd -q -t …/e2e-bin.tar.zst, so a silently truncated archive (seen in run 38059192473) no longer turns a queue leg red.

Risk: low. CI-only; mirrors the existing upload wrapper's continue-on-error/outcome pattern. Worst case is one extra ~94 MB download plus 10 s on a real failure.

Look here

• download-artifact/action.yml:26-72 — step gating: only attempt 1 is soft; attempt 2 verify gated on attempt 2 download succeeding
• ci.yml:1290-1297 and ci.yml:2265-2272 — the two consumers (cover e2e, e2e-windows/macos/full and every cargo-vex-matrix leg)
• test_ci_e2e_archive.py:122, :150 — runs each real verify command against good/truncated/missing archives; pins every step's if:

Verified. Read the full diff and compared with .github/actions/upload-artifact; checked zstd is already used by the Unpack step on every OS (so verify works on Windows/macOS too). Ran python3 -m unittest discover -s scripts/tests: 327 OK (6 skipped), zstd present so the archive tests ran. Merges cleanly with #1386. CI 54/54 (36 success, 18 skipped, 0 failed); Cursor Bugbot success.

Changes I made. None.

Open questions. None. Other download-artifact consumers (docker-base, compat workflows) are deliberately left for a follow-up.

Auto-merge is armed: approving sends it straight to the merge queue.

---

Generated by Claude Code