Skip to content

ci: prune unused Go build-cache entries before saving - #4676

Merged
aheritier merged 6 commits into
mainfrom
ci/measure-go-cache
Oct 21, 2026
Merged

aheritier merged 6 commits into
mainfrom
ci/measure-go-cache

Conversation

@dgageot

@dgageot dgageot commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Save only Go build-cache entries used or created by the current job, instead of carrying obsolete artifacts forward for Go's default five-day retention window.
  • Run pruning in a Node 24 post action before actions/cache saves. Keep PR jobs restore-only; leave module caching unchanged.
  • Start a separate v3 build-cache namespace. Cache misses remain ordinary recompilation, not correctness failures.

Measured results

Controlled run: same source change, same runners/modules, both caches restored using Actions Cache. Unit and binary tests forced uncached. Times include cache restore, build, and tests; exclude common tool/module setup.

Cache Run 1 Run 2 Mean
Current full cache 201s 224s 212.5s
Pruned cache 135s 119s 127s

About 40% faster in this Linux comparison, not a guarantee for every job.

  • Compressed cache: 2.99GB → ~460MB; expanded: 18.6GB → 2.63GB.
  • Restore alone: 62–115s → 7–18s.
  • Initial three-way comparison: full 199/234s, pruned 132/132s, no build cache 334/240s. This initial comparison used artifact delivery for the pruned arm; the table above removes that transport difference.

Validation

  • task build, task test, task lint passed locally.
  • 21 Node tests cover timestamps, symlinks, paths, action state, and actual Go package/executable reuse. Node 24 and 26 passed locally; all 21 passed on hosted Linux and Windows.
  • Actionlint (existing $/ syntax exception), workflow invariants and zizmor passed.
  • Production-action integration is green on Linux and Windows: warm, mark/prune, save, exact restore, source invalidation, build, and full tests. Confirms post-pruning runs before cache save. Resulting caches are ~463MB on both platforms.
  • Temporary benchmark workflow removed from the final patch; the linked runs retain the evidence. No production Go code changed.

Kept in draft for review. Expect a one-time cold build for each namespace on main; subsequent snapshots remain trimmed. Linux measurements are promising, not a guaranteed Windows speedup or whole-workflow saving.

@aheritier aheritier added area/ci CI/CD workflows and pipeline kind/chore Maintenance, deps, CI, tooling (maps to chore: commit prefix) labels Oct 20, 2026
First run showed timings but artifact-vs-cache was a confound. This
imports the prior pruned artifact into a unique Actions Cache entry
and compares identical cache restores (baseline vs pruned) after the
same source change invalidating the TUI, with binary tests pinned to
-count=1.
@dgageot

dgageot commented Oct 20, 2026

Copy link
Copy Markdown
Member Author

First hosted comparison is green: run 39504417698. Restore + build + forced-uncached tests: full cache 199s/234s; pruned 132s/132s; no cache 334s/240s. Inventory: full 18.60GB expanded, pruned 2.63GB; compressed 2.99GB vs 458MB. This is exploratory: pruned used artifact storage and was freshly warmed. The next revision compares both via Actions Cache after the same TUI source invalidation, with binary tests uncached too. Still draft; no production cache policy changed yet.

@dgageot

dgageot commented Oct 20, 2026

Copy link
Copy Markdown
Member Author

Second controlled run is green: 39505994975. Both arms now use Actions Cache, invalidate the same TUI source, and force unit/binary tests to execute. Full: 201s / 224s. Pruned: 135s / 119s (mean 40% faster). Restore: full 62s / 115s, pruned 18s / 7s. Implemented main-only pre-save pruning with a separate v3 build-cache namespace (module cache unchanged). Current revision validates the actual Node24 post action and exact candidate save/restore on both Linux and Windows; PR remains draft.

@dgageot dgageot changed the title ci: measure and reduce Go cache restore overhead ci: prune unused Go build-cache entries before saving Oct 20, 2026
@dgageot

dgageot commented Oct 20, 2026

Copy link
Copy Markdown
Member Author

Final integration is green on both Linux and Windows: https://github.com/docker/docker-agent/actions/runs/39509360263 . Both platforms passed all 21 helper tests, saved trimmed caches (~463MB), restored the exact new cache, and passed builds/full tests after source invalidation. Verified post-prune precedes cache save. The regular CI run is also green: https://github.com/docker/docker-agent/actions/runs/39509360336 . Temporary experimental workflow removed; PR remains draft. Final patch changes only cache infrastructure/tests, not production Go code.

@dgageot
dgageot marked this pull request as ready for review October 20, 2026 22:52
@dgageot
dgageot requested a review from a team as a code owner October 20, 2026 22:52
@aheritier

Copy link
Copy Markdown
Collaborator

@dgageot don't hesitate to involve @mpailloncy on such tuning if needed. He did it at scale in the past.

@aheritier
aheritier added this pull request to the merge queue Oct 21, 2026
Merged via the queue into main with commit 3f52e05 Oct 21, 2026
16 checks passed
@aheritier
aheritier deleted the ci/measure-go-cache branch October 21, 2026 08:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/ci CI/CD workflows and pipeline kind/chore Maintenance, deps, CI, tooling (maps to chore: commit prefix)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants