freshness: the checks could not act on a result

.github/workflows/build-deploy.yml

@@ -316,10 +316,20 @@ jobs:
       # ca-certificates-mozilla moved from 3.127-1 to 3.128-1 inside it.
       #
       # The bootstrap stage is already in the builder from the step above, so
-      # this exports files rather than installing anything again. A cache miss
-      # would resolve against newer databases, and gen-evidence still names any
-      # package it cannot account for, so the check is unchanged either way.
-      # HISTORY/evidence-race.md.
+      # this exports files rather than installing anything again.
+      #
+      # ⛔ The build args are part of that stage's cache key, so this step is
+      # handed the same three the step above was. BUILD_DATE in particular
+      # changes every run, and a step that omits it computes a key the build
+      # step never wrote, reuses a previous run's bootstrap layer, and exports
+      # the databases that layer was installed from. The image then holds
+      # today's packages and the evidence is joined against an older set, which
+      # gen-evidence reports as installed packages in no database. Handed the
+      # same args, the key matches and the layer is the one this run built.
+      #
+      # A cache miss would resolve against newer databases, and gen-evidence
+      # still names any package it cannot account for, so the check is
+      # unchanged either way. HISTORY/evidence-race.md.
       - name: Export the databases the build resolved against
         uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
         with:
@@ -328,6 +338,8 @@ jobs:
           target: dbsnapshot
           build-args: |
             IMAGE_VERSION=${{ needs.resolve.outputs.version }}
+            SOURCE_COMMIT=${{ github.sha }}
+            BUILD_DATE=${{ needs.resolve.outputs.build_date }}
           outputs: type=local,dest=/tmp/dbsnapshot
           cache-from: type=gha,scope=${{ matrix.docker_arch }}
 
@@ -585,8 +597,11 @@ jobs:
           GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         run: |
           set -uo pipefail
-          scripts/check-schedules-fired
-          rc=$?
+          # An exit of 3 is a normal result, not a failure. Under the runner's
+          # -e it would end this block before the capture below, so the command
+          # goes in an || list and the status is read from there.
+          rc=0
+          scripts/check-schedules-fired || rc=$?
           case "$rc" in
             0) echo "::notice::every schedule has fired within its own tolerance" ;;
             3)

.github/workflows/freshness-image-pins.yml

@@ -38,9 +38,12 @@ jobs:
         id: check
         run: |
           set -uo pipefail
-          # 0 every pin current, 3 at least one behind, 1 the check could not run.
-          scripts/check-image-pins | tee /tmp/pins.txt
-          rc="${PIPESTATUS[0]}"
+          # 0 every pin current, 3 at least one behind, 1 the check could not
+          # run. An exit of 3 is a result, so the pipeline is put in an ||
+          # list: under the runner's -e and pipefail a bare 3 would end the
+          # block before PIPESTATUS could be read.
+          rc=0
+          scripts/check-image-pins | tee /tmp/pins.txt || rc=$?
           echo "rc=$rc" >> "$GITHUB_OUTPUT"
           case "$rc" in
             0) echo "::notice::every pinned image is current" ;;
@@ -56,8 +59,9 @@ jobs:
           branch="freshness/image-pins-$(date -u +%Y%m%d)"
           git switch -c "$branch"
           echo "branch=$branch" >> "$GITHUB_OUTPUT"
-          scripts/check-image-pins --apply | tee /tmp/applied.txt
-          rc="${PIPESTATUS[0]}"
+          # The apply exits 3 on a bump as well, so it is guarded the same way.
+          rc=0
+          scripts/check-image-pins --apply | tee /tmp/applied.txt || rc=$?
           if [ "$rc" != "3" ]; then
             echo "applying the bumps did not report a bump, refusing to open a pull request" >&2
             exit 1

.github/workflows/freshness-keyring.yml

@@ -68,9 +68,12 @@ jobs:
         run: |
           set -uo pipefail
           # 0 current, 3 behind, 4 gone, 5 the pinned bytes changed, 1 the check
-          # could not run.
-          scripts/check-keyring-pin "$PIN" | tee /tmp/check.txt
-          rc="${PIPESTATUS[0]}"
+          # could not run. Every one of those is a result, so the pipeline is
+          # put in an || list: under the runner's -e and pipefail a bare 3, 4
+          # or 5 would end the block before PIPESTATUS could be read, and the
+          # annotation below would never print.
+          rc=0
+          scripts/check-keyring-pin "$PIN" | tee /tmp/check.txt || rc=$?
           echo "rc=$rc" >> "$GITHUB_OUTPUT"
           case "$rc" in
             0) echo "::notice::${PIN} is current" ;;
@@ -108,8 +111,10 @@ jobs:
           set -uo pipefail
           branch="freshness/${KEYRING}-$(date -u +%Y%m%d)"
           git switch -c "$branch"
-          scripts/check-keyring-pin --apply "$PIN" | tee /tmp/apply.txt
-          rc="${PIPESTATUS[0]}"
+          # The apply is a result too, and it exits 3, 4 or 5 rather than 0.
+          # Same || list as the check step, for the same reason.
+          rc=0
+          scripts/check-keyring-pin --apply "$PIN" | tee /tmp/apply.txt || rc=$?
           case "$rc" in
             3 | 4 | 5) ;;
             *)
@@ -143,24 +148,30 @@ jobs:
           # installs but does not verify produces an empty root rather than a
           # build error.
           ver="$(date -u +%Y.%m.%d)"
+          # Read once. The dbsnapshot build below has to be handed the same
+          # value or its cache key is not the one the image build wrote, and it
+          # exports a previous run's databases instead of this build's.
+          build_date="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
           docker build --platform "$PLATFORM" \
             --build-arg "IMAGE_VERSION=$ver" \
             --build-arg "SOURCE_COMMIT=${{ github.sha }}" \
-            --build-arg "BUILD_DATE=$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
+            --build-arg "BUILD_DATE=$build_date" \
             -t "freshness-check:$DOCKER_ARCH" .
 
           # The databases the build resolved against, so the evidence is not
           # joined against a set that moved. HISTORY/evidence-race.md.
           docker build --platform "$PLATFORM" --target dbsnapshot \
             --build-arg "IMAGE_VERSION=$ver" \
+            --build-arg "SOURCE_COMMIT=${{ github.sha }}" \
+            --build-arg "BUILD_DATE=$build_date" \
             --output "type=local,dest=/tmp/dbsnapshot" .
 
           echo "::group::image suite"
           # ⛔ CONTAINER_RUNTIME is not optional. The runner image carries both
           # docker and podman, and gen-evidence prefers podman when it finds
           # one. The image was built with docker and lives in docker's store, so
           # podman tries to pull it from Docker Hub and quay.io and fails.
-          SOURCE_COMMIT="${{ github.sha }}" BUILD_DATE="$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
+          SOURCE_COMMIT="${{ github.sha }}" BUILD_DATE="$build_date" \
             DB_SNAPSHOT=/tmp/dbsnapshot CONTAINER_RUNTIME=docker \
             scripts/gen-evidence "$DOCKER_ARCH" "freshness-check:$DOCKER_ARCH" "$PLATFORM" \
               "/tmp/evidence-$DOCKER_ARCH.json"

.github/workflows/freshness-publish.yml

@@ -65,8 +65,13 @@ jobs:
         id: recency
         run: |
           set -uo pipefail
-          scripts/check-publish-recency
-          rc=$?
+          # A non-zero exit is the answer this check is asked for, so the
+          # command is put in an || list. Under the runner's -e a bare command
+          # that exits 3 would end the block before the status could be read,
+          # the annotation below would never print, and the step outcome would
+          # be red for the wrong reason.
+          rc=0
+          scripts/check-publish-recency || rc=$?
           echo "rc=$rc" >> "$GITHUB_OUTPUT"
           case "$rc" in
             0) echo "::notice::the publish is current" ;;
@@ -80,8 +85,9 @@ jobs:
           GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         run: |
           set -uo pipefail
-          scripts/check-schedules-fired
-          rc=$?
+          # Same shape as the recency check above, and for the same reason.
+          rc=0
+          scripts/check-schedules-fired || rc=$?
           echo "rc=$rc" >> "$GITHUB_OUTPUT"
           case "$rc" in
             0) echo "::notice::every schedule has fired within its own tolerance" ;;

Dockerfile

@@ -11,7 +11,7 @@
 # against what that tag resolves to now. A Dockerfile instruction takes no
 # trailing comment, so the marker sits on its own line directly above.
 # tag: latest
-FROM --platform=$BUILDPLATFORM docker.io/library/archlinux@sha256:b860afd5823683f7ea389ba5f00d812f4fe55f6f286dea329d2abeefa535e309 AS bootstrap
+FROM --platform=$BUILDPLATFORM docker.io/library/archlinux@sha256:3c64b538588416593f69a1aebd2f2ec0f2541b29159f32556fa7def2167147a0 AS bootstrap
 
 ARG TARGETARCH
 ARG TARGETVARIANT

HISTORY/CONTINUE.md

@@ -693,6 +693,9 @@ manual cross reference remap.
 | 50 | The dangling `.tmp/` citations, and a test that stops them coming back | `tests/static/97-scratch-citations.sh` |
 | 51 | This file moved into the repository, so it is no longer one wiped directory away from being lost | `HISTORY/CONTINUE.md` |
 | 52 | The first release, `v0.1.0`: 50 assets, eight architectures, six families, verified by downloading three of them and checking them against the published `SHA256SUMS`. It took three runs and the two that failed published nothing | run `33213236059`, `HISTORY/releases.md` |
+| 53 | The freshness checks could never act on a result. A run block executes under `-e`, so a check exiting 3, 4 or 5 ended the step before the status was read, the annotation was never printed and the remediation never fired. Seven sites now capture with `cmd \|\| rc=$?`, and a third shape in the pipeline-trap test reports the old one | `HISTORY/freshness-status-capture.md`, `tests/static/25-pipeline-traps.sh` |
+| 54 | The dbsnapshot export was never reading this run's bootstrap layer. A stage's build args are its cache key and `BUILD_DATE` changes every run, so the export reused an older day's databases and riscv64 failed `Record the evidence` from 2026-10-05. Both steps are now handed the same args, and the test fails if the two sets drift | `HISTORY/evidence-race.md`, `tests/static/68-evidence-snapshot.sh` |
+| 55 | `gen-mirrorlist all` no longer regenerates an architecture the build matrix has disabled, which is what the monthly mirror freshness run failed on for the three PowerPC ports: ArchPOWER's anchor refuses a GitHub runner with 403 | `tests/static/75-architecture-set.sh` |
 
 ---
 

HISTORY/evidence-race.md

@@ -181,6 +181,11 @@ with `target: dbsnapshot` and `outputs: type=local,dest=/tmp/dbsnapshot`. The
 bootstrap stage is already in the builder from the step above, so it exports
 files rather than installing anything again.
 
+⛔ **That last sentence was only true once the two steps were handed the same
+build args.** A stage's build args are part of its cache key, and `BUILD_DATE`
+changes every run, so the export was not reading the layer the build step had
+just written. The measurement and the fix are below.
+
 ## Measurements
 
 Built 2026-08-28 on `linux/amd64`, from the tree this document ships in.
@@ -265,15 +270,59 @@ Its log shows `CACHED` on every layer of the bootstrap stage and one
 15 seconds on the run that failed. Reading `extra.db` locally replaces fetching
 8.9 MB, and the export adds a second back. The point is correctness, not speed.
 
+## The arg mismatch, 2026-10-13
+
+⚠ **The export was not hitting this build's cache.** On the riscv64 job the
+bootstrap stage ran on every image build and reported `CACHED` on every export:
+
+```bash
+for r in 38255111643 38135773513 38047830800; do
+  jid="$(gh api repos/pkgforge-dev/docker-archlinux/actions/runs/"$r"/jobs \
+    --jq '.jobs[] | select(.name=="Build riscv64") | .id')"
+  gh run view --job "$jid" --log | grep -A1 '\[bootstrap  9/11\]'
+done
+```
+
+| step, every day | line |
+| --- | --- |
+| image build | `#18 [bootstrap 9/11] RUN <<EOS ... ==> Setting up target directory...` |
+| export | `#11 [bootstrap 9/11] CACHED` |
+
+⛔ **The cause is the build args.** The image build was handed `IMAGE_VERSION`,
+`SOURCE_COMMIT` and `BUILD_DATE`; the export was handed `IMAGE_VERSION` alone.
+A different arg set is a different cache key, so the export found an entry from
+an earlier day and exported those databases. The image held today's packages,
+the join had a hole, and the run failed on it:
+
+```
+gen-evidence: these installed packages are in no repository database:
+  libsysprof-capture 50.0-6
+  lmdb 0.9.35-1
+  nftables 1:1.1.7-3
+gen-evidence: /tmp/dbsnapshot is not the snapshot this image was installed from
+```
+
+⭐ **The fix is to hand both steps the same args.** `Build and push by digest`
+and `Export the databases the build resolved against` now pass the same three.
+`freshness-keyring.yml` builds an image and an export in one step and had the
+same mismatch; it now reads `BUILD_DATE` once so the two builds cannot straddle
+a second either. `tests/static/68-evidence-snapshot.sh` assertion 5 fails if the
+two arg sets drift apart, which is the one input the tree can check.
+
+⚠ **A matching key is necessary, not sufficient.** The export still reports
+`CACHED`; what changed is that it is now this run's entry. When the build step
+misses, the export misses with it, and gen-evidence still names anything it
+cannot account for.
+
 ## What this does not fix
 
-⚠ **A cache miss on the export.** The exported databases are the ones the image
-was installed from only when the export hits the layer cache the build filled.
-Nothing in the tree can check that. What happens on a miss is the old behaviour:
-gen-evidence still names every package it cannot account for, and the run fails
-rather than passing quietly. The diagnostic distinguishes the two causes, because
-a snapshot that has a hole did not come from this build, and telling the reader
-that upstream moved would send them to the wrong place.
+⚠ **A cache miss with matching args.** The exported databases are the ones the
+image was installed from because both steps are handed the same key. If that
+key misses, both miss and both resolve at the same moment against the same
+mirrors, which is the pre-snapshot behaviour with a much smaller window. What
+is not measurable from the tree is whether the build step and the export step
+landed on the same second; `tests/static/68-evidence-snapshot.sh` checks the
+inputs, not the outcome.
 
 ⚠ **The four architectures still resolve at four different times.** Each build
 job runs its own `pacman -Sy`, so one run is four package sets rather than one.
@@ -290,9 +339,10 @@ shorter is not recorded anywhere.
 | --- | --- |
 | the Dockerfile has a `dbsnapshot` stage, taking content from the bootstrap stage | `tests/static/68-evidence-snapshot.sh` 1, 2 |
 | the databases are copied out before the directory is emptied | `68` 3 |
-| the build job exports the target, and reads it back from the same path | `68` 4, 5 |
-| a `DB_SNAPSHOT` that cannot be used is refused, and named | `68` 6, 7 |
-| a usable one is accepted | `68` 8 |
+| the build job exports the target, and reads it back from the same path | `68` 4, 6 |
+| the export step is handed the same build args as the build step | `68` 5 |
+| a `DB_SNAPSHOT` that cannot be used is refused, and named | `68` 7, 8 |
+| a usable one is accepted | `68` 9 |
 | the evidence is refused when a package cannot be accounted for | `scripts/gen-evidence`, and `tests/image/40-evidence.sh` on the file it writes |
 
 Every one of those was seen to fail. `HISTORY/tests-seen-to-fail.md`.

HISTORY/freshness-status-capture.md

@@ -0,0 +1,83 @@
+# The freshness checks could not act on a result
+
+⭐ **Measured 2026-10-12.** Every freshness workflow runs a check that exits
+non-zero on a normal result. The step read that status on the line after the
+command, and the command's exit ended the step before the line ran.
+
+## What the 2026-10-12 run shows
+
+`Freshness, port keyring pins`, run `38245829268`, the `archlinux-lcpu` job:
+
+```bash
+gh run view --job 114799312447 --log \
+  | grep -E 'result: GONE|Process completed'
+```
+
+```
+result: GONE, the pinned package no longer answers, this port is broken now
+##[error]Process completed with exit code 4.
+```
+
+⭐ The check ran and answered correctly. The step had two lines after it:
+
+```bash
+scripts/check-keyring-pin "$PIN" | tee /tmp/check.txt
+rc="${PIPESTATUS[0]}"
+```
+
+Neither ran. The `::error` annotation was never printed, and
+`steps.check.outputs.rc` was empty, so every
+`if: steps.check.outputs.rc != '0'` recovery step was skipped.
+
+## The mechanism
+
+GitHub Actions runs every `shell: bash` block as
+`bash --noprofile --norc -e -o pipefail`. A block may add `set -u` (and these
+do, `set -uo pipefail`) but `-e` stays on. A check that exits 3, 4 or 5 and the
+capture of that status therefore cannot sit in that order: the shell leaves at
+the check.
+
+⛔ **The interface is not the defect.** Exiting 3, 4 or 5 is what
+`tests/static/95-publish-watchdog.sh` and the freshness workflows were built
+around. The step was reading that interface with the one construct `-e`
+forbids.
+
+## What changed
+
+The command goes in a `||` list, where a non-zero status is the result the list
+was written for:
+
+| file | before | after |
+| --- | --- | --- |
+| `freshness-keyring.yml` | `cmd \| tee ...; rc="${PIPESTATUS[0]}"` | `rc=0; cmd \| tee ... \|\| rc=$?` |
+| `freshness-keyring.yml`, apply | same shape | same fix |
+| `freshness-image-pins.yml` | same shape | same fix |
+| `freshness-image-pins.yml`, apply | same shape | same fix |
+| `freshness-publish.yml`, recency | `cmd; rc=$?` | `rc=0; cmd \|\| rc=$?` |
+| `freshness-publish.yml`, schedules | `cmd; rc=$?` | same fix |
+| `build-deploy.yml`, watchdog | `cmd; rc=$?` | same fix |
+
+⭐ `cmd || rc=$?` is already the shape the tests use, and it is not one of the
+shapes `tests/static/20-no-swallowed-errors.sh` forbids. `set +e` and `|| true`
+would have let the check run and then discarded its answer, which is the defect
+that policy exists for.
+
+## What each red run was trying to say
+
+| workflow | the result that was lost |
+| --- | --- |
+| `freshness-keyring.yml` | the loong64 pin had been gone from its mirror since 2026-09-06, and the job no longer applied the bump |
+| `freshness-image-pins.yml` | the base image digest had moved, and the job no longer applied it |
+| `freshness-publish.yml` | the publish had stopped, but the run said `the recency check could not run` instead of `the newest dated index tag is older than the schedule allows` |
+| `build-deploy.yml` | the watchdog produced a bare red mark and no annotation |
+
+## What is checked by a test
+
+`tests/static/25-pipeline-traps.sh` gained a third shape. It reports a line that
+reads `rc=$?` or `PIPESTATUS` unless the command is guarded by `||`. The one
+file it does not read is itself, because the regular expression is its own
+source. `HISTORY/tests-seen-to-fail.md`.
+
+⚠ **What it cannot see** is a step that reads the status through some other
+variable name. `rc` is the name every step here uses; a step that invents
+another would pass the scan.