Skip to content

fix(routing): pipefail + grep -q false negatives (CI flake) - #328

Closed
MendixMau wants to merge 2 commits into
masterfrom
fix/render-routing-pipefail-flake
Closed

MendixMau wants to merge 2 commits into
masterfrom
fix/render-routing-pipefail-flake

Conversation

@MendixMau

Copy link
Copy Markdown
Owner

Problem

A reviewer saw bin/render-routing.sh --check report UNROUTED skills/iterative-build-loop.md once, even though that skill has a routing row. Re-running the check passed.

Cause (confirmed). bin/lib/skill-routing.sh routing_unrouted_skills runs under the caller's set -euo pipefail and does printf '%s\n' "$routed" | grep -qxF "$rel". grep -q exits on its first match. The routed-path list is about 4.8 KB, which is more than one stdio buffer, so printf writes it in two chunks. If grep has already exited when the second chunk is written, printf gets SIGPIPE (141). With pipefail, the pipeline then fails, so a skill that was found is reported as not found. Whether this happens depends on process timing.

Evidence

  • Minimal repro under set -o pipefail: a large input with the match first. The pipe returns 141 and the here-string returns 0.
  • Real table, master code: 15 of 100 calls of routing_unrouted_skills named a routed skill. Each run named a different skill (tool-output-is-not-ground-truth.md, change-intake.md, mdl-cookbook-microflows.md, …). A loop of the full render-routing.sh --check also failed 3 times before I stopped it.
  • Real table, this branch: 0 of 100.

Fix

Every producer | grep -q that runs under pipefail is now a here-string (grep -q -- PAT <<<"$x"). This covers scripts that set pipefail themselves and libraries sourced by a pipefail script. Owned files are excluded; see below. Bash writes a here-string's full input before grep starts, so no producer can be killed mid-write. There are 28 sites in 12 scripts:

  • bin/lib/skill-routing.sh: routing_unrouted_skills and routing_stale_exemptions. The second had the opposite failure: an exemption that was also routed could go unreported.
  • bin/lib/obligation-check.sh: _ob_has_denominator, which could produce a false "no denominator", and the reverse check (named / NOROUTE).
  • bin/lib/triage-template.sh: the pristine test, sourced by sync-project.sh.
  • bin/gate-check.sh: the stage-0 sign-off placeholder test and the stage-4 closing-rows checks ($rows, the per-module verify-module.sh test, the last row).
  • bin/check-portability.sh, project-bin/_common.sh (mxtk_version_in_name), check-design-portability.sh, check-page-shell.sh, close-task.sh, check-sp-health.sh, and the tasklist probe in app-restart.sh / test-stack-up.sh.

Some of these sites can't race in practice because they pipe one short line. I changed them anyway: the swap is 1:1 and keeps the fixture's static guard clean.

Left as is: the find_artifact … | grep -q . lines in gate-check.sh. find_artifact ends in -print -quit, so it writes one line and has finished before grep exits.

Semantics are the same. A here-string adds a trailing newline, which grep ignores. An empty value gives a single empty line, and none of the patterns can match an empty line.

Not done: owned files (WAVE2-RULES)

These files still contain the pattern and run under pipefail. The proposed patch is in the hand-back to the lead:

  • bin/sync-project.sh: lines 101, 345, 427, 1361
  • project-bin/review-module.sh: line 284
  • project-bin/conformance-check.sh: lines 206 and 208

project-bin/exec.sh line 1441 does not run under pipefail, so it is not affected.

Fixture

tests/wave2/test-pipefail-grep-q.sh <bin/lib/skill-routing.sh> makes the race deterministic. The routing table has the matching row first, then about 1 MB of padding, far more than the 64 KB pipe buffer.

  • T1: a routed skill is not reported as unrouted.
  • T2: an exemption that is also routed is still reported.
  • T3: a static guard that the 12 fixed scripts contain no | grep -q pipeline.

Result: fails 10 of 10 runs on master's code and passes on this branch.

Checks run

  • New fixture: 3/3 passing (master: 0/3).
  • Related existing fixtures, all passing: test-bug03-gates (31 ok), test-bug02-register (8), test-triage-report (66), test-obligation-fidelity (6), test-obligation-final-look (6), test-obligation-valid-at (3), test-design-reaches-app (43), test-page-shell-body (18), test-app-restart (44), test-stack-up-local (23), test-mxbuild-version-match (18), test-render-routing (5), test-routing-tier (12).
  • bin/render-routing.sh --check passes (rc 0). No routing or skill text changed, so the baseline word count is unchanged.
  • bin/check-scripts.sh: 109/109 shell and 21/21 Node files parse. bin/check-portability.sh is clean (214 files). check-no-private-citations is clean. The leak guard is clean. check-pr-discipline is clean.
  • Full suites were not run, per the repo rules.

Field run

No consuming-project field run yet; one is still owed. The evidence above comes from this repo's own routing table, which is the instrument's real input: 15/100 failing before the fix, 0/100 after. The tasklist (Windows) sites have not been run on Windows.

Risks

  • Low. Each change is a mechanical rewrite of the pipeline into a here-string.
  • Bash 3.2 supports here-strings.
  • check-portability still treats <<< inside $(…) as a non-heredoc and reports these lines clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw


Generated by Claude Code

`grep -q` exits on its first match; a producer still writing into the pipe
then dies of SIGPIPE (141), and under `set -o pipefail` the pipeline reports
failure, so "found" reads as "not found". render-routing.sh --check runs
routing_unrouted_skills under `set -euo pipefail` and pipes a ~4.8 KB
routed-path list (two stdio writes) into `grep -qxF`; the race made it
report routed skills as UNROUTED at random.

Measured on the real table: 15/100 calls of routing_unrouted_skills named
a routed skill on master, 0/100 after.

Replace every `producer | grep -q` that runs under pipefail (directly, or
sourced by a pipefail script) in the non-owned scripts with a here-string,
which bash writes in full before grep starts: 28 sites in 12 scripts.
gate-check's `find_artifact ... | grep -q .` lines are left: find_artifact
ends in `-print -quit`, one line, no race.

New fixture tests/wave2/test-pipefail-grep-q.sh puts the match first and
1 MB after it, making the race deterministic: fails 10/10 on master,
passes after.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw
MendixMau pushed a commit that referenced this pull request Oct 13, 2026
Review: MERGE — all 28 sites semantically equivalent (trailing newline, lost producer status, NUL bytes, -- checked per site); render-routing --check 0/30 flakes on the branch vs 15/100 false UNROUTED on master.
test-pipefail-grep-q.sh declares bin/lib/skill-routing.sh as its subject;
run-all.sh only searched bin/, project-bin/, skills/, hooks and the root,
so CI reported "declares subject 'skill-routing.sh', which is not in the
toolkit". bin/lib/ is searched right after bin/; no file name exists in
both bin/lib/ and any later candidate, so no other fixture's subject moves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw
MendixMau pushed a commit that referenced this pull request Oct 13, 2026
#328 was merged into integration at 530b217 while its CI was red on
exactly this (test-pipefail-grep-q.sh subject unresolved by run-all).
Fix 2827a6d: CI green on that head.

Review: independent review (agent aa0e9e6) of 2827a6d — MERGE; no other
fixture's subject resolution changes; no CHANGELOG line required.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw
MendixMau added a commit that referenced this pull request Oct 14, 2026
)

Every constituent PR was reviewed by a fresh agent that did not write it (CLAUDE.md "Merging a PR"); only MERGE verdicts joined. CI green on each constituent head and on this head (40814fc); full suites run on the branch.

Review lines (per merged PR):
- #315 context trim — MERGE; 360/385 removed lines found verbatim in their new homes, rest read by hand; baseline 79,997 -> 76,259.
- #316 independent-review + word-budget rules — MERGE-AFTER-FIX (3 rounds), fixes applied as specified.
- #318 page-fidelity displayed text — MERGE after fix round + confirmation.
- #317 interaction-model skill (#291 part 1) — MERGE after fix round + confirmation; no client names leaked.
- #321 PROOF-OF-LOOK per section; final-look not due before derived Stage 6 — MERGE after fix round + confirmation.
- #307 graph-sweep guard; per-BRD conformance ledgers — MERGE after two fix rounds + confirmation.
- #325 extraction-quality section; token-burn fixes — MERGE after fix round + confirmation.
- #310 e2e OQL by alias, negative-span liveness, APP-DOWN, #302 credential regression fix — MERGE after fix round + 3 confirmations.
- #323 inbox promotions — MERGE after fix round + confirmation.
- #308 lint gate BLIND on reverted stock rules — MERGE (FIRST-SEEN reverted) after two fix rounds + confirmation.
- #328 pipefail + grep -q flake — MERGE; 0/30 flakes vs 15/100 on master. Follow-up: run-all resolves bin/lib/ subjects (reviewed, MERGE).
- #319 test-stack-up command-line ownership proof — MERGE after fix round + 2 confirmations.
- #327 change-intake core + on-demand detail; CRLF-safe change log — MERGE after fix round + confirmation.
- #320 follow-ups to #290/#292; Dispatching split out — MERGE after fix round + confirmation.
- #326 status delivery rounds (existing-app mode) — MERGE after fix round + confirmation.
- #312 lint batch 4: SEC014, toolkit CONV009 — MERGE after review + 2 fix rounds + confirmation; field run on mxcli 0.26.
- #324 exec.sh change classes, shared stale-lock takeover, SEC014 note echo — MERGE (joint review); follow-up owed for a takeover race after a mid-takeover crash.
- #322 MxTest after-startup guard, mxcli-run pid check — MERGE (joint review).
- #314 doctor property-set exit code; sync announces its bound — MERGE (joint review).
- #330 review follow-ups (exit 4/5 docs, pipefail, NUL strip) — MERGE-AFTER-FIX, fixes applied; takeover re-check dropped.

Field runs are still owed for most instruments in this batch (stated in each PR).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw
@MendixMau

Copy link
Copy Markdown
Owner Author

Landed on master as part of the integration squash #329 (commit 31e90ae), after independent review — its Review: line is in that commit message. Closing as superseded.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants