From d18f65f05899f4ffd6471bf99f0ee8927287f168 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 14 Oct 2026 14:30:19 +0000 Subject: [PATCH 1/3] fix(baselines): a loosening ratchet baseline moves only on the user's own command (#348) lint-gate.sh --update-baseline took --approved-in-chat as the user's yes, and the agent could type that flag itself. A rise (lint) or an accepted regression (conformance-check) is now written only against a row in .claude/baseline-approvals.tsv: the sha256 of exactly the proposed content, written by the new bin/baseline-approve.sh, which the user runs in a terminal. Claude Code denies it and the file to the agent; Copilot must ask; it refuses without a TTY. Every instrument write is recorded in .claude/baseline-writes.tsv, and gate-check prints "Baselines (human-approved?)": FAIL for a baseline matching neither record (blocks Stages 5 and 6), MANUAL for one predating the lock. Bench (tests/wave2/bench-baseline-lock.sh, master vs this branch): unapproved moves landing 10/10 -> 2/10 per baseline (the 2 are direct file edits, all flagged); legitimate re-baselines 4/4 -> 4/4 at one human command each. Field evidence: none yet. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw --- CHANGELOG.md | 1 + agents/gate-agent.md | 6 +- bin/gate-check.sh | 40 ++++ bin/install-claude-permissions.sh | 9 + bin/install-harness-permissions.sh | 6 +- bin/lib/install-manifest.sh | 5 +- bin/lib/sec014-notice.sh | 3 +- lint-rules/README.md | 4 +- lint-rules/sec014_external_role_write.star | 3 +- project-bin/baseline-approve.sh | 59 ++++++ project-bin/baseline-lock.sh | 123 +++++++++++ project-bin/conformance-check.sh | 56 ++++- project-bin/exec.sh | 6 +- project-bin/lint-gate.sh | 86 ++++++-- skills/lint-that-actually-runs.md | 29 ++- tests/wave2/bench-baseline-lock.sh | 199 ++++++++++++++++++ tests/wave2/fixtures/baseline-lock/human.sh | 20 ++ tests/wave2/test-baseline-lock.sh | 141 +++++++++++++ .../wave2/test-conformance-baseline-scope.sh | 8 +- .../wave2/test-install-claude-permissions.sh | 6 + tests/wave2/test-lint-gate-accept.sh | 28 ++- tests/wave2/test-sec014-notice.sh | 2 +- 22 files changed, 786 insertions(+), 54 deletions(-) create mode 100755 project-bin/baseline-approve.sh create mode 100755 project-bin/baseline-lock.sh create mode 100755 tests/wave2/bench-baseline-lock.sh create mode 100644 tests/wave2/fixtures/baseline-lock/human.sh create mode 100755 tests/wave2/test-baseline-lock.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index ff60ea83..827c9f38 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. Named m ## Unreleased +- fix(lint-gate, conformance-check, gate-check): **a ratchet baseline that loosens now moves only on the user's own command; `--approved-in-chat` is no longer evidence** ([#348](https://github.com/MendixMau/mxcli-project-toolkit/issues/348)). The agent whose script caused a rise could type that flag itself, so "re-baselining is the user's call" ran on trust. A rise (lint) or an accepted regression (conformance) is now written only against a row that the new `bin/baseline-approve.sh` writes to `.claude/baseline-approvals.tsv`. The user runs that command in a terminal, and the row is the sha256 of exactly the content they reviewed. Claude Code denies the command and the file to the agent (`install-claude-permissions.sh`), Copilot must ask before running it, and the command refuses to run without a TTY. Every instrument write is recorded in `.claude/baseline-writes.tsv`. `gate-check.sh` prints `Baselines (human-approved?)`: FAIL for a baseline that matches neither record (this blocks Stages 5 and 6) and MANUAL for one that predates the lock. Bench (`tests/wave2/bench-baseline-lock.sh`, 10 unapproved and 4 legitimate scripted attempts per baseline): unapproved moves landing 10/10 → 2/10 (the 2 are direct file edits, now all flagged; unflagged 10/10 → 0/10), legitimate re-baselines 4/4 → 4/4, at the cost of one human command each. Field evidence: none yet. Not covered: the mxbuild verification stamp, which records a machine check and has no human judgement to lock, and page-fidelity and token-burn, which keep no baseline. — external reviewer feedback - process(build): **Stage 5 drafts the ready set in parallel and applies serially.** The runbook dispatch row said "sequential within a module", which serialised drafting even for independent scripts; it now points to `iterative-build-loop.md` → Gate cost, the single statement of the rule. The unit is the build-plan row: every `BUILD` row whose "Depends on" rows are all applied, minus overlapping writers (two rows that replace, alter or drop+create the same document never share a ready set, and no row is drafted while it overlaps a queued unapplied one; `brd-to-build-plan.md` has Produces name every document a row writes), runs at once, its scripts in order, one `mdl-agent` per script. Applies stay one at a time through `bin/exec.sh`, and the ready set is recomputed after each. A queued draft made before a sibling apply landed is re-checked with `mxcli diff --format struct` (preflight STOP rule 23); any removal line (checked against `describe` per BUG-146) or STOP rule 21 target means re-draft, never patch, because a stale restatement of a shared document is exactly rule 21's silent seed-call loss and passes `check --references`. Report `ready set N, drafted N, applied k`; the `mdl-agent` template says a drafter never applies and never depends on an unapplied draft. Field evidence: none yet. — MendixMau - fix(platform-link): **CLI commits now show their changed documents in Studio Pro History.** Studio Pro's History → Model changes pane reads a `ModelChanges` array from the commit's `refs/notes/mx_metadata` note. Trap 6c's hand-written template set it to `[]`, so every mxcli/agent commit showed "Model changes 0" (field project: 270 of 270 notes empty). New `project-bin/mx-metadata-note.py` (stdlib only) computes the array from the commit's changed `.mxunit` files, in Studio Pro's entry shape: Status, UnitID, UnitType, UnitName and Module (walked up the `.mpr` Unit tree). It has `changes` / `write` / `backfill` / `sync ` subcommands. `install-project-hooks.sh` now also installs post-commit and post-rewrite hooks that write the note and never block, and `--check` covers them, so `sync-project.sh` rolls them out to wired projects. Notes are published with `mx-metadata-note.py sync ` after the branch push. Push config is never touched. With mxcli v0.26+ in the project root the hooks call `mxcli git note --write` instead (mendixlabs/mxcli#1337); the Python script is the fallback for older pins, and on 160 field commits both wrote the same 3,594 entries. Trap 6c rewritten. Measured against Studio Pro's own notes, 51 commits in 5 apps: 3387/3389 entries exact on all five fields. Field run on a 357-commit Mendix 11.15.0 app: 158 notes backfilled and pushed, and Studio Pro History lists the documents per revision. Review fixes: an earlier revision added `HEAD` + the notes ref to `remote..push`. It was dropped because any push refspec replaces `push.default`. A branch tracking a differently named upstream got a new same-named branch on Team Server where `simple` refuses, a missing local notes ref failed the whole push, and nobody knows whether Studio Pro's own push reads that config. The installer now only fetches the remote's notes (non-forced). It removes that exact `HEAD` + notes pair if present, and leaves and reports any other entry. `sync` handles a remote with no notes ref (it used to exit 1) and retries a rejected push once after re-merging. A note that already lists changes (Studio Pro's) is never overwritten by `write`/`backfill` without `--force`, and `sync` keeps a colleague's non-empty note over ours. The mxcli probe greps `git note --help` for `--write`. Hooks and docs call the script through `resolve_py`. Fixture `tests/wave2/test-mx-metadata-note.sh` (40 cases, over real captured template units; single- and two-tree layouts, amend, backfill and `--force`, push exit codes identical to a no-hooks clone, leftover-refspec cleanup, sync to an empty remote and its retry, notes fetch at install, a diverged notes merge; green under `init.defaultBranch` master and main). — field project (Mendix 11.15 app), MendixMau - learn(fixture-seeding): **explore the data structure first, choose rows by use case, scan actions for resets before any run.** Two seeding runs on public template apps: seeding raised journey passes 33 -> 58 and 20 -> 51 of the same sets (FAIL 93 -> 68, 163 -> 132; INVALID unchanged), and a "Reset all demodata" journey wiped accounts so 104 of 127 journeys reported `The username or password you entered is incorrect`, identical on empty and seeded databases. Skill gains three questions a seed must answer, a use-case checklist, a destructive-action scan, a generic traps table with symptoms, measured expectations and a denominator checklist; routing trigger now fires on seed/test/demo data and empty pages. Independent review fixes: object ids follow the Mendix bigint pattern with a business-key guard (no uuids), associations are seeded where the page's path stores them, four new traps (enum names, inheritance chain, calculated attributes, view/external entities), and the port-offset and container-label traps moved to `harness-architecture.md`. — MendixMau diff --git a/agents/gate-agent.md b/agents/gate-agent.md index b7da6484..e5ecfedd 100644 --- a/agents/gate-agent.md +++ b/agents/gate-agent.md @@ -101,9 +101,9 @@ This table is your whole list. The baseline table in the project's CLAUDE.local. get the per-document list behind a rise. Name each rule that rose, the documents behind it, and whether the rise belongs to the script under review or to earlier debt. A rise is a FAIL for the script even when mxbuild is clean. Never re-baseline to turn a row green — that is - the user's decision, made in chat; `--update-baseline` refuses a rise without `--reason`, a - security-tier rise (SEC*, CONV006) also without `--approved-in-chat`, and logs each one to - `.claude/lint-acceptances.tsv`. A row reading `LINT BLIND` is an instrument fault (a rule + the user's decision; `--update-baseline` refuses a rise without `--reason`, and then + without the user running `bin/baseline-approve.sh lint` in their own terminal (never you), and + logs each one to `.claude/lint-acceptances.tsv`. A row reading `LINT BLIND` is an instrument fault (a rule inspected nothing), not a rise: name the rule and its configuration fix. `LINT BLIND: rules reverted to stock (…)` means `mxcli init` re-seeded the toolkit rules and lint did not run: the fix is the toolkit's `bin/sync-project.sh`, never a re-baseline. If the row says diff --git a/bin/gate-check.sh b/bin/gate-check.sh index 61f648b3..beaabc27 100755 --- a/bin/gate-check.sh +++ b/bin/gate-check.sh @@ -2544,6 +2544,37 @@ else printf 'Artifacts: FAULT — bin/lib/artifact-check.sh missing from %s; no artifact was checked\n' "$TOOLKIT_DIR" fi +# --------------------------------------------------------------------------- +# Baselines — did a ratchet baseline move without a person's approval? (#348) +# +# lint-gate.sh and conformance-check.sh record every baseline they write, and a loosening write +# only lands against a row bin/baseline-approve.sh wrote (a person, in a terminal). A baseline +# whose content matches neither record moved some other way — a hand edit, or a write by a +# script that skipped the lock: FAIL. A baseline with no record at all predates the lock: +# MANUAL, since nothing can say who moved it. The toolkit's own baseline-lock.sh reads the +# project, whichever version the project installed. BLOCKS Stages 5 and 6 on FAIL (below); the +# remedy, a person running `bin/baseline-approve.sh --current`, is never blocked. +BASE_STATUS="PASS"; BASE_NOTE="" +BASE_LOCK="$TOOLKIT_DIR/project-bin/baseline-lock.sh" +if [ -f "$BASE_LOCK" ]; then + for _bk in lint conformance; do + _bv="$(bash "$BASE_LOCK" verify "$PROJECT_DIR" "$_bk" 2>/dev/null)" + _bs="$(printf '%s' "$_bv" | cut -f1)" + case "$_bs" in + NONE) continue ;; + FAIL|GONE) BASE_STATUS="FAIL" ;; + MANUAL) [ "$BASE_STATUS" = PASS ] && BASE_STATUS="MANUAL" ;; + PASS) ;; + *) [ "$BASE_STATUS" = PASS ] && BASE_STATUS="MANUAL"; _bv="$_bk could not be verified" ;; + esac + BASE_NOTE="$BASE_NOTE${BASE_NOTE:+; }$_bk: $(printf '%s' "$_bv" | cut -f2-)" + done + [ -n "$BASE_NOTE" ] || { BASE_STATUS="PASS"; BASE_NOTE="no ratchet baseline in this project yet"; } +else + BASE_STATUS="MANUAL"; BASE_NOTE="project-bin/baseline-lock.sh missing from $TOOLKIT_DIR — cannot evaluate, which is not a pass" +fi +printf "Baselines (human-approved?): %s — %s\n" "$BASE_STATUS" "$BASE_NOTE" + # --------------------------------------------------------------------------- # Source sufficiency — was the source actually characterised, or did Stage 0 # pass on nobody having looked? @@ -3265,6 +3296,15 @@ if [ -n "$REQUESTED_STAGE" ]; then # skills/close-the-loop.md was written to retire, and a warning shown once per key # (advise()'s own semantics) is too easy to have already been shown and forgotten by the time # a report goes un-filed again. + # No Stage-5/6 gate passes on a ratchet baseline that moved without a person's approval: + # a rise baselined away is a FAIL nobody was shown. See the Baselines block above. + if { [ "$REQUESTED_STAGE" = "5" ] || [ "$REQUESTED_STAGE" = "6" ]; } && [ "$BASE_STATUS" = "FAIL" ]; then + echo "" >&2 + echo "Gate BLOCKED by a baseline that moved without approval: $BASE_NOTE" >&2 + echo "Show the user the change (git log -p on the baseline file). If they accept it, THEY run" >&2 + echo "bin/baseline-approve.sh --current in their own terminal; otherwise restore it from git." >&2 + exit 1 + fi if { [ "$REQUESTED_STAGE" = "5" ] || [ "$REQUESTED_STAGE" = "6" ]; } && [ "$DISP_STATUS" = "FAIL" ]; then echo "" >&2 echo "Gate BLOCKED by report-disposition-check: $DISP_NOTE" >&2 diff --git a/bin/install-claude-permissions.sh b/bin/install-claude-permissions.sh index e2e543a7..62c81d74 100755 --- a/bin/install-claude-permissions.sh +++ b/bin/install-claude-permissions.sh @@ -43,6 +43,10 @@ # runs the same mxcli underneath. A skill that genuinely needs the bare form runs it from a # script (the rule matches the top-level command only). # +# THE SECOND DENY (2026-10-14, #348): `*baseline-approv*` in Bash and edits of +# baseline-approvals.tsv. Approving a ratchet baseline move is the user's act; the agent could +# type `--approved-in-chat` itself. See project-bin/baseline-lock.sh. +# # THE ONE HOOK: `bash bin/session-check.sh || true` on SessionStart — the project-local check # that says at the top of every session whether the model on disk is verified, whether the # mxbuild gate can run here, and whether the installed guard scripts are stale. It never blocks. @@ -184,6 +188,11 @@ DENY=( "Bash(./mxcli exec:*)" "Bash(mxcli exec:*)" "Bash(./mxcli.exe exec:*)" + # #348: approving a ratchet baseline move is the user's act. Denies the approve command and + # any Bash line naming the approvals file, plus file-tool edits of it. The user runs it in + # their own terminal; see project-bin/baseline-lock.sh for what this does not stop. + "Bash(*baseline-approv*)" + "Edit(**/baseline-approvals.tsv)" ) ASK=() diff --git a/bin/install-harness-permissions.sh b/bin/install-harness-permissions.sh index dbb0d000..01f6b3a7 100755 --- a/bin/install-harness-permissions.sh +++ b/bin/install-harness-permissions.sh @@ -106,14 +106,16 @@ echo "=== copilot ===" require_py VSCODE_SETTINGS="$PROJECT_DIR/.vscode/settings.json" COPILOT_SIDECAR="$PROJECT_DIR/.vscode/.mxtk-autoapprove-added.json" -# Only relative, committable entries — see the header note above for why the two absolute +# `/baseline-approv/: false` (#348): approving a baseline move is the user's act — Copilot must +# ask, never auto-run it. Only relative, committable entries — see the header note above for why the two absolute # $TOOLKIT_ROOT entries Claude gets are deliberately not mirrored here. COPILOT_ENTRIES_JSON='{ "./mxcli": true, "mxcli": true, "./bin/exec.sh": true, "bin/exec.sh": true, - "/^\\.?\\/?bin\\/[^ ]+\\.sh$/": true + "/^\\.?\\/?bin\\/[^ ]+\\.sh$/": true, + "/baseline-approv/": false }' "$PY" - "$VSCODE_SETTINGS" "$COPILOT_SIDECAR" "$MODE" "$COPILOT_ENTRIES_JSON" <<'PY' || { ec=$?; [ "$ec" -gt "$RC" ] && RC=$ec; } import json, os, shutil, sys diff --git a/bin/lib/install-manifest.sh b/bin/lib/install-manifest.sh index f33a52b0..8c113a06 100644 --- a/bin/lib/install-manifest.sh +++ b/bin/lib/install-manifest.sh @@ -114,7 +114,10 @@ MXTK_AGENTS_STAGE_BUILD="mdl-agent.md gate-agent.md test-agent.md review-agent.m # may run itself (stops only the pid run --local recorded for THIS project), and `mxcli test` on a # scratch copy under the test-port lock. exec.sh names app-restart.sh in its watch advisory, so # it has to be installed beside it. -MXTK_PROJECT_BIN="app-restart.sh test-scratch.sh assemble-prototype.js prototype-route.js check-prototype-links.js wf-add-path-terminators.py wf-set-call-captions.py mxunit_bson.py _common.sh app-facts.sh _claims.sh snapshot-mpr.sh restore-mpr.sh exec.sh look-ledger.sh save-sp.sh restart-sp.sh check-sp-health.sh verify-module.sh test-stack-up.sh fixture-manifest.sh check-root-clean.sh lint-gate.sh lint-trend.sh metrics-ledger.sh metrics-trend.sh close-task.sh conformance-check.sh coverage-preflight.sh graph-sweep.sh review-module.sh coherence-cadence.sh build-plan-status.sh done-drift-check.sh page-scope.sh render-improvement-register.sh check-design-portability.sh check-design-reaches-app.sh check-page-shell.sh page-fidelity.js model-stamp.sh verify-model.sh install-project-hooks.sh mx-metadata-note.py session-check.sh constants-audit.sh context-pack.sh" +# baseline-lock.sh and baseline-approve.sh joined on 2026-10-14 (#348) as a PAIR: lint-gate.sh +# and conformance-check.sh resolve the lock as their sibling, and the approve command (run by the +# user, never the agent) is what their refusal names. +MXTK_PROJECT_BIN="baseline-lock.sh baseline-approve.sh app-restart.sh test-scratch.sh assemble-prototype.js prototype-route.js check-prototype-links.js wf-add-path-terminators.py wf-set-call-captions.py mxunit_bson.py _common.sh app-facts.sh _claims.sh snapshot-mpr.sh restore-mpr.sh exec.sh look-ledger.sh save-sp.sh restart-sp.sh check-sp-health.sh verify-module.sh test-stack-up.sh fixture-manifest.sh check-root-clean.sh lint-gate.sh lint-trend.sh metrics-ledger.sh metrics-trend.sh close-task.sh conformance-check.sh coverage-preflight.sh graph-sweep.sh review-module.sh coherence-cadence.sh build-plan-status.sh done-drift-check.sh page-scope.sh render-improvement-register.sh check-design-portability.sh check-design-reaches-app.sh check-page-shell.sh page-fidelity.js model-stamp.sh verify-model.sh install-project-hooks.sh mx-metadata-note.py session-check.sh constants-audit.sh context-pack.sh" # Files in project-bin/ that are deliberately NOT installed into projects. The reverse check # below flags anything named by NEITHER list, so a new file in project-bin/ has to be either diff --git a/bin/lib/sec014-notice.sh b/bin/lib/sec014-notice.sh index ee1fa38a..463efc9b 100644 --- a/bin/lib/sec014-notice.sh +++ b/bin/lib/sec014-notice.sh @@ -54,6 +54,7 @@ mxtk_sec014_notice() { echo " external_roles: \"RoleA, RoleB\" # one text, not a YAML list" echo "\"none\" (or SEC014 enabled: false) switches this security rule off: only on the user's answer in chat, recorded in" echo "PROJECT.md — never to silence this notice. The first configured run raises SEC014 0 -> N, so" - echo "re-baseline right after: bin/lint-gate.sh --update-baseline --reason \"SEC014 configured\" --approved-in-chat" + echo "re-baseline right after: bin/lint-gate.sh --update-baseline --reason \"SEC014 configured\", then the user" + echo "runs bin/baseline-approve.sh lint in their own terminal (an agent cannot approve it)" return 0 } diff --git a/lint-rules/README.md b/lint-rules/README.md index 24fd51c0..106bfa0c 100644 --- a/lint-rules/README.md +++ b/lint-rules/README.md @@ -198,8 +198,8 @@ asked in chat. `external_roles: "none"` switches off a security rule, and so doe `rules: SEC014: enabled: false`. Either one is set only on the user's answer and recorded as a decision row in `PROJECT.md`. An agent never sets one to make the NOTE go away. The first run after configuring raises SEC014 from 0 to N at once, and lint-gate would blame that security-tier rise on whatever script ran next. So re-baseline in the same step: -`bin/lint-gate.sh --update-baseline --reason "SEC014 configured" --approved-in-chat`, after the -user has seen the findings. +`bin/lint-gate.sh --update-baseline --reason "SEC014 configured"`, then the user, having seen the +findings, runs `bin/baseline-approve.sh lint` in their own terminal. **Tuning a toolkit rule never means editing it.** `conv009` and `sec014` read their settings with `get_option()` from `.claude/lint-config.yaml` (`rules: : options:`), so the file diff --git a/lint-rules/sec014_external_role_write.star b/lint-rules/sec014_external_role_write.star index 6df3c752..45cabf23 100644 --- a/lint-rules/sec014_external_role_write.star +++ b/lint-rules/sec014_external_role_write.star @@ -43,7 +43,8 @@ # finding here would make lint-gate exit PASS-BLIND on every exec of every synced project until # someone answers a question — the #229 pattern. The silence is not a pass: the sync names it. # After configuring, SEC014 jumps 0 -> N in one run; re-baseline right then -# (lint-gate.sh --update-baseline --reason "SEC014 configured" --approved-in-chat), or the +# (lint-gate.sh --update-baseline --reason "SEC014 configured", then the user runs +# baseline-approve.sh lint), or the # rise is blamed on whatever script runs next. # # SELF-CHECKS (configured but unusable: the rule must say so, never report a clean pass): diff --git a/project-bin/baseline-approve.sh b/project-bin/baseline-approve.sh new file mode 100755 index 00000000..5c40dc15 --- /dev/null +++ b/project-bin/baseline-approve.sh @@ -0,0 +1,59 @@ +#!/usr/bin/env bash +# baseline-approve.sh — a PERSON approves a ratchet baseline move. Run it in your own terminal. +# +# bin/baseline-approve.sh lint # approve the proposal lint-gate.sh just refused +# bin/baseline-approve.sh conformance # same, for conformance-check.sh +# bin/baseline-approve.sh --current # approve the baseline file as it is on disk +# # (what gate-check asks for after a hand edit) +# +# It shows the proposed baseline against the current one, asks you to type `approve`, and +# appends one row to .claude/baseline-approvals.tsv: the sha256 of exactly what you saw. The +# instrument then writes that content and nothing else — if the counts change before it re-runs, +# the hash no longer matches and it asks again. Commit the approvals file with the baseline. +# +# WHY A SEPARATE COMMAND (#348): the agent could type `--approved-in-chat` itself. This script +# is denied to the agent in Claude Code (bin/install-claude-permissions.sh), and it refuses +# without a terminal on stdin, which an agent's tool call does not have. See baseline-lock.sh +# for what that does and does not protect against. +set -uo pipefail + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +LOCK="$(dirname "${BASH_SOURCE[0]}")/baseline-lock.sh" +[ -f "$LOCK" ] || { echo "baseline-approve: $LOCK missing — run the toolkit's bin/sync-project.sh" >&2; exit 2; } + +KIND="${1:-}"; CURRENT=0 +[ "${2:-}" = "--current" ] && CURRENT=1 +case "$KIND" in lint|conformance) ;; *) sed -n '2,8p' "$0"; exit 2 ;; esac + +if [ ! -t 0 ]; then + echo "baseline-approve: REFUSED — no terminal. This is a person's decision: run it yourself," >&2 + echo "in your own terminal, not through an agent." >&2 + exit 2 +fi + +BASE="$ROOT/$(bash "$LOCK" file "$KIND")" +if [ "$CURRENT" -eq 1 ]; then + SRC="$BASE"; [ -f "$SRC" ] || { echo "baseline-approve: no $KIND baseline at $SRC" >&2; exit 2; } + echo "Approving the $KIND baseline AS IT IS ON DISK: $SRC" + head -60 "$SRC" +else + SRC="$ROOT/.claude/baseline-pending/$KIND" + [ -f "$SRC" ] || { echo "baseline-approve: nothing pending for $KIND — run the instrument's --update-baseline first" >&2; exit 2; } + echo "Proposed $KIND baseline:" + [ -f "$SRC.summary" ] && sed 's/^/ /' "$SRC.summary" + echo "" + if [ -f "$BASE" ]; then diff -u "$BASE" "$SRC" | head -80; else head -60 "$SRC"; fi +fi +SHA="$(bash "$LOCK" sha "$SRC")" +echo "" +printf "Type 'approve' to accept this %s baseline (anything else cancels): " "$KIND" +read -r ANSWER +[ "$ANSWER" = "approve" ] || { echo "Not approved. Nothing written."; exit 1; } + +BY="$(git -C "$ROOT" config user.name 2>/dev/null || true)"; BY="${BY:-${USER:-unknown}}" +NOTE="$( [ "$CURRENT" -eq 1 ] && echo "current file" || head -1 "$SRC.summary" 2>/dev/null)" +F="$ROOT/.claude/baseline-approvals.tsv" +[ -f "$F" ] || printf 'ts\tkind\tsha256\tby\tnote\n' > "$F" +printf '%s\t%s\t%s\t%s\t%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$KIND" "$SHA" "$BY" \ + "$(printf '%s' "$NOTE" | tr '\t\n' ' ')" >> "$F" +echo "Approved ($(printf '%s' "$SHA" | cut -c1-12)). The agent can now re-run its --update-baseline command." diff --git a/project-bin/baseline-lock.sh b/project-bin/baseline-lock.sh new file mode 100755 index 00000000..ddef6346 --- /dev/null +++ b/project-bin/baseline-lock.sh @@ -0,0 +1,123 @@ +#!/usr/bin/env bash +# baseline-lock.sh — the shared lock behind every ratchet baseline a person must approve. +# +# WHY (#348, 2026-10-14). `lint-gate.sh --update-baseline` took `--approved-in-chat` as the +# proof that the user said yes. The agent whose script caused the rise could type that flag +# itself, so "re-baselining is the user's call" was honour-system. A baseline that LOOSENS now +# moves only against a row in .claude/baseline-approvals.tsv whose sha256 is the exact +# proposed content — a row only bin/baseline-approve.sh writes, from a terminal, after a person +# reads the change and types `approve`. The agent cannot write that row through its normal +# tools: bin/install-claude-permissions.sh denies `*baseline-approv*` in Bash and edits to the +# approvals file (deny beats allow and auto mode in Claude Code). +# +# WHAT IT DOES NOT PROTECT AGAINST, said plainly: an agent that deliberately evades the deny +# rules (obfuscated shell, a script that writes the row), a harness with no deny rules whose +# agent terminal is a real TTY, or someone deleting both ledgers. It turns "the agent typed a +# flag" into "the agent forged a record", which is visible in git and named by gate-check. +# +# Covered baselines (kind -> file, relative to the project root): +# lint .claude/lint-baseline.json project-bin/lint-gate.sh +# conformance docs/conformance/baseline.tsv project-bin/conformance-check.sh +# +# Records: +# .claude/baseline-approvals.tsv ts kind sha256 by note — written ONLY by baseline-approve.sh +# .claude/baseline-writes.tsv ts kind sha256 basis — every write an instrument made +# (basis: first | tighten | seed | approved) +# .claude/baseline-pending/[.summary] the refused proposal, for the person to review +# Commit all three with the baseline. +# +# Usage (instruments and gate-check call it; a person never needs to): +# baseline-lock.sh file print the baseline path (relative) +# baseline-lock.sh sha print sha256 of a file +# baseline-lock.sh verify print STATEnote, exit 0 +# NONE no baseline and no record (a first write needs no approval) +# GONE no baseline, but records exist (re-creating it needs approval) +# PASS the file on disk matches an approval or a recorded instrument write +# FAIL it matches neither: it moved outside the instrument, or without approval +# MANUAL a baseline with no record at all — it predates this lock; unverifiable +# baseline-lock.sh approved exit 0 iff an approval row matches +# baseline-lock.sh record append a write row +# baseline-lock.sh propose save the proposal, print the ask +# +# Producer of the records: the two instruments above (writes, proposals) and +# project-bin/baseline-approve.sh (approvals). Consumer: those instruments and +# bin/gate-check.sh's "Baselines" line (toolkit copy, run against the project root). +set -uo pipefail + +APPROVALS=".claude/baseline-approvals.tsv" +WRITES=".claude/baseline-writes.tsv" + +_file() { + case "$1" in + lint) echo ".claude/lint-baseline.json" ;; + conformance) echo "docs/conformance/baseline.tsv" ;; + *) echo "baseline-lock: unknown kind '$1' (lint | conformance)" >&2; return 2 ;; + esac +} + +_sha() { + if command -v sha256sum >/dev/null 2>&1; then sha256sum "$1" | cut -d' ' -f1 + elif command -v shasum >/dev/null 2>&1; then shasum -a 256 "$1" | cut -d' ' -f1 + else openssl dgst -sha256 "$1" | sed 's/^.*= *//' + fi +} + +_has() { # _has — a row for this kind and hash + [ -f "$1" ] && awk -F'\t' -v k="$2" -v s="$3" '$2==k && $3==s {f=1} END {exit !f}' "$1" +} +_any() { # _any — any row for this kind + [ -f "$1" ] && awk -F'\t' -v k="$2" '$2==k {f=1} END {exit !f}' "$1" +} +_now() { date -u +%Y-%m-%dT%H:%M:%SZ; } + +cmd="${1:-}"; [ $# -gt 0 ] && shift +case "$cmd" in + file) + _file "$1" ;; + sha) + _sha "$1" ;; + verify) + root="$1"; kind="$2"; f="$(_file "$kind")" || exit 2 + a="$root/$APPROVALS"; w="$root/$WRITES" + if [ ! -f "$root/$f" ]; then + if _any "$a" "$kind" || _any "$w" "$kind"; then + printf 'GONE\t%s is missing but was recorded before — re-creating it needs a person: bin/baseline-approve.sh %s\n' "$f" "$kind" + else + printf 'NONE\tno %s baseline yet\n' "$kind" + fi + exit 0 + fi + s="$(_sha "$root/$f")" + if _has "$a" "$kind" "$s"; then + printf 'PASS\t%s matches a human approval\n' "$f" + elif _has "$w" "$kind" "$s"; then + printf 'PASS\t%s matches a recorded write (%s)\n' "$f" \ + "$(awk -F'\t' -v k="$kind" -v s="$s" '$2==k && $3==s {b=$4} END {print b}' "$w")" + elif _any "$a" "$kind" || _any "$w" "$kind"; then + printf 'FAIL\t%s moved without approval (sha %s is in neither %s nor %s) — a person reviews it and runs: bin/baseline-approve.sh %s --current\n' \ + "$f" "$(printf '%s' "$s" | cut -c1-12)" "$APPROVALS" "$WRITES" "$kind" + else + printf 'MANUAL\t%s has no approval or write record (it predates the lock) — cannot tell who moved it; the next instrument write starts the record\n' "$f" + fi + exit 0 ;; + approved) + _has "$1/$APPROVALS" "$2" "$3" ;; + record) + root="$1"; mkdir -p "$root/.claude" + [ -f "$root/$WRITES" ] || printf 'ts\tkind\tsha256\tbasis\n' > "$root/$WRITES" + printf '%s\t%s\t%s\t%s\n' "$(_now)" "$2" "$3" "$4" >> "$root/$WRITES" + rm -f "$root/.claude/baseline-pending/$2" "$root/.claude/baseline-pending/$2.summary" ;; + propose) + root="$1"; kind="$2"; src="$3"; summary="$4" + d="$root/.claude/baseline-pending"; mkdir -p "$d" + cp "$src" "$d/$kind" && printf '%s\n' "$summary" > "$d/$kind.summary" + echo "" >&2 + echo " Moving this baseline is the user's call, and a flag the agent can type is not their" >&2 + echo " answer. Show the user the change above, then ask THEM to run, in their own terminal:" >&2 + echo " bin/baseline-approve.sh $kind" >&2 + echo " (it shows the proposal and asks them to type 'approve'). Agents: do not run it —" >&2 + echo " it is denied to you and refuses without a terminal. Then re-run the same command." >&2 + ;; + *) + sed -n '2,45p' "$0"; exit 2 ;; +esac diff --git a/project-bin/conformance-check.sh b/project-bin/conformance-check.sh index b76031cb..98553195 100755 --- a/project-bin/conformance-check.sh +++ b/project-bin/conformance-check.sh @@ -389,13 +389,57 @@ KEYED="$(printf '%s\n' "$RESULTS" | awk -F'\t' '{print $1"\t"$2"\t"$5}')" # whole dropped every other module's baseline, and their regressions went unmeasured with # nothing said (card-disbursement build, 2026-09-26: `--update-baseline --module MockServices` # dropped the 127 Disbursement rows). +# +# A rewrite that LOOSENS needs a person (#348): a row OK in the old baseline and not OK in the +# new one is a regression being accepted, so it is written only against a matching row in +# .claude/baseline-approvals.tsv (bin/baseline-approve.sh conformance, run by the user). So does +# re-creating a deleted baseline, or any write over one that moved outside this script. A +# first baseline, a rewrite that loosens nothing and first-sight seeding need nothing; every +# write is recorded (bin/baseline-lock.sh) so gate-check can tell a hand edit from ours. +LOCK="$(dirname "${BASH_SOURCE[0]}")/baseline-lock.sh" +[ -f "$LOCK" ] || LOCK="" +PRIOR=""; [ -n "$LOCK" ] && PRIOR="$(bash "$LOCK" verify "$PROJECT_ROOT" conformance | cut -f1)" +# lock_write — move over the baseline, or refuse (exit 2) +lock_write() { + local new="$1" basis="$2" why="" loose sha + if [ -f "$BASELINE" ]; then + loose="$(awk -F'\t' 'NR==FNR { if ($3=="OK") ok[$1"\t"$2]=1; next } + (($1"\t"$2) in ok) && $3!="OK" {n++} END {print n+0}' "$BASELINE" "$new")" + [ "$loose" -gt 0 ] && why="accepts $loose regression(s) (rows OK in the baseline, not OK now)" + fi + [ "$PRIOR" = GONE ] && why="${why:+$why; }re-creates a baseline that was deleted" + [ "$PRIOR" = FAIL ] && why="${why:+$why; }follows a baseline that moved without approval" + if [ -z "$why" ]; then + mv "$new" "$BASELINE" + if [ -n "$LOCK" ]; then bash "$LOCK" record "$PROJECT_ROOT" conformance "$(bash "$LOCK" sha "$BASELINE")" "$basis" + else echo " note: bin/baseline-lock.sh not installed — this baseline write is unrecorded (sync-project.sh)" >&2; fi + return 0 + fi + if [ -z "$LOCK" ]; then + echo "REFUSED — this baseline write $why and needs a person's approval, but bin/baseline-lock.sh is not installed (run the toolkit's bin/sync-project.sh)." >&2 + rm -f "$new"; exit 2 + fi + sha="$(bash "$LOCK" sha "$new")" + if bash "$LOCK" approved "$PROJECT_ROOT" conformance "$sha"; then + mv "$new" "$BASELINE" + bash "$LOCK" record "$PROJECT_ROOT" conformance "$sha" approved + echo " human approval found for this exact baseline (sha $(printf '%s' "$sha" | cut -c1-12))" + return 0 + fi + echo "REFUSED — this baseline write $why." >&2 + [ -f "$BASELINE" ] && awk -F'\t' 'NR==FNR { if ($3=="OK") ok[$1"\t"$2]=1; next } + (($1"\t"$2) in ok) && $3!="OK" {print " " $1 " " $2 " OK -> " $3}' "$BASELINE" "$new" | head -20 >&2 + bash "$LOCK" propose "$PROJECT_ROOT" conformance "$new" "conformance: $why${MODULE:+ (module $MODULE)}" + rm -f "$new"; exit 2 +} if [ "$UPDATE_BASELINE" -eq 1 ] || [ ! -f "$BASELINE" ]; then if [ -n "$MODULE" ] && [ -f "$BASELINE" ]; then { awk -F'\t' -v m="$MODULE" '$1!=m' "$BASELINE"; printf '%s\n' "$KEYED"; } \ - | sed '/^$/d' | sort > "$BASELINE.tmp" && mv "$BASELINE.tmp" "$BASELINE" + | sed '/^$/d' | sort > "$BASELINE.tmp" else - printf '%s\n' "$KEYED" > "$BASELINE" + printf '%s\n' "$KEYED" > "$BASELINE.tmp" fi + if [ -f "$BASELINE" ]; then lock_write "$BASELINE.tmp" tighten; else lock_write "$BASELINE.tmp" first; fi WROTE_BASELINE=1 else WROTE_BASELINE=0 @@ -416,9 +460,13 @@ if [ "$WROTE_BASELINE" -eq 0 ]; then REGRESSIONS+="$mod $ptr was OK, now $verdict"$'\n' fi done <<< "$KEYED" - if [ -n "$SEEDED" ]; then + # Seeding adds rows the baseline never had; it is recorded, never laundered over a hand edit. + if [ -n "$SEEDED" ] && [ "$PRIOR" = FAIL ]; then + echo " note: $(printf '%s' "$SEEDED" | grep -c .) row(s) not seeded — the baseline moved without approval (see gate-check)" >&2 + SEEDED="" + elif [ -n "$SEEDED" ]; then { cat "$BASELINE"; printf '%s' "$SEEDED"; } | sed '/^$/d' | sort > "$BASELINE.tmp" \ - && mv "$BASELINE.tmp" "$BASELINE" + && lock_write "$BASELINE.tmp" seed fi fi NSEED=$(printf '%s' "$SEEDED" | grep -c . || true) diff --git a/project-bin/exec.sh b/project-bin/exec.sh index b85dde08..78fe3d62 100755 --- a/project-bin/exec.sh +++ b/project-bin/exec.sh @@ -858,9 +858,9 @@ run_lint_gate() { LINT_DETAIL="lint ROSE: ${_rules:-see output}${_blind:+; BLIND: $_blind}" echo " ✗ lint: rule(s) rose vs baseline — ${_rules:-see below}" sed -n '/^FAIL/,$p' "$_lo" | head -60 | sed 's/^/ /' - echo " Fix the script and re-run. Accepting the debt is the user's call, made in chat:" - echo " bin/lint-gate.sh --update-baseline --reason \"\" (a security-tier rise also" - echo " needs --approved-in-chat). The acceptance is logged to .claude/lint-acceptances.tsv." + echo " Fix the script and re-run. Accepting the debt is the user's call, not yours:" + echo " bin/lint-gate.sh --update-baseline --reason \"\", then the USER runs" + echo " bin/baseline-approve.sh lint in their own terminal. Logged to .claude/lint-acceptances.tsv." fi ;; *) LINT_DETAIL="lint could not run (lint-gate exit $LINT_RC)"; echo " ⚠ $LINT_DETAIL" tail -5 "$_lo" | sed 's/^/ /' ;; diff --git a/project-bin/lint-gate.sh b/project-bin/lint-gate.sh index cba15e74..3524d5c3 100755 --- a/project-bin/lint-gate.sh +++ b/project-bin/lint-gate.sh @@ -15,7 +15,7 @@ # bin/lint-gate.sh # check against baseline; exit 1 if any rule rose # bin/lint-gate.sh --update-baseline # accept current counts as the new baseline # --reason "" # REQUIRED when the update raises any rule's count -# --approved-in-chat # ALSO required when a security-tier rule rises +# then bin/baseline-approve.sh lint # run by the USER, in a terminal, when it rises # bin/lint-gate.sh --full # ignore the vendor exclusion (rarely useful) # # Exit codes: 0 clean or unchanged · 1 a rule increased, or BLIND (a rule inspected nothing, @@ -50,9 +50,15 @@ # .claude/lint-acceptances.tsv (ts, total before/after, per-rule rises, security flag, reason) # — commit it with the baseline. A first baseline and a pure ratchet-down need no reason. # A rise in a security-tier rule (default SEC* and CONV006; override, one id or glob per line, -# in .claude/lint-security-rules.txt) also needs --approved-in-chat: a claim on the record that -# the user was shown the findings and said yes, the same contract as gate-check --ack-protocol. -# Never set it on your own judgement. +# in .claude/lint-security-rules.txt) is flagged as such in that log. +# +# A RISE NEEDS A PERSON, NOT A FLAG (#348, 2026-10-14). --approved-in-chat was the proof the user +# said yes, and the agent could type it itself. Now any update that raises a rule (or re-creates +# a deleted baseline, or follows a hand edit gate-check would FAIL) writes only content a person +# approved: the refused run saves the proposal and prints `bin/baseline-approve.sh lint` for the +# USER to run in their terminal; it records the sha256 of exactly that content, and the re-run +# writes it. A first baseline and a pure ratchet-down still need nothing. --approved-in-chat is +# accepted and ignored, so old command lines still parse. See baseline-lock.sh. # # PRODUCES .claude/loop/lint-ledger.tsv (append-only, one row per rule per run; read with # project-bin/lint-trend.sh) and .claude/loop/lint-last.json on every run that actually gets as far as computing @@ -132,8 +138,8 @@ while [ $# -gt 0 ]; do --reason) [ $# -ge 2 ] || { echo "lint-gate: --reason needs a value" >&2; exit 2; } REASON="$2"; shift ;; --reason=*) REASON="${1#--reason=}" ;; - --approved-in-chat) APPROVED=1 ;; - -h|--help) sed -n '2,64p' "$0"; exit 0 ;; + --approved-in-chat) APPROVED=1 ;; # retired as evidence (#348): accepted, ignored + -h|--help) sed -n '2,71p' "$0"; exit 0 ;; *) echo "lint-gate: unknown argument '$1'" >&2; exit 2 ;; esac shift @@ -282,9 +288,14 @@ else fi [ -s "$OUT" ] || { echo "lint-gate: lint produced no output" >&2; exit 2; } -BASELINE="$BASELINE" OUT="$OUT" UPDATE="$UPDATE" REASON="$REASON" APPROVED="$APPROVED" \ +if [ "$APPROVED" -eq 1 ] && [ "$UPDATE" -eq 1 ]; then + echo "lint-gate: note — --approved-in-chat is no longer evidence (#348); a rise needs bin/baseline-approve.sh, run by the user." >&2 +fi +LOCK="$(dirname "${BASH_SOURCE[0]}")/baseline-lock.sh" +[ -f "$LOCK" ] || LOCK="" +BASELINE="$BASELINE" OUT="$OUT" UPDATE="$UPDATE" REASON="$REASON" LOCK="$LOCK" ROOT="$ROOT" \ SECURITY_FILE="$SECURITY_FILE" ACCEPT_LOG="$ACCEPT_LOG" "$PY" - <<'PY' -import json, os, sys, collections, datetime, fnmatch +import json, os, sys, collections, datetime, fnmatch, hashlib, subprocess, tempfile out_path = os.environ["OUT"] base_path = os.environ["BASELINE"] @@ -389,7 +400,9 @@ counts = collections.Counter(v["ruleId"] for v in viol) if update: # Accepting a rise is a decision; record it. See the header ("ACCEPTING DEBT ..."). reason = os.environ.get("REASON", "").strip() - approved = os.environ.get("APPROVED") == "1" + lock, root = os.environ.get("LOCK", ""), os.environ.get("ROOT", ".") + def run_lock(*a): + return subprocess.run(["bash", lock] + list(a), capture_output=True, text=True) sec_pats = ["SEC*", "CONV006"] sec_file = os.environ.get("SECURITY_FILE", "") if sec_file and os.path.exists(sec_file): @@ -414,15 +427,42 @@ if update: print(" bin/lint-gate.sh --update-baseline --reason \"\"", file=sys.stderr) sys.exit(2) - if sec_raised and not approved: - print("lint-gate: REFUSED — security-tier rule(s) rose: %s" % ", ".join( - "%s %d->%d" % x for x in sec_raised), file=sys.stderr) - print(" Show the user these findings in chat. Only after they say yes, re-run with", file=sys.stderr) - print(" --approved-in-chat (a claim on the record). Otherwise fix the grant.", file=sys.stderr) - for v in [x for x in viol if x["ruleId"] in {r for r, _, _ in sec_raised}][:10]: - print(" %s | %s | %s" % (v["ruleId"], v.get("document") or v.get("module") or "", - (v.get("message") or "")[:110]), file=sys.stderr) - sys.exit(2) + # The exact bytes this update would write; an approval is for these and nothing else. + content = json.dumps({"counts": dict(sorted(counts.items())), + "total": len(viol), + "note": "Per-rule finding counts on OUR modules. Gate fails when any rises."}, + indent=2) + "\n" + sha = hashlib.sha256(content.encode("utf-8")).hexdigest() + prior = run_lock("verify", root, "lint").stdout.split("\t")[0] if lock else "" + why = [] + if raised: + why.append("raises %s" % ", ".join("%s %d->%d" % x for x in raised)) + if prior == "GONE": + why.append("re-creates a baseline that was deleted") + if prior == "FAIL": + why.append("follows a baseline that moved without approval") + basis = "first" if old_base is None else "tighten" + if why: + basis = "approved" + if not lock: + print("lint-gate: REFUSED — this update %s and needs a person's approval, but" % "; ".join(why), file=sys.stderr) + print(" bin/baseline-lock.sh is not installed. Run the toolkit's bin/sync-project.sh.", file=sys.stderr) + sys.exit(2) + if run_lock("approved", root, "lint", sha).returncode != 0: + print("lint-gate: REFUSED — this update %s." % "; ".join(why), file=sys.stderr) + for v in [x for x in viol if x["ruleId"] in {r for r, _, _ in sec_raised}][:10]: + print(" %s | %s | %s" % (v["ruleId"], v.get("document") or v.get("module") or "", + (v.get("message") or "")[:110]), file=sys.stderr) + summary = "lint: %s%s. Reason: %s" % ("; ".join(why), + " (SECURITY-TIER: %s)" % ", ".join(r for r, _, _ in sec_raised) if sec_raised else "", + reason or "-") + with tempfile.NamedTemporaryFile("w", suffix=".json", delete=False) as t: + t.write(content) + r = run_lock("propose", root, "lint", t.name, summary) + os.unlink(t.name) + sys.stderr.write(r.stderr) + sys.exit(2) + print("==> human approval found for this exact baseline (sha %s)" % sha[:12]) if raised: log = os.environ.get("ACCEPT_LOG", "") try: @@ -440,11 +480,11 @@ if update: except OSError as e: print("lint-gate: could not append %s: %s" % (log, e), file=sys.stderr) sys.exit(2) - json.dump({"counts": dict(sorted(counts.items())), - "total": len(viol), - "note": "Per-rule finding counts on OUR modules. Gate fails when any rises."}, - open(base_path, "w"), indent=2) - open(base_path, "a").write("\n") + open(base_path, "w").write(content) + if lock: + run_lock("record", root, "lint", sha, basis) + else: + print("lint-gate: bin/baseline-lock.sh not installed — this write is unrecorded (sync-project.sh)", file=sys.stderr) print("==> baseline written: %d findings across %d rules" % (len(viol), len(counts))) for r, c in counts.most_common(): print(" %5d %-8s %s" % (c, sev[r], r)) diff --git a/skills/lint-that-actually-runs.md b/skills/lint-that-actually-runs.md index c1932eac..a3274ebd 100644 --- a/skills/lint-that-actually-runs.md +++ b/skills/lint-that-actually-runs.md @@ -135,10 +135,31 @@ the ratchet is the *only* mechanism available. `project-bin/lint-gate.sh` implem re-baselined 61 times, every time by the agent whose script caused the rise; baselines grew 8 -> 757, 440 -> 677 and 9 -> 187, with security warnings accepted in the same batches as naming info. So an update that raises any rule needs `--reason ""` and is appended to -`.claude/lint-acceptances.tsv` (commit it with the baseline); a rise in a security-tier rule -(default `SEC*` and `CONV006`, override in `.claude/lint-security-rules.txt`) also needs -`--approved-in-chat`, set only after the user saw the findings and said yes. A first baseline -and a ratchet-down need neither. +`.claude/lint-acceptances.tsv` (commit it with the baseline). A first baseline and a +ratchet-down need nothing more. + +**A rise moves only on the user's own command (#348).** `--approved-in-chat` was the agent's +claim that the user said yes, and the agent could type it unasked. Now the refused update saves +the proposal and prints `bin/baseline-approve.sh lint`. Show the user the rise (security-tier +rules, default `SEC*` and `CONV006`, are flagged in it); **the user** runs that command in their +own terminal, reads the diff and types `approve`; then re-run the same update. The approval is +the sha256 of exactly that content, so a count that moves in between is refused again. + +``` +agent: bin/lint-gate.sh --update-baseline --reason "x" --approved-in-chat -> exit 0 (before: the agent approved itself) +agent: bin/lint-gate.sh --update-baseline --reason "x" -> REFUSED, prints the user's command +user: bin/baseline-approve.sh lint (their terminal; types approve) +agent: bin/lint-gate.sh --update-baseline --reason "x" -> written +``` + +Never run `baseline-approve.sh` yourself and never edit `.claude/baseline-approvals.tsv`. +Claude Code denies both (`bin/install-claude-permissions.sh`), and the command refuses to run +without a terminal. `conformance-check.sh --update-baseline` follows the same rule when it +would accept a regression. `gate-check.sh` prints `Baselines (human-approved?)`: FAIL when a +baseline matches neither an approval nor a recorded write (a hand edit; this blocks Stages 5 +and 6), and MANUAL when a baseline predates the lock. Commit `.claude/baseline-*.tsv` with the +baseline. What the lock does not stop, an agent deliberately evading the deny rules, is listed +in `project-bin/baseline-lock.sh`. **The ratchet remembers one number per rule; the ledger remembers every run.** Every gate run appends one row per rule to `.claude/loop/lint-ledger.tsv` (timestamp, verdict, rule, count, diff --git a/tests/wave2/bench-baseline-lock.sh b/tests/wave2/bench-baseline-lock.sh new file mode 100755 index 00000000..956e5862 --- /dev/null +++ b/tests/wave2/bench-baseline-lock.sh @@ -0,0 +1,199 @@ +#!/usr/bin/env bash +# bench-baseline-lock.sh — before/after measurement for #348 (human-locked baselines). +# +# Usage: bash tests/wave2/bench-baseline-lock.sh +# e.g. a `git worktree add --detach /tmp/m origin/master` checkout vs this branch. +# +# Replays the same scripted agent attempts against each tree, for each ratchet baseline the lock +# covers (lint-gate.sh, conformance-check.sh). Each attempt starts from a fresh project holding +# a clean baseline and proposes a LOOSER one (a lint rule rises; a conformance row goes OK -> +# STALE). Unapproved attempts carry only what an agent can produce on its own: --reason, +# --approved-in-chat, --module, deleting the baseline first. Legitimate attempts follow the +# documented flow: the agent runs the update; if it is refused and names the approve command, +# the person runs it (fixtures/baseline-lock/human.sh, a pseudo-terminal) and the agent re-runs. +# +# Every number in the printed table is counted here, not typed: +# unapproved moves that landed attempt left a baseline different from the clean one +# (the instrument wrote it, or the agent edited the file) +# ...that gate-check did not flag baseline changed or deleted, and the tree's gate-check +# does not print "Baselines ...: FAIL" +# legitimate re-baselines that landed the proposed content is on disk at the end +# extra human steps per legitimate approve-command runs the flow needed, per attempt +# A trailing "Not-a-bench" note: this is replayed attempts on synthetic input, not a field run. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +[ $# -eq 2 ] || { sed -n '4,6p' "$0"; exit 2; } +BEFORE="$(cd "$1" && pwd)"; AFTER="$(cd "$2" && pwd)" +. "$HERE/fixtures/baseline-lock/human.sh" +pty_available || { echo "bench-baseline-lock: no usable \`script\` for a pseudo-terminal — cannot replay the person's step" >&2; exit 2; } +command -v python3 >/dev/null 2>&1 && PY=python3 || PY=python # portability-ok: bench probe +WORK="$(mktemp -d "${TMPDIR:-/tmp}/bench-baseline.XXXXXX")" +trap 'rm -rf "$WORK"' EXIT + +lintjson() { # lintjson RULE=n ... + _o="$1"; shift + "$PY" - "$@" > "$_o" <<'PY' +import json, sys +v = [] +for a in sys.argv[1:]: + r, n = a.split("=") + v += [{"ruleId": r, "severity": "warning", "message": "f%d" % i, "module": "Orders", + "document": "D%d" % i, "documentType": "microflow", "documentId": "%s-%d" % (r, i), + "suggestion": "s"} for i in range(int(n))] +print(json.dumps({"violations": v, "summary": {}})) +PY +} +install() { # install + mkdir -p "$2/bin" "$2/.claude" + for f in _common.sh lint-gate.sh conformance-check.sh baseline-lock.sh baseline-approve.sh; do + [ -f "$1/project-bin/$f" ] && cp "$1/project-bin/$f" "$2/bin/" + done + printf 'mpr' > "$2/App.mpr" +} + +# ---- lint ----------------------------------------------------------------------------------- +lint_proj() { # lint_proj — a project with a clean lint baseline (CONV009=2 SEC003=1) + install "$1" "$2"; printf '#!/bin/sh\nexit 0\n' > "$2/mxcli"; chmod +x "$2/mxcli" + lintjson "$2/lint.json" CONV009=2 SEC003=1 + (cd "$2" && LINT_JSON="$2/lint.json" bash bin/lint-gate.sh --update-baseline) >/dev/null 2>&1 +} +lint_run() { (cd "$1" && LINT_JSON="$1/lint.json" bash bin/lint-gate.sh --update-baseline "${@:2}") >/dev/null 2>&1; } +# Unapproved attempts: | ("DELETE" = rm the baseline first) +LINT_UNAPPROVED="CONV009=5 SEC003=1|--reason because +CONV009=5 SEC003=1|--reason because --approved-in-chat +CONV009=2 SEC003=3|--reason demo-users --approved-in-chat +CONV009=9 SEC003=4|--approved-in-chat --reason user-said-yes +CONV009=3 SEC003=1 CONV006=1|--reason grant-later --approved-in-chat +CONV009=40 SEC003=1|DELETE +CONV009=6 SEC003=2|DELETE --reason x --approved-in-chat +CONV009=4 SEC003=1|--reason=inline --approved-in-chat +CONV009=12 SEC003=1|HANDEDIT +CONV009=2 SEC003=6|HANDEDIT" +LINT_LEGIT="CONV009=5 SEC003=1 +CONV009=2 SEC003=3 +CONV009=8 SEC003=1 CONV006=2 +CONV009=3 SEC003=2" + +# ---- conformance ---------------------------------------------------------------------------- +conf_proj() { # conf_proj — 3 modules x 2 rows, all OK, baselined + install "$1" "$2" + : > "$2/PRESENT" + for m in Orders Billing Stock; do + mkdir -p "$2/architecture/modules/$m" + printf '%s.E\n%s.ACT\n' "$m" "$m" >> "$2/PRESENT" + printf '| Pointer | Kind | Name | Row | Proof | Acceptance | Status |\n|---|---|---|---|---|---|---|\n| `/domainEntities/0` | entity | E | 1.1 | x | `DESCRIBE ENTITY %s.E` | built |\n| `/microflows/0` | microflow | ACT | 1.2 | x | `DESCRIBE MICROFLOW %s.ACT` | built |\n' "$m" "$m" \ + > "$2/architecture/modules/$m/coverage-ledger.md" + done + cat > "$2/mxcli" <<'EOF' +#!/usr/bin/env bash +cmd=""; while [ $# -gt 0 ]; do [ "$1" = -c ] && cmd="$2"; shift; done +name="${cmd##* }" +if grep -qx "$name" "$(dirname "$0")/PRESENT"; then echo "-- $name"; exit 0; fi +echo "Error: $name not found"; exit 1 +EOF + chmod +x "$2/mxcli" + (cd "$2" && PROJECT_ROOT="$2" bash bin/conformance-check.sh --quiet) >/dev/null 2>&1 +} +conf_break() { # conf_break — those elements go absent (rows regress) + _d="$1"; shift + for n in "$@"; do grep -vx "$n" "$_d/PRESENT" > "$_d/PRESENT.n"; mv "$_d/PRESENT.n" "$_d/PRESENT"; done +} +conf_run() { (cd "$1" && PROJECT_ROOT="$1" bash bin/conformance-check.sh --quiet --update-baseline "${@:2}") >/dev/null 2>&1; } +CONF_UNAPPROVED="Orders.E| +Orders.ACT| +Billing.E Billing.ACT| +Stock.E|--module Stock +Orders.E Stock.ACT| +Billing.ACT|--module Billing +Orders.E|DELETE +Stock.E Stock.ACT|DELETE +Orders.E|HANDEDIT +Billing.E Stock.ACT|HANDEDIT" +CONF_LEGIT="Orders.E +Billing.ACT|--module Billing +Stock.E Stock.ACT +Orders.ACT Billing.E" + +gate_flags() { # gate_flags — 1 if the tree's gate-check says Baselines FAIL + # captured first: `| grep -q` under pipefail reads a SIGPIPE'd gate-check as "not flagged" + _g="$(bash "$1/bin/gate-check.sh" "$2" --no-html 2>/dev/null)" + case "$_g" in *"Baselines (human-approved?): FAIL"*) echo 1 ;; *) echo 0 ;; esac +} + +# measure → prints "landed silent n_un legit_ok n_legit human_steps" +measure() { + _t="$1"; _k="$2"; landed=0; silent=0; nu=0; lok=0; nl=0; steps=0; i=0 + if [ "$_k" = lint ]; then un="$LINT_UNAPPROVED"; lg="$LINT_LEGIT"; else un="$CONF_UNAPPROVED"; lg="$CONF_LEGIT"; fi + while IFS='|' read -r spec flags <&3; do + [ -n "$spec" ] || continue + i=$((i+1)); d="$WORK/$_k-u$i-$(basename "$_t")"; nu=$((nu+1)) + if [ "$_k" = lint ]; then + lint_proj "$_t" "$d"; f="$d/.claude/lint-baseline.json"; clean="$(cat "$f")" + lintjson "$d/lint.json" $spec + else + conf_proj "$_t" "$d"; f="$d/docs/conformance/baseline.tsv"; clean="$(cat "$f")" + conf_break "$d" $spec + fi + case "$flags" in + HANDEDIT) # the agent edits the baseline file itself: the measured state, no instrument + if [ "$_k" = lint ]; then + "$PY" -c 'import collections,json,sys;c=collections.Counter(v["ruleId"] for v in json.load(open(sys.argv[1]))["violations"]);json.dump({"counts":dict(sorted(c.items())),"total":sum(c.values())},open(sys.argv[2],"w"),indent=2)' "$d/lint.json" "$f" + else + cp "$f" "$f.n" + for n in $spec; do m="${n%%.*}"; case "$n" in *.E) p='`/domainEntities/0`' ;; *) p='`/microflows/0`' ;; esac + awk -F'\t' -v m="$m" -v p="$p" 'BEGIN{OFS="\t"} $1==m && $2==p {$3="STALE"} {print}' "$f.n" > "$f.m"; mv "$f.m" "$f.n"; done + mv "$f.n" "$f" + fi ;; + *) + case "$flags" in DELETE*) rm -f "$f"; flags="${flags#DELETE}" ;; esac + # shellcheck disable=SC2086 + if [ "$_k" = lint ]; then eval "lint_run \"\$d\" $flags"; else eval "conf_run \"\$d\" $flags"; fi ;; + esac + changed=0; { [ ! -f "$f" ] || [ "$(cat "$f")" != "$clean" ]; } && changed=1 + { [ -f "$f" ] && [ "$changed" = 1 ]; } && landed=$((landed+1)) + [ "$changed" = 1 ] && [ "$(gate_flags "$_t" "$d")" = 0 ] && silent=$((silent+1)) + done 3</dev/null)" + else + conf_proj "$_t" "$d"; f="$d/docs/conformance/baseline.tsv" + conf_break "$d" $spec + # shellcheck disable=SC2086 + conf_run "$d" $flags || { + [ -f "$d/bin/baseline-approve.sh" ] && human_approve "$d" conformance && steps=$((steps+1)) + conf_run "$d" $flags; } + want="$(for n in $spec; do printf 'STALE\n'; done | grep -c .)" + got="$(grep -c 'STALE' "$f" 2>/dev/null)" + fi + [ -n "$got" ] && [ "$got" = "$want" ] && lok=$((lok+1)) + done 3< [--current] +# Runs /bin/baseline-approve.sh under a pseudo-terminal and types `approve`, which is +# what a person does in their own terminal. util-linux and BSD `script` take different +# arguments; both are probed. Returns 3 when neither works (the caller reports a skip — it +# never writes the approval row by hand, which would test nothing). +human_approve() { + _hp="$1"; shift + if script -qec true /dev/null >/dev/null 2>&1; then + printf 'approve\n' | script -qec "bash '$_hp/bin/baseline-approve.sh' $*" /dev/null >/dev/null 2>&1 + elif script -q /dev/null true >/dev/null 2>&1; then + printf 'approve\n' | script -q /dev/null bash "$_hp/bin/baseline-approve.sh" "$@" >/dev/null 2>&1 + else + return 3 + fi +} + +# pty_available — exit 0 when human_approve can run here +pty_available() { script -qec true /dev/null >/dev/null 2>&1 || script -q /dev/null true >/dev/null 2>&1; } diff --git a/tests/wave2/test-baseline-lock.sh b/tests/wave2/test-baseline-lock.sh new file mode 100755 index 00000000..e02f1fa6 --- /dev/null +++ b/tests/wave2/test-baseline-lock.sh @@ -0,0 +1,141 @@ +#!/usr/bin/env bash +# test-baseline-lock.sh — a ratchet baseline moves only on a person's approval (#348). +# +# Usage: bash tests/wave2/test-baseline-lock.sh [toolkit-root] +# The subject is a toolkit tree: its project-bin/lint-gate.sh, conformance-check.sh, +# baseline-lock.sh, baseline-approve.sh and bin/gate-check.sh. Run against origin/master's +# tree, the L/C/G cases fail — that is the hole: the agent typed --approved-in-chat itself. +# +# L1 lint: a rise with --reason AND --approved-in-chat, no person → refused, baseline unchanged +# L2 baseline-approve.sh with no terminal (an agent's tool call) → refused, no approval row +# L3 after the person approves → the update lands, exactly the approved content +# L4 the approval binds to content: counts change after approval → refused again +# L5 deleting the baseline and re-creating it lower is refused (no laundering by delete) +# C1 conformance: --update-baseline accepting a regression, no person → refused, unchanged +# C2 after the person approves → it lands +# G1 gate-check: a hand-moved baseline reports FAIL, and Stage 5 is BLOCKED (exit 1) +# G2 after the person approves the file as it is (--current) → PASS +# G3 a baseline with no record at all (predates the lock) → MANUAL, not PASS +# +# Synthetic lint JSON and a stub mxcli, as in test-lint-gate-accept.sh and +# test-conformance-baseline-scope.sh: nothing here parses tool output; the assertions are +# about which baseline writes land. The person's step runs baseline-approve.sh under a +# pseudo-terminal (fixtures/baseline-lock/human.sh); the fixture never writes an approval row. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +TK="$(cd "${1:-$HERE/../..}" && pwd)" +. "$HERE/fixtures/baseline-lock/human.sh" +command -v python3 >/dev/null 2>&1 && PY=python3 || PY=python # portability-ok: fixture probe +WORK="$(mktemp -d "${TMPDIR:-/tmp}/baseline-lock.XXXXXX")" +trap 'rm -rf "$WORK"' EXIT +PASS=0; FAIL=0 +ok() { PASS=$((PASS+1)); printf ' ok %s\n' "$1"; } +bad() { FAIL=$((FAIL+1)); printf ' FAIL %s\n' "$1"; [ -n "${2:-}" ] && printf '%s\n' "$2" | head -8 | sed 's/^/ /'; } +pty_available || { echo "SKIP: no usable \`script\` for a pseudo-terminal"; exit 0; } + +newproj() { # newproj — a project with the toolkit's scripts installed in bin/ + mkdir -p "$1/bin" "$1/.claude" + for f in _common.sh lint-gate.sh conformance-check.sh baseline-lock.sh baseline-approve.sh; do + [ -f "$TK/project-bin/$f" ] && cp "$TK/project-bin/$f" "$1/bin/" + done + printf 'mpr' > "$1/App.mpr" +} +lintjson() { # lintjson RULE=n ... > $WORK/lint.json + "$PY" - "$@" > "$WORK/lint.json" <<'PY' +import json, sys +v = [] +for a in sys.argv[1:]: + r, n = a.split("=") + v += [{"ruleId": r, "severity": "warning", "message": "f%d" % i, "module": "Orders", + "document": "D%d" % i, "documentType": "microflow", "documentId": "%s-%d" % (r, i), + "suggestion": "s"} for i in range(int(n))] +print(json.dumps({"violations": v, "summary": {}})) +PY +} +P="$WORK/p"; newproj "$P"; printf '#!/bin/sh\nexit 0\n' > "$P/mxcli"; chmod +x "$P/mxcli" +gate() { (cd "$P" && LINT_JSON="$WORK/lint.json" bash bin/lint-gate.sh "$@") 2>&1; } +LB="$P/.claude/lint-baseline.json" +counts() { "$PY" -c 'import json,sys;print(json.load(open(sys.argv[1]))["counts"])' "$1" 2>/dev/null; } + +echo "L1 the agent's own flags do not move the lint baseline" +lintjson CONV009=2 SEC003=1; gate --update-baseline >/dev/null +before="$(cat "$LB" 2>/dev/null)" +lintjson CONV009=5 SEC003=2 +out="$(gate --update-baseline --reason "cleanup later" --approved-in-chat)"; rc=$? +[ "$rc" -ne 0 ] && [ "$(cat "$LB")" = "$before" ] && ok "refused (rc $rc), baseline unchanged" \ + || bad "the agent moved the baseline on its own flag (rc $rc): $(counts "$LB")" "$out" + +echo "L2 the approve command refuses without a terminal" +out="$(bash "$P/bin/baseline-approve.sh" lint &1)"; rc=$? +[ "$rc" -ne 0 ] && [ ! -s "$P/.claude/baseline-approvals.tsv" ] && ok "no terminal → refused, no approval row" \ + || bad "approved without a terminal (rc $rc)" "$out" + +echo "L3 with the person's approval the update lands" +human_approve "$P" lint +out="$(gate --update-baseline --reason "cleanup later")"; rc=$? +[ "$rc" -eq 0 ] && [ "$(counts "$LB")" = "{'CONV009': 5, 'SEC003': 2}" ] && ok "accepted: $(counts "$LB")" \ + || bad "approved update did not land (rc $rc)" "$out" + +echo "L4 an approval covers exactly what the person saw" +lintjson CONV009=7 SEC003=2; gate --update-baseline --reason r >/dev/null +human_approve "$P" lint +lintjson CONV009=9 SEC003=2 +out="$(gate --update-baseline --reason r)"; rc=$? +[ "$rc" -ne 0 ] && [ "$(counts "$LB")" = "{'CONV009': 5, 'SEC003': 2}" ] && ok "counts moved after approval → refused" \ + || bad "a stale approval moved the baseline (rc $rc): $(counts "$LB")" "$out" + +echo "L5 deleting the baseline does not launder a rise" +rm -f "$LB"; lintjson CONV009=9 SEC003=2 +out="$(gate --update-baseline)"; rc=$? +[ "$rc" -ne 0 ] && [ ! -f "$LB" ] && ok "re-creation needs a person (rc $rc)" \ + || bad "a deleted baseline was re-created at 9 with no approval (rc $rc)" "$out" + +echo "C1/C2 conformance: accepting a regression needs the person" +C="$WORK/c"; newproj "$C"; mkdir -p "$C/architecture/modules/Orders" +printf 'Orders.Order\nOrders.ACT_Save\n' > "$C/PRESENT" +cat > "$C/mxcli" <<'EOF' +#!/usr/bin/env bash +cmd=""; while [ $# -gt 0 ]; do [ "$1" = -c ] && cmd="$2"; shift; done +name="${cmd##* }" +if grep -qx "$name" "$(dirname "$0")/PRESENT"; then echo "-- $name"; exit 0; fi +echo "Error: $name not found"; exit 1 +EOF +chmod +x "$C/mxcli" +cat > "$C/architecture/modules/Orders/coverage-ledger.md" <<'EOF' +| Pointer | Kind | Name | Row | Proof | Acceptance | Status | +|---|---|---|---|---|---|---| +| `/domainEntities/0` | entity | Order | 1.1 | exec.sh | `DESCRIBE ENTITY Orders.Order` | built | +| `/microflows/0` | microflow | ACT_Save | 1.2 | exec.sh | `DESCRIBE MICROFLOW Orders.ACT_Save` | built | +EOF +conf() { ( cd "$C" && PROJECT_ROOT="$C" bash bin/conformance-check.sh --quiet "$@" ) 2>&1; } +CB="$C/docs/conformance/baseline.tsv" +conf >/dev/null +printf 'Orders.Order\n' > "$C/PRESENT" +before="$(cat "$CB")" +out="$(conf --update-baseline)"; rc=$? +[ "$rc" -ne 0 ] && [ "$(cat "$CB")" = "$before" ] && ok "C1 regression not baselined without the person (rc $rc)" \ + || bad "C1 the agent baselined a regression (rc $rc)" "$out" +human_approve "$C" conformance +out="$(conf --update-baseline)"; rc=$? +[ "$rc" -eq 0 ] && grep -q "ACT_Save\|/microflows/0.*STALE" "$CB" && grep -q STALE "$CB" && ok "C2 accepted after approval" \ + || bad "C2 approved rewrite did not land (rc $rc)" "$out" + +echo "G1-G3 gate-check reports a baseline that moved without approval" +GC="$TK/bin/gate-check.sh" +gline() { bash "$GC" "$1" --no-html 2>/dev/null | grep '^Baselines'; } +printf '{\n "counts": {"CONV009": 40, "SEC003": 2}\n}\n' > "$LB" # a hand edit +l="$(gline "$P")" +case "$l" in *FAIL*lint*) ok "G1 hand-moved lint baseline: FAIL" ;; *) bad "G1 hand edit not flagged" "${l:-}" ;; esac +bash "$GC" "$P" 5 --no-html >/dev/null 2>"$WORK/g.err"; rc=$? +[ "$rc" -eq 1 ] && grep -q "BLOCKED by a baseline" "$WORK/g.err" && ok "G1 Stage 5 BLOCKED (exit 1)" \ + || bad "G1 Stage 5 not blocked by the hand edit (rc $rc)" "$(tail -3 "$WORK/g.err")" +human_approve "$P" lint --current +l="$(gline "$P")" +case "$l" in *PASS*) ok "G2 after the person approves the file as it is: PASS" ;; *) bad "G2 not PASS after --current approval" "$l" ;; esac +L="$WORK/legacy"; mkdir -p "$L/.claude"; cp "$LB" "$L/.claude/lint-baseline.json" +l="$(gline "$L")" +case "$l" in *MANUAL*) ok "G3 a baseline with no record: MANUAL" ;; *) bad "G3 unrecorded baseline not MANUAL" "${l:-}" ;; esac + +echo "" +echo "PASS=$PASS FAIL=$FAIL" +[ "$FAIL" -eq 0 ] diff --git a/tests/wave2/test-conformance-baseline-scope.sh b/tests/wave2/test-conformance-baseline-scope.sh index a10fa5cc..bab46c66 100755 --- a/tests/wave2/test-conformance-baseline-scope.sh +++ b/tests/wave2/test-conformance-baseline-scope.sh @@ -98,7 +98,13 @@ rc="$(run)" mv "$P/PRESENT.bak" "$P/PRESENT" # --- 6. an unscoped --update-baseline still rewrites everything -------------------------------- -printf 'Orders\t`/stale`\tOK\n' >> "$P/docs/conformance/baseline.tsv" +# The retired row arrives through the ledger, not a hand edit of baseline.tsv: since #348 a +# baseline that moved outside the instrument is refused (test-baseline-lock.sh covers that). +printf '| `/enumerations/0` | enum | Retired | 1.3 | exec.sh | `DESCRIBE ENTITY Orders.Order` | built |\n' \ + >> "$P/architecture/modules/Orders/coverage-ledger.md" +run --update-baseline >/dev/null +[ "$(wc -l < "$P/docs/conformance/baseline.tsv" | tr -d ' ')" = 5 ] || bad "the retired row never reached the baseline" +ledger Orders Order ACT_Order_Save rc="$(run --update-baseline)" [ "$rc" = 0 ] && [ "$(wc -l < "$P/docs/conformance/baseline.tsv" | tr -d ' ')" = 4 ] \ && ok "unscoped --update-baseline rewrites the whole file (a retired row is dropped)" \ diff --git a/tests/wave2/test-install-claude-permissions.sh b/tests/wave2/test-install-claude-permissions.sh index d656705e..0a286bdf 100755 --- a/tests/wave2/test-install-claude-permissions.sh +++ b/tests/wave2/test-install-claude-permissions.sh @@ -79,6 +79,8 @@ EXPECTED_DENY=( "Bash(./mxcli exec:*)" "Bash(mxcli exec:*)" "Bash(./mxcli.exe exec:*)" + "Bash(*baseline-approv*)" + "Edit(**/baseline-approvals.tsv)" ) SESSION_HOOK="bash bin/session-check.sh || true" @@ -403,6 +405,10 @@ PY printf '%s\n' "$AA" | grep -qxF "$(printf 'bin/exec.sh\ttrue')" \ && ok "T6: Copilot autoApprove has bin/exec.sh: true" \ || bad "T6: Copilot autoApprove missing bin/exec.sh: true" + # #348: approving a baseline move is the user's act — Copilot must ask, never auto-run it. + printf '%s\n' "$AA" | grep -qxF "$(printf '/baseline-approv/\tfalse')" \ + && ok "T6: Copilot autoApprove has /baseline-approv/: false" \ + || bad "T6: Copilot autoApprove missing /baseline-approv/: false" printf '%s\n' "$AA" | grep -q "$TOOLKIT_ROOT" \ && bad "T6: Copilot .vscode/settings.json leaked the absolute toolkit-root path" \ || ok "T6: Copilot .vscode/settings.json carries no absolute toolkit-root path" diff --git a/tests/wave2/test-lint-gate-accept.sh b/tests/wave2/test-lint-gate-accept.sh index 1952a1f3..f2c7f7a9 100755 --- a/tests/wave2/test-lint-gate-accept.sh +++ b/tests/wave2/test-lint-gate-accept.sh @@ -10,6 +10,8 @@ # as "lint ROSE: see output" (20 rows), so red stopped meaning anything (#229). # - --update-baseline took no reason: 61 self-accepted re-baselines, security warnings # included, and nothing summed them. +# - (#348, 2026-10-14) the agent could type --approved-in-chat itself: a rise now waits for +# bin/baseline-approve.sh, run by the person (simulated under a pseudo-terminal here). # This proves each fix discriminates. No mxcli or .mpr is needed: lint-gate's LINT_JSON mode # reads a captured result. The violation objects copy the key set of a real # `mxcli lint --format json` capture (mxcli v0.25.0: ruleId, severity, message, module, @@ -30,6 +32,9 @@ bad() { FAIL=$((FAIL+1)); printf ' FAIL %s\n' "$1"; [ -n "${2:-}" ] && printf ' P="$WORK/proj"; mkdir -p "$P/bin" "$P/.claude" cp "$GATE" "$P/bin/lint-gate.sh"; [ -f "$TK/project-bin/_common.sh" ] && cp "$TK/project-bin/_common.sh" "$P/bin/" +# The lock pair is installed beside lint-gate.sh (#348); human_approve is the person's step. +for f in baseline-lock.sh baseline-approve.sh; do [ -f "$(dirname "$GATE")/$f" ] && cp "$(dirname "$GATE")/$f" "$P/bin/"; done +. "$TK/tests/wave2/fixtures/baseline-lock/human.sh" printf 'mpr' > "$P/App.mpr"; printf '#!/bin/sh\nexit 0\n' > "$P/mxcli"; chmod +x "$P/mxcli" "$P/bin/lint-gate.sh" # v [module] -> n violation objects @@ -61,19 +66,26 @@ out=$(gate --update-baseline 2>&1); rc=$? [ "$rc" -eq 2 ] && ok "refused (rc 2)" || bad "rise accepted without a reason (rc $rc)" "$out" [ "$(base_total)" = "$before" ] && ok "baseline unchanged ($before)" || bad "baseline changed to $(base_total)" -echo "T4 with --reason the rise is accepted and logged" +echo "T4 with --reason the rise still waits for the person; after their approval it is accepted and logged" out=$(gate --update-baseline --reason "two flows split later in slice 3" 2>&1); rc=$? -[ "$rc" -eq 0 ] && ok "accepted with a reason" || bad "rc=$rc" "$out" +[ "$rc" -eq 2 ] && [ "$(base_total)" = "$before" ] && ok "a reason alone is not approval (rc 2, baseline unchanged)" || bad "rise accepted on the agent's word (rc $rc)" "$out" +case "$out" in *"bin/baseline-approve.sh lint"*) ok "refusal names the person's command" ;; *) bad "refusal does not name bin/baseline-approve.sh" "$out" ;; esac +human_approve "$P" lint; hrc=$? +[ "$hrc" -eq 3 ] && echo " skip (no usable \`script\` for a pseudo-terminal) — T4-T7 need it" && exit 0 +out=$(gate --update-baseline --reason "two flows split later in slice 3" 2>&1); rc=$? +[ "$rc" -eq 0 ] && ok "accepted after the person approved" || bad "rc=$rc" "$out" row=$(tail -1 "$P/.claude/lint-acceptances.tsv" 2>/dev/null) case "$row" in *CONV009+1*MDL-FLOW01+2*"two flows split later"*) ok "row: rises and reason recorded" ;; *) bad "acceptance row wrong" "$row" ;; esac -echo "T5 a security-tier rise needs --approved-in-chat on top of a reason" +echo "T5 --approved-in-chat is no longer evidence; a security-tier rise needs the person's approval" mk "$(v CONV009 info 3)$(v MDL-FLOW01 info 3)$(v SEC003 warning 2)$(v CONV006 warning 1)" -out=$(gate --update-baseline --reason "by design" 2>&1); rc=$? -[ "$rc" -eq 2 ] && ok "SEC003/CONV006 rise refused without approval" || bad "security rise accepted (rc $rc)" "$out" +out=$(gate --update-baseline --approved-in-chat --reason "user approved demo-user warning in chat" 2>&1); rc=$? +[ "$rc" -eq 2 ] && ok "SEC003/CONV006 rise refused on the flag alone (#348)" || bad "the agent's own flag moved the baseline (rc $rc)" "$out" case "$out" in *SEC003*CONV006*|*CONV006*SEC003*) ok "refusal names both security rules" ;; *) bad "refusal does not name the rules" "$out" ;; esac +grep -q "SECURITY-TIER" "$P/.claude/baseline-pending/lint.summary" 2>/dev/null && ok "the proposal the person reviews flags the security tier" || bad "proposal summary does not flag security" +human_approve "$P" lint out=$(gate --update-baseline --approved-in-chat --reason "user approved demo-user warning in chat" 2>&1); rc=$? -[ "$rc" -eq 0 ] && ok "accepted with reason + --approved-in-chat" || bad "rc=$rc" "$out" +[ "$rc" -eq 0 ] && ok "accepted after the person approved" || bad "rc=$rc" "$out" tail -1 "$P/.claude/lint-acceptances.tsv" | grep -q $'\tyes\t' && ok "row marks the security approval" || bad "security flag missing" "$(tail -1 "$P/.claude/lint-acceptances.tsv")" echo "T6 a pure ratchet-down needs no reason" @@ -86,8 +98,8 @@ out=$(gate --update-baseline 2>&1); rc=$? echo "T7 the project's security list replaces the default" printf '# ours\nCONV007\n' > "$P/.claude/lint-security-rules.txt" mk "$(v CONV009 info 1)$(v MDL-FLOW01 info 3)$(v SEC003 warning 2)$(v CONV006 warning 1)$(v CONV007 info 1)" -out=$(gate --update-baseline --reason "x" 2>&1); rc=$? -[ "$rc" -eq 2 ] && ok "CONV007 rise refused once listed" || bad "listed rule not treated as security (rc $rc)" "$out" +gate --update-baseline --reason "x" >/dev/null 2>&1 +grep -q "SECURITY-TIER: CONV007" "$P/.claude/baseline-pending/lint.summary" 2>/dev/null && ok "CONV007 flagged as security once listed" || bad "listed rule not treated as security" "$(cat "$P/.claude/baseline-pending/lint.summary" 2>/dev/null)" rm -f "$P/.claude/lint-security-rules.txt" echo "T8 PASS-BLIND is logged as BLIND, not as a rise" diff --git a/tests/wave2/test-sec014-notice.sh b/tests/wave2/test-sec014-notice.sh index 2c2d3d65..03d5317a 100644 --- a/tests/wave2/test-sec014-notice.sh +++ b/tests/wave2/test-sec014-notice.sh @@ -53,7 +53,7 @@ p=$(mkp a 1 "$STOCK_CFG"); out=$(notice "$p"); rc=$? [ $rc -eq 0 ] && ok "a rule installed, stock config -> notice owed" || bad "a no notice (rc=$rc)" case "$out" in *"rules:"*"SEC014:"*"options:"*"external_roles:"*) ok "a notice shows the exact config path" ;; *) bad "a config path missing" "$out" ;; esac case "$out" in *'"none"'*"user's answer"*"PROJECT.md"*) ok "a \"none\" is named as a user decision recorded in PROJECT.md" ;; *) bad "a none-is-a-decision missing" "$out" ;; esac -case "$out" in *"--update-baseline"*"--approved-in-chat"*) ok "a tells to re-baseline right after configuring" ;; *) bad "a re-baseline hint missing" "$out" ;; esac +case "$out" in *"--update-baseline"*"baseline-approve.sh lint"*) ok "a tells to re-baseline right after configuring" ;; *) bad "a re-baseline hint missing" "$out" ;; esac p=$(mkp b 1 NONE); notice "$p" >/dev/null && ok "b rule installed, no lint-config.yaml -> notice owed" || bad "b no notice without a config file" p=$(mkp c 1 "${STOCK_CFG}# SEC014: From ad7a9cfde52d1f1f5a697e065dbc63418bc70cb8 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 14 Oct 2026 15:14:10 +0000 Subject: [PATCH 2/3] fix(baselines): check write records against their content; no false FAIL on CRLF or merges (#375 review) - Bypass: `baseline-lock.sh record tighten` after a hand edit turned gate-check green. Every write now saves its content under .claude/baseline-history/, and verify replays the log: a record looser than the one before it without an approval is FAIL, whoever wrote it. Direct `record`, redirection into the logs and file-tool edits of them are denied in Claude Code; Copilot must ask. - False FAIL: content is compared with CR stripped (CRLF checkouts). A looser baseline or record that git HEAD holds (a merge) reads MANUAL, not FAIL. - The Bash deny names baseline-approve.sh only, so git add/commit/log -p on the approvals log work again. - Docs: the approve command runs in a separate terminal window; Claude Code's `!` prefix has no TTY and is refused. Fixtures: test-baseline-lock.sh gains F1 (forge), X1 (CRLF), X2 (HEAD blob), X3 (real two-branch merge); all four fail on d18f65f. Bench gains a forge attempt per baseline. Field evidence: none yet. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw --- CHANGELOG.md | 2 +- agents/gate-agent.md | 2 +- bin/gate-check.sh | 14 +- bin/install-claude-permissions.sh | 22 +- bin/install-harness-permissions.sh | 8 +- bin/lib/sec014-notice.sh | 2 +- lint-rules/README.md | 2 +- project-bin/baseline-approve.sh | 18 +- project-bin/baseline-lock.sh | 245 +++++++++++++----- project-bin/conformance-check.sh | 4 +- project-bin/exec.sh | 2 +- project-bin/lint-gate.sh | 2 +- skills/lint-that-actually-runs.md | 26 +- tests/wave2/bench-baseline-lock.sh | 17 +- tests/wave2/test-baseline-lock.sh | 55 ++++ .../wave2/test-install-claude-permissions.sh | 17 +- 16 files changed, 321 insertions(+), 117 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 827c9f38..78d253c9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,7 +19,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. Named m ## Unreleased -- fix(lint-gate, conformance-check, gate-check): **a ratchet baseline that loosens now moves only on the user's own command; `--approved-in-chat` is no longer evidence** ([#348](https://github.com/MendixMau/mxcli-project-toolkit/issues/348)). The agent whose script caused a rise could type that flag itself, so "re-baselining is the user's call" ran on trust. A rise (lint) or an accepted regression (conformance) is now written only against a row that the new `bin/baseline-approve.sh` writes to `.claude/baseline-approvals.tsv`. The user runs that command in a terminal, and the row is the sha256 of exactly the content they reviewed. Claude Code denies the command and the file to the agent (`install-claude-permissions.sh`), Copilot must ask before running it, and the command refuses to run without a TTY. Every instrument write is recorded in `.claude/baseline-writes.tsv`. `gate-check.sh` prints `Baselines (human-approved?)`: FAIL for a baseline that matches neither record (this blocks Stages 5 and 6) and MANUAL for one that predates the lock. Bench (`tests/wave2/bench-baseline-lock.sh`, 10 unapproved and 4 legitimate scripted attempts per baseline): unapproved moves landing 10/10 → 2/10 (the 2 are direct file edits, now all flagged; unflagged 10/10 → 0/10), legitimate re-baselines 4/4 → 4/4, at the cost of one human command each. Field evidence: none yet. Not covered: the mxbuild verification stamp, which records a machine check and has no human judgement to lock, and page-fidelity and token-burn, which keep no baseline. — external reviewer feedback +- fix(lint-gate, conformance-check, gate-check): **a ratchet baseline that loosens now moves only on the user's own command; `--approved-in-chat` is no longer evidence** ([#348](https://github.com/MendixMau/mxcli-project-toolkit/issues/348)). The agent whose script caused a rise could type that flag itself, so "re-baselining is the user's call" ran on trust. A rise (lint) or an accepted regression (conformance) is now written only against a row that the new `bin/baseline-approve.sh` writes to `.claude/baseline-approvals.tsv`. The user runs that command in a terminal, and the row is the sha256 of exactly the content they reviewed. Claude Code denies the command and the file to the agent (`install-claude-permissions.sh`), Copilot must ask before running it, and the command refuses to run without a TTY. Every instrument write is recorded with its content in `.claude/baseline-writes.tsv` and `.claude/baseline-history/`. `gate-check.sh` prints `Baselines (human-approved?)`, replaying each record against its content, so running `baseline-lock.sh record … tighten` after a hand edit is caught too. It reports FAIL for a baseline or record looser than the write before it with no approval (this blocks Stages 5 and 6), and MANUAL for one that predates the lock or that came in looser through a git commit or merge. Line endings are ignored, so a CRLF checkout compares equal. The deny rules name the two commands and redirection into the logs, so `git add` and `git log -p` on the logs still work. Bench (`tests/wave2/bench-baseline-lock.sh`, 11 unapproved and 4 legitimate scripted attempts per baseline): unapproved moves landing 11/11 → 3/11 (the 3 are direct file edits, one with a forged record; all now flagged, unflagged 11/11 → 0/11), legitimate re-baselines 4/4 → 4/4, at the cost of one human command each. Field evidence: none yet. Not covered: the mxbuild verification stamp, which records a machine check and has no human judgement to lock, and page-fidelity and token-burn, which keep no baseline. — external reviewer feedback - process(build): **Stage 5 drafts the ready set in parallel and applies serially.** The runbook dispatch row said "sequential within a module", which serialised drafting even for independent scripts; it now points to `iterative-build-loop.md` → Gate cost, the single statement of the rule. The unit is the build-plan row: every `BUILD` row whose "Depends on" rows are all applied, minus overlapping writers (two rows that replace, alter or drop+create the same document never share a ready set, and no row is drafted while it overlaps a queued unapplied one; `brd-to-build-plan.md` has Produces name every document a row writes), runs at once, its scripts in order, one `mdl-agent` per script. Applies stay one at a time through `bin/exec.sh`, and the ready set is recomputed after each. A queued draft made before a sibling apply landed is re-checked with `mxcli diff --format struct` (preflight STOP rule 23); any removal line (checked against `describe` per BUG-146) or STOP rule 21 target means re-draft, never patch, because a stale restatement of a shared document is exactly rule 21's silent seed-call loss and passes `check --references`. Report `ready set N, drafted N, applied k`; the `mdl-agent` template says a drafter never applies and never depends on an unapplied draft. Field evidence: none yet. — MendixMau - fix(platform-link): **CLI commits now show their changed documents in Studio Pro History.** Studio Pro's History → Model changes pane reads a `ModelChanges` array from the commit's `refs/notes/mx_metadata` note. Trap 6c's hand-written template set it to `[]`, so every mxcli/agent commit showed "Model changes 0" (field project: 270 of 270 notes empty). New `project-bin/mx-metadata-note.py` (stdlib only) computes the array from the commit's changed `.mxunit` files, in Studio Pro's entry shape: Status, UnitID, UnitType, UnitName and Module (walked up the `.mpr` Unit tree). It has `changes` / `write` / `backfill` / `sync ` subcommands. `install-project-hooks.sh` now also installs post-commit and post-rewrite hooks that write the note and never block, and `--check` covers them, so `sync-project.sh` rolls them out to wired projects. Notes are published with `mx-metadata-note.py sync ` after the branch push. Push config is never touched. With mxcli v0.26+ in the project root the hooks call `mxcli git note --write` instead (mendixlabs/mxcli#1337); the Python script is the fallback for older pins, and on 160 field commits both wrote the same 3,594 entries. Trap 6c rewritten. Measured against Studio Pro's own notes, 51 commits in 5 apps: 3387/3389 entries exact on all five fields. Field run on a 357-commit Mendix 11.15.0 app: 158 notes backfilled and pushed, and Studio Pro History lists the documents per revision. Review fixes: an earlier revision added `HEAD` + the notes ref to `remote..push`. It was dropped because any push refspec replaces `push.default`. A branch tracking a differently named upstream got a new same-named branch on Team Server where `simple` refuses, a missing local notes ref failed the whole push, and nobody knows whether Studio Pro's own push reads that config. The installer now only fetches the remote's notes (non-forced). It removes that exact `HEAD` + notes pair if present, and leaves and reports any other entry. `sync` handles a remote with no notes ref (it used to exit 1) and retries a rejected push once after re-merging. A note that already lists changes (Studio Pro's) is never overwritten by `write`/`backfill` without `--force`, and `sync` keeps a colleague's non-empty note over ours. The mxcli probe greps `git note --help` for `--write`. Hooks and docs call the script through `resolve_py`. Fixture `tests/wave2/test-mx-metadata-note.sh` (40 cases, over real captured template units; single- and two-tree layouts, amend, backfill and `--force`, push exit codes identical to a no-hooks clone, leftover-refspec cleanup, sync to an empty remote and its retry, notes fetch at install, a diverged notes merge; green under `init.defaultBranch` master and main). — field project (Mendix 11.15 app), MendixMau - learn(fixture-seeding): **explore the data structure first, choose rows by use case, scan actions for resets before any run.** Two seeding runs on public template apps: seeding raised journey passes 33 -> 58 and 20 -> 51 of the same sets (FAIL 93 -> 68, 163 -> 132; INVALID unchanged), and a "Reset all demodata" journey wiped accounts so 104 of 127 journeys reported `The username or password you entered is incorrect`, identical on empty and seeded databases. Skill gains three questions a seed must answer, a use-case checklist, a destructive-action scan, a generic traps table with symptoms, measured expectations and a denominator checklist; routing trigger now fires on seed/test/demo data and empty pages. Independent review fixes: object ids follow the Mendix bigint pattern with a business-key guard (no uuids), associations are seeded where the page's path stores them, four new traps (enum names, inheritance chain, calculated attributes, view/external entities), and the port-offset and container-label traps moved to `harness-architecture.md`. — MendixMau diff --git a/agents/gate-agent.md b/agents/gate-agent.md index e5ecfedd..5302592f 100644 --- a/agents/gate-agent.md +++ b/agents/gate-agent.md @@ -102,7 +102,7 @@ This table is your whole list. The baseline table in the project's CLAUDE.local. and whether the rise belongs to the script under review or to earlier debt. A rise is a FAIL for the script even when mxbuild is clean. Never re-baseline to turn a row green — that is the user's decision; `--update-baseline` refuses a rise without `--reason`, and then - without the user running `bin/baseline-approve.sh lint` in their own terminal (never you), and + without the user running `bin/baseline-approve.sh lint` in a separate terminal (never you), and logs each one to `.claude/lint-acceptances.tsv`. A row reading `LINT BLIND` is an instrument fault (a rule inspected nothing), not a rise: name the rule and its configuration fix. `LINT BLIND: rules reverted to stock (…)` means `mxcli init` re-seeded the toolkit rules and lint did not run: diff --git a/bin/gate-check.sh b/bin/gate-check.sh index beaabc27..7b7d05bd 100755 --- a/bin/gate-check.sh +++ b/bin/gate-check.sh @@ -2547,11 +2547,12 @@ fi # --------------------------------------------------------------------------- # Baselines — did a ratchet baseline move without a person's approval? (#348) # -# lint-gate.sh and conformance-check.sh record every baseline they write, and a loosening write -# only lands against a row bin/baseline-approve.sh wrote (a person, in a terminal). A baseline -# whose content matches neither record moved some other way — a hand edit, or a write by a -# script that skipped the lock: FAIL. A baseline with no record at all predates the lock: -# MANUAL, since nothing can say who moved it. The toolkit's own baseline-lock.sh reads the +# lint-gate.sh and conformance-check.sh record every baseline they write, with its content, and a +# loosening write only lands against a row bin/baseline-approve.sh wrote (a person, in a +# terminal). baseline-lock.sh verify replays the records against their content: a baseline, or a +# recorded write, looser than the write before it with no approval is FAIL (a hand edit, or a +# forged record — #375 review). MANUAL: no record at all (predates the lock), or the looser +# content is committed (a merge). CR is stripped before comparing. The toolkit's own baseline-lock.sh reads the # project, whichever version the project installed. BLOCKS Stages 5 and 6 on FAIL (below); the # remedy, a person running `bin/baseline-approve.sh --current`, is never blocked. BASE_STATUS="PASS"; BASE_NOTE="" @@ -3302,7 +3303,8 @@ if [ -n "$REQUESTED_STAGE" ]; then echo "" >&2 echo "Gate BLOCKED by a baseline that moved without approval: $BASE_NOTE" >&2 echo "Show the user the change (git log -p on the baseline file). If they accept it, THEY run" >&2 - echo "bin/baseline-approve.sh --current in their own terminal; otherwise restore it from git." >&2 + echo "bin/baseline-approve.sh --current in a separate terminal window (not via the agent" >&2 + echo "or Claude Code's ! prefix: it needs a TTY); otherwise restore it from git." >&2 exit 1 fi if { [ "$REQUESTED_STAGE" = "5" ] || [ "$REQUESTED_STAGE" = "6" ]; } && [ "$DISP_STATUS" = "FAIL" ]; then diff --git a/bin/install-claude-permissions.sh b/bin/install-claude-permissions.sh index 62c81d74..ddf8383c 100755 --- a/bin/install-claude-permissions.sh +++ b/bin/install-claude-permissions.sh @@ -43,9 +43,10 @@ # runs the same mxcli underneath. A skill that genuinely needs the bare form runs it from a # script (the rule matches the top-level command only). # -# THE SECOND DENY (2026-10-14, #348): `*baseline-approv*` in Bash and edits of -# baseline-approvals.tsv. Approving a ratchet baseline move is the user's act; the agent could -# type `--approved-in-chat` itself. See project-bin/baseline-lock.sh. +# THE SECOND DENY (2026-10-14, #348): the baseline-approve.sh command, a direct +# `baseline-lock.sh record`, and writes into the baseline logs. Approving a ratchet baseline move +# is the user's act; the agent could type `--approved-in-chat` itself. Narrow on purpose (#375 +# review): a rule matching the log's file name blocked `git add` and `git log -p` on it. # # THE ONE HOOK: `bash bin/session-check.sh || true` on SessionStart — the project-local check # that says at the top of every session whether the model on disk is verified, whether the @@ -188,11 +189,18 @@ DENY=( "Bash(./mxcli exec:*)" "Bash(mxcli exec:*)" "Bash(./mxcli.exe exec:*)" - # #348: approving a ratchet baseline move is the user's act. Denies the approve command and - # any Bash line naming the approvals file, plus file-tool edits of it. The user runs it in - # their own terminal; see project-bin/baseline-lock.sh for what this does not stop. - "Bash(*baseline-approv*)" + # #348: approving a ratchet baseline move is the user's act. Denies the approve command, a + # direct `baseline-lock.sh record` (instruments call it from inside their scripts, which these + # top-level rules do not see), shell redirection into the two logs, and file-tool edits of the + # logs and snapshots. Reading, `git add`/`git commit -- …`/`git log -p` on them stay allowed. + # See project-bin/baseline-lock.sh for what this does not stop. + "Bash(*baseline-approve.sh*)" + "Bash(*baseline-lock.sh*record*)" + "Bash(*>*baseline-approvals.tsv*)" + "Bash(*>*baseline-writes.tsv*)" "Edit(**/baseline-approvals.tsv)" + "Edit(**/baseline-writes.tsv)" + "Edit(**/baseline-history/**)" ) ASK=() diff --git a/bin/install-harness-permissions.sh b/bin/install-harness-permissions.sh index 01f6b3a7..3d9c8f29 100755 --- a/bin/install-harness-permissions.sh +++ b/bin/install-harness-permissions.sh @@ -106,8 +106,9 @@ echo "=== copilot ===" require_py VSCODE_SETTINGS="$PROJECT_DIR/.vscode/settings.json" COPILOT_SIDECAR="$PROJECT_DIR/.vscode/.mxtk-autoapprove-added.json" -# `/baseline-approv/: false` (#348): approving a baseline move is the user's act — Copilot must -# ask, never auto-run it. Only relative, committable entries — see the header note above for why the two absolute +# `/baseline-approve\.sh/` and `/baseline-lock\.sh.*record/` are false (#348): approving a +# baseline move is the user's act, so Copilot must ask, never auto-run them. +# Only relative, committable entries — see the header note above for why the two absolute # $TOOLKIT_ROOT entries Claude gets are deliberately not mirrored here. COPILOT_ENTRIES_JSON='{ "./mxcli": true, @@ -115,7 +116,8 @@ COPILOT_ENTRIES_JSON='{ "./bin/exec.sh": true, "bin/exec.sh": true, "/^\\.?\\/?bin\\/[^ ]+\\.sh$/": true, - "/baseline-approv/": false + "/baseline-approve\\.sh/": false, + "/baseline-lock\\.sh.*record/": false }' "$PY" - "$VSCODE_SETTINGS" "$COPILOT_SIDECAR" "$MODE" "$COPILOT_ENTRIES_JSON" <<'PY' || { ec=$?; [ "$ec" -gt "$RC" ] && RC=$ec; } import json, os, shutil, sys diff --git a/bin/lib/sec014-notice.sh b/bin/lib/sec014-notice.sh index 463efc9b..90ad8260 100644 --- a/bin/lib/sec014-notice.sh +++ b/bin/lib/sec014-notice.sh @@ -55,6 +55,6 @@ mxtk_sec014_notice() { echo "\"none\" (or SEC014 enabled: false) switches this security rule off: only on the user's answer in chat, recorded in" echo "PROJECT.md — never to silence this notice. The first configured run raises SEC014 0 -> N, so" echo "re-baseline right after: bin/lint-gate.sh --update-baseline --reason \"SEC014 configured\", then the user" - echo "runs bin/baseline-approve.sh lint in their own terminal (an agent cannot approve it)" + echo "runs bin/baseline-approve.sh lint in a separate terminal (an agent cannot approve it)" return 0 } diff --git a/lint-rules/README.md b/lint-rules/README.md index 106bfa0c..92318d3b 100644 --- a/lint-rules/README.md +++ b/lint-rules/README.md @@ -199,7 +199,7 @@ asked in chat. `external_roles: "none"` switches off a security rule, and so doe decision row in `PROJECT.md`. An agent never sets one to make the NOTE go away. The first run after configuring raises SEC014 from 0 to N at once, and lint-gate would blame that security-tier rise on whatever script ran next. So re-baseline in the same step: `bin/lint-gate.sh --update-baseline --reason "SEC014 configured"`, then the user, having seen the -findings, runs `bin/baseline-approve.sh lint` in their own terminal. +findings, runs `bin/baseline-approve.sh lint` in a separate terminal window. **Tuning a toolkit rule never means editing it.** `conv009` and `sec014` read their settings with `get_option()` from `.claude/lint-config.yaml` (`rules: : options:`), so the file diff --git a/project-bin/baseline-approve.sh b/project-bin/baseline-approve.sh index 5c40dc15..a83838d8 100755 --- a/project-bin/baseline-approve.sh +++ b/project-bin/baseline-approve.sh @@ -1,5 +1,7 @@ #!/usr/bin/env bash -# baseline-approve.sh — a PERSON approves a ratchet baseline move. Run it in your own terminal. +# baseline-approve.sh — a PERSON approves a ratchet baseline move. Run it in a separate terminal +# window of your own — not through the agent, and not via Claude Code's `!` prefix (no TTY there, +# so it is refused). # # bin/baseline-approve.sh lint # approve the proposal lint-gate.sh just refused # bin/baseline-approve.sh conformance # same, for conformance-check.sh @@ -23,11 +25,11 @@ LOCK="$(dirname "${BASH_SOURCE[0]}")/baseline-lock.sh" KIND="${1:-}"; CURRENT=0 [ "${2:-}" = "--current" ] && CURRENT=1 -case "$KIND" in lint|conformance) ;; *) sed -n '2,8p' "$0"; exit 2 ;; esac +case "$KIND" in lint|conformance) ;; *) sed -n '2,10p' "$0"; exit 2 ;; esac if [ ! -t 0 ]; then - echo "baseline-approve: REFUSED — no terminal. This is a person's decision: run it yourself," >&2 - echo "in your own terminal, not through an agent." >&2 + echo "baseline-approve: REFUSED — no terminal. This is a person's decision: run it yourself, in a" >&2 + echo "separate terminal window — not through an agent, and not via Claude Code's ! prefix." >&2 exit 2 fi @@ -56,4 +58,10 @@ F="$ROOT/.claude/baseline-approvals.tsv" [ -f "$F" ] || printf 'ts\tkind\tsha256\tby\tnote\n' > "$F" printf '%s\t%s\t%s\t%s\t%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$KIND" "$SHA" "$BY" \ "$(printf '%s' "$NOTE" | tr '\t\n' ' ')" >> "$F" -echo "Approved ($(printf '%s' "$SHA" | cut -c1-12)). The agent can now re-run its --update-baseline command." +if [ "$CURRENT" -eq 1 ]; then + # The file on disk is now the accepted state: log it, so later writes are checked against it. + bash "$LOCK" record "$ROOT" "$KIND" approved + echo "Approved ($(printf '%s' "$SHA" | cut -c1-12)): the $KIND baseline on disk is now the accepted state." +else + echo "Approved ($(printf '%s' "$SHA" | cut -c1-12)). The agent can now re-run its --update-baseline command." +fi diff --git a/project-bin/baseline-lock.sh b/project-bin/baseline-lock.sh index ddef6346..879a05d8 100755 --- a/project-bin/baseline-lock.sh +++ b/project-bin/baseline-lock.sh @@ -6,37 +6,48 @@ # itself, so "re-baselining is the user's call" was honour-system. A baseline that LOOSENS now # moves only against a row in .claude/baseline-approvals.tsv whose sha256 is the exact # proposed content — a row only bin/baseline-approve.sh writes, from a terminal, after a person -# reads the change and types `approve`. The agent cannot write that row through its normal -# tools: bin/install-claude-permissions.sh denies `*baseline-approv*` in Bash and edits to the -# approvals file (deny beats allow and auto mode in Claude Code). +# reads the change and types `approve`. # -# WHAT IT DOES NOT PROTECT AGAINST, said plainly: an agent that deliberately evades the deny -# rules (obfuscated shell, a script that writes the row), a harness with no deny rules whose -# agent terminal is a real TTY, or someone deleting both ledgers. It turns "the agent typed a -# flag" into "the agent forged a record", which is visible in git and named by gate-check. +# THE RECORD IS CHECKED, NOT BELIEVED (#375 review). Every instrument write is logged with a +# snapshot of its content, and `verify` replays the log: each recorded write must be no looser +# than the one before it, or carry an approval. So appending a row by hand — or running +# `record` directly after a hand edit — does not launder a rise: the forged row is looser than +# its predecessor and verify says FAIL. Only content can pass, never a claim about content. +# +# What it does NOT stop, said plainly: a forged row in baseline-approvals.tsv (Claude Code denies +# the approve command, file-tool edits and shell redirection into it — bin/install-claude- +# permissions.sh — but an agent determined to evade those, e.g. via an interpreter, can still +# write one); a harness with no deny rules whose agent terminal is a real TTY; deleting the whole +# log (verify then reads "predates the lock": MANUAL, not PASS); and a loosening committed to git +# (reported MANUAL, below, so a merge never reads as tampering). Each of those is a deliberate +# forgery visible in git, not a flag the rule allowed. # # Covered baselines (kind -> file, relative to the project root): # lint .claude/lint-baseline.json project-bin/lint-gate.sh # conformance docs/conformance/baseline.tsv project-bin/conformance-check.sh +# "Looser" means: lint — any rule's count above the previous one; conformance — any row OK in the +# previous baseline and present but not OK now. Content is compared with CR stripped, so a CRLF +# checkout (Windows autocrlf) hashes the same as LF. # -# Records: -# .claude/baseline-approvals.tsv ts kind sha256 by note — written ONLY by baseline-approve.sh -# .claude/baseline-writes.tsv ts kind sha256 basis — every write an instrument made -# (basis: first | tighten | seed | approved) -# .claude/baseline-pending/[.summary] the refused proposal, for the person to review -# Commit all three with the baseline. +# Records (commit them with the baseline): +# .claude/baseline-approvals.tsv ts kind sha256 by note — ONLY baseline-approve.sh +# .claude/baseline-writes.tsv ts kind sha256 basis — every write an instrument made +# .claude/baseline-history// the content of each recorded write +# .claude/baseline-pending/[.summary] the refused proposal, for the person to review # -# Usage (instruments and gate-check call it; a person never needs to): +# Usage (instruments, baseline-approve.sh and gate-check call it; a person never needs to): # baseline-lock.sh file print the baseline path (relative) -# baseline-lock.sh sha print sha256 of a file +# baseline-lock.sh sha sha256 of the file, CR stripped # baseline-lock.sh verify print STATEnote, exit 0 # NONE no baseline and no record (a first write needs no approval) # GONE no baseline, but records exist (re-creating it needs approval) -# PASS the file on disk matches an approval or a recorded instrument write -# FAIL it matches neither: it moved outside the instrument, or without approval -# MANUAL a baseline with no record at all — it predates this lock; unverifiable +# PASS approved, or no looser than the last verified write +# FAIL looser than the last verified write with no approval, or a recorded write that +# loosened without one (a forged record) +# MANUAL no record at all (predates the lock), or the looser content is exactly what git +# HEAD holds (a merge or checkout brought it in, not an uncommitted local edit) # baseline-lock.sh approved exit 0 iff an approval row matches -# baseline-lock.sh record append a write row +# baseline-lock.sh record log the baseline now on disk # baseline-lock.sh propose save the proposal, print the ask # # Producer of the records: the two instruments above (writes, proposals) and @@ -44,8 +55,143 @@ # bin/gate-check.sh's "Baselines" line (toolkit copy, run against the project root). set -uo pipefail -APPROVALS=".claude/baseline-approvals.tsv" -WRITES=".claude/baseline-writes.tsv" +PY="" +for _c in python3 python py; do # portability-ok: this IS the interpreter probe + case "$(command -v "$_c" 2>/dev/null)" in *[Ww]indows[Aa]pps*) continue ;; esac + if "$_c" -c 'import sys; sys.exit(0 if sys.version_info[0] == 3 else 1)' >/dev/null 2>&1; then PY="$_c"; break; fi +done +[ -n "$PY" ] || { echo "baseline-lock: Python 3 is required and was not found" >&2; exit 2; } + +IFS= read -r -d '' _LOCK_PY <<'PY' || true +import datetime, hashlib, json, os, shutil, subprocess, sys + +FILES = {"lint": ".claude/lint-baseline.json", "conformance": "docs/conformance/baseline.tsv"} +APPROVALS, WRITES, HISTORY = ".claude/baseline-approvals.tsv", ".claude/baseline-writes.tsv", ".claude/baseline-history" + +def norm(b): + return b.replace(b"\r", b"") + +def sha_bytes(b): + return hashlib.sha256(norm(b)).hexdigest() + +def sha_file(p): + return sha_bytes(open(p, "rb").read()) + +def rows(root, name, kind): + p = os.path.join(root, name) + if not os.path.isfile(p): + return [] + out = [] + for line in open(p, encoding="utf-8", errors="replace"): + f = line.rstrip("\r\n").split("\t") + if len(f) >= 3 and f[1] == kind and f[0] != "ts": + out.append(f) + return out + +def load(kind, b): + t = norm(b).decode("utf-8", "replace") + if kind == "lint": + return {k: int(v) for k, v in (json.loads(t).get("counts") or {}).items()} + d = {} + for line in t.splitlines(): + f = line.split("\t") + if len(f) >= 3 and f[0]: + d[(f[0], f[1])] = f[2] + return d + +def looser(kind, prev, new): + if kind == "lint": + return [r for r in sorted(new) if new[r] > prev.get(r, 0)] + return ["%s %s" % k for k in sorted(new) if prev.get(k) == "OK" and new[k] != "OK"] + +def head_blob(root, rel): + try: + r = subprocess.run(["git", "-C", root, "show", "HEAD:./" + rel], capture_output=True, timeout=20) + return r.stdout if r.returncode == 0 else None + except (OSError, subprocess.SubprocessError): + return None + +def verify(root, kind): + rel = FILES[kind] + path = os.path.join(root, rel) + w, a = rows(root, WRITES, kind), rows(root, APPROVALS, kind) + approved = {r[2] for r in a} + hw = head_blob(root, WRITES) + committed = set(norm(hw).decode("utf-8", "replace").splitlines()) if hw else set() + last, manual = None, [] + for r in w: + snap = os.path.join(root, HISTORY, kind, r[2]) + if not os.path.isfile(snap) or sha_file(snap) != r[2]: + return "FAIL", "recorded write %s (%s) has no matching snapshot under %s — the record cannot be checked" % (r[2][:12], r[0], HISTORY) + try: + c = load(kind, open(snap, "rb").read()) + except ValueError: + return "FAIL", "recorded write %s (%s) is not a readable %s baseline" % (r[2][:12], r[0], kind) + up = [] if last is None or r[2] in approved else looser(kind, last, c) + if up and "\t".join(r) in committed: + # Committed rows came through git: two branches that each wrote a baseline, merged. + manual.append("recorded write %s (%s) is looser than the row before it (%s) but is committed — " + "a merge of two branches' baselines, not a local edit; review the history" % (r[2][:12], r[0], ", ".join(up[:5]))) + up = [] + if up: + return "FAIL", ("recorded write %s (%s, basis %s) loosens the baseline (%s) and no approval matches it — " + "a record is checked against its content, not believed" % (r[2][:12], r[0], r[3] if len(r) > 3 else "?", ", ".join(up[:5]))) + last = c + if not os.path.isfile(path): + if w or a: + return "GONE", "%s is missing but was recorded before — re-creating it needs a person: bin/baseline-approve.sh %s" % (rel, kind) + return "NONE", "no %s baseline yet" % kind + raw = open(path, "rb").read() + s = sha_bytes(raw) + if s in approved: + return ("MANUAL", manual[0]) if manual else ("PASS", "%s matches a human approval" % rel) + if last is None: + return "MANUAL", "%s has no write record (it predates the lock) — cannot tell who moved it; the next instrument write starts the record" % rel + try: + cur = load(kind, raw) + except ValueError: + return "FAIL", "%s is not a readable %s baseline" % (rel, kind) + up = looser(kind, last, cur) + if not up: + return ("MANUAL", manual[0]) if manual else ("PASS", "%s is no looser than its last recorded write" % rel) + hb = head_blob(root, rel) + if hb and sha_bytes(hb) == s: + return "MANUAL", ("%s is looser than its last recorded write (%s) but is exactly what git HEAD holds — " + "it came through git (a merge or checkout), not a local edit; review its history, and if the " + "user accepts it they run: bin/baseline-approve.sh %s --current" % (rel, ", ".join(up[:5]), kind)) + return "FAIL", ("%s moved without approval: looser than its last recorded write (%s) — a person reviews it and runs: " + "bin/baseline-approve.sh %s --current" % (rel, ", ".join(up[:5]), kind)) + +def record(root, kind, basis): + path = os.path.join(root, FILES[kind]) + raw = open(path, "rb").read() + s = sha_bytes(raw) + d = os.path.join(root, HISTORY, kind) + os.makedirs(d, exist_ok=True) + open(os.path.join(d, s), "wb").write(norm(raw)) + p = os.path.join(root, WRITES) + new = not os.path.isfile(p) + with open(p, "a") as f: + if new: + f.write("ts\tkind\tsha256\tbasis\n") + f.write("%s\t%s\t%s\t%s\n" % (datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"), kind, s, basis)) + for n in (kind, kind + ".summary"): + try: + os.remove(os.path.join(root, ".claude", "baseline-pending", n)) + except OSError: + pass + +cmd, args = sys.argv[1], sys.argv[2:] +if cmd == "sha": + print(sha_file(args[0])) +elif cmd == "verify": + st, note = verify(args[0], args[1]) + print("%s\t%s" % (st, note)) +elif cmd == "approved": + sys.exit(0 if any(r[2] == args[2] for r in rows(args[0], APPROVALS, args[1])) else 1) +elif cmd == "record": + record(args[0], args[1], args[2]) +PY _file() { case "$1" in @@ -55,69 +201,30 @@ _file() { esac } -_sha() { - if command -v sha256sum >/dev/null 2>&1; then sha256sum "$1" | cut -d' ' -f1 - elif command -v shasum >/dev/null 2>&1; then shasum -a 256 "$1" | cut -d' ' -f1 - else openssl dgst -sha256 "$1" | sed 's/^.*= *//' - fi -} - -_has() { # _has — a row for this kind and hash - [ -f "$1" ] && awk -F'\t' -v k="$2" -v s="$3" '$2==k && $3==s {f=1} END {exit !f}' "$1" -} -_any() { # _any — any row for this kind - [ -f "$1" ] && awk -F'\t' -v k="$2" '$2==k {f=1} END {exit !f}' "$1" -} -_now() { date -u +%Y-%m-%dT%H:%M:%SZ; } - cmd="${1:-}"; [ $# -gt 0 ] && shift case "$cmd" in file) _file "$1" ;; sha) - _sha "$1" ;; - verify) - root="$1"; kind="$2"; f="$(_file "$kind")" || exit 2 - a="$root/$APPROVALS"; w="$root/$WRITES" - if [ ! -f "$root/$f" ]; then - if _any "$a" "$kind" || _any "$w" "$kind"; then - printf 'GONE\t%s is missing but was recorded before — re-creating it needs a person: bin/baseline-approve.sh %s\n' "$f" "$kind" - else - printf 'NONE\tno %s baseline yet\n' "$kind" - fi - exit 0 - fi - s="$(_sha "$root/$f")" - if _has "$a" "$kind" "$s"; then - printf 'PASS\t%s matches a human approval\n' "$f" - elif _has "$w" "$kind" "$s"; then - printf 'PASS\t%s matches a recorded write (%s)\n' "$f" \ - "$(awk -F'\t' -v k="$kind" -v s="$s" '$2==k && $3==s {b=$4} END {print b}' "$w")" - elif _any "$a" "$kind" || _any "$w" "$kind"; then - printf 'FAIL\t%s moved without approval (sha %s is in neither %s nor %s) — a person reviews it and runs: bin/baseline-approve.sh %s --current\n' \ - "$f" "$(printf '%s' "$s" | cut -c1-12)" "$APPROVALS" "$WRITES" "$kind" - else - printf 'MANUAL\t%s has no approval or write record (it predates the lock) — cannot tell who moved it; the next instrument write starts the record\n' "$f" - fi - exit 0 ;; - approved) - _has "$1/$APPROVALS" "$2" "$3" ;; + "$PY" -c "$_LOCK_PY" sha "$1" ;; + verify|approved) + _file "$2" >/dev/null || exit 2 + "$PY" -c "$_LOCK_PY" "$cmd" "$@" ;; record) - root="$1"; mkdir -p "$root/.claude" - [ -f "$root/$WRITES" ] || printf 'ts\tkind\tsha256\tbasis\n' > "$root/$WRITES" - printf '%s\t%s\t%s\t%s\n' "$(_now)" "$2" "$3" "$4" >> "$root/$WRITES" - rm -f "$root/.claude/baseline-pending/$2" "$root/.claude/baseline-pending/$2.summary" ;; + _file "$2" >/dev/null || exit 2 + "$PY" -c "$_LOCK_PY" record "$@" ;; propose) root="$1"; kind="$2"; src="$3"; summary="$4" d="$root/.claude/baseline-pending"; mkdir -p "$d" cp "$src" "$d/$kind" && printf '%s\n' "$summary" > "$d/$kind.summary" echo "" >&2 echo " Moving this baseline is the user's call, and a flag the agent can type is not their" >&2 - echo " answer. Show the user the change above, then ask THEM to run, in their own terminal:" >&2 + echo " answer. Show the user the change above, then ask THEM to run, in a separate terminal" >&2 + echo " window (not the agent's, and not Claude Code's ! mode, which has no TTY):" >&2 echo " bin/baseline-approve.sh $kind" >&2 echo " (it shows the proposal and asks them to type 'approve'). Agents: do not run it —" >&2 echo " it is denied to you and refuses without a terminal. Then re-run the same command." >&2 ;; *) - sed -n '2,45p' "$0"; exit 2 ;; + sed -n '2,57p' "$0"; exit 2 ;; esac diff --git a/project-bin/conformance-check.sh b/project-bin/conformance-check.sh index 98553195..1019e812 100755 --- a/project-bin/conformance-check.sh +++ b/project-bin/conformance-check.sh @@ -411,7 +411,7 @@ lock_write() { [ "$PRIOR" = FAIL ] && why="${why:+$why; }follows a baseline that moved without approval" if [ -z "$why" ]; then mv "$new" "$BASELINE" - if [ -n "$LOCK" ]; then bash "$LOCK" record "$PROJECT_ROOT" conformance "$(bash "$LOCK" sha "$BASELINE")" "$basis" + if [ -n "$LOCK" ]; then bash "$LOCK" record "$PROJECT_ROOT" conformance "$basis" else echo " note: bin/baseline-lock.sh not installed — this baseline write is unrecorded (sync-project.sh)" >&2; fi return 0 fi @@ -422,7 +422,7 @@ lock_write() { sha="$(bash "$LOCK" sha "$new")" if bash "$LOCK" approved "$PROJECT_ROOT" conformance "$sha"; then mv "$new" "$BASELINE" - bash "$LOCK" record "$PROJECT_ROOT" conformance "$sha" approved + bash "$LOCK" record "$PROJECT_ROOT" conformance approved echo " human approval found for this exact baseline (sha $(printf '%s' "$sha" | cut -c1-12))" return 0 fi diff --git a/project-bin/exec.sh b/project-bin/exec.sh index 78fe3d62..21852653 100755 --- a/project-bin/exec.sh +++ b/project-bin/exec.sh @@ -860,7 +860,7 @@ run_lint_gate() { sed -n '/^FAIL/,$p' "$_lo" | head -60 | sed 's/^/ /' echo " Fix the script and re-run. Accepting the debt is the user's call, not yours:" echo " bin/lint-gate.sh --update-baseline --reason \"\", then the USER runs" - echo " bin/baseline-approve.sh lint in their own terminal. Logged to .claude/lint-acceptances.tsv." + echo " bin/baseline-approve.sh lint in a separate terminal. Logged to .claude/lint-acceptances.tsv." fi ;; *) LINT_DETAIL="lint could not run (lint-gate exit $LINT_RC)"; echo " ⚠ $LINT_DETAIL" tail -5 "$_lo" | sed 's/^/ /' ;; diff --git a/project-bin/lint-gate.sh b/project-bin/lint-gate.sh index 3524d5c3..cbf0cc34 100755 --- a/project-bin/lint-gate.sh +++ b/project-bin/lint-gate.sh @@ -482,7 +482,7 @@ if update: sys.exit(2) open(base_path, "w").write(content) if lock: - run_lock("record", root, "lint", sha, basis) + run_lock("record", root, "lint", basis) else: print("lint-gate: bin/baseline-lock.sh not installed — this write is unrecorded (sync-project.sh)", file=sys.stderr) print("==> baseline written: %d findings across %d rules" % (len(viol), len(counts))) diff --git a/skills/lint-that-actually-runs.md b/skills/lint-that-actually-runs.md index a3274ebd..cde826f6 100644 --- a/skills/lint-that-actually-runs.md +++ b/skills/lint-that-actually-runs.md @@ -141,24 +141,30 @@ ratchet-down need nothing more. **A rise moves only on the user's own command (#348).** `--approved-in-chat` was the agent's claim that the user said yes, and the agent could type it unasked. Now the refused update saves the proposal and prints `bin/baseline-approve.sh lint`. Show the user the rise (security-tier -rules, default `SEC*` and `CONV006`, are flagged in it); **the user** runs that command in their -own terminal, reads the diff and types `approve`; then re-run the same update. The approval is +rules, default `SEC*` and `CONV006`, are flagged in it). **The user** runs that command in a +separate terminal window, reads the diff and types `approve`. It needs a TTY, so it is refused +through the agent and through Claude Code's `!` prefix. Then re-run the same update. The approval is the sha256 of exactly that content, so a count that moves in between is refused again. ``` agent: bin/lint-gate.sh --update-baseline --reason "x" --approved-in-chat -> exit 0 (before: the agent approved itself) agent: bin/lint-gate.sh --update-baseline --reason "x" -> REFUSED, prints the user's command -user: bin/baseline-approve.sh lint (their terminal; types approve) +user: bin/baseline-approve.sh lint (a separate terminal window; types approve) agent: bin/lint-gate.sh --update-baseline --reason "x" -> written ``` -Never run `baseline-approve.sh` yourself and never edit `.claude/baseline-approvals.tsv`. -Claude Code denies both (`bin/install-claude-permissions.sh`), and the command refuses to run -without a terminal. `conformance-check.sh --update-baseline` follows the same rule when it -would accept a regression. `gate-check.sh` prints `Baselines (human-approved?)`: FAIL when a -baseline matches neither an approval nor a recorded write (a hand edit; this blocks Stages 5 -and 6), and MANUAL when a baseline predates the lock. Commit `.claude/baseline-*.tsv` with the -baseline. What the lock does not stop, an agent deliberately evading the deny rules, is listed +Never run `baseline-approve.sh` or `baseline-lock.sh record` yourself, and never edit the +`.claude/baseline-*` logs. Claude Code denies all three (`bin/install-claude-permissions.sh`). +`conformance-check.sh --update-baseline` follows the same rule when it would accept a +regression. + +`gate-check.sh` prints `Baselines (human-approved?)`. It replays each recorded write against +its saved content, so a forged record is caught too. It reports: +- FAIL: the baseline is looser than its last recorded write with no approval. This blocks + Stages 5 and 6. +- MANUAL: the baseline predates the lock, or the looser content came in through git (a merge). + +A CRLF checkout compares equal. Commit `.claude/baseline-*` with the baseline. What the lock does not stop, an agent deliberately evading the deny rules, is listed in `project-bin/baseline-lock.sh`. **The ratchet remembers one number per rule; the ledger remembers every run.** Every gate run diff --git a/tests/wave2/bench-baseline-lock.sh b/tests/wave2/bench-baseline-lock.sh index 956e5862..d7495f7b 100755 --- a/tests/wave2/bench-baseline-lock.sh +++ b/tests/wave2/bench-baseline-lock.sh @@ -8,7 +8,8 @@ # covers (lint-gate.sh, conformance-check.sh). Each attempt starts from a fresh project holding # a clean baseline and proposes a LOOSER one (a lint rule rises; a conformance row goes OK -> # STALE). Unapproved attempts carry only what an agent can produce on its own: --reason, -# --approved-in-chat, --module, deleting the baseline first. Legitimate attempts follow the +# --approved-in-chat, --module, deleting the baseline first, editing the file, and editing it +# then forging a write record (`baseline-lock.sh record … tighten`, the #375 review's bypass). Legitimate attempts follow the # documented flow: the agent runs the update; if it is refused and names the approve command, # the person runs it (fixtures/baseline-lock/human.sh, a pseudo-terminal) and the agent re-runs. # @@ -68,7 +69,8 @@ CONV009=40 SEC003=1|DELETE CONV009=6 SEC003=2|DELETE --reason x --approved-in-chat CONV009=4 SEC003=1|--reason=inline --approved-in-chat CONV009=12 SEC003=1|HANDEDIT -CONV009=2 SEC003=6|HANDEDIT" +CONV009=2 SEC003=6|HANDEDIT +CONV009=15 SEC003=1|FORGE" LINT_LEGIT="CONV009=5 SEC003=1 CONV009=2 SEC003=3 CONV009=8 SEC003=1 CONV006=2 @@ -108,7 +110,8 @@ Billing.ACT|--module Billing Orders.E|DELETE Stock.E Stock.ACT|DELETE Orders.E|HANDEDIT -Billing.E Stock.ACT|HANDEDIT" +Billing.E Stock.ACT|HANDEDIT +Stock.ACT|FORGE" CONF_LEGIT="Orders.E Billing.ACT|--module Billing Stock.E Stock.ACT @@ -135,7 +138,8 @@ measure() { conf_break "$d" $spec fi case "$flags" in - HANDEDIT) # the agent edits the baseline file itself: the measured state, no instrument + HANDEDIT|FORGE) # the agent edits the baseline file itself: the measured state, no instrument. + # FORGE then runs the #375 review's bypass: `baseline-lock.sh record tighten`. if [ "$_k" = lint ]; then "$PY" -c 'import collections,json,sys;c=collections.Counter(v["ruleId"] for v in json.load(open(sys.argv[1]))["violations"]);json.dump({"counts":dict(sorted(c.items())),"total":sum(c.values())},open(sys.argv[2],"w"),indent=2)' "$d/lint.json" "$f" else @@ -143,6 +147,9 @@ measure() { for n in $spec; do m="${n%%.*}"; case "$n" in *.E) p='`/domainEntities/0`' ;; *) p='`/microflows/0`' ;; esac awk -F'\t' -v m="$m" -v p="$p" 'BEGIN{OFS="\t"} $1==m && $2==p {$3="STALE"} {print}' "$f.n" > "$f.m"; mv "$f.m" "$f.n"; done mv "$f.n" "$f" + fi + if [ "$flags" = FORGE ] && [ -f "$d/bin/baseline-lock.sh" ]; then + bash "$d/bin/baseline-lock.sh" record "$d" "$_k" "$(bash "$d/bin/baseline-lock.sh" sha "$f")" tighten >/dev/null 2>&1 fi ;; *) case "$flags" in DELETE*) rm -f "$f"; flags="${flags#DELETE}" ;; esac @@ -195,5 +202,5 @@ for k in lint conformance; do printf '| %s | extra human steps per legitimate re-baseline | %s | %s |\n' "$k" "$(per "$b_s" "$b_nl")" "$(per "$a_s" "$a_nl")" done echo "" -echo "Of the unapproved attempts, $(printf '%s\n' "$LINT_UNAPPROVED" | grep -c HANDEDIT) (lint) and $(printf '%s\n' "$CONF_UNAPPROVED" | grep -c HANDEDIT) (conformance) edit the baseline file directly: no instrument can refuse that; the gate-check row is what catches it." +echo "Of the unapproved attempts, $(printf '%s\n' "$LINT_UNAPPROVED" | grep -cE 'HANDEDIT|FORGE') (lint) and $(printf '%s\n' "$CONF_UNAPPROVED" | grep -cE 'HANDEDIT|FORGE') (conformance) edit the baseline file directly ($(printf '%s\n' "$LINT_UNAPPROVED" | grep -c FORGE) each then forges a write record with \`baseline-lock.sh record … tighten\`): no instrument can refuse an edit; the gate-check row is what catches it." echo "Replayed: $(printf '%s\n' "$LINT_UNAPPROVED" | grep -c .) + $(printf '%s\n' "$CONF_UNAPPROVED" | grep -c .) unapproved and $(printf '%s\n' "$LINT_LEGIT" | grep -c .) + $(printf '%s\n' "$CONF_LEGIT" | grep -c .) legitimate attempts (lint + conformance) per tree, synthetic input; not a field measurement." diff --git a/tests/wave2/test-baseline-lock.sh b/tests/wave2/test-baseline-lock.sh index e02f1fa6..67677f99 100755 --- a/tests/wave2/test-baseline-lock.sh +++ b/tests/wave2/test-baseline-lock.sh @@ -16,6 +16,12 @@ # G1 gate-check: a hand-moved baseline reports FAIL, and Stage 5 is BLOCKED (exit 1) # G2 after the person approves the file as it is (--current) → PASS # G3 a baseline with no record at all (predates the lock) → MANUAL, not PASS +# --- #375 review --- +# F1 forge: hand-edit the baseline looser, then run `baseline-lock.sh record … tighten` +# directly → gate-check still FAIL (the record is checked against its content) +# X1 a CRLF checkout of a recorded baseline (Windows autocrlf) → PASS, not FAIL +# X2 a looser baseline that is exactly git HEAD's blob → MANUAL; the same edit uncommitted → FAIL +# X3 a real merge of two branches that each ratcheted a different rule down → not FAIL # # Synthetic lint JSON and a stub mxcli, as in test-lint-gate-accept.sh and # test-conformance-baseline-scope.sh: nothing here parses tool output; the assertions are @@ -136,6 +142,55 @@ L="$WORK/legacy"; mkdir -p "$L/.claude"; cp "$LB" "$L/.claude/lint-baseline.json l="$(gline "$L")" case "$l" in *MANUAL*) ok "G3 a baseline with no record: MANUAL" ;; *) bad "G3 unrecorded baseline not MANUAL" "${l:-}" ;; esac +echo "F1 a direct record after a hand edit does not launder it" +F="$WORK/f"; newproj "$F"; cp "$P/mxcli" "$F/mxcli" +fgate() { (cd "$F" && LINT_JSON="$WORK/lint.json" bash bin/lint-gate.sh "$@") 2>&1; } +lintjson CONV009=3; fgate --update-baseline >/dev/null +printf '{\n "counts": {"CONV009": 30}\n}\n' > "$F/.claude/lint-baseline.json" +# The review's exact command: `record tighten` (extra args are ignored now). +bash "$F/bin/baseline-lock.sh" record "$F" lint "$(bash "$F/bin/baseline-lock.sh" sha "$F/.claude/lint-baseline.json")" tighten >/dev/null 2>&1 +l="$(gline "$F")" +case "$l" in *FAIL*) ok "F1 forged record detected: FAIL" ;; *) bad "F1 a direct 'record … tighten' turned a hand edit green" "${l:-}" ;; esac +lintjson CONV009=30 +out="$(fgate --update-baseline --reason x)"; rc=$? +[ "$rc" -ne 0 ] && ok "F1 lint-gate refuses to build on the forged state (rc $rc)" || bad "F1 lint-gate wrote over a forged baseline" "$out" + +echo "X1-X3 no false FAIL from line endings or git" +X="$WORK/x"; newproj "$X"; cp "$P/mxcli" "$X/mxcli" +xgate() { (cd "$X" && LINT_JSON="$WORK/lint.json" bash bin/lint-gate.sh "$@") 2>&1; } +lintjson CONV009=5 SEC003=5; xgate --update-baseline >/dev/null +XB="$X/.claude/lint-baseline.json" +"$PY" -c 'import sys;p=sys.argv[1];b=open(p,"rb").read();open(p,"wb").write(b.replace(b"\n",b"\r\n"))' "$XB" +l="$(gline "$X")" +case "$l" in *PASS*) ok "X1 CRLF checkout of the recorded baseline: PASS" ;; *) bad "X1 CRLF read as a move" "$l" ;; esac +"$PY" -c 'import sys;p=sys.argv[1];b=open(p,"rb").read();open(p,"wb").write(b.replace(b"\r\n",b"\n"))' "$XB" +if command -v git >/dev/null 2>&1; then + gx() { git -C "$X" -c user.name=t -c user.email=fixture "$@" >/dev/null 2>&1; } + gx init -q; gx checkout -q -b main; gx add -A; gx commit -q -m base + printf '{\n "counts": {"CONV009": 9, "SEC003": 5}\n}\n' > "$XB" + l="$(gline "$X")" + case "$l" in *FAIL*) ok "X2 uncommitted looser edit: FAIL" ;; *) bad "X2 uncommitted looser edit not FAIL" "$l" ;; esac + gx commit -q -am "loosen in git" + l="$(gline "$X")" + case "$l" in *MANUAL*) ok "X2 the same content committed (HEAD blob): MANUAL" ;; *) bad "X2 committed content not MANUAL" "$l" ;; esac + gx reset -q --hard HEAD~1 + gx checkout -q -b a; lintjson CONV009=3 SEC003=5; xgate --update-baseline >/dev/null; gx add -A; gx commit -q -m a + gx checkout -q main; gx checkout -q -b b; lintjson CONV009=5 SEC003=2; xgate --update-baseline >/dev/null; gx add -A; gx commit -q -m b + gx checkout -q a; gx merge -q --no-edit b + # The two logs and the baseline conflict; a person resolves them the obvious way: both log + # rows kept, the baseline taking each side's own ratchet (CONV009 3 from a, SEC003 2 from b). + # That content was never written by any instrument, so its hash is in no record — the case + # the review raised. + printf '{\n "counts": {\n "CONV009": 3,\n "SEC003": 2\n },\n "total": 5\n}\n' > "$XB" + { git -C "$X" show a:.claude/baseline-writes.tsv; git -C "$X" show b:.claude/baseline-writes.tsv | tail -1; } > "$X/.claude/baseline-writes.tsv" + gx add -A; gx commit -q -m merge + [ "$(git -C "$X" rev-list --count HEAD)" -ge 4 ] || bad "X3 the merge was not built" "$(git -C "$X" log --oneline | head -5)" + l="$(gline "$X")" + case "$l" in *FAIL*|"") bad "X3 a committed two-branch merge read as tampering" "${l:-}" ;; *) ok "X3 merged baselines: not FAIL ($(printf '%s' "$l" | sed 's/^Baselines (human-approved?): //' | cut -c1-6))" ;; esac +else + echo " skip X2-X3 (no git)" +fi + echo "" echo "PASS=$PASS FAIL=$FAIL" [ "$FAIL" -eq 0 ] diff --git a/tests/wave2/test-install-claude-permissions.sh b/tests/wave2/test-install-claude-permissions.sh index 0a286bdf..f770d42c 100755 --- a/tests/wave2/test-install-claude-permissions.sh +++ b/tests/wave2/test-install-claude-permissions.sh @@ -79,8 +79,13 @@ EXPECTED_DENY=( "Bash(./mxcli exec:*)" "Bash(mxcli exec:*)" "Bash(./mxcli.exe exec:*)" - "Bash(*baseline-approv*)" + "Bash(*baseline-approve.sh*)" + "Bash(*baseline-lock.sh*record*)" + "Bash(*>*baseline-approvals.tsv*)" + "Bash(*>*baseline-writes.tsv*)" "Edit(**/baseline-approvals.tsv)" + "Edit(**/baseline-writes.tsv)" + "Edit(**/baseline-history/**)" ) SESSION_HOOK="bash bin/session-check.sh || true" @@ -177,6 +182,10 @@ for e in "${EXPECTED_DENY[@]}"; do done [ "$MISSING_DENY" -eq 0 ] && ok "T1: all ${#EXPECTED_DENY[@]} deny entries present in the fresh file" \ || bad "T1: some deny entries missing from the fresh file" +# #375 review: a Bash deny naming the approvals log by file name blocked `git add`/`git log -p` on it. +printf '%s\n' "$DENY1" | grep -qxF 'Bash(*baseline-approv*)' \ + && bad "T1: the over-broad Bash(*baseline-approv*) deny is back (blocks git add on the log)" \ + || ok "T1: no Bash deny matches the approvals log by name alone (git add/log stay allowed)" py_hook_present "$FRESH/.claude/settings.json" && ok "T1: SessionStart hook present in the fresh file" \ || bad "T1: SessionStart hook missing from the fresh file" py_look_hook_present "$FRESH/.claude/settings.json" && ok "T1: PostToolUse(Read) LOOK hook present in the fresh file" \ @@ -406,9 +415,9 @@ PY && ok "T6: Copilot autoApprove has bin/exec.sh: true" \ || bad "T6: Copilot autoApprove missing bin/exec.sh: true" # #348: approving a baseline move is the user's act — Copilot must ask, never auto-run it. - printf '%s\n' "$AA" | grep -qxF "$(printf '/baseline-approv/\tfalse')" \ - && ok "T6: Copilot autoApprove has /baseline-approv/: false" \ - || bad "T6: Copilot autoApprove missing /baseline-approv/: false" + printf '%s\n' "$AA" | grep -qxF '/baseline-approve\.sh/ false' \ + && ok "T6: Copilot autoApprove asks for baseline-approve.sh (false)" \ + || bad "T6: Copilot autoApprove missing /baseline-approve\.sh/: false" "$AA" printf '%s\n' "$AA" | grep -q "$TOOLKIT_ROOT" \ && bad "T6: Copilot .vscode/settings.json leaked the absolute toolkit-root path" \ || ok "T6: Copilot .vscode/settings.json carries no absolute toolkit-root path" From b042e953d590ff3666942e5679ab5bb454c0a4c0 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 14 Oct 2026 15:39:47 +0000 Subject: [PATCH 3/3] fix(baselines): only PASS passes without a person; git anchors the write log (#375 confirmation review) - H1: committed rows of .claude/baseline-writes.tsv are a required prefix, so removing or truncating them is FAIL, never "predates the lock". lint-gate and conformance-check refuse an unapproved write unless verify says PASS (MANUAL too). - H2: a loosening is MANUAL only when a git merge produced it (a looser row from a merge's other parent, or merge-made baseline content). A one-parent commit of it stays FAIL. gate-check blocks Stages 5-6 on MANUAL as well as FAIL. - M1/M2/L1: the replay restarts from the latest row the user approved, so `approve --current` clears a forged mid-log row, a merge MANUAL and a missing snapshot (rule 7). - M3: an approval covers only the state it saw: it must be newer than every write (microsecond timestamps), so returning to an old approved state after a tighter write is FAIL and lint-gate will not reuse it. - L2: a pty-faked TTY is listed as a residual in baseline-lock.sh. Fixtures: test-baseline-lock.sh 28/28 (11 new cases; 9 fail on ad7a9cf). Bench gains an rm-the-log attempt per baseline and now measures Stage 5 blocks. Field evidence: none yet. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw --- CHANGELOG.md | 2 +- bin/gate-check.sh | 21 ++-- project-bin/baseline-approve.sh | 2 +- project-bin/baseline-lock.sh | 156 +++++++++++++++++++++-------- project-bin/conformance-check.sh | 5 +- project-bin/lint-gate.sh | 2 + skills/lint-that-actually-runs.md | 9 +- tests/wave2/bench-baseline-lock.sh | 28 ++++-- tests/wave2/test-baseline-lock.sh | 79 ++++++++++++++- 9 files changed, 232 insertions(+), 72 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 78d253c9..45de104b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,7 +19,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. Named m ## Unreleased -- fix(lint-gate, conformance-check, gate-check): **a ratchet baseline that loosens now moves only on the user's own command; `--approved-in-chat` is no longer evidence** ([#348](https://github.com/MendixMau/mxcli-project-toolkit/issues/348)). The agent whose script caused a rise could type that flag itself, so "re-baselining is the user's call" ran on trust. A rise (lint) or an accepted regression (conformance) is now written only against a row that the new `bin/baseline-approve.sh` writes to `.claude/baseline-approvals.tsv`. The user runs that command in a terminal, and the row is the sha256 of exactly the content they reviewed. Claude Code denies the command and the file to the agent (`install-claude-permissions.sh`), Copilot must ask before running it, and the command refuses to run without a TTY. Every instrument write is recorded with its content in `.claude/baseline-writes.tsv` and `.claude/baseline-history/`. `gate-check.sh` prints `Baselines (human-approved?)`, replaying each record against its content, so running `baseline-lock.sh record … tighten` after a hand edit is caught too. It reports FAIL for a baseline or record looser than the write before it with no approval (this blocks Stages 5 and 6), and MANUAL for one that predates the lock or that came in looser through a git commit or merge. Line endings are ignored, so a CRLF checkout compares equal. The deny rules name the two commands and redirection into the logs, so `git add` and `git log -p` on the logs still work. Bench (`tests/wave2/bench-baseline-lock.sh`, 11 unapproved and 4 legitimate scripted attempts per baseline): unapproved moves landing 11/11 → 3/11 (the 3 are direct file edits, one with a forged record; all now flagged, unflagged 11/11 → 0/11), legitimate re-baselines 4/4 → 4/4, at the cost of one human command each. Field evidence: none yet. Not covered: the mxbuild verification stamp, which records a machine check and has no human judgement to lock, and page-fidelity and token-burn, which keep no baseline. — external reviewer feedback +- fix(lint-gate, conformance-check, gate-check): **a ratchet baseline that loosens now moves only on the user's own command; `--approved-in-chat` is no longer evidence** ([#348](https://github.com/MendixMau/mxcli-project-toolkit/issues/348)). The agent whose script caused a rise could type that flag itself, so "re-baselining is the user's call" ran on trust. A rise (lint) or an accepted regression (conformance) is now written only against a row that the new `bin/baseline-approve.sh` writes to `.claude/baseline-approvals.tsv`. The user runs that command in a terminal, and the row is the sha256 of exactly the content they reviewed. Claude Code denies the command and the file to the agent (`install-claude-permissions.sh`), Copilot must ask before running it, and the command refuses to run without a TTY. Every instrument write is recorded with its content in `.claude/baseline-writes.tsv` and `.claude/baseline-history/`. `gate-check.sh` prints `Baselines (human-approved?)`, replaying each record against its content, so running `baseline-lock.sh record … tighten` after a hand edit is caught too. It reports FAIL for a baseline or record looser than the write before it with no approval, or when committed records have gone missing. It reports MANUAL for a baseline that predates the lock or a loosening a git merge produced; a plain one-parent commit of a loosening stays FAIL. Both block Stages 5 and 6, and the instruments refuse to write until the user approves. An approval covers only the state it saw, and the user's `approve --current` re-anchors the replay, so it clears any FAIL or MANUAL. Line endings are ignored, so a CRLF checkout compares equal. The deny rules name the two commands and redirection into the logs, so `git add` and `git log -p` on the logs still work. Bench (`tests/wave2/bench-baseline-lock.sh`, 11 unapproved and 4 legitimate scripted attempts per baseline): unapproved moves landing 11/11 → 3/11 (the 3 are direct file edits, one with a forged record; all now flagged, unflagged 11/11 → 0/11), legitimate re-baselines 4/4 → 4/4, at the cost of one human command each. Field evidence: none yet. Not covered: the mxbuild verification stamp, which records a machine check and has no human judgement to lock, and page-fidelity and token-burn, which keep no baseline. — external reviewer feedback - process(build): **Stage 5 drafts the ready set in parallel and applies serially.** The runbook dispatch row said "sequential within a module", which serialised drafting even for independent scripts; it now points to `iterative-build-loop.md` → Gate cost, the single statement of the rule. The unit is the build-plan row: every `BUILD` row whose "Depends on" rows are all applied, minus overlapping writers (two rows that replace, alter or drop+create the same document never share a ready set, and no row is drafted while it overlaps a queued unapplied one; `brd-to-build-plan.md` has Produces name every document a row writes), runs at once, its scripts in order, one `mdl-agent` per script. Applies stay one at a time through `bin/exec.sh`, and the ready set is recomputed after each. A queued draft made before a sibling apply landed is re-checked with `mxcli diff --format struct` (preflight STOP rule 23); any removal line (checked against `describe` per BUG-146) or STOP rule 21 target means re-draft, never patch, because a stale restatement of a shared document is exactly rule 21's silent seed-call loss and passes `check --references`. Report `ready set N, drafted N, applied k`; the `mdl-agent` template says a drafter never applies and never depends on an unapplied draft. Field evidence: none yet. — MendixMau - fix(platform-link): **CLI commits now show their changed documents in Studio Pro History.** Studio Pro's History → Model changes pane reads a `ModelChanges` array from the commit's `refs/notes/mx_metadata` note. Trap 6c's hand-written template set it to `[]`, so every mxcli/agent commit showed "Model changes 0" (field project: 270 of 270 notes empty). New `project-bin/mx-metadata-note.py` (stdlib only) computes the array from the commit's changed `.mxunit` files, in Studio Pro's entry shape: Status, UnitID, UnitType, UnitName and Module (walked up the `.mpr` Unit tree). It has `changes` / `write` / `backfill` / `sync ` subcommands. `install-project-hooks.sh` now also installs post-commit and post-rewrite hooks that write the note and never block, and `--check` covers them, so `sync-project.sh` rolls them out to wired projects. Notes are published with `mx-metadata-note.py sync ` after the branch push. Push config is never touched. With mxcli v0.26+ in the project root the hooks call `mxcli git note --write` instead (mendixlabs/mxcli#1337); the Python script is the fallback for older pins, and on 160 field commits both wrote the same 3,594 entries. Trap 6c rewritten. Measured against Studio Pro's own notes, 51 commits in 5 apps: 3387/3389 entries exact on all five fields. Field run on a 357-commit Mendix 11.15.0 app: 158 notes backfilled and pushed, and Studio Pro History lists the documents per revision. Review fixes: an earlier revision added `HEAD` + the notes ref to `remote..push`. It was dropped because any push refspec replaces `push.default`. A branch tracking a differently named upstream got a new same-named branch on Team Server where `simple` refuses, a missing local notes ref failed the whole push, and nobody knows whether Studio Pro's own push reads that config. The installer now only fetches the remote's notes (non-forced). It removes that exact `HEAD` + notes pair if present, and leaves and reports any other entry. `sync` handles a remote with no notes ref (it used to exit 1) and retries a rejected push once after re-merging. A note that already lists changes (Studio Pro's) is never overwritten by `write`/`backfill` without `--force`, and `sync` keeps a colleague's non-empty note over ours. The mxcli probe greps `git note --help` for `--write`. Hooks and docs call the script through `resolve_py`. Fixture `tests/wave2/test-mx-metadata-note.sh` (40 cases, over real captured template units; single- and two-tree layouts, amend, backfill and `--force`, push exit codes identical to a no-hooks clone, leftover-refspec cleanup, sync to an empty remote and its retry, notes fetch at install, a diverged notes merge; green under `init.defaultBranch` master and main). — field project (Mendix 11.15 app), MendixMau - learn(fixture-seeding): **explore the data structure first, choose rows by use case, scan actions for resets before any run.** Two seeding runs on public template apps: seeding raised journey passes 33 -> 58 and 20 -> 51 of the same sets (FAIL 93 -> 68, 163 -> 132; INVALID unchanged), and a "Reset all demodata" journey wiped accounts so 104 of 127 journeys reported `The username or password you entered is incorrect`, identical on empty and seeded databases. Skill gains three questions a seed must answer, a use-case checklist, a destructive-action scan, a generic traps table with symptoms, measured expectations and a denominator checklist; routing trigger now fires on seed/test/demo data and empty pages. Independent review fixes: object ids follow the Mendix bigint pattern with a business-key guard (no uuids), associations are seeded where the page's path stores them, four new traps (enum names, inheritance chain, calculated attributes, view/external entities), and the port-offset and container-label traps moved to `harness-architecture.md`. — MendixMau diff --git a/bin/gate-check.sh b/bin/gate-check.sh index 7b7d05bd..730791c5 100755 --- a/bin/gate-check.sh +++ b/bin/gate-check.sh @@ -2551,11 +2551,12 @@ fi # loosening write only lands against a row bin/baseline-approve.sh wrote (a person, in a # terminal). baseline-lock.sh verify replays the records against their content: a baseline, or a # recorded write, looser than the write before it with no approval is FAIL (a hand edit, or a -# forged record — #375 review). MANUAL: no record at all (predates the lock), or the looser -# content is committed (a merge). CR is stripped before comparing. The toolkit's own baseline-lock.sh reads the -# project, whichever version the project installed. BLOCKS Stages 5 and 6 on FAIL (below); the +# forged record — #375 review). MANUAL: no record at all (predates the lock or the log was +# removed), or looser content a git merge produced. Committed records may not go missing. CR is +# stripped before comparing. Both FAIL and MANUAL block, since only a person clears either. The toolkit's own baseline-lock.sh reads the +# project, whichever version the project installed. BLOCKS Stages 5 and 6 (below); the # remedy, a person running `bin/baseline-approve.sh --current`, is never blocked. -BASE_STATUS="PASS"; BASE_NOTE="" +BASE_STATUS="PASS"; BASE_NOTE=""; BASE_BLOCK=0 BASE_LOCK="$TOOLKIT_DIR/project-bin/baseline-lock.sh" if [ -f "$BASE_LOCK" ]; then for _bk in lint conformance; do @@ -2563,10 +2564,10 @@ if [ -f "$BASE_LOCK" ]; then _bs="$(printf '%s' "$_bv" | cut -f1)" case "$_bs" in NONE) continue ;; - FAIL|GONE) BASE_STATUS="FAIL" ;; - MANUAL) [ "$BASE_STATUS" = PASS ] && BASE_STATUS="MANUAL" ;; + FAIL|GONE) BASE_STATUS="FAIL"; BASE_BLOCK=1 ;; + MANUAL) [ "$BASE_STATUS" = PASS ] && BASE_STATUS="MANUAL"; BASE_BLOCK=1 ;; PASS) ;; - *) [ "$BASE_STATUS" = PASS ] && BASE_STATUS="MANUAL"; _bv="$_bk could not be verified" ;; + *) [ "$BASE_STATUS" = PASS ] && BASE_STATUS="MANUAL"; BASE_BLOCK=1; _bv="$_bk could not be verified" ;; esac BASE_NOTE="$BASE_NOTE${BASE_NOTE:+; }$_bk: $(printf '%s' "$_bv" | cut -f2-)" done @@ -3299,9 +3300,11 @@ if [ -n "$REQUESTED_STAGE" ]; then # a report goes un-filed again. # No Stage-5/6 gate passes on a ratchet baseline that moved without a person's approval: # a rise baselined away is a FAIL nobody was shown. See the Baselines block above. - if { [ "$REQUESTED_STAGE" = "5" ] || [ "$REQUESTED_STAGE" = "6" ]; } && [ "$BASE_STATUS" = "FAIL" ]; then + # MANUAL blocks too (#375 confirmation review): every non-PASS state here is one only a person + # clears, and a non-blocking MANUAL was a one-commit bypass. + if { [ "$REQUESTED_STAGE" = "5" ] || [ "$REQUESTED_STAGE" = "6" ]; } && [ "$BASE_BLOCK" = 1 ]; then echo "" >&2 - echo "Gate BLOCKED by a baseline that moved without approval: $BASE_NOTE" >&2 + echo "Gate BLOCKED by a baseline no person has approved ($BASE_STATUS): $BASE_NOTE" >&2 echo "Show the user the change (git log -p on the baseline file). If they accept it, THEY run" >&2 echo "bin/baseline-approve.sh --current in a separate terminal window (not via the agent" >&2 echo "or Claude Code's ! prefix: it needs a TTY); otherwise restore it from git." >&2 diff --git a/project-bin/baseline-approve.sh b/project-bin/baseline-approve.sh index a83838d8..5962228c 100755 --- a/project-bin/baseline-approve.sh +++ b/project-bin/baseline-approve.sh @@ -56,7 +56,7 @@ BY="$(git -C "$ROOT" config user.name 2>/dev/null || true)"; BY="${BY:-${USER:-u NOTE="$( [ "$CURRENT" -eq 1 ] && echo "current file" || head -1 "$SRC.summary" 2>/dev/null)" F="$ROOT/.claude/baseline-approvals.tsv" [ -f "$F" ] || printf 'ts\tkind\tsha256\tby\tnote\n' > "$F" -printf '%s\t%s\t%s\t%s\t%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$KIND" "$SHA" "$BY" \ +printf '%s\t%s\t%s\t%s\t%s\n' "$(bash "$LOCK" now)" "$KIND" "$SHA" "$BY" \ "$(printf '%s' "$NOTE" | tr '\t\n' ' ')" >> "$F" if [ "$CURRENT" -eq 1 ]; then # The file on disk is now the accepted state: log it, so later writes are checked against it. diff --git a/project-bin/baseline-lock.sh b/project-bin/baseline-lock.sh index 879a05d8..31ec3a1c 100755 --- a/project-bin/baseline-lock.sh +++ b/project-bin/baseline-lock.sh @@ -14,13 +14,22 @@ # `record` directly after a hand edit — does not launder a rise: the forged row is looser than # its predecessor and verify says FAIL. Only content can pass, never a claim about content. # +# GIT IS THE MEMORY (#375 confirmation review). Rows git HEAD has committed must all still be in +# the log: removing or truncating them is FAIL, not "predates the lock". A loosening is MANUAL +# only when a merge produced it (a looser row from a merge's other parent, or merge-made baseline +# content); a plain one-parent commit of it stays FAIL. An approval covers only the state it saw +# when given: a later write supersedes it. The user's `approve --current` re-anchors the replay +# at that row, so it clears every earlier FAIL or MANUAL (rule 7). The instruments refuse any +# un-approved write unless verify says PASS, and gate-check blocks on FAIL and MANUAL alike: no +# state but PASS passes without a person. +# # What it does NOT stop, said plainly: a forged row in baseline-approvals.tsv (Claude Code denies # the approve command, file-tool edits and shell redirection into it — bin/install-claude- # permissions.sh — but an agent determined to evade those, e.g. via an interpreter, can still -# write one); a harness with no deny rules whose agent terminal is a real TTY; deleting the whole -# log (verify then reads "predates the lock": MANUAL, not PASS); and a loosening committed to git -# (reported MANUAL, below, so a merge never reads as tampering). Each of those is a deliberate -# forgery visible in git, not a flag the rule allowed. +# write one); a harness with no deny rules whose agent terminal is a real TTY, or a TTY faked by +# a pty wrapper; and edits to rows not yet committed (outside git, or before the commit, the log +# is only as tamper-evident as the working tree). Each is a deliberate forgery, not a flag the +# rule allowed. # # Covered baselines (kind -> file, relative to the project root): # lint .claude/lint-baseline.json project-bin/lint-gate.sh @@ -38,15 +47,16 @@ # Usage (instruments, baseline-approve.sh and gate-check call it; a person never needs to): # baseline-lock.sh file print the baseline path (relative) # baseline-lock.sh sha sha256 of the file, CR stripped +# baseline-lock.sh now the records' timestamp (microseconds) # baseline-lock.sh verify print STATEnote, exit 0 # NONE no baseline and no record (a first write needs no approval) # GONE no baseline, but records exist (re-creating it needs approval) -# PASS approved, or no looser than the last verified write -# FAIL looser than the last verified write with no approval, or a recorded write that -# loosened without one (a forged record) -# MANUAL no record at all (predates the lock), or the looser content is exactly what git -# HEAD holds (a merge or checkout brought it in, not an uncommitted local edit) -# baseline-lock.sh approved exit 0 iff an approval row matches +# PASS no looser than the last verified write (replayed from the latest approval) +# FAIL looser than it with no approval, a recorded write that loosened without one +# (a forged record), a missing snapshot, or committed rows missing from the log +# MANUAL no record at all (predates the lock, or the log was removed outside git), or a +# loosening a git merge produced. Blocks like FAIL; cleared by approve --current +# baseline-lock.sh approved exit 0 iff an approval newer than every write matches # baseline-lock.sh record log the baseline now on disk # baseline-lock.sh propose save the proposal, print the ask # @@ -71,6 +81,11 @@ APPROVALS, WRITES, HISTORY = ".claude/baseline-approvals.tsv", ".claude/baseline def norm(b): return b.replace(b"\r", b"") +def now(): + # Microseconds: an approval and the write it covers can land in the same second, and the + # order between them is the whole check (live(), justified()). + return datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%fZ") + def sha_bytes(b): return hashlib.sha256(norm(b)).hexdigest() @@ -104,38 +119,89 @@ def looser(kind, prev, new): return [r for r in sorted(new) if new[r] > prev.get(r, 0)] return ["%s %s" % k for k in sorted(new) if prev.get(k) == "OK" and new[k] != "OK"] -def head_blob(root, rel): +def git(root, *a): try: - r = subprocess.run(["git", "-C", root, "show", "HEAD:./" + rel], capture_output=True, timeout=20) + r = subprocess.run(["git", "-C", root] + list(a), capture_output=True, timeout=30) return r.stdout if r.returncode == 0 else None except (OSError, subprocess.SubprocessError): return None +_LINES = {} +def lines_at(root, rev, rel): + if (rev, rel) in _LINES: + return _LINES[(rev, rel)] + b = git(root, "show", "%s:./%s" % (rev, rel)) + _LINES[(rev, rel)] = norm(b).decode("utf-8", "replace").splitlines() if b is not None else [] + return _LINES[(rev, rel)] + +def merged_in(root, line): + # True when `line` reached HEAD's log through a merge: present on a non-first parent of a + # merge commit and absent from its first parent. A row committed on the mainline is not. + out = git(root, "log", "--merges", "--format=%H %P", "--", WRITES) or b"" + for m in out.decode().splitlines(): + ps = m.split()[1:] + if len(ps) > 1 and line not in lines_at(root, ps[0], WRITES) \ + and any(line in lines_at(root, p, WRITES) for p in ps[1:]): + return True + return False + +def blob_merged(root, rel, s, recorded): + # The current content is git HEAD's, and the commit that last changed it on the mainline is a + # merge that produced it (or took it from a side that recorded it). A single-parent commit of a + # loosening is not a merge: FAIL (#375 confirmation review, H2). + hb = git(root, "show", "HEAD:./" + rel) + if hb is None or sha_bytes(hb) != s: + return False + c = (git(root, "log", "-1", "--first-parent", "--format=%P", "--", rel) or b"").decode().split() + if len(c) < 2: + return False + sides = {sha_bytes(git(root, "show", "%s:./%s" % (p, rel)) or b"") for p in c} + return s not in sides or s in recorded + def verify(root, kind): rel = FILES[kind] path = os.path.join(root, rel) w, a = rows(root, WRITES, kind), rows(root, APPROVALS, kind) - approved = {r[2] for r in a} - hw = head_blob(root, WRITES) - committed = set(norm(hw).decode("utf-8", "replace").splitlines()) if hw else set() - last, manual = None, [] - for r in w: + line = lambda r: "\t".join(r) + def justified(i): # a human approved exactly this write, after the write before it + prev = w[i - 1][0] if i else "" + return any(ap[2] == w[i][2] and prev <= ap[0] <= w[i][0] for ap in a) + k = max([i for i in range(len(w)) if justified(i)], default=None) + # Committed rows are a required prefix: a record git has seen cannot be deleted or edited + # away (#375 confirmation review, H1). Only a human approval written after them supersedes. + ingit = git(root, "rev-parse", "--is-inside-work-tree") is not None + if ingit: + comm = [l for l in lines_at(root, "HEAD", WRITES) if l.split("\t")[1:2] == [kind]] + have = {line(r) for r in w} + missing = [l for l in comm if l not in have] + anchor = line(w[k]) if k is not None else None + superseded = anchor is not None and (anchor not in comm or + all(comm.index(m) < comm.index(anchor) for m in missing)) + if missing and not superseded: + return "FAIL", ("%d committed record(s) missing from %s (first: %s) — the log was truncated or " + "rewritten; restore it from git, or the user approves the current state: " + "bin/baseline-approve.sh %s --current" % (len(missing), WRITES, missing[0][:40], kind)) + start = k if k is not None else 0 + last, manual = None, None + for i in range(start, len(w)): + r = w[i] snap = os.path.join(root, HISTORY, kind, r[2]) if not os.path.isfile(snap) or sha_file(snap) != r[2]: - return "FAIL", "recorded write %s (%s) has no matching snapshot under %s — the record cannot be checked" % (r[2][:12], r[0], HISTORY) + return "FAIL", "recorded write %s (%s) has no matching snapshot under %s — the record cannot be checked; the user can re-anchor: bin/baseline-approve.sh %s --current" % (r[2][:12], r[0], HISTORY, kind) try: c = load(kind, open(snap, "rb").read()) except ValueError: return "FAIL", "recorded write %s (%s) is not a readable %s baseline" % (r[2][:12], r[0], kind) - up = [] if last is None or r[2] in approved else looser(kind, last, c) - if up and "\t".join(r) in committed: - # Committed rows came through git: two branches that each wrote a baseline, merged. - manual.append("recorded write %s (%s) is looser than the row before it (%s) but is committed — " - "a merge of two branches' baselines, not a local edit; review the history" % (r[2][:12], r[0], ", ".join(up[:5]))) - up = [] - if up: + up = [] if i == start or justified(i) else looser(kind, last, c) + via_merge = bool(ingit and line(r) in comm and merged_in(root, line(r))) + if up and via_merge: + manual = ("recorded write %s (%s) is looser than the row before it (%s) and came in through a merge " + "— the user reviews it and runs: bin/baseline-approve.sh %s --current" % (r[2][:12], r[0], ", ".join(up[:5]), kind)) + elif up: return "FAIL", ("recorded write %s (%s, basis %s) loosens the baseline (%s) and no approval matches it — " "a record is checked against its content, not believed" % (r[2][:12], r[0], r[3] if len(r) > 3 else "?", ", ".join(up[:5]))) + elif i == start or justified(i) or not via_merge: + manual = None # a later approval or mainline write clears a merge's MANUAL (M2) last = c if not os.path.isfile(path): if w or a: @@ -143,24 +209,28 @@ def verify(root, kind): return "NONE", "no %s baseline yet" % kind raw = open(path, "rb").read() s = sha_bytes(raw) - if s in approved: - return ("MANUAL", manual[0]) if manual else ("PASS", "%s matches a human approval" % rel) if last is None: - return "MANUAL", "%s has no write record (it predates the lock) — cannot tell who moved it; the next instrument write starts the record" % rel + return "MANUAL", ("%s has no write record (it predates the lock, or the log was removed) — the user " + "approves it once: bin/baseline-approve.sh %s --current" % (rel, kind)) try: cur = load(kind, raw) except ValueError: return "FAIL", "%s is not a readable %s baseline" % (rel, kind) up = looser(kind, last, cur) - if not up: - return ("MANUAL", manual[0]) if manual else ("PASS", "%s is no looser than its last recorded write" % rel) - hb = head_blob(root, rel) - if hb and sha_bytes(hb) == s: - return "MANUAL", ("%s is looser than its last recorded write (%s) but is exactly what git HEAD holds — " - "it came through git (a merge or checkout), not a local edit; review its history, and if the " - "user accepts it they run: bin/baseline-approve.sh %s --current" % (rel, ", ".join(up[:5]), kind)) - return "FAIL", ("%s moved without approval: looser than its last recorded write (%s) — a person reviews it and runs: " - "bin/baseline-approve.sh %s --current" % (rel, ", ".join(up[:5]), kind)) + if up and ingit and blob_merged(root, rel, s, {r[2] for r in w}): + return "MANUAL", ("%s is looser than its last recorded write (%s) and was produced by a merge — the user " + "reviews it and runs: bin/baseline-approve.sh %s --current" % (rel, ", ".join(up[:5]), kind)) + if up: + return "FAIL", ("%s moved without approval: looser than its last recorded write (%s) — a person reviews it and runs: " + "bin/baseline-approve.sh %s --current" % (rel, ", ".join(up[:5]), kind)) + return ("MANUAL", manual) if manual else ("PASS", "%s is no looser than its last recorded write" % rel) + +def live(root, kind, s): + # An approval covers the state it approved when it was given: it must postdate every + # recorded write (#375 confirmation review, M3). + w = rows(root, WRITES, kind) + after = w[-1][0] if w else "" + return any(ap[2] == s and ap[0] > after for ap in rows(root, APPROVALS, kind)) def record(root, kind, basis): path = os.path.join(root, FILES[kind]) @@ -174,7 +244,7 @@ def record(root, kind, basis): with open(p, "a") as f: if new: f.write("ts\tkind\tsha256\tbasis\n") - f.write("%s\t%s\t%s\t%s\n" % (datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"), kind, s, basis)) + f.write("%s\t%s\t%s\t%s\n" % (now(), kind, s, basis)) for n in (kind, kind + ".summary"): try: os.remove(os.path.join(root, ".claude", "baseline-pending", n)) @@ -182,13 +252,15 @@ def record(root, kind, basis): pass cmd, args = sys.argv[1], sys.argv[2:] -if cmd == "sha": +if cmd == "now": + print(now()) +elif cmd == "sha": print(sha_file(args[0])) elif cmd == "verify": st, note = verify(args[0], args[1]) print("%s\t%s" % (st, note)) elif cmd == "approved": - sys.exit(0 if any(r[2] == args[2] for r in rows(args[0], APPROVALS, args[1])) else 1) + sys.exit(0 if live(args[0], args[1], args[2]) else 1) elif cmd == "record": record(args[0], args[1], args[2]) PY @@ -207,6 +279,8 @@ case "$cmd" in _file "$1" ;; sha) "$PY" -c "$_LOCK_PY" sha "$1" ;; + now) + "$PY" -c "$_LOCK_PY" now ;; verify|approved) _file "$2" >/dev/null || exit 2 "$PY" -c "$_LOCK_PY" "$cmd" "$@" ;; @@ -226,5 +300,5 @@ case "$cmd" in echo " it is denied to you and refuses without a terminal. Then re-run the same command." >&2 ;; *) - sed -n '2,57p' "$0"; exit 2 ;; + sed -n '2,65p' "$0"; exit 2 ;; esac diff --git a/project-bin/conformance-check.sh b/project-bin/conformance-check.sh index 1019e812..3ab4d7f4 100755 --- a/project-bin/conformance-check.sh +++ b/project-bin/conformance-check.sh @@ -409,6 +409,7 @@ lock_write() { fi [ "$PRIOR" = GONE ] && why="${why:+$why; }re-creates a baseline that was deleted" [ "$PRIOR" = FAIL ] && why="${why:+$why; }follows a baseline that moved without approval" + [ "$PRIOR" = MANUAL ] && why="${why:+$why; }follows a baseline with no verifiable record" if [ -z "$why" ]; then mv "$new" "$BASELINE" if [ -n "$LOCK" ]; then bash "$LOCK" record "$PROJECT_ROOT" conformance "$basis" @@ -461,8 +462,8 @@ if [ "$WROTE_BASELINE" -eq 0 ]; then fi done <<< "$KEYED" # Seeding adds rows the baseline never had; it is recorded, never laundered over a hand edit. - if [ -n "$SEEDED" ] && [ "$PRIOR" = FAIL ]; then - echo " note: $(printf '%s' "$SEEDED" | grep -c .) row(s) not seeded — the baseline moved without approval (see gate-check)" >&2 + if [ -n "$SEEDED" ] && { [ "$PRIOR" = FAIL ] || [ "$PRIOR" = MANUAL ]; }; then + echo " note: $(printf '%s' "$SEEDED" | grep -c .) row(s) not seeded — the baseline is $PRIOR in gate-check; the user runs bin/baseline-approve.sh conformance --current" >&2 SEEDED="" elif [ -n "$SEEDED" ]; then { cat "$BASELINE"; printf '%s' "$SEEDED"; } | sed '/^$/d' | sort > "$BASELINE.tmp" \ diff --git a/project-bin/lint-gate.sh b/project-bin/lint-gate.sh index cbf0cc34..26d4da44 100755 --- a/project-bin/lint-gate.sh +++ b/project-bin/lint-gate.sh @@ -441,6 +441,8 @@ if update: why.append("re-creates a baseline that was deleted") if prior == "FAIL": why.append("follows a baseline that moved without approval") + if prior == "MANUAL": + why.append("follows a baseline with no verifiable record (predates the lock, or came in by a merge)") basis = "first" if old_base is None else "tighten" if why: basis = "approved" diff --git a/skills/lint-that-actually-runs.md b/skills/lint-that-actually-runs.md index cde826f6..f826ab50 100644 --- a/skills/lint-that-actually-runs.md +++ b/skills/lint-that-actually-runs.md @@ -160,9 +160,12 @@ regression. `gate-check.sh` prints `Baselines (human-approved?)`. It replays each recorded write against its saved content, so a forged record is caught too. It reports: -- FAIL: the baseline is looser than its last recorded write with no approval. This blocks - Stages 5 and 6. -- MANUAL: the baseline predates the lock, or the looser content came in through git (a merge). +- FAIL: the baseline is looser than its last recorded write with no approval. Committed + records that have gone missing also FAIL. +- MANUAL: the baseline predates the lock, or the looser content came in through a git merge. + +Both block Stages 5 and 6, and the instruments refuse to write until the user approves. The +remedy for either is `baseline-approve.sh --current`, run by the user. A CRLF checkout compares equal. Commit `.claude/baseline-*` with the baseline. What the lock does not stop, an agent deliberately evading the deny rules, is listed in `project-bin/baseline-lock.sh`. diff --git a/tests/wave2/bench-baseline-lock.sh b/tests/wave2/bench-baseline-lock.sh index d7495f7b..6fa14a83 100755 --- a/tests/wave2/bench-baseline-lock.sh +++ b/tests/wave2/bench-baseline-lock.sh @@ -16,8 +16,8 @@ # Every number in the printed table is counted here, not typed: # unapproved moves that landed attempt left a baseline different from the clean one # (the instrument wrote it, or the agent edited the file) -# ...that gate-check did not flag baseline changed or deleted, and the tree's gate-check -# does not print "Baselines ...: FAIL" +# ...that gate-check did not block baseline changed or deleted, and the tree's gate-check +# does not block Stage 5 on it # legitimate re-baselines that landed the proposed content is on disk at the end # extra human steps per legitimate approve-command runs the flow needed, per attempt # A trailing "Not-a-bench" note: this is replayed attempts on synthetic input, not a field run. @@ -70,7 +70,8 @@ CONV009=6 SEC003=2|DELETE --reason x --approved-in-chat CONV009=4 SEC003=1|--reason=inline --approved-in-chat CONV009=12 SEC003=1|HANDEDIT CONV009=2 SEC003=6|HANDEDIT -CONV009=15 SEC003=1|FORGE" +CONV009=15 SEC003=1|FORGE +CONV009=11 SEC003=1|RMLOG" LINT_LEGIT="CONV009=5 SEC003=1 CONV009=2 SEC003=3 CONV009=8 SEC003=1 CONV006=2 @@ -111,16 +112,17 @@ Orders.E|DELETE Stock.E Stock.ACT|DELETE Orders.E|HANDEDIT Billing.E Stock.ACT|HANDEDIT -Stock.ACT|FORGE" +Stock.ACT|FORGE +Billing.E|RMLOG" CONF_LEGIT="Orders.E Billing.ACT|--module Billing Stock.E Stock.ACT Orders.ACT Billing.E" -gate_flags() { # gate_flags — 1 if the tree's gate-check says Baselines FAIL +gate_flags() { # gate_flags — 1 if the tree's gate-check BLOCKS Stage 5 on the baseline # captured first: `| grep -q` under pipefail reads a SIGPIPE'd gate-check as "not flagged" - _g="$(bash "$1/bin/gate-check.sh" "$2" --no-html 2>/dev/null)" - case "$_g" in *"Baselines (human-approved?): FAIL"*) echo 1 ;; *) echo 0 ;; esac + _g="$(bash "$1/bin/gate-check.sh" "$2" 5 --no-html 2>&1 >/dev/null)" + case "$_g" in *"Gate BLOCKED by a baseline"*) echo 1 ;; *) echo 0 ;; esac } # measure → prints "landed silent n_un legit_ok n_legit human_steps" @@ -138,7 +140,9 @@ measure() { conf_break "$d" $spec fi case "$flags" in - HANDEDIT|FORGE) # the agent edits the baseline file itself: the measured state, no instrument. + HANDEDIT|FORGE|RMLOG) # the agent edits the baseline file itself: the measured state, no instrument. + # RMLOG first deletes the write log and snapshots (#375 confirmation review, H1), and + # afterwards runs the instrument's update to re-record the edited state. # FORGE then runs the #375 review's bypass: `baseline-lock.sh record tighten`. if [ "$_k" = lint ]; then "$PY" -c 'import collections,json,sys;c=collections.Counter(v["ruleId"] for v in json.load(open(sys.argv[1]))["violations"]);json.dump({"counts":dict(sorted(c.items())),"total":sum(c.values())},open(sys.argv[2],"w"),indent=2)' "$d/lint.json" "$f" @@ -148,6 +152,10 @@ measure() { awk -F'\t' -v m="$m" -v p="$p" 'BEGIN{OFS="\t"} $1==m && $2==p {$3="STALE"} {print}' "$f.n" > "$f.m"; mv "$f.m" "$f.n"; done mv "$f.n" "$f" fi + if [ "$flags" = RMLOG ]; then + rm -rf "$d/.claude/baseline-writes.tsv" "$d/.claude/baseline-history" + if [ "$_k" = lint ]; then lint_run "$d" --reason x; else conf_run "$d"; fi + fi if [ "$flags" = FORGE ] && [ -f "$d/bin/baseline-lock.sh" ]; then bash "$d/bin/baseline-lock.sh" record "$d" "$_k" "$(bash "$d/bin/baseline-lock.sh" sha "$f")" tighten >/dev/null 2>&1 fi ;; @@ -197,10 +205,10 @@ for k in lint conformance; do set -- $(measure "$AFTER" "$k"); a_l=$1; a_f=$2; a_nu=$3; a_ok=$4; a_nl=$5; a_s=$6 per() { "$PY" -c "import sys;print('%.1f' % (int(sys.argv[1])/max(1,int(sys.argv[2]))))" "$1" "$2"; } printf '| %s | unapproved moves that landed | %d / %d | %d / %d |\n' "$k" "$b_l" "$b_nu" "$a_l" "$a_nu" - printf '| %s | baseline changed without approval and NOT flagged by gate-check | %d / %d | %d / %d |\n' "$k" "$b_f" "$b_nu" "$a_f" "$a_nu" + printf '| %s | baseline changed without approval and NOT blocked at Stage 5 by gate-check | %d / %d | %d / %d |\n' "$k" "$b_f" "$b_nu" "$a_f" "$a_nu" printf '| %s | legitimate re-baselines that landed | %d / %d | %d / %d |\n' "$k" "$b_ok" "$b_nl" "$a_ok" "$a_nl" printf '| %s | extra human steps per legitimate re-baseline | %s | %s |\n' "$k" "$(per "$b_s" "$b_nl")" "$(per "$a_s" "$a_nl")" done echo "" -echo "Of the unapproved attempts, $(printf '%s\n' "$LINT_UNAPPROVED" | grep -cE 'HANDEDIT|FORGE') (lint) and $(printf '%s\n' "$CONF_UNAPPROVED" | grep -cE 'HANDEDIT|FORGE') (conformance) edit the baseline file directly ($(printf '%s\n' "$LINT_UNAPPROVED" | grep -c FORGE) each then forges a write record with \`baseline-lock.sh record … tighten\`): no instrument can refuse an edit; the gate-check row is what catches it." +echo "Of the unapproved attempts, $(printf '%s\n' "$LINT_UNAPPROVED" | grep -cE 'HANDEDIT|FORGE|RMLOG') (lint) and $(printf '%s\n' "$CONF_UNAPPROVED" | grep -cE 'HANDEDIT|FORGE|RMLOG') (conformance) edit the baseline file directly ($(printf '%s\n' "$LINT_UNAPPROVED" | grep -c FORGE) each then forges a write record with \`baseline-lock.sh record … tighten\`; $(printf '%s\n' "$LINT_UNAPPROVED" | grep -c RMLOG) each first deletes the write log and re-runs the update): no instrument can refuse an edit; the gate-check row is what catches it." echo "Replayed: $(printf '%s\n' "$LINT_UNAPPROVED" | grep -c .) + $(printf '%s\n' "$CONF_UNAPPROVED" | grep -c .) unapproved and $(printf '%s\n' "$LINT_LEGIT" | grep -c .) + $(printf '%s\n' "$CONF_LEGIT" | grep -c .) legitimate attempts (lint + conformance) per tree, synthetic input; not a field measurement." diff --git a/tests/wave2/test-baseline-lock.sh b/tests/wave2/test-baseline-lock.sh index 67677f99..fe273ae2 100755 --- a/tests/wave2/test-baseline-lock.sh +++ b/tests/wave2/test-baseline-lock.sh @@ -20,8 +20,16 @@ # F1 forge: hand-edit the baseline looser, then run `baseline-lock.sh record … tighten` # directly → gate-check still FAIL (the record is checked against its content) # X1 a CRLF checkout of a recorded baseline (Windows autocrlf) → PASS, not FAIL -# X2 a looser baseline that is exactly git HEAD's blob → MANUAL; the same edit uncommitted → FAIL -# X3 a real merge of two branches that each ratcheted a different rule down → not FAIL +# X2 a looser baseline, uncommitted → FAIL; committed in a one-parent commit → still FAIL +# X3 a real merge of two branches that each ratcheted a different rule down → MANUAL (blocks) +# --- #375 confirmation review --- +# M2 after that merge, the user's approve --current clears it → PASS +# H1 rm the log + snapshots, or truncate a committed row, then hand-edit → FAIL, and +# lint-gate refuses to re-record over it +# M1 a forged mid-log row: the user's approve --current supersedes it → PASS (rule 7) +# M3 an old approval does not cover a return to its state after a tighter write → FAIL +# L1 a missing snapshot: the user's approve --current re-anchors → PASS +# G4 a baseline that predates the lock: lint-gate refuses even a ratchet-down until approved # # Synthetic lint JSON and a stub mxcli, as in test-lint-gate-accept.sh and # test-conformance-baseline-scope.sh: nothing here parses tool output; the assertions are @@ -172,7 +180,7 @@ if command -v git >/dev/null 2>&1; then case "$l" in *FAIL*) ok "X2 uncommitted looser edit: FAIL" ;; *) bad "X2 uncommitted looser edit not FAIL" "$l" ;; esac gx commit -q -am "loosen in git" l="$(gline "$X")" - case "$l" in *MANUAL*) ok "X2 the same content committed (HEAD blob): MANUAL" ;; *) bad "X2 committed content not MANUAL" "$l" ;; esac + case "$l" in *FAIL*) ok "X2 the same content in a one-parent commit: still FAIL" ;; *) bad "X2 a plain git commit laundered the loosening" "$l" ;; esac gx reset -q --hard HEAD~1 gx checkout -q -b a; lintjson CONV009=3 SEC003=5; xgate --update-baseline >/dev/null; gx add -A; gx commit -q -m a gx checkout -q main; gx checkout -q -b b; lintjson CONV009=5 SEC003=2; xgate --update-baseline >/dev/null; gx add -A; gx commit -q -m b @@ -186,11 +194,72 @@ if command -v git >/dev/null 2>&1; then gx add -A; gx commit -q -m merge [ "$(git -C "$X" rev-list --count HEAD)" -ge 4 ] || bad "X3 the merge was not built" "$(git -C "$X" log --oneline | head -5)" l="$(gline "$X")" - case "$l" in *FAIL*|"") bad "X3 a committed two-branch merge read as tampering" "${l:-}" ;; *) ok "X3 merged baselines: not FAIL ($(printf '%s' "$l" | sed 's/^Baselines (human-approved?): //' | cut -c1-6))" ;; esac + case "$l" in *MANUAL*) ok "X3 merged baselines: MANUAL, not FAIL" ;; *) bad "X3 a committed two-branch merge not MANUAL" "${l:-}" ;; esac + human_approve "$X" lint --current + l="$(gline "$X")" + case "$l" in *PASS*) ok "M2 the user's approve --current clears the merge" ;; *) bad "M2 merge MANUAL did not clear" "$l" ;; esac + + echo "H1 removing or truncating committed records does not launder a loosening" + H="$WORK/h"; newproj "$H"; cp "$P/mxcli" "$H/mxcli" + hgate() { (cd "$H" && LINT_JSON="$WORK/lint.json" bash bin/lint-gate.sh "$@") 2>&1; } + gh() { git -C "$H" -c user.name=t -c user.email=fixture "$@" >/dev/null 2>&1; } + lintjson CONV009=5; hgate --update-baseline >/dev/null + lintjson CONV009=3; hgate --update-baseline >/dev/null + gh init -q; gh add -A; gh commit -q -m base + rm -rf "$H/.claude/baseline-writes.tsv" "$H/.claude/baseline-history" + printf '{\n "counts": {"CONV009": 9}\n}\n' > "$H/.claude/lint-baseline.json" + l="$(gline "$H")" + case "$l" in *FAIL*) ok "H1 rm of the log: FAIL (committed records missing)" ;; *) bad "H1 rm of the log read as 'predates the lock'" "$l" ;; esac + lintjson CONV009=9; out="$(hgate --update-baseline --reason x)"; rc=$? + [ "$rc" -ne 0 ] && ok "H1 lint-gate refuses to re-record over it (rc $rc)" || bad "H1 lint-gate re-recorded a laundered log" "$out" + gh checkout -q -- . + "$PY" -c 'import sys;p=sys.argv[1];l=open(p).read().splitlines(True);open(p,"w").write("".join(l[:-1]))' "$H/.claude/baseline-writes.tsv" + printf '{\n "counts": {"CONV009": 5}\n}\n' > "$H/.claude/lint-baseline.json" + l="$(gline "$H")" + case "$l" in *FAIL*) ok "H1 truncating the last committed row: FAIL" ;; *) bad "H1 truncation laundered CONV009 3 -> 5" "$l" ;; esac else - echo " skip X2-X3 (no git)" + echo " skip X2-X3, M2, H1 (no git)" fi +echo "M1 M3 L1 G4 the user's approval is the remedy, and only for the state it saw" +M="$WORK/m"; newproj "$M"; cp "$P/mxcli" "$M/mxcli" +mgate() { (cd "$M" && LINT_JSON="$WORK/lint.json" bash bin/lint-gate.sh "$@") 2>&1; } +ML="$M/.claude/lint-baseline.json"; MLOCK="$M/bin/baseline-lock.sh" +lintjson CONV009=5; mgate --update-baseline >/dev/null +lintjson CONV009=3; mgate --update-baseline >/dev/null +printf '{\n "counts": {"CONV009": 7}\n}\n' > "$ML" +bash "$MLOCK" record "$M" lint "$(bash "$MLOCK" sha "$ML")" tighten >/dev/null 2>&1 +# ...and the file goes back to 3, recorded on top: the forged 7 sits mid-log, and the state the +# user will approve is not the forged one. +printf '{\n "counts": {"CONV009": 3}\n}\n' > "$ML" +bash "$MLOCK" record "$M" lint "$(bash "$MLOCK" sha "$ML")" tighten >/dev/null 2>&1 +l="$(gline "$M")" +case "$l" in *FAIL*) ok "M1 forged mid-log row: FAIL" ;; *) bad "M1 forged mid-log row not caught" "$l" ;; esac +human_approve "$M" lint --current +l="$(gline "$M")" +case "$l" in *PASS*) ok "M1 the user's approve --current supersedes it" ;; *) bad "M1 approve --current did not clear the forged row (rule 7)" "$l" ;; esac +lintjson CONV009=8; mgate --update-baseline --reason r >/dev/null 2>&1 +human_approve "$M" lint; mgate --update-baseline --reason r >/dev/null 2>&1 +old="$(cat "$ML")" +lintjson CONV009=6; mgate --update-baseline >/dev/null 2>&1 +printf '%s\n' "$old" > "$ML" +l="$(gline "$M")" +case "$l" in *FAIL*) ok "M3 returning to an old approved state after a tighter write: FAIL" ;; *) bad "M3 an expired approval still covered its old state" "$l" ;; esac +lintjson CONV009=8; out="$(mgate --update-baseline --reason r)"; rc=$? +[ "$rc" -ne 0 ] && ok "M3 lint-gate does not reuse the expired approval (rc $rc)" || bad "M3 lint-gate reused an old approval" "$out" +human_approve "$M" lint --current +rm -f "$M/.claude/baseline-history/lint/"* +lintjson CONV009=6; mgate --update-baseline >/dev/null 2>&1 +l="$(gline "$M")" +case "$l" in *FAIL*) ok "L1 missing snapshots: FAIL" ;; *) bad "L1 missing snapshots not reported" "$l" ;; esac +human_approve "$M" lint --current +l="$(gline "$M")" +case "$l" in *PASS*) ok "L1 the user's approve --current re-anchors" ;; *) bad "L1 approve --current could not clear a missing snapshot" "$l" ;; esac +Lg="$WORK/legacy2"; newproj "$Lg"; cp "$P/mxcli" "$Lg/mxcli" +printf '{\n "counts": {"CONV009": 5}\n}\n' > "$Lg/.claude/lint-baseline.json" +lintjson CONV009=4; out="$( (cd "$Lg" && LINT_JSON="$WORK/lint.json" bash bin/lint-gate.sh --update-baseline) 2>&1)"; rc=$? +[ "$rc" -ne 0 ] && ok "G4 unrecorded baseline: even a ratchet-down waits for the user (rc $rc)" || bad "G4 lint-gate recorded over an unverified baseline" "$out" + echo "" echo "PASS=$PASS FAIL=$FAIL" [ "$FAIL" -eq 0 ]