Skip to content

e2e: read OQL rows by alias; liveness for negative-only span claims; APP-DOWN between journeys - #310

Closed
MendixMau wants to merge 10 commits into
masterfrom
fix/e2e-oql-rows
Closed

MendixMau wants to merge 10 commits into
masterfrom
fix/e2e-oql-rows

Conversation

@MendixMau

@MendixMau MendixMau commented Oct 13, 2026 •

Copy link
Copy Markdown
Owner

This PR fixes #154 and #150, adds the harness half of #285, and carries a follow-up to the already-merged #302 (Refs #286). It also makes the normalizer read the per-BRD conformance reports introduced in #307.

#154: OQL rows were read by position (Closes #154)

Problem. helpers.js built each row with Object.values(r), and oqlScalar returned rows[0][0]. mxcli v0.26.0 does three things that break that:

  • it prints the keys alphabetically;
  • it leaves out a column that is null in every row;
  • over a view entity, it adds an ID key beside the alias.

So columns shifted, and a view's count was read as its ID. The brace counter also ignored strings, so a value containing ] or } made the helper return null.

Fix.

  • Rows are keyed by the SELECT aliases, in SELECT order, with explicit nulls.
  • oqlScalar returns the first SELECT alias, or null rather than a guess.
  • New oqlRows.
  • JSON slicing is now string-aware.
  • The admin preflight uses the same parser.

Behaviour change for projects. oqlScalar used to return the alphabetically-first key. Check multi-column seeds, captures and baselines that feed {{vars}}.

Golden. Captured from the real binary against a stub admin API (tests/wave2/fixtures/oql-json/). test-oql-rows.sh passes 15/15; all 15 fail on master.

#150: negative-only span claims (Closes #150)

Fix. Rung 2 is now traceRung(). A negative-only claim is judged only when there is evidence that tracing is live:

  • the declared spans.liveness span, or
  • any service span since this journey's own start (2 s clock-skew margin).

Without that evidence the claim reads INVALID. Three more INVALID cases are now named instead of failing silently or throwing:

  • a project whose otel.js is older and lacks liveness();
  • a spans block that declares neither field;
  • the old array shape or {}.

The trace-negative mutant injects the forbidden name after the liveness check.

Also changed.

  • validate-journeys.js accepts negative-only claims and checks liveness and regex validity.
  • journey-examples.md and journey-proof.md are updated.
  • example.journey.json is rewritten into the runner's shape: 0 validator problems, and all 7 mutants supported.

#285, harness half: APP-DOWN between journeys (Refs #285)

Fix. The runner probes baseUrl before each journey (3 tries).

  • If the app is down before a journey, that journey is recorded as APP-DOWN INVALID and not walked.
  • If it is down after a journey, that journey's FAIL rows are flagged.
  • APP_PROBE=0 turns the probe off.

The upstream bundler race is not addressed here.

#302 follow-up (Refs #286), plus #307 per-BRD conformance

  • Credentials. config.js now reports passSource. Only the template placeholder Demo2025! is refused. A password the project set in project.config.js is accepted, the env still wins, and ALLOW_CONFIG_CREDENTIALS=1 still overrides. Before this, every journeys rung would have failed after the next sync.
  • SIGINT/SIGTERM now write the findings slot as aborted: "signal …".
  • report-normalize.js:
    • setup rows keep kind setup; the schema enum is updated.
    • It honours aborted: a top-level aborted field, kept rows flagged notAResult, and the instrument marked as a fault.
    • It reads the per-BRD report-<date>-<BRD>.tsv files from Fix graph-sweep/app-facts catalog guard and per-BRD conformance ledgers (#259, #260) #307: newest date wins, and every BRD file of that date is aggregated. Single-file projects are unchanged, and a stale older base file is never mixed in.
  • CHANGELOG names ALLOW_CONFIG_CREDENTIALS and the 403 verdict change: data rungs used to go INVALID with exit 1; the suite now aborts with exit 2.

Checks run

Check Result
test-journey-preflight.sh 75/75
test-report-normalize-two-tree.sh 9/9; the new per-BRD case fails on master
test-journey-liveness.sh 36/36
test-oql-rows.sh 15/15
test-validate-journeys-spans.sh 12/12
test-journey-control-exit.sh 7/7
test-journey-persona.sh 4/4
test-otel-capture-until.sh 4/4
report-normalize.js --selftest all OK
journey-runner.selftest.js OK
journey-rung4-scope.test.js 28/28
validate-journeys.js --selftest OK
check-scripts, portability, private citations, leak guard, pr-discipline clean

test-validate-journeys-spans.sh failed once under heavy load. The cause: under pipefail, printf | grep -q can be killed by SIGPIPE and flip a check. The new fixtures now use here-strings; 20 reruns were clean.

Field-run status

Still owed for every part. Nothing here has run against a live runtime or a real suite.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw

…APP-DOWN between journeys

#154: helpers.js built OQL rows with Object.values(r) and oqlScalar read
rows[0][0]. mxcli v0.26.0 prints keys alphabetically, omits an all-null
column and, over a view entity, adds ID beside the alias, so columns
shifted and a view's count read as its ID. The brace counter ignored
strings, so a value holding ] or } returned null. Rows are now keyed by
the parsed SELECT aliases (SELECT order, explicit nulls), oqlScalar reads
the first alias, new oqlRows, string-aware slicing. Golden: the released
v0.26.0 binary's verbatim output against a stub admin API
(tests/wave2/fixtures/oql-json/); test-oql-rows.sh 15/15, 0/15 on master.

#150 remainder: a mustNotFire-only step with zero spans was INVALID on
every run. Rung 2 (now traceRung) waits the full window and requires a
declared spans.liveness span or any service span since the journey began
(otel.js liveness) before judging; the trace-negative mutant injects the
forbidden name after that check.

#285 harness half: probe baseUrl before each journey (APP-DOWN, not
walked) and after each (flag its FAIL rows). APP_PROBE=0 disables.
test-journey-liveness.sh 27/27 over the captured Jaeger trace.

No live-app field run yet.

Closes #154
Refs #150 #285

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw
…s a loud INVALID

#150 item 3: a spans block declaring neither ordered nor mustNotFire
(e.g. the array shape [{mf,status}]) is now recorded INVALID by
traceRung instead of being skipped silently.

examples/validate-journeys.js rejected a mustNotFire-only claim as
"NEVER checked", which contradicted the runner since the liveness change.
It now accepts it, validates the new spans.liveness field (read only for
a negative-only claim) and rejects a mustNotFire that is not a regex;
selftest 19 rules red, 3 clean cases green. skills/journey-examples.md
documents the liveness rule and the INVALID for an empty spans block.

Fixtures: test-validate-journeys-spans.sh 12/12 (8 fail on master's
validator); test-journey-liveness.sh 30/30.

Closes #150

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw
The shipped example had drifted from journey-runner.js: name/role instead
of id/persona, a `value` seed instead of `sql`, page/expectPage/
expectText/click instead of actions + ui.ready/textPresent, spans as an
array of {mf,status} (now a loud INVALID), and SQL-style
<module>$<entity> queries where the runner sends OQL. validate-journeys.js
reported 3 problems on it; now 0. All 7 positive-control mutants are
supported on it; journey-runner.selftest.js (which falls back to it) and
test-journey-preflight.sh (back-compat cases) pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw
…hape named

- journey-runner.js: the negative-only liveness window starts at this
  journey's own start (LIVENESS_SKEW_MS = 2 s clock-skew margin), not 30 s
  earlier where the previous journey's spans counted.
- A project otel.js older than the runner (hand-edited, kept by sync)
  has no liveness(): the step reads INVALID naming the stale otel.js
  instead of a TypeError.
- spans as an array (old example shape) or {} gets an INVALID that names
  the shape; validate-journeys.js names the array shape too.
- Comment cited a non-existent fixture; "first cell" wording -> "first
  SELECT alias" (journey-examples.md, validate-journeys.js,
  full.journey.json); "16 rules" -> 19.
- CHANGELOG: callout that oqlScalar now returns the first SELECT alias.

Fixtures: test-journey-liveness.sh 36/36 (cases for the skew window, the
stale otel.js and the named shapes), test-validate-journeys-spans.sh 12/12.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw
…domain line

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw
…ings, setup kind

- config.js reports passSource (env / project.config.js / template
  placeholder); the preflight refuses only the template placeholder
  (Demo2025!, or credentials() source 'template default'). As shipped,
  #302 refused every non-env password, though the template tells projects
  to put their real one in DEFAULT_PASS and nothing loads an env file
  before verify-module — every journeys rung would FAULT after a sync.
  An admin object without passSource keeps the #302 rule.
- SIGINT/SIGTERM write the findings slot as aborted ("signal SIGINT")
  instead of leaving the previous run's file there.
- report-normalize.js: rung 'setup' rows keep kind 'setup' (were 'ui').
- CHANGELOG names ALLOW_CONFIG_CREDENTIALS and the 403 verdict change
  (data rungs INVALID exit 1 -> suite aborts exit 2).

test-journey-preflight.sh 74/74 (9 new cases fail on master: project-set
password runs, placeholder refuses, env wins, SIGINT end to end);
report-normalize --selftest new setup-kind case.

Refs #286

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw
…credentials branch

- report-normalize.js: a findings file with `aborted` (preflight refusal or
  a signal) yields a report with top-level `aborted`, every kept journey
  row flagged notAResult/abortedRun, and the journey-runner instrument a
  fault — an aborted run never reads as results. --selftest cases; and
  test-journey-preflight.sh runs the real runner's aborted file through
  the real normalizer (fails on master's normalizer).
- skills/report-schema.md: kind enum gains `setup`; `aborted` documented.
- config.js: drop the `credentials().source === 'template default'`
  branch no template emits; the Demo2025! placeholder rule stays.

Refs #286

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw
conformance-check now writes one report per BRD (#307), but
report-normalize.js kept only report-<date>.tsv, so per-BRD modules never
reached report.json and a stale base file could read as measured. It now
takes the newest date among report-<date>[-<BRD>].tsv and concatenates
every file of that date; single-file projects are unchanged.
pickConformanceFiles is exported; --selftest cases and case 3 of
test-report-normalize-two-tree.sh (fails on master: reads the stale file).

Also: the new fixtures use here-strings instead of `printf | grep -q`,
which under pipefail can SIGPIPE and flip a check (seen once under load).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw
MendixMau pushed a commit that referenced this pull request Oct 13, 2026
- agents/review-agent.md: the sweep faults on empty refs or refs with no
  RefKind, not on missing call edges.
- verify-module.sh: resolve_ledger_hint accepts a directory LEDGER_FILE and
  a module with only the per-BRD directory, instead of falling back to
  architecture/coverage-ledger.md. test-verify-ledger-hint.sh (4).
- review-module.sh: a module with only <M>/coverage-ledger/*.md is a ledger
  (conformance and coverage rungs, BRD names read from the per-BRD files);
  the reuse log/summary field names the report files actually read.
  test-review-conformance-per-brd.sh case 1b (6 checks).
- graph-sweep.sh: fixture for the "no orphan-read kinds" WARNING branch and
  its absence when an action edge exists (test-catalog-edge-guard.sh, 20).
- CHANGELOG: per-BRD reports reach docs/report.json via #310.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw
…ured golden

- pickConformanceFiles accepts report-<date>-<any suffix>.tsv: the
  narrower class dropped "report-<date>-F003 shipping.tsv" (a BRD id with
  a space, which #307's own fixture writes) and fell back to an older
  base file — the stale-report bug again.
- collectConformance strips a leading "<BRD>:" from the pointer, so a
  per-BRD row keeps requirement "/x" (was null for every per-BRD row);
  the BRD stays in the check id, so one pointer in two BRDs stays two.
- Golden: tests/wave2/fixtures/conformance-per-brd/ is conformance-check.sh's
  real output (PR #307, 8fc7c6a) on that PR's per-BRD fixture layout;
  case 3 of test-report-normalize-two-tree.sh reads it (fails on master
  and on the previous commit). report-schema.md: any suffix of the newest
  date is joined.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw
MendixMau pushed a commit that referenced this pull request Oct 13, 2026
…DOWN; #302 credential fix; aborted reports; per-BRD conformance in report.json (closes #154 #150, refs #285 #286)

Review: MERGE (initial review + fix round + 3 confirmation rounds) — project-set DEFAULT_PASS runs again (urgent #302 regression fixed); normal report.json shape unchanged except 'aborted': null; per-BRD golden reproduced byte-for-byte from #307's conformance-check.
MendixMau added a commit that referenced this pull request Oct 14, 2026
)

Every constituent PR was reviewed by a fresh agent that did not write it (CLAUDE.md "Merging a PR"); only MERGE verdicts joined. CI green on each constituent head and on this head (40814fc); full suites run on the branch.

Review lines (per merged PR):
- #315 context trim — MERGE; 360/385 removed lines found verbatim in their new homes, rest read by hand; baseline 79,997 -> 76,259.
- #316 independent-review + word-budget rules — MERGE-AFTER-FIX (3 rounds), fixes applied as specified.
- #318 page-fidelity displayed text — MERGE after fix round + confirmation.
- #317 interaction-model skill (#291 part 1) — MERGE after fix round + confirmation; no client names leaked.
- #321 PROOF-OF-LOOK per section; final-look not due before derived Stage 6 — MERGE after fix round + confirmation.
- #307 graph-sweep guard; per-BRD conformance ledgers — MERGE after two fix rounds + confirmation.
- #325 extraction-quality section; token-burn fixes — MERGE after fix round + confirmation.
- #310 e2e OQL by alias, negative-span liveness, APP-DOWN, #302 credential regression fix — MERGE after fix round + 3 confirmations.
- #323 inbox promotions — MERGE after fix round + confirmation.
- #308 lint gate BLIND on reverted stock rules — MERGE (FIRST-SEEN reverted) after two fix rounds + confirmation.
- #328 pipefail + grep -q flake — MERGE; 0/30 flakes vs 15/100 on master. Follow-up: run-all resolves bin/lib/ subjects (reviewed, MERGE).
- #319 test-stack-up command-line ownership proof — MERGE after fix round + 2 confirmations.
- #327 change-intake core + on-demand detail; CRLF-safe change log — MERGE after fix round + confirmation.
- #320 follow-ups to #290/#292; Dispatching split out — MERGE after fix round + confirmation.
- #326 status delivery rounds (existing-app mode) — MERGE after fix round + confirmation.
- #312 lint batch 4: SEC014, toolkit CONV009 — MERGE after review + 2 fix rounds + confirmation; field run on mxcli 0.26.
- #324 exec.sh change classes, shared stale-lock takeover, SEC014 note echo — MERGE (joint review); follow-up owed for a takeover race after a mid-takeover crash.
- #322 MxTest after-startup guard, mxcli-run pid check — MERGE (joint review).
- #314 doctor property-set exit code; sync announces its bound — MERGE (joint review).
- #330 review follow-ups (exit 4/5 docs, pipefail, NUL strip) — MERGE-AFTER-FIX, fixes applied; takeover re-check dropped.

Field runs are still owed for most instruments in this batch (stated in each PR).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw
@MendixMau

Copy link
Copy Markdown
Owner Author

Landed on master as part of the integration squash #329 (commit 31e90ae), after independent review — its Review: line is in that commit message. Closing as superseded.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants