e2e: read OQL rows by alias; liveness for negative-only span claims; APP-DOWN between journeys

CHANGELOG.md

@@ -17,6 +17,9 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are.
 
 ## Unreleased
 
+- fix(journeys): **the #302 preflight no longer refuses a password the project set in `project.config.js`; only the template placeholder is refused.** As shipped, #302 refused every run whose `TEST_PASS`/`APP_PASS` was not in the environment unless `ALLOW_CONFIG_CREDENTIALS=1` was set. But `project.config.template.js` tells every project to put its real demo password in `DEFAULT_PASS`, nothing in the toolkit loads an env file before `verify-module.sh` runs the journeys, and `sync-project.sh` refreshes unedited runner/config copies. So after the next sync every journeys rung would have faulted with rc 2. Now `config.js` reports `passSource` (`env` / `project.config.js` / `template placeholder`) and only the template's placeholder (`Demo2025!`) is refused. The env still wins, and `ALLOW_CONFIG_CREDENTIALS=1` still overrides. **Verdict change to know about from #302:** a refused admin credential (HTTP 401/403) used to leave every data rung INVALID with exit 1; the suite now aborts before the first journey with exit 2. A SIGINT/SIGTERM now writes the findings slot as `aborted: "signal SIGINT"` instead of leaving the previous run's file there. `report-normalize.js` keeps `rung: 'setup'` rows as kind `setup` instead of folding them into `ui` (`report-schema.md` enum updated). It also honours `aborted` in a findings file: the report gets a top-level `aborted`, every kept journey row is flagged `notAResult`, and the journey-runner instrument is a fault, so an aborted run never reads as results. It also reads the per-BRD conformance reports `report-<date>-<BRD>.tsv` that conformance-check now writes ([#307](https://github.com/MendixMau/mxcli-project-toolkit/pull/307)). It used to keep only `report-<date>.tsv`, so per-BRD modules never reached report.json and a stale base file could show as measured. It now takes the newest date among `report-<date>[-<any suffix>].tsv` and aggregates every file of that date, including BRD ids that contain a space. A per-BRD pointer `<BRD>:`/x`` keeps its requirement `/x`. Single-file projects are unchanged. Golden input: conformance-check.sh's real per-BRD output on #307's own fixture layout (`tests/wave2/fixtures/conformance-per-brd/`), plus `--selftest` cases and case 3 of `test-report-normalize-two-tree.sh`. Fixtures: `test-journey-preflight.sh` 74 cases (9 new, all failing on the #302 code: project-set password runs, placeholder refuses, env wins, SIGINT writes aborted findings end to end) and `report-normalize.js --selftest` cases (setup kind, aborted run). Refs [#286](https://github.com/MendixMau/mxcli-project-toolkit/issues/286) — MendixMau
+- fix(e2e): **`helpers.js` reads OQL rows by SELECT alias, never by position.** `mxcli oql --direct --json` prints keys alphabetically, leaves out a column that is null in every row, and over a view entity adds `ID` beside the alias. `Object.values(r)` therefore shifted columns, and `oqlScalar` returned a view row's ID instead of its count; a string value holding `]` or `}` truncated the brace-counting slice, so a real answer read as "no measurement" ([#154](https://github.com/MendixMau/mxcli-project-toolkit/issues/154), card-disbursement build). Rows are now keyed by the parsed SELECT aliases in SELECT order with explicit nulls, `oqlScalar` reads the first alias (null rather than a guess when the list has no alias), new `oqlRows`, string-aware JSON slicing; the admin preflight uses the same parser. **Behaviour change to check in your project:** `oqlScalar` now returns the **first SELECT alias**, where it used to return the alphabetically-first key. A multi-column seed, scope capture or baseline query whose value flows into `{{vars}}` may now resolve to a different column. Put the column you mean first, or make the query single-column. Golden: the released mxcli v0.26.0 binary's verbatim stdout/stderr for four queries against a stub admin API (`tests/wave2/fixtures/oql-json/`); fixture `tests/wave2/test-oql-rows.sh` (15 cases, 0/15 on the old helper). No live-app field run yet — MendixMau
+- fix(e2e): **a negative-only span claim is judged against a liveness reference, and an app that dies mid-suite reads APP-DOWN, not FAIL.** A `mustNotFire`-only step (Cancel on a reset confirmation) captures zero spans, which also happens when Jaeger is down, so it was INVALID on every run and its control mutant was unsupported ([#150](https://github.com/MendixMau/mxcli-project-toolkit/issues/150)). Rung 2 now waits the full window and requires a declared `spans.liveness` span or any service span since the journey began (`otel.js liveness`; the window starts at the journey's own start with a 2 s clock-skew margin); no evidence stays INVALID. An `otel.js` older than the runner, i.e. a hand-edited copy kept by sync, reads INVALID naming the stale file instead of throwing a TypeError. The `trace-negative` mutant injects the forbidden name after that check. A `spans` block declaring neither `ordered` nor `mustNotFire` is now a loud INVALID instead of a silent skip; the array shape `[{mf,status}]` and `{}` are named as such. `examples/validate-journeys.js` accepts a negative-only claim, which it used to reject as "NEVER checked" (contradicting the runner). It also checks the new `spans.liveness` field (only read for a negative-only claim) and a `mustNotFire` that is not a valid regex. `skills/journey-examples.md` documents the liveness rule. Fixture `tests/wave2/test-validate-journeys-spans.sh` (12 cases, 8 fail on the old validator). The shipped `project-tests/e2e/example.journey.json` is rewritten into the shape the runner reads. Its `name`/`role`, `value` seed, `page`/`expectPage`/`expectText`/`click` steps, array-shaped `spans` and SQL-style `<module>$<entity>` queries had drifted from it: the validator reported 3 problems on it, now 0, and all 7 control mutants are supported on it. Between journeys the runner probes `baseUrl` (3 tries); down before a journey → `APP-DOWN` INVALID, not walked; down after one → its FAIL rows are flagged ([#285](https://github.com/MendixMau/mxcli-project-toolkit/issues/285) harness half; a watch rebuild took a runtime down mid-suite on 2026-10-12). `APP_PROBE=0` disables it. Fixture `tests/wave2/test-journey-liveness.sh` (36 cases over the captured Jaeger trace and a real local HTTP server); main-loop wiring not fixtured (needs a browser), no field run yet — MendixMau
 - learn(testing-shape, test-result-audit): **three test-strength rules, each with a denominator: refusals probed on the server, every undeclared 4xx a failure, one clean-database final run** ([#172](https://github.com/MendixMau/mxcli-project-toolkit/issues/172)). A field comparison of two builds against one frozen spec found the toolkit build checked permission refusals only in the client (a hidden menu item), logged unexpected 400/422 responses as information, and reached 20/20 by stitching three runs. `testing-shape.md` §7 gains three report lines (`N of N restricted pairs probed server-side`, `undeclared 4xx/5xx: 0 of M requests — each fails its step`, `one clean-DB run <id>: N of N scenarios incl. timers, not stitched`); `test-result-audit.md` gains check 2e, which carries the reasons and the field evidence and asks for those numbers. Both baseline skills stay inside the 80,000-word baseline budget (79,997): the reasons live in the on-demand audit skill, and two redundant sentences in `testing-shape.md` §4's verdict paragraphs were folded (the empty-set corollary is stated in the Trace row, `harness-architecture.md` and `report-schema.md`). The backing inbox note is trimmed to its unpromoted "Keep" items. — MendixMau
 - learn(learned-detection-gaps, module-review): **two UI defects every model gate passes: a lazy-loaded combo box filter hides its preset, and a theme that pins row heights by position lets the next row cover an inserted widget** ([#299](https://github.com/MendixMau/mxcli-project-toolkit/issues/299)). Both passed check, mxbuild, lint and the scripted checks on a 127-module production app and were found only in the running app. Two register rows with the first oracle and the pre-build check (pluggable properties that change *when* data loads; grep the theme for fixed `height` on positional row classes before an insert), and two LOOK lines in `module-review.md` §4c with denominators (`N of N filters show their preset on first load`, `N of N widgets inserted into an existing container fully visible`). — jopterhorst
 - fix(app-restart, test-stack-up): **a port held by another process before `run --local` is named and reported as "instrument blocked" (exit 4), never killed** ([#262](https://github.com/MendixMau/mxcli-project-toolkit/issues/262)). Studio Pro had started its own runtime on the app/admin ports a restart needed; `mxcli run --local` then failed with "Authentication failed" (the admin API answering was Studio Pro's), the next journey ran against Studio Pro's runtime, and four more runs were NOT RUN. `app-restart.sh` now checks the app port and the admin port (record's `adminPort`, else app + 10) before stopping anything — a holder outside the recorded process tree exits 4 with its name, pid and parent — and again after the stop, where anything still holding a port is by definition not ours. `test-stack-up.sh` runs the same check wherever it would tell you to start `run --local` (`--no-docker`, no Docker daemon, `--check` with a leftover record), on the record's ports, `RUN_LOCAL_APP_PORT`/`RUN_LOCAL_ADMIN_PORT`, or mxcli's defaults 8080/8090. Holder lookup: lsof, then ss, then netstat + tasklist on Windows; unnamed holders still block when nothing of ours is running. Fixtures: `test-app-restart.sh` +2 cases (14 assertions, all fail on the old script), `test-stack-up-local.sh` T7 (4 of 6 fail on the old script). — AlvinnYeo

examples/journeys/full.journey.json

@@ -39,8 +39,8 @@
       "name": "supplierCode",
       "sql": "SELECT SupplierCode AS c FROM Procurement.Supplier WHERE Active = true LIMIT 1",
       "_note": [
-        "Every OQL column must be aliased — helpers.oqlScalar reads the first cell of the first",
-        "row, and mxcli rejects an unaliased SELECT. A seed that returns null or an empty",
+        "Every OQL column must be aliased — helpers.oqlScalar reads the first SELECT alias of the",
+        "first row, and mxcli rejects an unaliased SELECT. A seed that returns null or an empty",
         "string aborts the journey as INVALID (instrument absent), never FAIL (feature broken)."
       ]
     },

examples/validate-journeys.js

@@ -14,7 +14,7 @@
 // rule below encodes a way a journey can be wrong that the runner cannot tell
 // apart from a defect: a placeholder that reaches the page as the literal string
 // "{{supplierCode}}", an OQL check with no bar that is recorded INVALID forever, a
-// `mustNotFire` the runner never evaluates.
+// `spans` block that declares nothing the runner reads.
 //
 // WHAT IT IS NOT
 // It cannot tell you whether a selector, entity, microflow or requirement pointer
@@ -51,7 +51,7 @@ function validate(j, brds) {
     if (!s.sql) add(at, 'no `sql`');
     if (s.name && seen.has(s.name)) add(at, `duplicate seed name "${s.name}"`);
     if (s.sql && !/\bAS\b/i.test(s.sql))
-      add(at, 'SQL has no column alias — oqlScalar reads the first cell and mxcli rejects an unaliased SELECT');
+      add(at, 'SQL has no column alias — oqlScalar reads the first SELECT alias and mxcli rejects an unaliased SELECT');
     for (const k of [...String(s.sql || '').matchAll(/\{\{(\w+)\}\}/g)].map(m => m[1]))
       if (!seen.has(k)) add(at, `references {{${k}}}, which is not an EARLIER seed — seeds resolve in array order`);
     if (s.name) seen.add(s.name);
@@ -107,13 +107,29 @@ function validate(j, brds) {
         if (typeof t !== 'string')
           add(`${at}.ui.${key}`, 'contains a non-string — comment objects inside these arrays become assertions');
 
+    // A negative-only claim (`mustNotFire` with no `ordered`) is evaluated since #150: the
+    // runner judges it once tracing is shown live — the step's `liveness` span, or any span from
+    // the service since the journey began. `liveness` is read only for such a claim.
     const sp = st.spans;
-    if (sp) {
+    if (Array.isArray(sp)) {
+      add(`${at}.spans`, 'is an array — the old shape [{ mf, status }], which the runner never read (it records it INVALID); use { "ordered": [...], "mustNotFire": [...] }');
+    } else if (sp) {
       const ordered = sp.ordered || [];
-      if (!Array.isArray(ordered) || !ordered.length) {
-        if ((sp.mustNotFire || []).length)
-          add(`${at}.spans`, 'declares `mustNotFire` with no non-empty `ordered` — the runner evaluates the whole trace block only when `ordered` is non-empty, so this claim is NEVER checked');
-        else add(`${at}.spans`, 'is present but declares nothing the runner reads');
+      const mustNot = sp.mustNotFire || [];
+      if (!Array.isArray(ordered)) add(`${at}.spans.ordered`, 'must be an array');
+      if (!Array.isArray(mustNot)) add(`${at}.spans.mustNotFire`, 'must be an array');
+      const hasOrdered = Array.isArray(ordered) && ordered.length > 0;
+      const hasMustNot = Array.isArray(mustNot) && mustNot.length > 0;
+      if (!hasOrdered && !hasMustNot)
+        add(`${at}.spans`, 'is present but declares nothing the runner reads (neither `ordered` nor `mustNotFire`) — the runner records it INVALID');
+      for (const n of hasMustNot ? mustNot : []) {
+        try { new RegExp(n); } catch (e) { add(`${at}.spans.mustNotFire`, `"${n}" is not a valid regex — the runner compiles it with new RegExp()`); }
+      }
+      if (sp.liveness !== undefined) {
+        if (typeof sp.liveness !== 'string' || !sp.liveness.trim())
+          add(`${at}.spans.liveness`, 'must be a non-empty string: a microflow name or span operation the step is known to produce');
+        else if (hasOrdered)
+          add(`${at}.spans.liveness`, 'is only read for a negative-only claim (`mustNotFire` with no `ordered`) — with `ordered`, the claimed microflows are the liveness evidence');
       }
       for (const n of ordered)
         if (!String(n).includes('.'))
@@ -206,7 +222,12 @@ function selftest() {
     ['outcome with no sql', j => { delete j.outcome.sql; return j; }, 'reads `outcome.sql`'],
     ['ready given a bare widget name', j => { j.steps[0].ui.ready = 'dvThing'; return j; }, 'is not a CSS selector'],
     ['checks given a selector', j => { j.steps[0].ui.checks = ['.mx-name-btnGo']; return j; }, 'looks like a selector'],
-    ['mustNotFire with no ordered', j => { j.steps[0].spans = { mustNotFire: ['ACT_Del'] }; return j; }, 'NEVER checked'],
+    ['negative-only spans claim is accepted', j => { j.steps[0].spans = { mustNotFire: ['ACT_Del'] }; return j; }, null],
+    ['negative-only claim with liveness is accepted', j => { j.steps[0].spans = { ordered: [], mustNotFire: ['ACT_Del'], liveness: 'M.DS_Page' }; return j; }, null],
+    ['spans block declaring nothing', j => { j.steps[0].spans = { ordered: [] }; return j; }, 'declares nothing the runner reads'],
+    ['liveness not a string', j => { j.steps[0].spans = { mustNotFire: ['ACT_Del'], liveness: 5 }; return j; }, 'must be a non-empty string'],
+    ['liveness beside ordered', j => { j.steps[0].spans.liveness = 'M.DS_Page'; return j; }, 'only read for a negative-only claim'],
+    ['mustNotFire not a regex', j => { j.steps[0].spans.mustNotFire = ['ACT_(Del']; return j; }, 'not a valid regex'],
     ['unqualified ordered span', j => { j.steps[0].spans.ordered = ['ACT_Go']; return j; }, 'is unqualified'],
     ['unknown action', j => { j.steps[0].actions[0].do = 'swipe'; return j; }, 'unknown action'],
     ['wait using settleMs', j => { j.steps[0].actions = [{ do: 'wait', settleMs: 500 }]; return j; }, 'reads `ms`'],
@@ -224,7 +245,7 @@ function selftest() {
     console.log(`${ok ? 'ok  ' : 'FAIL'} ${name}${ok ? '' : `\n       got: ${problems.join('\n            ') || '(no problems reported)'}`}`);
   }
   console.log(bad ? `\n${bad} rule(s) could not be shown failing — do not trust a green from this tool.`
-                  : `\n${cases.length - 1} rules each watched going red; the clean case stays green.`);
+                  : `\n${cases.filter(c => c[2] !== null).length} rules each watched going red; ${cases.filter(c => c[2] === null).length} clean case(s) stay green.`);
   return bad ? 1 : 0;
 }
 

project-tests/e2e/config.js

@@ -81,6 +81,21 @@ const PORT = STACK.port;
 // and page-audit.js cannot end up running as two different identities.
 const CREDS = PROJ.credentials();
 
+// Where the test password came from, never the password itself (#278, #302 follow-up).
+//   env                  — TEST_PASS (or its APP_PASS alias)
+//   project.config.js    — a password the project set there (the template tells every
+//                          project to put its real demo password in DEFAULT_PASS)
+//   template placeholder — the value the shipped template carries, i.e. nobody set one
+// Only the last is refused by journey-runner's preflight. Refusing every non-env password
+// (as #302 first did) broke every project whose real password lives in project.config.js,
+// because nothing in the toolkit loads an env file before verify-module runs the journeys.
+const TEMPLATE_PASSWORDS = ['Demo2025!'];   // project.config.template.js DEFAULT_PASS
+function passwordSource({ env = process.env, creds = CREDS } = {}) {
+  if (env.TEST_PASS || env.APP_PASS) return 'env';
+  if (TEMPLATE_PASSWORDS.includes(creds.pass)) return 'template placeholder';
+  return 'project.config.js';
+}
+
 // ── M2EE admin credential: where it came from, never what it is ──────────────
 // Every data rung, seed and outcome query goes through `mxcli oql --direct` with this
 // token. It used to be `M2EE_ADMIN_PASS || '1'`: a literal default that silently differed
@@ -156,11 +171,13 @@ const cfg = {
   // is wrong: the login page shows the same generic message either way.
   user: CREDS.user,
   pass: CREDS.pass,
-  // Whether the test password came from the environment (TEST_PASS, or its APP_PASS alias)
-  // rather than project.config.js's default. journey-runner's preflight refuses to log in on a
-  // default (#278): a runner started without the project's env-loading step submitted the
-  // template password, then the admin fallback, and locked two demo accounts for ~5 min each.
+  // Whether the test password came from the environment (TEST_PASS, or its APP_PASS alias),
+  // and passSource (see passwordSource above). journey-runner's preflight refuses to log in on
+  // the TEMPLATE PLACEHOLDER only (#278): a runner started without the project's env-loading
+  // step submitted the template password, then the admin fallback, and locked two demo
+  // accounts for ~5 min each. A password the project set in project.config.js is accepted.
   passFromEnv: !!(process.env.TEST_PASS || process.env.APP_PASS),
+  passSource: passwordSource(),
   // Last-resort identity so a credential problem degrades to a reported finding
   // instead of a zero-coverage run. Set FALLBACK_USER='' to disable.
   fallbackUser: CREDS.fallbackUser,
@@ -213,4 +230,4 @@ function resolveTarget() {
   return t;
 }
 
-module.exports = { cfg, NAV, TARGETS, resolveTarget, resolveAdmin };
+module.exports = { cfg, NAV, TARGETS, resolveTarget, resolveAdmin, passwordSource, TEMPLATE_PASSWORDS };