diff --git a/CHANGELOG.md b/CHANGELOG.md index 20342404..c08dde8d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,9 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. ## Unreleased +- fix(journeys): **the #302 preflight no longer refuses a password the project set in `project.config.js`; only the template placeholder is refused.** As shipped, #302 refused every run whose `TEST_PASS`/`APP_PASS` was not in the environment unless `ALLOW_CONFIG_CREDENTIALS=1` was set. But `project.config.template.js` tells every project to put its real demo password in `DEFAULT_PASS`, nothing in the toolkit loads an env file before `verify-module.sh` runs the journeys, and `sync-project.sh` refreshes unedited runner/config copies. So after the next sync every journeys rung would have faulted with rc 2. Now `config.js` reports `passSource` (`env` / `project.config.js` / `template placeholder`) and only the template's placeholder (`Demo2025!`) is refused. The env still wins, and `ALLOW_CONFIG_CREDENTIALS=1` still overrides. **Verdict change to know about from #302:** a refused admin credential (HTTP 401/403) used to leave every data rung INVALID with exit 1; the suite now aborts before the first journey with exit 2. A SIGINT/SIGTERM now writes the findings slot as `aborted: "signal SIGINT"` instead of leaving the previous run's file there. `report-normalize.js` keeps `rung: 'setup'` rows as kind `setup` instead of folding them into `ui` (`report-schema.md` enum updated). It also honours `aborted` in a findings file: the report gets a top-level `aborted`, every kept journey row is flagged `notAResult`, and the journey-runner instrument is a fault, so an aborted run never reads as results. It also reads the per-BRD conformance reports `report--.tsv` that conformance-check now writes ([#307](https://github.com/MendixMau/mxcli-project-toolkit/pull/307)). It used to keep only `report-.tsv`, so per-BRD modules never reached report.json and a stale base file could show as measured. It now takes the newest date among `report-[-].tsv` and aggregates every file of that date, including BRD ids that contain a space. A per-BRD pointer `:`/x`` keeps its requirement `/x`. Single-file projects are unchanged. Golden input: conformance-check.sh's real per-BRD output on #307's own fixture layout (`tests/wave2/fixtures/conformance-per-brd/`), plus `--selftest` cases and case 3 of `test-report-normalize-two-tree.sh`. Fixtures: `test-journey-preflight.sh` 74 cases (9 new, all failing on the #302 code: project-set password runs, placeholder refuses, env wins, SIGINT writes aborted findings end to end) and `report-normalize.js --selftest` cases (setup kind, aborted run). Refs [#286](https://github.com/MendixMau/mxcli-project-toolkit/issues/286) — MendixMau +- fix(e2e): **`helpers.js` reads OQL rows by SELECT alias, never by position.** `mxcli oql --direct --json` prints keys alphabetically, leaves out a column that is null in every row, and over a view entity adds `ID` beside the alias. `Object.values(r)` therefore shifted columns, and `oqlScalar` returned a view row's ID instead of its count; a string value holding `]` or `}` truncated the brace-counting slice, so a real answer read as "no measurement" ([#154](https://github.com/MendixMau/mxcli-project-toolkit/issues/154), card-disbursement build). Rows are now keyed by the parsed SELECT aliases in SELECT order with explicit nulls, `oqlScalar` reads the first alias (null rather than a guess when the list has no alias), new `oqlRows`, string-aware JSON slicing; the admin preflight uses the same parser. **Behaviour change to check in your project:** `oqlScalar` now returns the **first SELECT alias**, where it used to return the alphabetically-first key. A multi-column seed, scope capture or baseline query whose value flows into `{{vars}}` may now resolve to a different column. Put the column you mean first, or make the query single-column. Golden: the released mxcli v0.26.0 binary's verbatim stdout/stderr for four queries against a stub admin API (`tests/wave2/fixtures/oql-json/`); fixture `tests/wave2/test-oql-rows.sh` (15 cases, 0/15 on the old helper). No live-app field run yet — MendixMau +- fix(e2e): **a negative-only span claim is judged against a liveness reference, and an app that dies mid-suite reads APP-DOWN, not FAIL.** A `mustNotFire`-only step (Cancel on a reset confirmation) captures zero spans, which also happens when Jaeger is down, so it was INVALID on every run and its control mutant was unsupported ([#150](https://github.com/MendixMau/mxcli-project-toolkit/issues/150)). Rung 2 now waits the full window and requires a declared `spans.liveness` span or any service span since the journey began (`otel.js liveness`; the window starts at the journey's own start with a 2 s clock-skew margin); no evidence stays INVALID. An `otel.js` older than the runner, i.e. a hand-edited copy kept by sync, reads INVALID naming the stale file instead of throwing a TypeError. The `trace-negative` mutant injects the forbidden name after that check. A `spans` block declaring neither `ordered` nor `mustNotFire` is now a loud INVALID instead of a silent skip; the array shape `[{mf,status}]` and `{}` are named as such. `examples/validate-journeys.js` accepts a negative-only claim, which it used to reject as "NEVER checked" (contradicting the runner). It also checks the new `spans.liveness` field (only read for a negative-only claim) and a `mustNotFire` that is not a valid regex. `skills/journey-examples.md` documents the liveness rule. Fixture `tests/wave2/test-validate-journeys-spans.sh` (12 cases, 8 fail on the old validator). The shipped `project-tests/e2e/example.journey.json` is rewritten into the shape the runner reads. Its `name`/`role`, `value` seed, `page`/`expectPage`/`expectText`/`click` steps, array-shaped `spans` and SQL-style `$` queries had drifted from it: the validator reported 3 problems on it, now 0, and all 7 control mutants are supported on it. Between journeys the runner probes `baseUrl` (3 tries); down before a journey → `APP-DOWN` INVALID, not walked; down after one → its FAIL rows are flagged ([#285](https://github.com/MendixMau/mxcli-project-toolkit/issues/285) harness half; a watch rebuild took a runtime down mid-suite on 2026-10-12). `APP_PROBE=0` disables it. Fixture `tests/wave2/test-journey-liveness.sh` (36 cases over the captured Jaeger trace and a real local HTTP server); main-loop wiring not fixtured (needs a browser), no field run yet — MendixMau - learn(testing-shape, test-result-audit): **three test-strength rules, each with a denominator: refusals probed on the server, every undeclared 4xx a failure, one clean-database final run** ([#172](https://github.com/MendixMau/mxcli-project-toolkit/issues/172)). A field comparison of two builds against one frozen spec found the toolkit build checked permission refusals only in the client (a hidden menu item), logged unexpected 400/422 responses as information, and reached 20/20 by stitching three runs. `testing-shape.md` §7 gains three report lines (`N of N restricted pairs probed server-side`, `undeclared 4xx/5xx: 0 of M requests — each fails its step`, `one clean-DB run : N of N scenarios incl. timers, not stitched`); `test-result-audit.md` gains check 2e, which carries the reasons and the field evidence and asks for those numbers. Both baseline skills stay inside the 80,000-word baseline budget (79,997): the reasons live in the on-demand audit skill, and two redundant sentences in `testing-shape.md` §4's verdict paragraphs were folded (the empty-set corollary is stated in the Trace row, `harness-architecture.md` and `report-schema.md`). The backing inbox note is trimmed to its unpromoted "Keep" items. — MendixMau - learn(learned-detection-gaps, module-review): **two UI defects every model gate passes: a lazy-loaded combo box filter hides its preset, and a theme that pins row heights by position lets the next row cover an inserted widget** ([#299](https://github.com/MendixMau/mxcli-project-toolkit/issues/299)). Both passed check, mxbuild, lint and the scripted checks on a 127-module production app and were found only in the running app. Two register rows with the first oracle and the pre-build check (pluggable properties that change *when* data loads; grep the theme for fixed `height` on positional row classes before an insert), and two LOOK lines in `module-review.md` §4c with denominators (`N of N filters show their preset on first load`, `N of N widgets inserted into an existing container fully visible`). — jopterhorst - fix(app-restart, test-stack-up): **a port held by another process before `run --local` is named and reported as "instrument blocked" (exit 4), never killed** ([#262](https://github.com/MendixMau/mxcli-project-toolkit/issues/262)). Studio Pro had started its own runtime on the app/admin ports a restart needed; `mxcli run --local` then failed with "Authentication failed" (the admin API answering was Studio Pro's), the next journey ran against Studio Pro's runtime, and four more runs were NOT RUN. `app-restart.sh` now checks the app port and the admin port (record's `adminPort`, else app + 10) before stopping anything — a holder outside the recorded process tree exits 4 with its name, pid and parent — and again after the stop, where anything still holding a port is by definition not ours. `test-stack-up.sh` runs the same check wherever it would tell you to start `run --local` (`--no-docker`, no Docker daemon, `--check` with a leftover record), on the record's ports, `RUN_LOCAL_APP_PORT`/`RUN_LOCAL_ADMIN_PORT`, or mxcli's defaults 8080/8090. Holder lookup: lsof, then ss, then netstat + tasklist on Windows; unnamed holders still block when nothing of ours is running. Fixtures: `test-app-restart.sh` +2 cases (14 assertions, all fail on the old script), `test-stack-up-local.sh` T7 (4 of 6 fail on the old script). — AlvinnYeo diff --git a/examples/journeys/full.journey.json b/examples/journeys/full.journey.json index 14e81794..59eebc2c 100644 --- a/examples/journeys/full.journey.json +++ b/examples/journeys/full.journey.json @@ -39,8 +39,8 @@ "name": "supplierCode", "sql": "SELECT SupplierCode AS c FROM Procurement.Supplier WHERE Active = true LIMIT 1", "_note": [ - "Every OQL column must be aliased — helpers.oqlScalar reads the first cell of the first", - "row, and mxcli rejects an unaliased SELECT. A seed that returns null or an empty", + "Every OQL column must be aliased — helpers.oqlScalar reads the first SELECT alias of the", + "first row, and mxcli rejects an unaliased SELECT. A seed that returns null or an empty", "string aborts the journey as INVALID (instrument absent), never FAIL (feature broken)." ] }, diff --git a/examples/validate-journeys.js b/examples/validate-journeys.js index c080deb2..2d87dd21 100755 --- a/examples/validate-journeys.js +++ b/examples/validate-journeys.js @@ -14,7 +14,7 @@ // rule below encodes a way a journey can be wrong that the runner cannot tell // apart from a defect: a placeholder that reaches the page as the literal string // "{{supplierCode}}", an OQL check with no bar that is recorded INVALID forever, a -// `mustNotFire` the runner never evaluates. +// `spans` block that declares nothing the runner reads. // // WHAT IT IS NOT // It cannot tell you whether a selector, entity, microflow or requirement pointer @@ -51,7 +51,7 @@ function validate(j, brds) { if (!s.sql) add(at, 'no `sql`'); if (s.name && seen.has(s.name)) add(at, `duplicate seed name "${s.name}"`); if (s.sql && !/\bAS\b/i.test(s.sql)) - add(at, 'SQL has no column alias — oqlScalar reads the first cell and mxcli rejects an unaliased SELECT'); + add(at, 'SQL has no column alias — oqlScalar reads the first SELECT alias and mxcli rejects an unaliased SELECT'); for (const k of [...String(s.sql || '').matchAll(/\{\{(\w+)\}\}/g)].map(m => m[1])) if (!seen.has(k)) add(at, `references {{${k}}}, which is not an EARLIER seed — seeds resolve in array order`); if (s.name) seen.add(s.name); @@ -107,13 +107,29 @@ function validate(j, brds) { if (typeof t !== 'string') add(`${at}.ui.${key}`, 'contains a non-string — comment objects inside these arrays become assertions'); + // A negative-only claim (`mustNotFire` with no `ordered`) is evaluated since #150: the + // runner judges it once tracing is shown live — the step's `liveness` span, or any span from + // the service since the journey began. `liveness` is read only for such a claim. const sp = st.spans; - if (sp) { + if (Array.isArray(sp)) { + add(`${at}.spans`, 'is an array — the old shape [{ mf, status }], which the runner never read (it records it INVALID); use { "ordered": [...], "mustNotFire": [...] }'); + } else if (sp) { const ordered = sp.ordered || []; - if (!Array.isArray(ordered) || !ordered.length) { - if ((sp.mustNotFire || []).length) - add(`${at}.spans`, 'declares `mustNotFire` with no non-empty `ordered` — the runner evaluates the whole trace block only when `ordered` is non-empty, so this claim is NEVER checked'); - else add(`${at}.spans`, 'is present but declares nothing the runner reads'); + const mustNot = sp.mustNotFire || []; + if (!Array.isArray(ordered)) add(`${at}.spans.ordered`, 'must be an array'); + if (!Array.isArray(mustNot)) add(`${at}.spans.mustNotFire`, 'must be an array'); + const hasOrdered = Array.isArray(ordered) && ordered.length > 0; + const hasMustNot = Array.isArray(mustNot) && mustNot.length > 0; + if (!hasOrdered && !hasMustNot) + add(`${at}.spans`, 'is present but declares nothing the runner reads (neither `ordered` nor `mustNotFire`) — the runner records it INVALID'); + for (const n of hasMustNot ? mustNot : []) { + try { new RegExp(n); } catch (e) { add(`${at}.spans.mustNotFire`, `"${n}" is not a valid regex — the runner compiles it with new RegExp()`); } + } + if (sp.liveness !== undefined) { + if (typeof sp.liveness !== 'string' || !sp.liveness.trim()) + add(`${at}.spans.liveness`, 'must be a non-empty string: a microflow name or span operation the step is known to produce'); + else if (hasOrdered) + add(`${at}.spans.liveness`, 'is only read for a negative-only claim (`mustNotFire` with no `ordered`) — with `ordered`, the claimed microflows are the liveness evidence'); } for (const n of ordered) if (!String(n).includes('.')) @@ -206,7 +222,12 @@ function selftest() { ['outcome with no sql', j => { delete j.outcome.sql; return j; }, 'reads `outcome.sql`'], ['ready given a bare widget name', j => { j.steps[0].ui.ready = 'dvThing'; return j; }, 'is not a CSS selector'], ['checks given a selector', j => { j.steps[0].ui.checks = ['.mx-name-btnGo']; return j; }, 'looks like a selector'], - ['mustNotFire with no ordered', j => { j.steps[0].spans = { mustNotFire: ['ACT_Del'] }; return j; }, 'NEVER checked'], + ['negative-only spans claim is accepted', j => { j.steps[0].spans = { mustNotFire: ['ACT_Del'] }; return j; }, null], + ['negative-only claim with liveness is accepted', j => { j.steps[0].spans = { ordered: [], mustNotFire: ['ACT_Del'], liveness: 'M.DS_Page' }; return j; }, null], + ['spans block declaring nothing', j => { j.steps[0].spans = { ordered: [] }; return j; }, 'declares nothing the runner reads'], + ['liveness not a string', j => { j.steps[0].spans = { mustNotFire: ['ACT_Del'], liveness: 5 }; return j; }, 'must be a non-empty string'], + ['liveness beside ordered', j => { j.steps[0].spans.liveness = 'M.DS_Page'; return j; }, 'only read for a negative-only claim'], + ['mustNotFire not a regex', j => { j.steps[0].spans.mustNotFire = ['ACT_(Del']; return j; }, 'not a valid regex'], ['unqualified ordered span', j => { j.steps[0].spans.ordered = ['ACT_Go']; return j; }, 'is unqualified'], ['unknown action', j => { j.steps[0].actions[0].do = 'swipe'; return j; }, 'unknown action'], ['wait using settleMs', j => { j.steps[0].actions = [{ do: 'wait', settleMs: 500 }]; return j; }, 'reads `ms`'], @@ -224,7 +245,7 @@ function selftest() { console.log(`${ok ? 'ok ' : 'FAIL'} ${name}${ok ? '' : `\n got: ${problems.join('\n ') || '(no problems reported)'}`}`); } console.log(bad ? `\n${bad} rule(s) could not be shown failing — do not trust a green from this tool.` - : `\n${cases.length - 1} rules each watched going red; the clean case stays green.`); + : `\n${cases.filter(c => c[2] !== null).length} rules each watched going red; ${cases.filter(c => c[2] === null).length} clean case(s) stay green.`); return bad ? 1 : 0; } diff --git a/project-tests/e2e/config.js b/project-tests/e2e/config.js index badbac59..d0c72b1b 100644 --- a/project-tests/e2e/config.js +++ b/project-tests/e2e/config.js @@ -81,6 +81,21 @@ const PORT = STACK.port; // and page-audit.js cannot end up running as two different identities. const CREDS = PROJ.credentials(); +// Where the test password came from, never the password itself (#278, #302 follow-up). +// env — TEST_PASS (or its APP_PASS alias) +// project.config.js — a password the project set there (the template tells every +// project to put its real demo password in DEFAULT_PASS) +// template placeholder — the value the shipped template carries, i.e. nobody set one +// Only the last is refused by journey-runner's preflight. Refusing every non-env password +// (as #302 first did) broke every project whose real password lives in project.config.js, +// because nothing in the toolkit loads an env file before verify-module runs the journeys. +const TEMPLATE_PASSWORDS = ['Demo2025!']; // project.config.template.js DEFAULT_PASS +function passwordSource({ env = process.env, creds = CREDS } = {}) { + if (env.TEST_PASS || env.APP_PASS) return 'env'; + if (TEMPLATE_PASSWORDS.includes(creds.pass)) return 'template placeholder'; + return 'project.config.js'; +} + // ── M2EE admin credential: where it came from, never what it is ────────────── // Every data rung, seed and outcome query goes through `mxcli oql --direct` with this // token. It used to be `M2EE_ADMIN_PASS || '1'`: a literal default that silently differed @@ -156,11 +171,13 @@ const cfg = { // is wrong: the login page shows the same generic message either way. user: CREDS.user, pass: CREDS.pass, - // Whether the test password came from the environment (TEST_PASS, or its APP_PASS alias) - // rather than project.config.js's default. journey-runner's preflight refuses to log in on a - // default (#278): a runner started without the project's env-loading step submitted the - // template password, then the admin fallback, and locked two demo accounts for ~5 min each. + // Whether the test password came from the environment (TEST_PASS, or its APP_PASS alias), + // and passSource (see passwordSource above). journey-runner's preflight refuses to log in on + // the TEMPLATE PLACEHOLDER only (#278): a runner started without the project's env-loading + // step submitted the template password, then the admin fallback, and locked two demo + // accounts for ~5 min each. A password the project set in project.config.js is accepted. passFromEnv: !!(process.env.TEST_PASS || process.env.APP_PASS), + passSource: passwordSource(), // Last-resort identity so a credential problem degrades to a reported finding // instead of a zero-coverage run. Set FALLBACK_USER='' to disable. fallbackUser: CREDS.fallbackUser, @@ -213,4 +230,4 @@ function resolveTarget() { return t; } -module.exports = { cfg, NAV, TARGETS, resolveTarget, resolveAdmin }; +module.exports = { cfg, NAV, TARGETS, resolveTarget, resolveAdmin, passwordSource, TEMPLATE_PASSWORDS }; diff --git a/project-tests/e2e/example.journey.json b/project-tests/e2e/example.journey.json index 27cd39e9..f25e08f8 100644 --- a/project-tests/e2e/example.journey.json +++ b/project-tests/e2e/example.journey.json @@ -20,63 +20,89 @@ "so a single good historical row carries the assertion. A data block that carries", "assocMustBeSet with no scope must be reported [WEAK EVIDENCE], never silently accepted.", "", - "Names below are placeholders. Replace // with your own." + "Names below are placeholders. Replace /// with your own.", + "Every key here is one journey-runner.js reads (examples/validate-journeys.js reports 0", + "problems on this file). The field-by-field tour is skills/journey-examples.md.", + "Add \"persona\": \"\" when the journey proves one role's access path — a run signed in as anyone else records the journey INVALID and does not walk it." ], - - "name": "Example", - "role": "", - "goal": "One sentence in the user's language. Not 'the microflow succeeds'.", - + "id": "J-EXAMPLE-01", + "title": "Example — open the overview, submit the form, see the row on its parent", + "requirement": [], "seeds": [ { "name": "parentKey", - "$comment": "Referenced later as {{parentKey}}. Seeds are the only values a step may assume exist.", - "value": "EXAMPLE-0001" + "$comment": "Referenced later as {{parentKey}}. Seeds are the only values a step may assume exist; they are read from the live DB, never hardcoded. Every OQL column is aliased.", + "sql": "SELECT p. AS k FROM . AS p LIMIT 1" } ], - "steps": [ { "name": "Open the overview and see the seeded row", - "page": ".", "$comment": "rung 1 landing guard + rung 2 what the screen says. Without the guard an assertion grades the PREVIOUS page.", - "expectPage": ".", - "expectText": ["{{parentKey}}"] + "actions": [ + { + "do": "nav", + "item": "" + } + ], + "ui": { + "ready": ".mx-name-", + "textPresent": [ + "{{parentKey}}" + ] + } }, { "name": "Submit the form", - "click": "