diff --git a/CHANGELOG.md b/CHANGELOG.md index c1a77322..b6b57890 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. Named m ## Unreleased +- learn(load-test): **new on-demand skill for load testing a page without correlating Mendix's client protocol by hand: Playwright records the real `/xas/` calls, a probe maps each opaque `runtimeOperation` to its microflow through Jaeger, and the chosen call is replayed with the session's own cookie and CSRF token.** Ten field traps (HTTP 200 on a failed action, caught REST errors under a green microflow span, the wrong call hammered, a recording that still held the previous page, a tracing check that failed open, a harness that throttled itself, harness vs app limits, requests vs users, sampled trace counts, empty data) and a four-line report that says what the run does not prove. Routed ondemand for lead, test and review; no instrument ships yet and the roadmap's "no load testing" line is reworded. Field evidence: a warehouse demo app, Mendix 11, 16 runs 2026-09-09 to 2026-10-08, bursts of 1000 and 2000 concurrent calls on one session — MendixMau - new(process-map): **Process map: one swimlane map per business process, each step on its wireframe once screens exist, every open question and ASSUMED decision pinned to the step it is about; drawn twice, a draft at Stage 2 close without screens (pass 1, shown at CAC-3) and the full map at Stage 3 with screens (pass 2, walked at the Stage 3 gate close-out).** `skills/process-map.md` (when, model, anchoring, build, verify by looking) and `project-bin/process-map.js` (`validate`, `anchor`, `build`; Node stdlib, deterministic) with its template. Named "map" because "process" in the progress report means the toolkit's own stages, while this shows the app's business process. With no wireframes, a step may name a screen that is not drawn yet or have none, and a person step stays a person step; once a wireframe exists `validate` fails a step page with no wireframe, a use case on no process and not standalone, a decision with one outcome, two steps in one cell. `anchor` places items mechanically (named microflow, listed or named use case, named screen, overlapping source lines) and keeps judged placements by fingerprint, so a reworded question is unplaced again instead of silently wrong; global buckets are allowed and counted. Wired into the Stage 3 Agent produces and Surface rows, a `process-map` opt-in manifest row (an existing model counts as opting in), the `swimlanes` opt-in row (the model discharges it), CAC-3 and CAC-4, and a check_stage_3 FAIL when the map is older than its model, its anchors or a wireframe. `tests/wave2/test-process-map.sh` on a synthetic order approval fixture with its own committed BRD, covering both passes. Field evidence: a requirements-driven rebuild POC, 2026-10-16, where the reviewer could not tell where the sixty questions fit; the map placed 102 items with 0 unplaced. — MendixMau - process(testing): **unit tests are chosen per flow, not per module: a module brief lists `Unit flows:` (one `- ACT_X: reason` line each) and `affected-tests.sh` runs only the listed flows the change touched.** Follow-up to #417 on the maintainer's policy that unit-testing everything is overkill (logic and integrations only, UI journeys catch the rest). Module done runs the `--module` modules' listed flows only; a list wins for its module even under `Unit tests default: yes`; bullets (`-`, `*`, `+`, `1.`, `1)`) may be separated by blank lines, carry `(note)` or `Module.` forms, or sit inline (`Unit flows: A: money, B: rest`), a `## Unit flows` heading works with or without a colon, a line that cannot be read warns with its text, and a Unit flows block that yields no flow warns; under default yes a test is dropped only when every module it names has a list that excludes it; CRLF briefs are read; `Unit tests: yes` or `Unit modules:` with no list keeps the module-level selection; a flow listed without a reason, with no test, with a different casing than the model, or not in the model (`SHOW MICROFLOWS IN`), and a module the model lacks, warn. Test: wave2 `test-affected-tests.sh` L10 (35 checks). Baseline unchanged at 75,194 words (budget 80,000) — MendixMau - fix(sync-project, doctor): **a project whose `.claude/settings.json` lacks the `look-ledger.sh seen` hook now gets it back on the next sync, and `doctor.sh` warns when it is missing.** In a lab project the look ledger read "0 of 31 pages seen" while 22 of them had screenshots: `install-claude-permissions.sh` (the only writer of the hook) had never run there, `sync-project.sh` only printed a warning in a long report, and `doctor.sh` checked the allow-list but not the two hooks, so `seen.tsv` stayed empty and any gate built on the ledger would have stalled the build. `install-claude-permissions.sh` gains `--hooks-only` (add the SessionStart and PostToolUse hooks, touch no allow, deny or ask entry, prune nothing, write nothing to `settings.local.json`); `sync-project.sh` runs it when the hooks are missing (`--dry-run` says what it would add); `doctor.sh` names the missing hook and the fix. `tests/wave2/test-look-ledger-hook-repair.sh` reproduces the gap, then proves detect, repair, idempotence and that a Read of a screenshot writes a seen row. Review follow-up: sync repairs only a project that installed the profile before (sidecar with a non-empty allow, or `.mxtk/permission-profile`; checked by `--consent-check`), otherwise it WARNs with the one-line `--hooks-only` command; `.mxtk/no-hook-repair` is the documented opt-out (sync and doctor); `--hooks-only` creates no empty allow/deny, leaves no `.bak`, keeps the file's indent and non-ASCII, and leaves a hand-formatted file untouched (exit 3, lines printed). — lab project field run (night 2026-10-16) diff --git a/README.md b/README.md index 95f9a546..59d5cbae 100644 --- a/README.md +++ b/README.md @@ -671,6 +671,7 @@ Every mxcli project has a `.ai-context/skills/` directory (bundled by `mxcli ini | Proving a module's user journey end-to-end — the deep form of step 3 PROVE; use whenever an instrument reports green and you cannot say what would have made it red | `skills/journey-proof.md` | | Writing or running e2e cases beyond the happy journeys (blank/whitespace required fields, over-length, duplicates, cancel, double submit, wrong state, wrong role, backend down, stale write, empty search) — every case cites a requirement first, UNSPECIFIED is re-scored against house rules, GAP counts carry the denominator; runs after journey-proof and before monkey-test | `skills/unhappy-path-testing.md` | | Running the fuzz/crash net on a module whose journeys are already green — and reading the result, which is NOT evidence the module works | `skills/monkey-test.md` | +| Load, performance or stress testing a page ("will it hold 1000 users", concurrent users, k6, JMeter) — record the real /xas/ calls with Playwright and replay them instead of correlating tokens by hand, and report what the number does NOT prove (requests vs users, empty data, whose limit was hit) | `skills/load-test.md` | | Writing or running a microflow unit test (.test.mdl, mxcli test, @expect) once the project or the user has decided a flow gets one — what is worth a test, equality-only asserts, --list before any run, a scratch copy so the live app survives, and a positive control that must fail | `skills/unit-test.md` | | UX audit and screenshot-loop discipline | `skills/learned-skill-ux-audit.md` | | Tracking scope delta between the BRD and the built state | `skills/learned-skill-scope-delta.md` | diff --git a/ROUTING.md b/ROUTING.md index e07ec03c..e2fe31e1 100644 --- a/ROUTING.md +++ b/ROUTING.md @@ -223,6 +223,7 @@ picks the row up. That is the whole procedure — there is no second list to rem | Proving a module's user journey end-to-end — the deep form of step 3 PROVE; use whenever an instrument reports green and you cannot say what would have made it red | `skills/journey-proof.md` | test,review | 5,6 | ondemand | | Writing or running e2e cases beyond the happy journeys (blank/whitespace required fields, over-length, duplicates, cancel, double submit, wrong state, wrong role, backend down, stale write, empty search) — every case cites a requirement first, UNSPECIFIED is re-scored against house rules, GAP counts carry the denominator; runs after journey-proof and before monkey-test | `skills/unhappy-path-testing.md` | test,review | 5,6 | ondemand | | Running the fuzz/crash net on a module whose journeys are already green — and reading the result, which is NOT evidence the module works | `skills/monkey-test.md` | test,review | 5,6 | ondemand | +| Load, performance or stress testing a page ("will it hold 1000 users", concurrent users, k6, JMeter) — record the real /xas/ calls with Playwright and replay them instead of correlating tokens by hand, and report what the number does NOT prove (requests vs users, empty data, whose limit was hit) | `skills/load-test.md` | lead,test,review | 5,6 | ondemand | | Writing or running a microflow unit test (.test.mdl, mxcli test, @expect) once the project or the user has decided a flow gets one — what is worth a test, equality-only asserts, --list before any run, a scratch copy so the live app survives, and a positive control that must fail | `skills/unit-test.md` | lead,mdl,test,review | 5,6 | ondemand | | UX audit and screenshot-loop discipline | `skills/learned-skill-ux-audit.md` | review | 6 | ondemand | | Tracking scope delta between the BRD and the built state | `skills/learned-skill-scope-delta.md` | review | 6 | ondemand | diff --git a/agents/review-agent.md b/agents/review-agent.md index 3118046d..21a2677c 100644 --- a/agents/review-agent.md +++ b/agents/review-agent.md @@ -71,6 +71,7 @@ This table is your whole list. The baseline table in the project's CLAUDE.local. | `skills/journey-proof.md` | Proving a module's user journey end-to-end | | `skills/unhappy-path-testing.md` | Writing or running e2e cases beyond the happy journeys (blank/whitespace required fields, over-length, duplicates, cancel, double submit, wrong state, wrong role, backend down, stale write, empty search) | | `skills/monkey-test.md` | Running the fuzz/crash net on a module whose journeys are already green | +| `skills/load-test.md` | Load, performance or stress testing a page ("will it hold 1000 users", concurrent users, k6, JMeter) | | `skills/unit-test.md` | Writing or running a microflow unit test (.test.mdl, mxcli test, @expect) once the project or the user has decided a flow gets one | | `skills/e2e-locators-that-lie.md` | An e2e rung reports a UI defect (widget never appeared, click did nothing, screen contradicts the data, fixture count wrong) | | `skills/learned-skill-ux-audit.md` | UX audit and screenshot-loop discipline | diff --git a/agents/test-agent.md b/agents/test-agent.md index e5ed79a9..88d69ae9 100644 --- a/agents/test-agent.md +++ b/agents/test-agent.md @@ -55,6 +55,7 @@ This table is your whole list. The baseline table in the project's CLAUDE.local. | `skills/journey-proof.md` | Proving a module's user journey end-to-end | | `skills/unhappy-path-testing.md` | Writing or running e2e cases beyond the happy journeys (blank/whitespace required fields, over-length, duplicates, cancel, double submit, wrong state, wrong role, backend down, stale write, empty search) | | `skills/monkey-test.md` | Running the fuzz/crash net on a module whose journeys are already green | +| `skills/load-test.md` | Load, performance or stress testing a page ("will it hold 1000 users", concurrent users, k6, JMeter) | | `skills/unit-test.md` | Writing or running a microflow unit test (.test.mdl, mxcli test, @expect) once the project or the user has decided a flow gets one | | `skills/e2e-locators-that-lie.md` | An e2e rung reports a UI defect (widget never appeared, click did nothing, screen contradicts the data, fixture count wrong) | | `skills/company-brain.md` | Setting up or wiring a COMPANY BRAIN | diff --git a/bin/gate-check.sh b/bin/gate-check.sh index 87fb892e..78a62a9a 100755 --- a/bin/gate-check.sh +++ b/bin/gate-check.sh @@ -1789,8 +1789,8 @@ stage_protocol_paths() { 2) echo "skills/interview-protocol.md skills/change-intake.md skills/change-intake-detail.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-brd.md skills/checkpoints/checkpoint-architecture.md skills/dispatching.md skills/image-transcription.md skills/small-project-tier.md skills/kb-generation.md skills/brd-generation.md skills/brd-validation.md" ;; 3) echo "skills/interview-protocol.md skills/change-intake.md skills/change-intake-detail.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-design.md skills/dispatching.md skills/small-project-tier.md skills/mendix-best-practices-index.md skills/decompose-app.md skills/mendix-service-seam.md skills/layering-review.md skills/architecture-blueprint.md skills/modularize-domain.md skills/design-artifacts.md skills/process-map.md skills/interaction-model.md skills/brd-to-build-plan.md skills/workflow-structure-rules.md skills/rest-between-mendix-apps.md skills/learned-mdl-cannot-express.md" ;; 4) echo "skills/interview-protocol.md skills/change-intake.md skills/change-intake-detail.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-build.md skills/agent-roles.md skills/dispatching.md skills/small-project-tier.md skills/module-brief.md skills/module-folder-convention.md skills/brd-to-build-plan.md skills/coverage-ledger.md skills/workflow-structure-rules.md skills/rest-integration-first-time-right.md skills/rest-between-mendix-apps.md skills/learned-constants-and-secrets.md" ;; - 5|build-ready) echo "skills/interview-protocol.md skills/change-intake.md skills/change-intake-detail.md skills/grill-mode.md skills/agent-roles.md skills/dispatching.md skills/module-brief.md skills/learned-mdl-preflight.md skills/module-folder-convention.md skills/learned-microflow-patterns.md skills/microflow-preflight.md skills/mendix-best-practices-index.md skills/ui-preflight-pages.md skills/design-spacing.md skills/learned-stylegallery.md skills/ui-loop.md skills/module-review.md skills/testing-shape.md skills/decompose-app.md skills/mendix-service-seam.md skills/microflow-loop-antipatterns.md skills/rehost-app.md skills/iterative-build-loop.md skills/mdl-cookbook-microflows.md skills/build/mdl/oneshot-mdl-method.md skills/learned-page-patterns.md skills/oneshot-page-structure-patterns.md skills/mendix-agents.md skills/mendix-agent-ui.md skills/mendix-agent-setup.md skills/fixture-seeding.md skills/journey-proof.md skills/unhappy-path-testing.md skills/monkey-test.md skills/unit-test.md skills/e2e-locators-that-lie.md skills/report-schema.md skills/harness-architecture.md skills/process-coherence-pass.md skills/lint-that-actually-runs.md skills/improvement-register.md skills/journey-examples.md skills/wiring-sweep.md skills/learned-workflow-patterns.md skills/workflow-structure-rules.md skills/rest-integration-first-time-right.md skills/rest-between-mendix-apps.md skills/bug-submission-checklist.md skills/empty-widget-triage.md skills/learned-sidebar-collapse-icons.md skills/learned-popup-navigation.md skills/learned-datagrid-customcontent-binding.md skills/learned-popup-feedback-pattern.md skills/learned-mdl-cannot-express.md skills/learned-css-that-never-applied.md skills/learned-detection-gaps.md skills/learned-dg2-patterns.md skills/learned-file-upload-widget.md skills/security-is-not-a-later-script.md skills/learned-local-db-confusion.md skills/full-harness-audit.md skills/test-result-audit.md skills/finding-disposition.md skills/preview-over-hub-tunnel.md skills/walking-skeleton.md skills/platform-link.md skills/teamserver-alignment.md skills/learned-constants-and-secrets.md" ;; - 6) echo "skills/interview-protocol.md skills/change-intake.md skills/change-intake-detail.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-cutover.md skills/dispatching.md skills/mendix-best-practices-index.md skills/module-review.md skills/testing-shape.md skills/existing-app-assurance.md skills/decompose-app.md skills/app-analysis.md skills/module-dependency-review.md skills/microflow-loop-antipatterns.md skills/rehost-app.md skills/qa-loop-goal-pattern.md skills/mendix-agent-setup.md skills/e2e-harness-base.md skills/learned-db-assertions.md skills/fixture-seeding.md skills/journey-proof.md skills/unhappy-path-testing.md skills/monkey-test.md skills/unit-test.md skills/e2e-locators-that-lie.md skills/learned-skill-ux-audit.md skills/learned-skill-scope-delta.md skills/report-schema.md skills/harness-architecture.md skills/process-coherence-pass.md skills/e2e-evidence-report.md skills/record-demo-video.md skills/demo-recording-polish.md skills/share-demo-package.md skills/lint-that-actually-runs.md skills/improvement-register.md skills/journey-examples.md skills/wiring-sweep.md skills/workflow-structure-rules.md skills/rest-between-mendix-apps.md skills/bug-submission-checklist.md skills/empty-widget-triage.md skills/anonymize-client-app-for-demo.md skills/learned-css-that-never-applied.md skills/learned-detection-gaps.md skills/learned-local-db-confusion.md skills/full-harness-audit.md skills/test-result-audit.md skills/finding-disposition.md skills/handoff-to-studio-pro.md skills/preview-over-hub-tunnel.md skills/platform-link.md skills/teamserver-alignment.md skills/learned-constants-and-secrets.md" ;; + 5|build-ready) echo "skills/interview-protocol.md skills/change-intake.md skills/change-intake-detail.md skills/grill-mode.md skills/agent-roles.md skills/dispatching.md skills/module-brief.md skills/learned-mdl-preflight.md skills/module-folder-convention.md skills/learned-microflow-patterns.md skills/microflow-preflight.md skills/mendix-best-practices-index.md skills/ui-preflight-pages.md skills/design-spacing.md skills/learned-stylegallery.md skills/ui-loop.md skills/module-review.md skills/testing-shape.md skills/decompose-app.md skills/mendix-service-seam.md skills/microflow-loop-antipatterns.md skills/rehost-app.md skills/iterative-build-loop.md skills/mdl-cookbook-microflows.md skills/build/mdl/oneshot-mdl-method.md skills/learned-page-patterns.md skills/oneshot-page-structure-patterns.md skills/mendix-agents.md skills/mendix-agent-ui.md skills/mendix-agent-setup.md skills/fixture-seeding.md skills/journey-proof.md skills/unhappy-path-testing.md skills/monkey-test.md skills/load-test.md skills/unit-test.md skills/e2e-locators-that-lie.md skills/report-schema.md skills/harness-architecture.md skills/process-coherence-pass.md skills/lint-that-actually-runs.md skills/improvement-register.md skills/journey-examples.md skills/wiring-sweep.md skills/learned-workflow-patterns.md skills/workflow-structure-rules.md skills/rest-integration-first-time-right.md skills/rest-between-mendix-apps.md skills/bug-submission-checklist.md skills/empty-widget-triage.md skills/learned-sidebar-collapse-icons.md skills/learned-popup-navigation.md skills/learned-datagrid-customcontent-binding.md skills/learned-popup-feedback-pattern.md skills/learned-mdl-cannot-express.md skills/learned-css-that-never-applied.md skills/learned-detection-gaps.md skills/learned-dg2-patterns.md skills/learned-file-upload-widget.md skills/security-is-not-a-later-script.md skills/learned-local-db-confusion.md skills/full-harness-audit.md skills/test-result-audit.md skills/finding-disposition.md skills/preview-over-hub-tunnel.md skills/walking-skeleton.md skills/platform-link.md skills/teamserver-alignment.md skills/learned-constants-and-secrets.md" ;; + 6) echo "skills/interview-protocol.md skills/change-intake.md skills/change-intake-detail.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-cutover.md skills/dispatching.md skills/mendix-best-practices-index.md skills/module-review.md skills/testing-shape.md skills/existing-app-assurance.md skills/decompose-app.md skills/app-analysis.md skills/module-dependency-review.md skills/microflow-loop-antipatterns.md skills/rehost-app.md skills/qa-loop-goal-pattern.md skills/mendix-agent-setup.md skills/e2e-harness-base.md skills/learned-db-assertions.md skills/fixture-seeding.md skills/journey-proof.md skills/unhappy-path-testing.md skills/monkey-test.md skills/load-test.md skills/unit-test.md skills/e2e-locators-that-lie.md skills/learned-skill-ux-audit.md skills/learned-skill-scope-delta.md skills/report-schema.md skills/harness-architecture.md skills/process-coherence-pass.md skills/e2e-evidence-report.md skills/record-demo-video.md skills/demo-recording-polish.md skills/share-demo-package.md skills/lint-that-actually-runs.md skills/improvement-register.md skills/journey-examples.md skills/wiring-sweep.md skills/workflow-structure-rules.md skills/rest-between-mendix-apps.md skills/bug-submission-checklist.md skills/empty-widget-triage.md skills/anonymize-client-app-for-demo.md skills/learned-css-that-never-applied.md skills/learned-detection-gaps.md skills/learned-local-db-confusion.md skills/full-harness-audit.md skills/test-result-audit.md skills/finding-disposition.md skills/handoff-to-studio-pro.md skills/preview-over-hub-tunnel.md skills/platform-link.md skills/teamserver-alignment.md skills/learned-constants-and-secrets.md" ;; 7) echo "skills/interview-protocol.md skills/change-intake.md skills/change-intake-detail.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-cutover.md skills/close-the-loop.md skills/share-demo-package.md skills/handoff-to-studio-pro.md skills/platform-link.md skills/teamserver-alignment.md skills/deploy-to-sandbox.md" ;; *) echo "" ;; # diff --git a/bin/lib/skill-routing.tsv b/bin/lib/skill-routing.tsv index 0461fcfd..27d8707a 100644 --- a/bin/lib/skill-routing.tsv +++ b/bin/lib/skill-routing.tsv @@ -174,6 +174,7 @@ fixture-seeding skills/fixture-seeding.md Seed data, test data, demo data or fix journey-proof skills/journey-proof.md Proving a module's user journey end-to-end — the deep form of step 3 PROVE; use whenever an instrument reports green and you cannot say what would have made it red test,review 5,6 ondemand verify unhappy-path-testing skills/unhappy-path-testing.md Writing or running e2e cases beyond the happy journeys (blank/whitespace required fields, over-length, duplicates, cancel, double submit, wrong state, wrong role, backend down, stale write, empty search) — every case cites a requirement first, UNSPECIFIED is re-scored against house rules, GAP counts carry the denominator; runs after journey-proof and before monkey-test test,review 5,6 ondemand verify monkey-test skills/monkey-test.md Running the fuzz/crash net on a module whose journeys are already green — and reading the result, which is NOT evidence the module works test,review 5,6 ondemand verify +load-test skills/load-test.md Load, performance or stress testing a page ("will it hold 1000 users", concurrent users, k6, JMeter) — record the real /xas/ calls with Playwright and replay them instead of correlating tokens by hand, and report what the number does NOT prove (requests vs users, empty data, whose limit was hit) lead,test,review 5,6 ondemand verify unit-test skills/unit-test.md Writing or running a microflow unit test (.test.mdl, mxcli test, @expect) once the project or the user has decided a flow gets one — what is worth a test, equality-only asserts, --list before any run, a scratch copy so the live app survives, and a positive control that must fail lead,mdl,test,review 5,6 ondemand verify e2e-locators-that-lie skills/e2e-locators-that-lie.md An e2e rung reports a UI defect (widget never appeared, click did nothing, screen contradicts the data, fixture count wrong) — check the locator before blaming the app: :visible, click order, poll, derived fixtures test,review 5,6 ondemand diagnose learned-skill-ux-audit skills/learned-skill-ux-audit.md UX audit and screenshot-loop discipline review 6 ondemand verify diff --git a/docs/toolkit-roadmap.md b/docs/toolkit-roadmap.md index 8447ee62..a0649274 100644 --- a/docs/toolkit-roadmap.md +++ b/docs/toolkit-roadmap.md @@ -321,8 +321,10 @@ Ideas with a reason and no design. Listed so they are not rediscovered. forbidden by the coverage skill, and generating a picture of a screen that already exists is exactly that. Wireframe what has no implementation yet, never draw one backwards from running code. -- A load testing capability. Measured as theatre without production data volumes, and the static - index audit answers the same question more honestly and in about a minute. +- A shipped load testing instrument. The method is now a skill (`skills/load-test.md`, added + 2026-10-16 from a field implementation); a generic `project-tests/e2e/loadtest.js` waits until + a second project runs the recipe. The theatre objection stands for empty databases, which is + why the skill makes the report state the data volume. - A randomised browser walker. The client has documented traps that make an untuned walker thrash rather than explore. The model level equivalent above is the version worth building. - Anything requiring a graph database or a vector store. The activity level schema is a good idea diff --git a/skills/load-test.md b/skills/load-test.md new file mode 100644 index 00000000..a4d72eb3 --- /dev/null +++ b/skills/load-test.md @@ -0,0 +1,144 @@ +# Skill: load-test — record the real page, replay it, and say what the number does not prove + +**Applies to:** any mxcli project with a runnable app, pipeline or not. On request only: a +load test is never a build-plan step and never a gate. +**Purpose:** answer "does this page hold up under N at once" without correlating Mendix's +client protocol by hand, and without a report that overclaims. +**Source:** a field implementation on a warehouse demo app (Mendix 11, `mxcli run --local` +with OTel to Jaeger), 16 recorded runs from 2026-09-09 to 2026-10-08, bursts of 1000 and +2000 concurrent calls. Every trap below was hit in those runs. + +**Instrument:** none ships in `project-tests/` yet. Build it in the project's `tests/e2e/` on +top of the shipped `helpers.js` (login, nav) and `otel.js` (`capture`, `liveness`). This file +holds the method and every judgement; the script fetches facts only (`skills-over-scripts.md`). + +--- + +## Why record and replay, not hand-written requests + +A Mendix page does not call a public API. The client posts to `/xas/`, and in Mendix 11 every +client-invoked microflow is action `runtimeOperation` with an opaque `operationId`. Writing +that payload by hand is a guess, and it changes between versions. Classic protocol tools hit +the same wall: dynamic ids and the CSRF token must be correlated, again after every release. + +The recipe skips correlation entirely: + +``` +1. Playwright logs a real persona in and opens the page (helpers.js login, navTo) +2. record every POST to /xas/ while the page loads (page.on('request')) +3. probe: fire each recorded call once, read Jaeger (otel.js capture) + → which microflow did it run, does it write? +4. replay the chosen call with that session's cookie and the + token: mx.session.sessionData.csrftoken (Mendix 11 has no + getCSRFToken), refuse to fire if empty, as helpers.js does (raw node:http) +5. ramp → burst → optional ladder → one lone request after +``` + +After a release you re-record, one command. Nothing to re-correlate. + +A k6 script fed from a HAR export works too, but the HAR carries one live session cookie: +replayed as-is, 1000 virtual users are one session, and the cookie expires mid-run. Each VU +must log in itself. Delete the HAR afterwards; it holds a live session. + +## When to run it + +| Situation | Run? | +|---|---| +| The user asks "will this hold N users", before a go-live or a sizing talk | **Yes**, on the page they name | +| Journeys for that page are red | No. A broken page under load measures the breakage. | +| As evidence a module is done | **No.** Load is not correctness. | +| Against a shared or production environment | Only with the owner's written yes. A burst is indistinguishable from an attack. | +| The chosen call writes (create, commit, move, submit, import) | **Refuse** unless the user explicitly allows writes on a throwaway database | + +## The traps, each paid for + +**1. HTTP 200 on a failed action.** Mendix answers 200 and puts the failure in the JSON body +under `error`. Counting status codes reported 0% errors over a broken service. Count +transport errors and application errors separately; parse every body. + +**2. A caught failure leaves the microflow green.** A handler that catches an upstream REST +error lets the page render and the microflow span end OK; only its activity spans are ERROR. +Walk activity-level spans, not just the microflow span. + +**3. The wrong call gets hammered.** Every recorded call reads `runtimeOperation`, so a label +regex cannot tell the grid's data fetch from a keep-alive. Measured: the most frequent call +was a 771-byte no-op; replaying it measured the HTTP dispatcher and reported it as the page. +Choose by the probe (which microflow ran), and when the probe finds nothing, by the biggest +response, and say in the report that the choice was not evidence-backed. + +**4. The tape holds the previous page.** Login and the page you parked on fire microflows too. +Measured: without clearing, six dashboard microflows topped the candidate list of a list-page +test. Park on another page, clear the recording, navigate, then prove you landed (a ready +selector, not caption text that sits in the nav on every page). + +**5. Tracing looks on when it is off.** Jaeger keeps a service name in its list long after the +process is gone. Two full runs printed `serverSide: null` because the check was "the name is +listed". The gate is: spans for this service arrived after the page was opened. No spans after +a real navigation is an **instrument FAULT** (exit 2), not a clean run. Usually the cause is a +`--trace-service` override; mxcli's default (the .mpr name) already matches. + +**6. The harness throttles itself.** Node's `fetch` pools sockets, so "1000 concurrent" became +"N concurrent, queued" and the app got the credit. Use `http.Agent({ maxSockets: Infinity })` +and fire the whole burst before awaiting any answer. No socket timeout: a 90 s answer is a +finding, a 30 s cut-off relabels it as a harness error. + +**7. Whose limit was hit.** On the ladder, classify every stop: + +| Error | Who broke | +|---|---| +| `ECONNRESET`, `ECONNREFUSED`, `EPIPE`, 5xx, error body | **the app** | +| `EADDRNOTAVAIL`, `EMFILE`, `ENFILE` | **the harness** (~28k ports per destination, TIME_WAIT ~60 s carries across rungs) | + +After each rung send one lone request. A rung that answered thousands and then cannot answer +one is a server that fell over late, and the rung's percentiles called it a pass. + +**8. Requests are not users.** One recorded cookie replayed by every worker is N concurrent +*requests* on one session. For users: log in K separate sessions and round-robin their +cookie and token pairs, from K **distinct** accounts. Two live sessions on one account can +degrade each other (measured: both lost their navigation menu), which reads as "the app got +slow". End every session afterwards with `helpers.logout` (the xas logout action, with the token); closing the browser context +leaves it alive against the runtime's session cap. + +**9. The trace store samples.** Jaeger's query caps at a few hundred traces; one burst makes +far more. Read server-side percentiles, never counts, and never read "0 ERROR spans" as "no +request failed" above the cap. + +**10. An empty database is a different app.** A p95 on 50 rows says nothing about 5 million. +State the row counts of the entities the page reads, and look at the query plan or missing +indexes before blaming the runtime. This is the honest half of "load testing is theatre". + +## Reading the result + +| Run says | Verdict | +|---|---| +| 0 transport, 0 app errors, lone request after answers, spans arrived | **PASS for this call, this data, this host**: nothing more | +| App errors > 0 or 5xx | **FAIL**, name the microflow from the probe | +| Stopped on `EADDRNOTAVAIL`/`EMFILE` | **Harness limit**, not a finding about the app | +| No spans after navigation | **FAULT**, no server-side claim may be made | +| Target picked by size, not by probe | Report it, with that caveat in the first line | + +Field numbers, for scale: bursts of 1000 on one session gave p95 between 0.2 s and 4.6 s +across runs with 0 errors; 2000 gave 1.2 to 11.5 s. All local, one account, one replayed call. +That is "the runtime absorbs 1000 calls at once on a laptop", not "1000 people can use it". + +## Reporting + +Four lines, every time: + +``` +load · → · · + burst : p50/p95/p99 <..> · transport · app errors · lone request after: + sessions: on distinct accounts [or: one session, so concurrent requests, not users] + Read as: . Not: production capacity. +``` + +Write the full run to `tests/e2e/artifacts/loadtest-.json`, with the recorded +calls, the probe mapping (operationId to microflow) and the chosen target, so a reader can +check what was hammered. + +## Related + +- `skills/testing-shape.md`: the module shape this sits outside of +- `skills/monkey-test.md`: the same "what a clean run is not allowed to mean" discipline +- `skills/e2e-harness-base.md`: login, persona and the no-unapproved-fix rule +- `skills/tool-output-is-not-ground-truth.md`: why a quiet run is not a passing one diff --git a/skills/testing-shape.md b/skills/testing-shape.md index 2b14dbf8..4f1b08c5 100644 --- a/skills/testing-shape.md +++ b/skills/testing-shape.md @@ -22,6 +22,7 @@ and the guard for each. Job 2 is why this skill exists; if you shorten it, do no | Why `$?` and tool output are not evidence | `tool-output-is-not-ground-truth.md` (shared toolkit) | | Does the whole journey hang together, not just each piece — a different axis, not a fifth rung | `process-coherence-pass.md` (shared toolkit) | | The full build→gate→prove→look→confirm→next loop this shape plugs into, incl. monkey testing | `module-review.md` (shared toolkit) | +| "Will this page hold N users": on request only, outside the module shape | `load-test.md` (shared toolkit) | > **Promotion note, closed 2026-08-18.** All three rows above now have a shared-toolkit copy, > so §4 — the load-bearing section — no longer cites anything a fresh project cannot open.