diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 4bb9a40a..158b02d7 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -29,6 +29,10 @@ - [ ] `script` (page-injected runtime) - [ ] `trace` (Trace mode) +## Capability matrix + +- [ ] Capability matrix updated (`pnpm docs:capabilities`) if this adds or changes user-facing behaviour + ## Notes for reviewers [//]: # (Anything non-obvious: design trade-offs, alternatives you ruled out, follow-ups. If this PR touches more than one adapter, say why the shared logic doesn't live in `core`.) diff --git a/CLAUDE.md b/CLAUDE.md index 533471c9..921868a1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -29,6 +29,7 @@ Run from repo root unless noted. | `pnpm demo:wdio` / `pnpm demo:nightwatch` / `pnpm demo:selenium` | Run the per-framework example projects. Useful for manual verification of UI or runtime changes. **No package pins a browser driver**, deliberately: a pinned `chromedriver` rots against whatever Chrome the developer has, and because pnpm puts the package's `node_modules/.bin` on `PATH`, Selenium Manager finds it there, prefers it over resolving, and on a major mismatch only *warns* before returning it anyway. The session then fails to start. With nothing on `PATH` both adapters resolve a matching driver (Nightwatch's Chrome service builder sets `requiresDriverBinary: false` and passes an unset `server_path` through to the same resolver), and nightwatch declares `chromedriver` an optional peer. | | `pnpm demo:wdio:mobile` / `:selenium:mobile` / `:nightwatch:mobile` / `:python:mobile` | The same, against Appium. All four build the same capability bag and drive the Clock app that ships with every Android system image (keying a duration on the timer keypad, never starting a timer), so a native example needs no `.apk`. `DEVTOOLS_MOBILE_PLATFORM=ios` runs the iOS spec instead — all four adapters — which drives Settings because the simulator ships no Clock, from a separate spec per platform rather than a branch. The simulator is chosen by udid and defaults to whichever is already booted, and an unmatched `IOS_DEVICE_NAME` is refused: naming one that does not exist makes the XCUITest driver create and boot it, every run, rather than fail. That refusal and the booted-simulator preflight are **local** policy — `xcrun simctl` enumerates local simulators and nothing else — so `IOS_UDID` and a non-local `APPIUM_HOST` both bypass them, or a real device and a cloud grid would be refused a run they were correctly configured for. `DEVTOOLS_MOBILE=web` drives the device's own browser on both platforms — Chrome on Android, Safari on iOS, which the XCUITest driver serves without a chromedriver. `examples/MOBILE.md` holds the prerequisites and the `DEVTOOLS_MOBILE` / `APPIUM_APP` switches; `DEVTOOLS_MODE=trace` flips any demo to trace mode. | | `pnpm dev` | Run all packages in parallel dev mode. | +| `pnpm docs:capabilities` | Render the per-adapter capability tables from `packages/conformance/src/capabilities.ts` into `README.md`; `--website ` also renders the docs-site tables, and `--check` fails instead of writing. | | `python3 packages/selenium-devtools-py/scripts/changes.py next-version` | The version a Python-adapter release would publish, from the fragments pending in `changes/`. `check --base ` is the CI gate; `apply` is what the release runs. | `selenium-devtools` exposes per-runner variants of its example via `pnpm --filter @wdio/selenium-devtools example:mocha` / `:mocha:allure` / `:jest` / `:cucumber`. @@ -48,6 +49,7 @@ Defined in root `tsconfig.json`: | `@wdio/devtools-script` / `*` | `packages/script/src/...` | | `@wdio/devtools-service` / `*` | `packages/service/src/...` | | `@wdio/selenium-devtools` / `*` | `packages/selenium-devtools/src/...` | +| `@wdio/nightwatch-devtools/*` | `packages/nightwatch-devtools/src/*` (subpaths only, for type imports such as the option types) | | `@wdio/devtools-shared` / `*` | `packages/shared/src/...` | | `@wdio/devtools-core` / `*` | `packages/core/src/...` | | `@wdio/devtools-trace` / `*` | `packages/trace/src/...` | @@ -259,6 +261,7 @@ Neither is hand-versioned: both assemble the version and the changelog at releas ### Documentation - User-facing docs live in two places that must stay in sync: this repo's `README.md` (+ per-package READMEs) and the **WebdriverIO devtools webpage** (`website/docs/devtools/**` in the `webdriverio/webdriverio` repo — e.g. `wdio/TraceMode.md`). When a change adds, removes, or alters user-facing behavior (a new option, CLI, flag, output, or workflow), update the README here **and** mirror it to the matching webpage doc in the same change. A docs PR that updates only one side isn't complete. +- The capability tables (the README's support, dashboard and mobile tables, the docs site's `CrossFrameworkSupport.md` and `MobileTesting.md`) are generated, never edited by hand. `packages/conformance/src/capabilities.ts` declares every capability with a cell per adapter, typed as a complete map over the conformance suite's adapter ids, so a new adapter does not compile until each capability declares it. A partial or unsupported cell carries a required reason, which renders as the footnote. A cell is `proven` only when it names conformance rules the adapter has no known gap against; it renders as "✅ tested", every other mark is a declaration. Two coverage checks keep a new feature from bypassing the matrix: every user-facing option (`capability-options.ts`, keyed on the real option types of the shared base and each JS adapter, plus the Python `enable()` keywords, pytest options and user-facing `DEVTOOLS_*` variables read from source by a test) names a capability row or says why it is not one, and every conformance rule backs a proven cell or is listed in `INTERNAL_RULES` with a reason. A test fails when the README tables are out of date; the docs-site copy is refreshed with `pnpm docs:capabilities --website ` in the same change. --- diff --git a/README.md b/README.md index 0a566cc8..36f66416 100644 --- a/README.md +++ b/README.md @@ -40,6 +40,45 @@ Want a portable artifact instead of a live UI (CI / agent diffing)? Switch to ** - **Jasmine**: Complete integration with grep-based filtering - **Cucumber**: Scenario-level and example-specific execution with feature:line targeting +**Dashboard and options by adapter:** + + + + +| Capability | WebdriverIO | Selenium | Nightwatch | Python | +|---|---|---|---|---| +| One row per command in the Actions list | ✅ tested | ✅ tested | ✅ tested | ✅ tested | +| Screenshot on every command row | ✅ | ✅ | ✅ | ✅ | +| Browser console logs | ✅ | ✅ | ✅ | ✅ | +| Source tab opens the test line that ran the command | ✅ | ✅ | ✅ | ✅ | +| Metadata tab (session, capabilities, device) | ✅ | ✅ | ✅ | ✅ | +| Performance timings on navigation rows | ✅ | ✅ | ✅ | ✅ | +| Failed command in the Errors tab, with the browser's message | ✅ tested | ✅ tested | ✅ tested | ✅ tested | +| Run, Rerun and Run all from the dashboard | ✅ | ✅ | ⚠️ no Run all ¹ | ✅ | +| Preserve & Rerun (compare a failed run with its rerun) | ✅ | ✅ | ✅ | ✅ | +| Dashboard window opens by itself in live mode, and can be turned off | ⚠️ always opens ² | ✅ `openUi: false` | ⚠️ always opens ³ | ✅ `DEVTOOLS_OPEN=0` | +| Turn off per-command screenshots (`captureScreenshots`) | ❌ ⁴ | ✅ | ❌ ⁴ | ❌ ⁴ | +| Force the test browser headless (`headless`) | ❌ ⁵ | ⚠️ Chrome only ⁶ | ❌ ⁵ | ❌ ⁵ | +| Safari kept frontmost so input lands (live mode) | ⚠️ once ⁷ | ✅ before each input | ⚠️ once ⁷ | ⚠️ once ⁷ | + +✅ tested: proven for that adapter by the cross-adapter conformance suite. Every other mark is declared. + +¹ The backend reruns a Nightwatch config through its framework path, which selects suites and tests but has no whole-run entry. + +² The service always opens the dashboard in live mode; `devtoolsCapabilities` picks its browser but cannot skip it. + +³ Nightwatch always opens the dashboard in live mode. + +⁴ No option: every command row takes a screenshot. + +⁵ No option: set headless in the browser capabilities, as for any run. + +⁶ The option adds `--headless=old` to `goog:chromeOptions` only, so other browsers stay headed; set their own headless capability. + +⁷ Brings Safari to the front once after the session starts (live mode, local macOS), so a click issued after another window took the front can still land nowhere. + + + ### 📊 Real-Time Visualization - **Live Browser Preview**: View the application under test in a scaled iframe with automatic screenshot updates - **Actions Timeline**: Command-by-command execution log with timestamps and parameters @@ -102,29 +141,68 @@ The dashboard runs in one of **two modes**, set per adapter via the shared `mode All three JavaScript adapters (`@wdio/devtools-service`, `@wdio/selenium-devtools`, `@wdio/nightwatch-devtools`) emit the **same normalized trace** through the shared `@wdio/devtools-core` capture library, so one archive format and one player serve every framework. -The trace **format and the player are identical** across those three, but **capture completeness varies** — WebdriverIO is the most complete; Selenium and Nightwatch cover the core flow with some gaps (e.g. inline-Allure per-test artifacts, retry-aware retention, Cucumber step nesting, auto BiDi). Each adapter's README lists its specifics. +The Python adapter (`selenium-devtools-py`) writes the same archive through the backend, so the trace **format and the player are identical** across all four, but **capture completeness varies**: WebdriverIO is the most complete, and the others cover the core flow with the gaps the table below names. Each adapter's README lists its specifics. -**Trace-mode support by adapter:** +**Support by adapter:** -| Capability | WebdriverIO | Selenium | Nightwatch | -|---|---|---|---| -| Trace mode + `show-trace` player | ✅ | ✅ | ✅ | -| DOM time-travel (mutation capture) | ✅ | ✅ ¹ | ✅ | -| Dense filmstrip (`filmstrip`) | ✅ CDP push | ✅ CDP push | ⚠️ polling only | -| Per-test granularity (`traceGranularity: 'test'`) | ✅ | ✅ | ⚠️ ² | -| Retry-aware retention (`tracePolicy`) | ✅ | ✅ | ⚠️ ³ | -| Cucumber step nesting | Scenario→Step ⁴ | ✅ full | Feature→Scenario ⁵ | -| Inline Allure attach | ✅ | ✅ | ⚠️ produce-only ⁶ | -| Assertion capture (`captureAssertions`) | ✅ | ✅ | ✅ | -| Auto BiDi capture | ✅ auto | ✅ auto | ⚠️ opt-in ⁷ | - -¹ With BiDi, Selenium registers the page collector at document start, so every document anchors its own DOM as WebdriverIO's does. Without BiDi it falls back to injecting the collector after each navigation, and there a navigation's snapshot can lag the command that triggered it. -² Nightwatch's Cucumber and exports-object interfaces get real per-test slicing; the BDD `describe/it` interface collapses to a single session-scoped slice keyed to the first test. -³ Only `retain-on-failure` works; the other retry-aware policies degrade to it because Nightwatch's `--retries` re-runs a testcase internally without re-firing the per-test hooks. -⁴ WebdriverIO does not yet carry feature-level ancestry, so its Cucumber nesting is Scenario→Step. -⁵ Nightwatch does not yet stamp per-step nesting (Feature→Scenario only). -⁶ Nightwatch has no live Allure attach API, so per-test `screenshot`/`video` are written to disk and listed in the manifest but not attached to an Allure test. -⁷ Opt-in via `bidi: true` + `webSocketUrl: true` in capabilities. + + + +| Capability | WebdriverIO | Selenium | Nightwatch | Python | +|---|---|---|---|---| +| Trace mode + `show-trace` player | ✅ | ✅ | ✅ | ✅ | +| DOM time-travel (mutation capture) | ✅ tested | ⚠️ with BiDi ¹ | ⚠️ with BiDi ¹ | ✅ tested | +| Network requests (browser sessions) | ✅ tested | ⚠️ with BiDi ² | ⚠️ with BiDi ² | ✅ tested | +| Screencast (`filmstrip` / video) | ✅ CDP push | ✅ CDP push | ⚠️ polling only ³ | ✅ CDP push | +| Per-test granularity (`traceGranularity: 'test'`) | ✅ | ✅ | ⚠️ ⁴ | ✅ | +| Retry-aware retention (`tracePolicy`) | ✅ | ✅ | ⚠️ `retain-on-failure` only ⁵ | ⚠️ `retain-on-failure` only ⁶ | +| Per-test `screenshot` / `video`, attached to Allure | ✅ | ✅ | ⚠️ produce-only ⁷ | ❌ ⁸ | +| Cucumber Feature→Scenario→Step nesting | ⚠️ Scenario→Step ⁹ | ✅ | ⚠️ Feature→Scenario ¹⁰ | ❌ ¹¹ | +| Assertion capture (`captureAssertions`) | ✅ | ⚠️ ¹² | ✅ | ✅ | +| BiDi console / network capture | ✅ auto | ✅ auto | ⚠️ opt-in ¹³ | ✅ auto | +| A11y tab + pick-locator overlay (trace player) | ✅ | ✅ | ✅ | ✅ | +| Transcript + Copy-for-LLM (trace player) | ✅ | ✅ | ✅ | ✅ | +| Unpacked trace output (`traceFormat: 'ndjson-directory'`) | ✅ | ✅ | ✅ | ❌ zip only ¹⁴ | +| Artifacts manifest for CI and reporters (`emitArtifactsManifest`) | ✅ | ✅ | ⚠️ opt-in ¹⁵ | ❌ ¹⁶ | +| Network without BiDi (Safari, Appium mobile browsers) | ⚠️ ¹⁷ | ⚠️ ¹⁷ | ⚠️ ¹⁷ | ⚠️ ¹⁷ | + +✅ tested: proven for that adapter by the cross-adapter conformance suite. Every other mark is declared. + +¹ Without BiDi (Safari, Appium) the page collector is injected after each navigation, so a page the test leaves within tens of milliseconds is never captured. + +² Without BiDi a page the test leaves within tens of milliseconds is never drained, so its requests are not listed. + +³ Nightwatch exposes no stable CDP handle, so frames come from a `takeScreenshot` poll every 200 ms. + +⁴ The BDD `describe/it` interface fires the plugin's hooks once per module, so it collapses to one slice keyed to the first test; Cucumber and exports-object get real per-test slices. + +⁵ Nightwatch's `--retries` reruns a testcase internally without firing the per-test hooks again, so the other policies behave like `retain-on-failure`. + +⁶ Nothing the Python adapter sends carries an attempt number, so the retry-aware policies behave like `retain-on-failure`. + +⁷ Nightwatch has no live Allure attach API, so per-test screenshots and videos are written to disk and listed in the manifest but not attached. + +⁸ The Python adapter has no `screenshot` / `video` options and no Allure attachment yet. + +⁹ The service records no feature ancestry for a scenario, so the trace nests Scenario→Step. + +¹⁰ Nightwatch steps are not stamped as nested groups, so the trace nests Feature→Scenario. + +¹¹ The Python adapter stamps no Feature, Scenario or Step groups, so a pytest-bdd run nests as plain tests. + +¹² `node:assert` rows pass and fail; jest-style `expect()` and chai expose no pass hook, so only failing matchers appear there. + +¹³ Nightwatch turns it on with `bidi: true` plus `webSocketUrl: true` in the capabilities. + +¹⁴ The Python adapter has no option for it; the backend it asks always writes a zip. + +¹⁵ Nightwatch has no live Allure signal to turn the manifest on by itself, so it stays opt-in. + +¹⁶ The Python adapter writes no artifacts manifest. + +¹⁷ Fetch and XHR calls carry headers and bodies; every other load comes from the page's Resource Timing buffer, with timing and size but no headers or body, and no status on Safari. + + In trace mode no DevTools UI window opens. At session end the adapter writes trace artifacts into a `test-results/` folder (created next to the resolved spec/config directory), suitable for offline replay, AI-agent diffing, or any consumer that prefers a portable artifact over a live UI. @@ -133,6 +211,7 @@ In trace mode no DevTools UI window opens. At session end the adapter writes tra | **WebdriverIO** | `services: [['devtools', { mode: 'trace' }]]` | | **Selenium** | `DevTools.configure({ mode: 'trace' })` (before importing `selenium-webdriver`) | | **Nightwatch** | `globals: nightwatchDevtools({ mode: 'trace' })` | +| **Python** | `pytest --devtools-trace`, or `devtools.enable(trace=True)` in a plain script | The trace artifact contains: - `trace.trace` — NDJSON `context-options` + `before`/`after` action events. When test hooks are available (Mocha's `it()` / Cucumber's `Scenario()`), each test becomes a `Tracing.tracingGroup` span — an open/close `before`/`after` pair with `method: "tracingGroup"` and `params.name` set to the test title. Child actions inside the group carry `parentId` pointing back to the group's `callId`, so timeline viewers render tests as labelled spans wrapping their commands. @@ -216,7 +295,25 @@ The discriminator is the **browser name in the capabilities**, not the device: m A hybrid app is the case capabilities alone cannot answer, because the answer changes mid-run. Appium reports `NATIVE_APP` or a webview context, and anything that is not `NATIVE_APP` counts as a webview — the `WEBVIEW_` prefix is a convention, and a driver naming its webview otherwise would have its DOM capture skipped. Following the context costs no round trip: the switch command carries its destination in its own arguments. -**Per-adapter:** all four adapters detect a native session and skip page-side capture on one. Only the **WebdriverIO** service additionally follows a hybrid app into its webview; Nightwatch, Selenium and the Python adapter answer from the startup capabilities, so a hybrid session's webview half is captured as if it were still native. +**Per-adapter:** + + + + +| Capability | WebdriverIO | Selenium | Nightwatch | Python | +|---|---|---|---|---| +| Network without BiDi (Safari, Appium mobile browsers) | ⚠️ ¹ | ⚠️ ¹ | ⚠️ ¹ | ⚠️ ¹ | +| Mobile browser sessions (Chrome on Android, Safari on iOS) | ✅ | ✅ | ✅ | ✅ | +| iOS (XCUITest): native apps and Safari | ✅ | ✅ | ✅ | ✅ | +| Native session detected, page-side capture skipped | ✅ | ✅ | ✅ | ✅ | +| Follows a hybrid app into its webview | ✅ | ⚠️ ² | ⚠️ ² | ⚠️ ² | +| Native app network through appium-interceptor (`appium:intercept`) | ✅ Android | ✅ Android | ✅ Android | ✅ Android | + +¹ Fetch and XHR calls carry headers and bodies; every other load comes from the page's Resource Timing buffer, with timing and size but no headers or body, and no status on Safari. + +² Decides from the startup capabilities, so a hybrid app's webview half is captured as if it were still native. + + **In the dashboard**, a phone capture is framed as the device — a full-height device column with the action list and dock beside it — in both live mode and the trace player. The trace's `context-options` records the platform, model and OS version, read back so the player labels frames without guessing. diff --git a/package.json b/package.json index da60a025..d8b1bf29 100644 --- a/package.json +++ b/package.json @@ -24,6 +24,7 @@ "demo:python:mobile": "python3 examples/selenium-py/mobile/${DEVTOOLS_MOBILE_PLATFORM:-android}/*.py", "dev": "pnpm --parallel dev", "preview": "pnpm --parallel preview", + "docs:capabilities": "tsx packages/conformance/src/capability-matrix.ts", "test": "vitest run", "test:watch": "vitest", "test:coverage": "vitest run --coverage", diff --git a/packages/conformance/src/capabilities.ts b/packages/conformance/src/capabilities.ts new file mode 100644 index 00000000..2d8c81a7 --- /dev/null +++ b/packages/conformance/src/capabilities.ts @@ -0,0 +1,450 @@ +import { RULES } from './constants.js' +import type { AdapterId, Capability, Cell, RuleId } from './types.js' + +// A Record, so a new adapter id fails to compile until it has a column title. +const ADAPTER_TITLES: Record = { + 'devtools-service': 'WebdriverIO', + 'selenium-devtools': 'Selenium', + 'nightwatch-devtools': 'Nightwatch', + 'selenium-devtools-py': 'Python' +} + +/** Column order of every rendered table. */ +export const ADAPTER_COLUMNS = ( + Object.entries(ADAPTER_TITLES) as [AdapterId, string][] +).map(([id, title]) => ({ id, title })) + +const SUPPORTED: Cell = { support: 'supported' } +const proven = (rules: RuleId[]): Cell => ({ support: 'proven', rules }) +const partial = (reason: string, label?: string): Cell => ({ + support: 'partial', + reason, + label +}) +const unsupported = (reason: string, label?: string): Cell => ({ + support: 'unsupported', + reason, + label +}) + +const DOM_RULES: RuleId[] = [ + RULES.documentAnchored, + RULES.anchorOnce, + RULES.anchorBirthTime, + RULES.rowReplaysPage +] +const ACTION_RULES: RuleId[] = [ + RULES.rowPerAction, + RULES.completionStamp, + RULES.navigationNamesDestination, + RULES.replaceTargetsRow +] +const NETWORK_RULES: RuleId[] = [ + RULES.requestListed, + RULES.requestListedOnce, + RULES.requestSourceServed +] +const ERROR_RULES: RuleId[] = [ + RULES.failureRecorded, + RULES.errorCarriesBrowserMessage +] + +const ALL_SUPPORTED: Record = { + 'devtools-service': SUPPORTED, + 'selenium-devtools': SUPPORTED, + 'nightwatch-devtools': SUPPORTED, + 'selenium-devtools-py': SUPPORTED +} + +const allProven = (rules: RuleId[]): Record => ({ + 'devtools-service': proven(rules), + 'selenium-devtools': proven(rules), + 'nightwatch-devtools': proven(rules), + 'selenium-devtools-py': proven(rules) +}) + +/** Rules that prove no capability cell on their own: a comparison between adapters, or a precondition of a feature rather than the feature. */ +export const INTERNAL_RULES: Partial> = { + [RULES.agreeOnRequests]: + 'Compares the adapters with one another; each adapter is judged by every-browser-request-listed.', + [RULES.agreeOnActions]: + 'Compares the adapters with one another; each adapter is judged by one-row-per-browser-action.', + [RULES.agreeOnErrors]: + 'Compares the adapters with one another; each adapter is judged by failed-command-has-error-row.', + [RULES.agreeOnPages]: + 'Compares the adapters with one another; each adapter is judged by row-replays-its-page.', + [RULES.agreeOnSnapshots]: + 'Compares the adapters with one another; each adapter is judged by every-document-anchored.', + [RULES.agreeOnRunControls]: + 'Compares the adapters with one another; each adapter is judged by run-controls-are-served.', + [RULES.metadataFirst]: + 'Checks that metadata reaches the dashboard before the first command, not the session, capabilities and device the Metadata tab shows.', + [RULES.runControlsServed]: + 'Checks that every run control the dashboard offers has a command to serve it, not that Run, Rerun and Run all work.' +} + +// appium-interceptor is an Android plugin; iOS app traffic needs a device or proxy capture outside the adapters. +const ANDROID_ONLY: Cell = { support: 'supported', label: 'Android' } + +const SAFARI_FOCUS_ONCE = + 'Brings Safari to the front once after the session starts (live mode, local macOS), so a click issued after another window took the front can still land nowhere.' +const NO_SCREENSHOT_SWITCH = 'No option: every command row takes a screenshot.' +const NO_HEADLESS_OPTION = + 'No option: set headless in the browser capabilities, as for any run.' + +const WITHOUT_BIDI_ANCHOR = + 'Without BiDi (Safari, Appium) the page collector is injected after each navigation, so a page the test leaves within tens of milliseconds is never captured.' +const WITHOUT_BIDI_DRAIN = + 'Without BiDi a page the test leaves within tens of milliseconds is never drained, so its requests are not listed.' +const NETWORK_FLOOR = + "Fetch and XHR calls carry headers and bodies; every other load comes from the page's Resource Timing buffer, with timing and size but no headers or body, and no status on Safari." +const STARTUP_CAPS_ONLY = + "Decides from the startup capabilities, so a hybrid app's webview half is captured as if it were still native." + +export const CAPABILITIES: Capability[] = [ + { + id: 'trace-mode', + name: 'Trace mode + `show-trace` player', + tables: ['readme-trace', 'cross-framework'], + cells: ALL_SUPPORTED + }, + { + id: 'dom-time-travel', + name: 'DOM time-travel (mutation capture)', + tables: ['readme-trace', 'cross-framework'], + cells: { + 'devtools-service': proven(DOM_RULES), + 'selenium-devtools': partial(WITHOUT_BIDI_ANCHOR, 'with BiDi'), + 'nightwatch-devtools': partial(WITHOUT_BIDI_ANCHOR, 'with BiDi'), + 'selenium-devtools-py': proven(DOM_RULES) + } + }, + { + id: 'network-requests', + name: 'Network requests (browser sessions)', + tables: ['readme-trace', 'cross-framework'], + cells: { + 'devtools-service': proven(NETWORK_RULES), + 'selenium-devtools': partial(WITHOUT_BIDI_DRAIN, 'with BiDi'), + 'nightwatch-devtools': partial(WITHOUT_BIDI_DRAIN, 'with BiDi'), + 'selenium-devtools-py': proven(NETWORK_RULES) + } + }, + { + id: 'filmstrip', + name: 'Screencast (`filmstrip` / video)', + tables: ['readme-trace', 'cross-framework'], + cells: { + 'devtools-service': { support: 'supported', label: 'CDP push' }, + 'selenium-devtools': { support: 'supported', label: 'CDP push' }, + 'nightwatch-devtools': partial( + 'Nightwatch exposes no stable CDP handle, so frames come from a `takeScreenshot` poll every 200 ms.', + 'polling only' + ), + 'selenium-devtools-py': { support: 'supported', label: 'CDP push' } + } + }, + { + id: 'per-test-granularity', + name: "Per-test granularity (`traceGranularity: 'test'`)", + tables: ['readme-trace', 'cross-framework'], + cells: { + 'devtools-service': SUPPORTED, + 'selenium-devtools': SUPPORTED, + 'nightwatch-devtools': partial( + "The BDD `describe/it` interface fires the plugin's hooks once per module, so it collapses to one slice keyed to the first test; Cucumber and exports-object get real per-test slices." + ), + 'selenium-devtools-py': SUPPORTED + } + }, + { + id: 'retry-retention', + name: 'Retry-aware retention (`tracePolicy`)', + tables: ['readme-trace', 'cross-framework'], + cells: { + 'devtools-service': SUPPORTED, + 'selenium-devtools': SUPPORTED, + 'nightwatch-devtools': partial( + "Nightwatch's `--retries` reruns a testcase internally without firing the per-test hooks again, so the other policies behave like `retain-on-failure`.", + '`retain-on-failure` only' + ), + 'selenium-devtools-py': partial( + 'Nothing the Python adapter sends carries an attempt number, so the retry-aware policies behave like `retain-on-failure`.', + '`retain-on-failure` only' + ) + } + }, + { + id: 'per-test-artifacts', + name: 'Per-test `screenshot` / `video`, attached to Allure', + tables: ['readme-trace', 'cross-framework'], + cells: { + 'devtools-service': SUPPORTED, + 'selenium-devtools': SUPPORTED, + 'nightwatch-devtools': partial( + 'Nightwatch has no live Allure attach API, so per-test screenshots and videos are written to disk and listed in the manifest but not attached.', + 'produce-only' + ), + 'selenium-devtools-py': unsupported( + 'The Python adapter has no `screenshot` / `video` options and no Allure attachment yet.' + ) + } + }, + { + id: 'cucumber-nesting', + name: 'Cucumber Feature→Scenario→Step nesting', + tables: ['readme-trace', 'cross-framework'], + cells: { + 'devtools-service': partial( + 'The service records no feature ancestry for a scenario, so the trace nests Scenario→Step.', + 'Scenario→Step' + ), + 'selenium-devtools': SUPPORTED, + 'nightwatch-devtools': partial( + 'Nightwatch steps are not stamped as nested groups, so the trace nests Feature→Scenario.', + 'Feature→Scenario' + ), + 'selenium-devtools-py': unsupported( + 'The Python adapter stamps no Feature, Scenario or Step groups, so a pytest-bdd run nests as plain tests.' + ) + } + }, + { + id: 'assertion-capture', + name: 'Assertion capture (`captureAssertions`)', + tables: ['readme-trace', 'cross-framework'], + cells: { + 'devtools-service': SUPPORTED, + 'selenium-devtools': partial( + '`node:assert` rows pass and fail; jest-style `expect()` and chai expose no pass hook, so only failing matchers appear there.' + ), + 'nightwatch-devtools': SUPPORTED, + 'selenium-devtools-py': SUPPORTED + } + }, + { + id: 'auto-bidi', + name: 'BiDi console / network capture', + tables: ['readme-trace', 'cross-framework'], + cells: { + 'devtools-service': { support: 'supported', label: 'auto' }, + 'selenium-devtools': { support: 'supported', label: 'auto' }, + 'nightwatch-devtools': partial( + 'Nightwatch turns it on with `bidi: true` plus `webSocketUrl: true` in the capabilities.', + 'opt-in' + ), + 'selenium-devtools-py': { support: 'supported', label: 'auto' } + } + }, + { + id: 'a11y-overlay', + name: 'A11y tab + pick-locator overlay (trace player)', + tables: ['readme-trace', 'cross-framework'], + cells: ALL_SUPPORTED + }, + { + id: 'transcript', + name: 'Transcript + Copy-for-LLM (trace player)', + tables: ['readme-trace', 'cross-framework'], + cells: ALL_SUPPORTED + }, + { + id: 'actions-list', + name: 'One row per command in the Actions list', + tables: ['readme-dashboard', 'cross-framework'], + cells: allProven(ACTION_RULES) + }, + { + id: 'command-screenshots', + name: 'Screenshot on every command row', + tables: ['readme-dashboard', 'cross-framework'], + cells: ALL_SUPPORTED + }, + { + id: 'console-logs', + name: 'Browser console logs', + tables: ['readme-dashboard', 'cross-framework'], + cells: ALL_SUPPORTED + }, + { + id: 'source-tab', + name: 'Source tab opens the test line that ran the command', + tables: ['readme-dashboard', 'cross-framework'], + cells: ALL_SUPPORTED + }, + { + id: 'metadata-tab', + name: 'Metadata tab (session, capabilities, device)', + tables: ['readme-dashboard', 'cross-framework'], + cells: ALL_SUPPORTED + }, + { + id: 'performance-timings', + name: 'Performance timings on navigation rows', + tables: ['readme-dashboard', 'cross-framework'], + cells: ALL_SUPPORTED + }, + { + id: 'failed-command-errors', + name: "Failed command in the Errors tab, with the browser's message", + tables: ['readme-dashboard', 'cross-framework'], + cells: allProven(ERROR_RULES) + }, + { + id: 'run-controls', + name: 'Run, Rerun and Run all from the dashboard', + tables: ['readme-dashboard', 'cross-framework'], + cells: { + 'devtools-service': SUPPORTED, + 'selenium-devtools': SUPPORTED, + 'nightwatch-devtools': partial( + 'The backend reruns a Nightwatch config through its framework path, which selects suites and tests but has no whole-run entry.', + 'no Run all' + ), + 'selenium-devtools-py': SUPPORTED + } + }, + { + id: 'preserve-rerun', + name: 'Preserve & Rerun (compare a failed run with its rerun)', + tables: ['readme-dashboard', 'cross-framework'], + cells: ALL_SUPPORTED + }, + { + id: 'dashboard-window', + name: 'Dashboard window opens by itself in live mode, and can be turned off', + tables: ['readme-dashboard', 'cross-framework'], + cells: { + 'devtools-service': partial( + 'The service always opens the dashboard in live mode; `devtoolsCapabilities` picks its browser but cannot skip it.', + 'always opens' + ), + 'selenium-devtools': { support: 'supported', label: '`openUi: false`' }, + 'nightwatch-devtools': partial( + 'Nightwatch always opens the dashboard in live mode.', + 'always opens' + ), + 'selenium-devtools-py': { + support: 'supported', + label: '`DEVTOOLS_OPEN=0`' + } + } + }, + { + id: 'screenshot-switch', + name: 'Turn off per-command screenshots (`captureScreenshots`)', + tables: ['readme-dashboard', 'cross-framework'], + cells: { + 'devtools-service': unsupported(NO_SCREENSHOT_SWITCH), + 'selenium-devtools': SUPPORTED, + 'nightwatch-devtools': unsupported(NO_SCREENSHOT_SWITCH), + 'selenium-devtools-py': unsupported(NO_SCREENSHOT_SWITCH) + } + }, + { + id: 'headless', + name: 'Force the test browser headless (`headless`)', + tables: ['readme-dashboard', 'cross-framework'], + cells: { + 'devtools-service': unsupported(NO_HEADLESS_OPTION), + 'selenium-devtools': partial( + 'The option adds `--headless=old` to `goog:chromeOptions` only, so other browsers stay headed; set their own headless capability.', + 'Chrome only' + ), + 'nightwatch-devtools': unsupported(NO_HEADLESS_OPTION), + 'selenium-devtools-py': unsupported(NO_HEADLESS_OPTION) + } + }, + { + id: 'safari-focus', + name: 'Safari kept frontmost so input lands (live mode)', + tables: ['readme-dashboard', 'cross-framework'], + cells: { + 'devtools-service': partial(SAFARI_FOCUS_ONCE, 'once'), + 'selenium-devtools': { support: 'supported', label: 'before each input' }, + 'nightwatch-devtools': partial(SAFARI_FOCUS_ONCE, 'once'), + 'selenium-devtools-py': partial(SAFARI_FOCUS_ONCE, 'once') + } + }, + { + id: 'trace-format', + name: "Unpacked trace output (`traceFormat: 'ndjson-directory'`)", + tables: ['readme-trace', 'cross-framework'], + cells: { + 'devtools-service': SUPPORTED, + 'selenium-devtools': SUPPORTED, + 'nightwatch-devtools': SUPPORTED, + 'selenium-devtools-py': unsupported( + 'The Python adapter has no option for it; the backend it asks always writes a zip.', + 'zip only' + ) + } + }, + { + id: 'artifacts-manifest', + name: 'Artifacts manifest for CI and reporters (`emitArtifactsManifest`)', + tables: ['readme-trace', 'cross-framework'], + cells: { + 'devtools-service': SUPPORTED, + 'selenium-devtools': SUPPORTED, + 'nightwatch-devtools': partial( + 'Nightwatch has no live Allure signal to turn the manifest on by itself, so it stays opt-in.', + 'opt-in' + ), + 'selenium-devtools-py': unsupported( + 'The Python adapter writes no artifacts manifest.' + ) + } + }, + { + id: 'network-without-bidi', + name: 'Network without BiDi (Safari, Appium mobile browsers)', + tables: ['readme-trace', 'readme-mobile', 'cross-framework', 'mobile'], + cells: { + 'devtools-service': partial(NETWORK_FLOOR), + 'selenium-devtools': partial(NETWORK_FLOOR), + 'nightwatch-devtools': partial(NETWORK_FLOOR), + 'selenium-devtools-py': partial(NETWORK_FLOOR) + } + }, + { + id: 'mobile-browser', + name: 'Mobile browser sessions (Chrome on Android, Safari on iOS)', + tables: ['readme-mobile', 'cross-framework', 'mobile'], + cells: ALL_SUPPORTED + }, + { + id: 'ios', + name: 'iOS (XCUITest): native apps and Safari', + tables: ['readme-mobile', 'cross-framework', 'mobile'], + cells: ALL_SUPPORTED + }, + { + id: 'native-detected', + name: 'Native session detected, page-side capture skipped', + tables: ['readme-mobile', 'cross-framework', 'mobile'], + cells: ALL_SUPPORTED + }, + { + id: 'hybrid-webview', + name: 'Follows a hybrid app into its webview', + tables: ['readme-mobile', 'cross-framework', 'mobile'], + cells: { + 'devtools-service': SUPPORTED, + 'selenium-devtools': partial(STARTUP_CAPS_ONLY), + 'nightwatch-devtools': partial(STARTUP_CAPS_ONLY), + 'selenium-devtools-py': partial(STARTUP_CAPS_ONLY) + } + }, + { + id: 'native-network', + name: 'Native app network through appium-interceptor (`appium:intercept`)', + tables: ['readme-mobile', 'cross-framework', 'mobile'], + cells: { + 'devtools-service': ANDROID_ONLY, + 'selenium-devtools': ANDROID_ONLY, + 'nightwatch-devtools': ANDROID_ONLY, + 'selenium-devtools-py': ANDROID_ONLY + } + } +] diff --git a/packages/conformance/src/capability-matrix.ts b/packages/conformance/src/capability-matrix.ts new file mode 100644 index 00000000..2ac9abcb --- /dev/null +++ b/packages/conformance/src/capability-matrix.ts @@ -0,0 +1,117 @@ +// `pnpm docs:capabilities [--check] [--website ]`: renders the capability tables. + +import { existsSync, readFileSync, writeFileSync } from 'node:fs' +import path from 'node:path' + +import { validateCapabilities } from './capability-validation.js' +import { CAPABILITY_TABLES, README_PAGE, REPO_ROOT } from './constants.js' +import { replaceBlock } from './render-capabilities.js' +import type { CapabilityTable, CapabilityTarget } from './types.js' + +export function targetsFor(websiteRoot?: string): CapabilityTarget[] { + return ( + Object.entries(CAPABILITY_TABLES) as [CapabilityTable, string][] + ).flatMap(([table, page]) => { + if (page === README_PAGE) { + return [{ table, file: path.join(REPO_ROOT, page) }] + } + return websiteRoot ? [{ table, file: path.join(websiteRoot, page) }] : [] + }) +} + +/** Each file with every one of its tables rendered; throws before anything is written if any target cannot be. */ +export function renderFiles( + targets: CapabilityTarget[] +): Map { + const files = new Map() + const problems: string[] = [] + for (const { table, file } of targets) { + if (!files.has(file)) { + if (!existsSync(file)) { + problems.push(`${file}: not found`) + continue + } + const current = readFileSync(file, 'utf8') + files.set(file, { current, next: current }) + } + const entry = files.get(file) + if (!entry) { + continue + } + try { + entry.next = replaceBlock(entry.next, table) + } catch (err) { + problems.push(`${file}: ${(err as Error).message}`) + } + } + if (problems.length) { + throw new Error(problems.join('\n')) + } + return files +} + +/** The files whose tables differ from the declared source, rewritten unless `check` is set. */ +export function syncTables( + targets: CapabilityTarget[], + check: boolean +): string[] { + const stale = [...renderFiles(targets)].filter( + ([, { current, next }]) => current !== next + ) + if (!check) { + for (const [file, { next }] of stale) { + writeFileSync(file, next) + } + } + return stale.map(([file]) => file) +} + +/** `--website ` or `--website=`; the value must not be another flag. */ +export function parseArgs(argv: string[]): { + check: boolean + websiteRoot?: string +} { + let check = false + let websiteRoot: string | undefined + for (let i = 0; i < argv.length; i++) { + const arg = argv[i] + if (arg === '--check') { + check = true + } else if (arg === '--website' || arg.startsWith('--website=')) { + const value = + arg === '--website' ? argv[++i] : arg.slice('--website='.length) + if (!value || value.startsWith('--')) { + throw new Error('--website needs the path of a webdriverio checkout') + } + websiteRoot = path.resolve(value) + } else { + throw new Error(`unknown argument ${arg}`) + } + } + return { check, websiteRoot } +} + +function main(argv: string[]): number { + try { + const problems = validateCapabilities() + if (problems.length) { + throw new Error(`capabilities.ts is invalid:\n ${problems.join('\n ')}`) + } + const { check, websiteRoot } = parseArgs(argv) + const stale = syncTables(targetsFor(websiteRoot), check) + for (const file of stale) { + console.log(`${check ? 'out of date' : 'updated'}: ${file}`) + } + return check && stale.length ? 1 : 0 + } catch (err) { + console.error(`docs:capabilities: ${(err as Error).message}`) + return 1 + } +} + +if ( + process.argv[1] && + path.resolve(process.argv[1]) === path.resolve(import.meta.filename) +) { + process.exitCode = main(process.argv.slice(2)) +} diff --git a/packages/conformance/src/capability-options.ts b/packages/conformance/src/capability-options.ts new file mode 100644 index 00000000..9cadd4ac --- /dev/null +++ b/packages/conformance/src/capability-options.ts @@ -0,0 +1,124 @@ +import type { BaseDevToolsOptions } from '@wdio/devtools-shared' +import type { ServiceOptions } from '@wdio/devtools-service' +import type { DevToolsOptions as SeleniumOptions } from '@wdio/selenium-devtools/types' +import type { DevToolsOptions as NightwatchOptions } from '@wdio/nightwatch-devtools/types' + +import type { AdapterId, Capability, OptionTarget } from './types.js' + +type OwnKeys = Exclude + +const WHERE_THE_BACKEND_LISTENS = { + notCapability: 'Where the dashboard backend listens.' +} + +// Keyed on the option types themselves, so a new option fails to compile until it is placed. +export const SHARED_OPTIONS: Record = { + port: WHERE_THE_BACKEND_LISTENS, + hostname: WHERE_THE_BACKEND_LISTENS, + screencast: { capability: 'filmstrip' }, + captureAssertions: { capability: 'assertion-capture' }, + mode: { capability: 'trace-mode' }, + traceFormat: { capability: 'trace-format' }, + traceGranularity: { capability: 'per-test-granularity' }, + tracePolicy: { capability: 'retry-retention' }, + filmstrip: { capability: 'filmstrip' }, + emitArtifactsManifest: { capability: 'artifacts-manifest' } +} + +export const SERVICE_OPTIONS: Record, OptionTarget> = { + screenshot: { capability: 'per-test-artifacts' }, + video: { capability: 'per-test-artifacts' }, + devtoolsCapabilities: { capability: 'dashboard-window' } +} + +export const SELENIUM_OPTIONS: Record< + OwnKeys, + OptionTarget +> = { + openUi: { capability: 'dashboard-window' }, + captureScreenshots: { capability: 'screenshot-switch' }, + rerunCommand: { capability: 'run-controls' }, + headless: { capability: 'headless' }, + screenshot: { capability: 'per-test-artifacts' }, + video: { capability: 'per-test-artifacts' } +} + +export const NIGHTWATCH_OPTIONS: Record< + OwnKeys, + OptionTarget +> = { + bidi: { capability: 'auto-bidi' }, + screenshot: { capability: 'per-test-artifacts' }, + video: { capability: 'per-test-artifacts' } +} + +const TURNS_THE_ADAPTER_ON = { + notCapability: 'Turns the adapter on.' +} + +// Internal handshake variables (rerun reuse, runner cwd) come from the shared contract and are set by the backend, not by users. +/** `enable()` keywords, pytest flags and ini keys, and `DEVTOOLS_*` variables; a test reads the real ones from the source. */ +export const PYTHON_OPTIONS: Record = { + host: WHERE_THE_BACKEND_LISTENS, + port: WHERE_THE_BACKEND_LISTENS, + webdriver_cls: { + notCapability: + 'Which WebDriver class to instrument, for a project that subclasses it.' + }, + trace: { capability: 'trace-mode' }, + filmstrip: { capability: 'filmstrip' }, + a11y: { capability: 'a11y-overlay' }, + trace_policy: { capability: 'retry-retention' }, + trace_granularity: { capability: 'per-test-granularity' }, + '--devtools': TURNS_THE_ADAPTER_ON, + '--devtools-trace': { capability: 'trace-mode' }, + '--devtools-trace-granularity': { capability: 'per-test-granularity' }, + '--devtools-trace-policy': { capability: 'retry-retention' }, + devtools: TURNS_THE_ADAPTER_ON, + devtools_trace: { capability: 'trace-mode' }, + devtools_trace_granularity: { capability: 'per-test-granularity' }, + devtools_trace_policy: { capability: 'retry-retention' }, + DEVTOOLS_HOST: WHERE_THE_BACKEND_LISTENS, + DEVTOOLS_PORT: WHERE_THE_BACKEND_LISTENS, + DEVTOOLS_BACKEND_CMD: { + notCapability: 'Which command launches the dashboard backend.' + }, + DEVTOOLS_ENABLE: TURNS_THE_ADAPTER_ON, + DEVTOOLS_BIDI: { capability: 'auto-bidi' }, + DEVTOOLS_OPEN: { capability: 'dashboard-window' }, + DEVTOOLS_TRACE: { capability: 'trace-mode' }, + DEVTOOLS_FILMSTRIP: { capability: 'filmstrip' }, + DEVTOOLS_A11Y: { capability: 'a11y-overlay' }, + DEVTOOLS_TRACE_POLICY: { capability: 'retry-retention' }, + DEVTOOLS_TRACE_GRANULARITY: { capability: 'per-test-granularity' } +} + +export const OPTION_MAPS: Record< + AdapterId | 'shared', + Record +> = { + shared: SHARED_OPTIONS, + 'devtools-service': SERVICE_OPTIONS, + 'selenium-devtools': SELENIUM_OPTIONS, + 'nightwatch-devtools': NIGHTWATCH_OPTIONS, + 'selenium-devtools-py': PYTHON_OPTIONS +} + +/** Every option names a declared capability, or says why it is not one. */ +export function optionProblems(capabilities: Capability[]): string[] { + const ids = new Set(capabilities.map((c) => c.id)) + return Object.entries(OPTION_MAPS).flatMap(([owner, options]) => + Object.entries(options).flatMap(([option, target]) => { + if ('capability' in target) { + return ids.has(target.capability) + ? [] + : [ + `option ${owner}.${option}: names unknown capability ${target.capability}` + ] + } + return target.notCapability.trim() + ? [] + : [`option ${owner}.${option}: not a capability, but no reason`] + }) + ) +} diff --git a/packages/conformance/src/capability-validation.ts b/packages/conformance/src/capability-validation.ts new file mode 100644 index 00000000..9fd19ab1 --- /dev/null +++ b/packages/conformance/src/capability-validation.ts @@ -0,0 +1,101 @@ +import { KNOWN_GAPS } from './allowlist.js' +import { + ADAPTER_COLUMNS, + CAPABILITIES, + INTERNAL_RULES +} from './capabilities.js' +import { optionProblems } from './capability-options.js' +import { RULES } from './constants.js' +import type { AdapterId, Capability, Cell, RuleId } from './types.js' + +/** Problems the declaration can have that its types cannot rule out; empty when it is sound. */ +export function validateCapabilities( + capabilities: Capability[] = CAPABILITIES, + internalRules: Partial> = INTERNAL_RULES +): string[] { + return [ + ...rowProblems(capabilities), + ...ruleCoverageProblems(capabilities, internalRules), + ...optionProblems(capabilities) + ] +} + +/** Per row: a cell for every adapter, a reason where one is due, real rules behind every proven cell. */ +export function rowProblems(capabilities: Capability[]): string[] { + const problems: string[] = [] + const known = new Set(Object.values(RULES)) + const ids = new Set() + for (const capability of capabilities) { + if (ids.has(capability.id)) { + problems.push(`${capability.id}: declared twice`) + } + ids.add(capability.id) + if (capability.tables.length === 0) { + problems.push(`${capability.id}: appears in no table`) + } + for (const { id: adapter } of ADAPTER_COLUMNS) { + const cell = capability.cells[adapter] + if (!cell) { + problems.push(`${capability.id}: no cell for ${adapter}`) + continue + } + problems.push( + ...cellProblems(cell, adapter, known).map( + (p) => `${capability.id} / ${adapter}: ${p}` + ) + ) + } + } + return problems +} + +/** Every conformance rule backs some proven cell, or is listed as internal with a reason. */ +export function ruleCoverageProblems( + capabilities: Capability[], + internalRules: Partial> +): string[] { + const proving = new Set( + capabilities.flatMap((c) => + Object.values(c.cells).flatMap((cell) => + cell.support === 'proven' ? cell.rules : [] + ) + ) + ) + return (Object.values(RULES) as RuleId[]).flatMap((rule) => { + const internal = internalRules[rule] + if (internal !== undefined && !internal.trim()) { + return [`rule ${rule}: internal without a reason`] + } + if (proving.has(rule) && internal !== undefined) { + return [`rule ${rule}: listed as internal but proves a cell`] + } + return proving.has(rule) || internal !== undefined + ? [] + : [`rule ${rule}: proves no capability and is not listed as internal`] + }) +} + +function cellProblems( + cell: Cell, + adapter: AdapterId, + known: Set +): string[] { + if (cell.support === 'partial' || cell.support === 'unsupported') { + return cell.reason.trim() ? [] : ['a reason is required'] + } + if (cell.support !== 'proven') { + return [] + } + if (cell.rules.length === 0) { + return ['proven by no rule'] + } + return cell.rules.flatMap((rule) => { + if (!known.has(rule)) { + return [`unknown rule ${rule}`] + } + const gap = KNOWN_GAPS.find( + (g) => g.adapter === adapter && g.rules.includes(rule) + ) + return gap ? [`${rule} is a known gap (bidi: ${gap.bidi})`] : [] + }) +} diff --git a/packages/conformance/src/constants.ts b/packages/conformance/src/constants.ts index b775d285..63947cdc 100644 --- a/packages/conformance/src/constants.ts +++ b/packages/conformance/src/constants.ts @@ -143,3 +143,22 @@ export const REPORT_INDEX = 'index.html' // Long enough to identify an argument, short enough to keep a timeline row on one line. export const TARGET_MAX_CHARS = 120 + +export const README_PAGE = 'README.md' + +/** Where each capability table is rendered, relative to this repo or to a webdriverio checkout. */ +export const CAPABILITY_TABLES = { + 'readme-trace': README_PAGE, + 'readme-dashboard': README_PAGE, + 'readme-mobile': README_PAGE, + 'cross-framework': 'website/docs/devtools/CrossFrameworkSupport.md', + mobile: 'website/docs/devtools/MobileTesting.md' +} as const + +export const SUPERSCRIPTS = ['⁰', '¹', '²', '³', '⁴', '⁵', '⁶', '⁷', '⁸', '⁹'] + +export const GENERATED_NOTICE = + '' + +export const LEGEND = + '✅ tested: proven for that adapter by the cross-adapter conformance suite. Every other mark is declared.' diff --git a/packages/conformance/src/render-capabilities.ts b/packages/conformance/src/render-capabilities.ts new file mode 100644 index 00000000..5366598c --- /dev/null +++ b/packages/conformance/src/render-capabilities.ts @@ -0,0 +1,119 @@ +import { ADAPTER_COLUMNS, CAPABILITIES } from './capabilities.js' +import { GENERATED_NOTICE, LEGEND, SUPERSCRIPTS } from './constants.js' +import type { Capability, CapabilityTable, Cell } from './types.js' + +export function superscript(n: number): string { + return String(n) + .split('') + .map((digit) => SUPERSCRIPTS[Number(digit)]) + .join('') +} + +export const markerFor = (table: CapabilityTable, edge: 'start' | 'end') => + `` + +function mark(cell: Cell): string { + switch (cell.support) { + case 'supported': + return '✅' + case 'proven': + return '✅ tested' + case 'partial': + return '⚠️' + case 'unsupported': + return '❌' + } +} + +/** One table, with each distinct reason numbered once in the order it first appears. */ +export function renderTable( + table: CapabilityTable, + capabilities: Capability[] = CAPABILITIES +): string { + const rows = capabilities.filter((c) => c.tables.includes(table)) + const reasons: string[] = [] + const footnote = (reason: string) => { + const index = reasons.indexOf(reason) + if (index !== -1) { + return index + 1 + } + reasons.push(reason) + return reasons.length + } + const cellText = (cell: Cell) => { + const parts = [mark(cell)] + if (cell.label) { + parts.push(cell.label) + } + if (cell.support === 'partial' || cell.support === 'unsupported') { + parts.push(superscript(footnote(cell.reason))) + } + return parts.join(' ') + } + const lines = [ + `| Capability | ${ADAPTER_COLUMNS.map((a) => a.title).join(' | ')} |`, + `|---|${ADAPTER_COLUMNS.map(() => '---').join('|')}|`, + ...rows.map( + (row) => + `| ${row.name} | ${ADAPTER_COLUMNS.map((a) => cellText(row.cells[a.id])).join(' | ')} |` + ) + ] + const hasProven = rows.some((row) => + ADAPTER_COLUMNS.some((a) => row.cells[a.id].support === 'proven') + ) + const blocks = [lines.join('\n')] + if (hasProven) { + blocks.push(LEGEND) + } + if (reasons.length) { + blocks.push(...reasons.map((r, i) => `${superscript(i + 1)} ${r}`)) + } + return blocks.join('\n\n') +} + +/** The full marker block, as it sits in a page. */ +export function renderBlock( + table: CapabilityTable, + capabilities?: Capability[] +): string { + return [ + markerFor(table, 'start'), + GENERATED_NOTICE, + '', + renderTable(table, capabilities), + '', + markerFor(table, 'end') + ].join('\n') +} + +/** Replaces the page's one marker block for this table; throws, naming the problem, when it has none, two, or a reversed pair. */ +export function replaceBlock( + page: string, + table: CapabilityTable, + capabilities?: Capability[] +): string { + const startMarker = markerFor(table, 'start') + const endMarker = markerFor(table, 'end') + const starts = page.split(startMarker).length - 1 + const ends = page.split(endMarker).length - 1 + if (starts === 0 && ends === 0) { + throw new Error( + `no ${table} table; insert the lines ${startMarker} and ${endMarker} where it belongs` + ) + } + if (starts !== 1 || ends !== 1) { + throw new Error( + `${starts} start and ${ends} end markers for ${table}; keep exactly one ${startMarker} and one ${endMarker}` + ) + } + const start = page.indexOf(startMarker) + const end = page.indexOf(endMarker) + if (end < start) { + throw new Error(`${endMarker} comes before ${startMarker}`) + } + return ( + page.slice(0, start) + + renderBlock(table, capabilities) + + page.slice(end + endMarker.length) + ) +} diff --git a/packages/conformance/src/types.ts b/packages/conformance/src/types.ts index 3d202aaf..2ffb0ff2 100644 --- a/packages/conformance/src/types.ts +++ b/packages/conformance/src/types.ts @@ -8,6 +8,7 @@ import type { import type { ACTION_METHODS, BIDI_MODES, + CAPABILITY_TABLES, RESPONSIBILITIES, RULES } from './constants.js' @@ -236,3 +237,29 @@ export interface Timeline { /** Violations with no subject, or whose subject is not on the timeline. */ unplaced: Violation[] } + +/** Where a capability row is rendered: a marker block in a markdown page. */ +export type CapabilityTable = keyof typeof CAPABILITY_TABLES + +export type Cell = + | { support: 'supported'; label?: string } + /** Supported, and checked for this adapter by the conformance suite through these rules. */ + | { support: 'proven'; rules: RuleId[]; label?: string } + | { support: 'partial'; reason: string; label?: string } + | { support: 'unsupported'; reason: string; label?: string } + +export interface Capability { + id: string + name: string + tables: CapabilityTable[] + cells: Record +} + +/** What a user-facing option controls: a capability row, or why it is not one. */ +export type OptionTarget = { capability: string } | { notCapability: string } + +/** One table in one page file. */ +export interface CapabilityTarget { + table: CapabilityTable + file: string +} diff --git a/packages/conformance/tests/capabilities.test.ts b/packages/conformance/tests/capabilities.test.ts new file mode 100644 index 00000000..d8f63058 --- /dev/null +++ b/packages/conformance/tests/capabilities.test.ts @@ -0,0 +1,303 @@ +import { describe, expect, it } from 'vitest' + +import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs' +import { tmpdir } from 'node:os' +import path from 'node:path' + +import { + ADAPTER_COLUMNS, + CAPABILITIES, + INTERNAL_RULES +} from '../src/capabilities.js' +import { optionProblems } from '../src/capability-options.js' +import { parseArgs, syncTables, targetsFor } from '../src/capability-matrix.js' +import { + rowProblems, + ruleCoverageProblems, + validateCapabilities +} from '../src/capability-validation.js' +import { CAPABILITY_TABLES, LEGEND, RULES } from '../src/constants.js' +import { + markerFor, + renderBlock, + renderTable, + replaceBlock, + superscript +} from '../src/render-capabilities.js' +import type { + AdapterId, + Capability, + CapabilityTable, + Cell, + RuleId +} from '../src/types.js' + +const allCells = (cell: Cell): Record => ({ + 'devtools-service': cell, + 'selenium-devtools': cell, + 'nightwatch-devtools': cell, + 'selenium-devtools-py': cell +}) + +const capability = (over: Partial = {}): Capability => ({ + id: 'probe', + name: 'Probe', + tables: ['cross-framework'], + cells: allCells({ support: 'supported' }), + ...over +}) + +describe('validateCapabilities', () => { + it('accepts the declared source', () => { + expect(validateCapabilities()).toEqual([]) + }) + + it('rejects an empty reason', () => { + const blank = capability({ + cells: { + ...allCells({ support: 'supported' }), + 'selenium-devtools': { support: 'partial', reason: ' ' } + } + }) + expect(rowProblems([blank])).toEqual([ + 'probe / selenium-devtools: a reason is required' + ]) + }) + + it('rejects a proven cell whose rule does not exist', () => { + const unknown = capability({ + cells: allCells({ support: 'proven', rules: ['no-such-rule' as RuleId] }) + }) + expect(rowProblems([unknown])).toContain( + 'probe / devtools-service: unknown rule no-such-rule' + ) + }) + + it('rejects a proven cell the adapter holds a known gap against', () => { + const gapped = capability({ + cells: allCells({ support: 'proven', rules: [RULES.documentAnchored] }) + }) + const problems = rowProblems([gapped]) + expect(problems).toContain( + 'probe / selenium-devtools: every-document-anchored is a known gap (bidi: off)' + ) + expect(problems.some((p) => p.startsWith('probe / devtools-service'))).toBe( + false + ) + }) + + it('rejects a missing adapter, a duplicate id and a row in no table', () => { + const { 'selenium-devtools-py': _python, ...three } = allCells({ + support: 'supported' + }) + const missing = capability({ cells: three as Record }) + expect(rowProblems([missing, capability({ tables: [] })])).toEqual([ + 'probe: no cell for selenium-devtools-py', + 'probe: declared twice', + 'probe: appears in no table' + ]) + }) +}) + +describe('rule coverage', () => { + it('flags a rule that proves nothing and is not internal', () => { + expect(ruleCoverageProblems([capability()], {})).toContain( + `rule ${RULES.metadataFirst}: proves no capability and is not listed as internal` + ) + }) + + it('flags an internal rule without a reason, or one that also proves a cell', () => { + const proven = capability({ + cells: allCells({ support: 'proven', rules: [RULES.metadataFirst] }) + }) + const problems = ruleCoverageProblems([proven], { + ...INTERNAL_RULES, + [RULES.metadataFirst]: 'Internal.', + [RULES.failureRecorded]: ' ' + }) + expect(problems).toContain( + `rule ${RULES.metadataFirst}: listed as internal but proves a cell` + ) + expect(problems).toContain( + `rule ${RULES.failureRecorded}: internal without a reason` + ) + }) +}) + +describe('option coverage', () => { + it('flags an option that names a capability nobody declared', () => { + expect(optionProblems([])).toContain( + 'option shared.mode: names unknown capability trace-mode' + ) + }) + + it('places every option of the declared source', () => { + expect(optionProblems(CAPABILITIES)).toEqual([]) + }) +}) + +describe('renderTable', () => { + it('has a column for every adapter', () => { + const header = renderTable('cross-framework').split('\n')[0] + expect(header).toBe( + `| Capability | ${ADAPTER_COLUMNS.map((a) => a.title).join(' | ')} |` + ) + }) + + it('marks proven cells and adds the legend only when a row has one', () => { + const proven = capability({ + cells: allCells({ support: 'proven', rules: [RULES.failureRecorded] }) + }) + expect(renderTable('cross-framework', [proven])).toContain( + '| Probe | ✅ tested | ✅ tested | ✅ tested | ✅ tested |' + ) + expect(renderTable('cross-framework', [proven])).toContain(LEGEND) + expect(renderTable('cross-framework', [capability()])).not.toContain(LEGEND) + }) + + it('numbers each distinct reason once and lists it under the table', () => { + const rows = [ + capability({ + cells: { + ...allCells({ support: 'supported' }), + 'selenium-devtools': { + support: 'partial', + label: 'with BiDi', + reason: 'Same.' + }, + 'nightwatch-devtools': { support: 'partial', reason: 'Same.' }, + 'selenium-devtools-py': { + support: 'unsupported', + label: 'n/a', + reason: 'Other.' + } + } + }) + ] + expect(renderTable('cross-framework', rows)).toBe( + [ + '| Capability | WebdriverIO | Selenium | Nightwatch | Python |', + '|---|---|---|---|---|', + '| Probe | ✅ | ⚠️ with BiDi ¹ | ⚠️ ¹ | ❌ n/a ² |', + '', + '¹ Same.', + '', + '² Other.' + ].join('\n') + ) + }) + + it('renders only the rows declared for that table', () => { + const mobile = renderTable('mobile') + for (const row of CAPABILITIES) { + expect(mobile.includes(`| ${row.name} |`)).toBe( + row.tables.includes('mobile') + ) + } + }) + + it('writes multi-digit footnote numbers', () => { + expect(superscript(12)).toBe('¹²') + }) +}) + +describe('replaceBlock', () => { + it('replaces only what sits between the markers', () => { + const page = [ + 'before', + '', + 'stale', + '', + 'after' + ].join('\n') + expect(replaceBlock(page, 'mobile')).toBe( + `before\n${renderBlock('mobile')}\nafter` + ) + }) + + it('names the marker lines to insert when the page has none', () => { + expect(() => replaceBlock('no table here', 'mobile')).toThrow( + `insert the lines ${markerFor('mobile', 'start')} and ${markerFor('mobile', 'end')}` + ) + }) + + it('refuses a duplicated block and a reversed pair, each with its own message', () => { + const block = `${markerFor('mobile', 'start')}\n${markerFor('mobile', 'end')}` + expect(() => replaceBlock(`${block}\n${block}`, 'mobile')).toThrow( + '2 start and 2 end markers' + ) + expect(() => + replaceBlock( + `${markerFor('mobile', 'end')}\n${markerFor('mobile', 'start')}`, + 'mobile' + ) + ).toThrow('comes before') + }) +}) + +describe('docs:capabilities arguments', () => { + it('takes the website path in either form', () => { + expect(parseArgs(['--website', 'a']).websiteRoot).toBe(path.resolve('a')) + expect(parseArgs(['--website=a', '--check'])).toEqual({ + check: true, + websiteRoot: path.resolve('a') + }) + }) + + it('rejects a flag or nothing as the website path', () => { + expect(() => parseArgs(['--website', '--check'])).toThrow('needs the path') + expect(() => parseArgs(['--website'])).toThrow('needs the path') + expect(() => parseArgs(['--website='])).toThrow('needs the path') + }) +}) + +describe('syncTables', () => { + const tables: CapabilityTable[] = ['readme-trace', 'readme-mobile'] + const blocks = tables + .map((t) => `${markerFor(t, 'start')}\nstale\n${markerFor(t, 'end')}`) + .join('\n') + + it('writes a file once with all of its tables, and writes nothing when any target fails', () => { + const dir = mkdtempSync(path.join(tmpdir(), 'capabilities-')) + try { + const good = path.join(dir, 'good.md') + const bad = path.join(dir, 'bad.md') + writeFileSync(good, blocks) + writeFileSync(bad, 'no markers') + const targets = [ + ...tables.map((table) => ({ table, file: good })), + { table: 'mobile' as const, file: bad } + ] + expect(() => syncTables(targets, false)).toThrow('bad.md') + expect(readFileSync(good, 'utf8')).toBe(blocks) + + expect(syncTables(targets.slice(0, 2), false)).toEqual([good]) + expect(readFileSync(good, 'utf8')).toBe( + tables.map((t) => renderBlock(t)).join('\n') + ) + } finally { + rmSync(dir, { recursive: true, force: true }) + } + }) + + it('reports a missing file before writing anything', () => { + expect(() => + syncTables([{ table: 'mobile', file: '/nonexistent/page.md' }], true) + ).toThrow('not found') + }) +}) + +describe('generated tables', () => { + it('every table has at least one row', () => { + for (const table of Object.keys(CAPABILITY_TABLES)) { + expect(CAPABILITIES.some((c) => c.tables.includes(table as never))).toBe( + true + ) + } + }) + + // Run `pnpm docs:capabilities` when this fails. + it('README.md matches the declared source', () => { + expect(syncTables(targetsFor(), true)).toEqual([]) + }) +}) diff --git a/packages/conformance/tests/capability-python-options.test.ts b/packages/conformance/tests/capability-python-options.test.ts new file mode 100644 index 00000000..b47c103e --- /dev/null +++ b/packages/conformance/tests/capability-python-options.test.ts @@ -0,0 +1,80 @@ +import { spawnSync } from 'node:child_process' +import path from 'node:path' +import { describe, expect, it } from 'vitest' + +import { PYTHON_OPTIONS } from '../src/capability-options.js' +import { REPO_ROOT } from '../src/constants.js' + +const PKG = path.join( + REPO_ROOT, + 'packages/selenium-devtools-py/src/selenium_devtools' +) +const hasPython = spawnSync('python3', ['--version']).status === 0 + +// Parsed with Python's own ast rather than imported, so the check needs no selenium install. +const READ_OPTIONS = ` +import ast, json, sys +init, plugin, constants = sys.argv[1:4] + +def tree(path): + return ast.parse(open(path).read()) + +enable = next((n for n in ast.walk(tree(init)) + if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef)) and n.name == "enable"), None) +if enable is None: + sys.exit("enable() not found in " + init) +a = enable.args +names = [p.arg for p in a.posonlyargs + a.args + a.kwonlyargs] +names += ["*" + a.vararg.arg] if a.vararg else [] +names += ["**" + a.kwarg.arg] if a.kwarg else [] + +for call in ast.walk(tree(plugin)): + if not (isinstance(call, ast.Call) and isinstance(call.func, ast.Attribute)): + continue + strings = [x.value for x in call.args if isinstance(x, ast.Constant) and isinstance(x.value, str)] + if call.func.attr == "addoption": + longs = [s for s in strings if s.startswith("--")] + names += longs or strings + elif call.func.attr == "addini" and strings: + names.append(strings[0]) + +for node in tree(constants).body: + if (isinstance(node, ast.Assign) and isinstance(node.value, ast.Constant) + and isinstance(node.value.value, str) and node.value.value.startswith("DEVTOOLS_") + and any(isinstance(t, ast.Name) and t.id.startswith("ENV_") for t in node.targets)): + names.append(node.value.value) + +print(json.dumps(names)) +` + +function pythonOptions(): string[] { + const run = spawnSync( + 'python3', + [ + '-I', + '-c', + READ_OPTIONS, + path.join(PKG, '__init__.py'), + path.join(PKG, 'pytest_plugin.py'), + path.join(PKG, 'constants.py') + ], + { encoding: 'utf8' } + ) + if (run.status !== 0) { + throw new Error( + `could not read the Python adapter's options: ${run.stderr}` + ) + } + return JSON.parse(run.stdout) as string[] +} + +describe('Python adapter options', () => { + it.skipIf(!hasPython)( + 'maps every enable() keyword, pytest option and DEVTOOLS_* variable, and nothing else', + () => { + const found = [...new Set(pythonOptions())].sort() + expect(found.length).toBeGreaterThan(10) + expect(Object.keys(PYTHON_OPTIONS).sort()).toEqual(found) + } + ) +}) diff --git a/tsconfig.json b/tsconfig.json index 5f4bb283..f488b249 100644 --- a/tsconfig.json +++ b/tsconfig.json @@ -42,7 +42,8 @@ "@wdio/devtools-service": ["./packages/service/src/index.ts"], "@wdio/devtools-service/*": ["./packages/service/src/*"], "@wdio/selenium-devtools": ["./packages/selenium-devtools/src/index.ts"], - "@wdio/selenium-devtools/*": ["./packages/selenium-devtools/src/*"] + "@wdio/selenium-devtools/*": ["./packages/selenium-devtools/src/*"], + "@wdio/nightwatch-devtools/*": ["./packages/nightwatch-devtools/src/*"] } }, // This file is both the `extends` base for every package config and, run