chore: run existing Playwright suite on Endform

[ostenbom]

No ticket: this fork PR prepares the requested Endform comparison.

What & why

Was: The full Chromium suite uses four native CI browser workers or the existing Azure runner.

Now: Endform runs the same suite against the same CI-local application services, with at most four concurrent tests.

Stacked on #1, targeting codex/playwright-baseline. Retarget to main after benchmark verification, keeping baseline evidence available and unmerged.

Where to look

• E2E workflow: unchanged services; Endform replaces the runner.
• Endform config: precise files, database environment, loopback HTTP/TCP, concurrency.
• PR caller: job-scoped OIDC permission.

Breaking changes

• This PR contains breaking changes

None. Internal test infrastructure; public APIs and self-hosted application behavior remain unchanged.

Migrations & env

• Existing ENTERPRISE_LICENSE_KEY repository secret is required and absent.
• GitHub connection to Endform organization 2G1ZCj7X is confirmed; CI uses OIDC.
• BASE_URL defaults to localhost; optional local POSTGRES_PORT defaults to CI port 5432.

How this was tested

Rerun: npx endform@latest test --organization-id 2G1ZCj7X from the repository root.

Coverage

Behaviour	How	Outcome
Suite selection	manual	Native discovery remains 134 Chromium tests across 49 files; test files unchanged.
Configuration	manual	TypeScript config check, YAML parsing, formatting, catalog checks pass.

Open gaps

• CI browser stages blocked by missing enterprise license; no timing comparison is valid yet.
• Local run times out during login; SpiceDB delivery is unavailable. CI browser execution remains unverified.
• Existing Cube client unit test times out on full and isolated runs.

Risks

• CI-local application capacity limits concurrency; four concurrent tests preserve the native baseline load.

---

Note

AI model used — unknown, reasoning effort unknown.

Integration and validation details

The baseline commit is 4965bf998f8bb8764397a16ffa64780a4f214091, preserving upstream application and tests at 87012a2049029adc36698f4a7853dd77606fbb19.

Baseline setup run: https://github.com/endformdev/formbricks/actions/runs/36840156772. It completed with failure at the existing enterprise-license preflight; the application and Playwright were not started. This is not a completed suite baseline.

There is no shard matrix or report-merging job. Endform is invoked once for all existing specs. Postgres, Valkey, SpiceDB, RustFS, migrations, rate-limit/cache integration checks, app startup and authorization outbox worker remain in place. Browser cache/install and Azure mode selection are removed only from this migrated E2E workflow. Unrelated jobs remain intact.

Existing fullyParallel: true, Chromium project, timeout, two CI retries, zero local retries and serial signup grouping remain unchanged. Traces change from on-first-retry to retain-on-failure. An equivalent static dotenv import fixes Endform's observed failure evaluating the original require call. Only two runtime-read images and the database environment are declared explicitly.

CLI resolved version: 0.81.3. Config-helper dependency: 0.81.1, because the repository's three-day supply-chain cooldown rejected 0.81.3 and its platform packages. The cooldown is preserved; execution uses the requested latest CLI.

Local dependencies and all 13 application/dependency builds completed. Initial migrations encountered an existing host Postgres on port 5432; an isolated Docker Postgres uses port 55432 without changing that server. Explicit database creation resolved the migration error: 177 schema migrations and 28 data baselines completed. The application returns HTTP 200 from /health and the authorization outbox worker is ready.

pnpm test completed with 26/27 tasks successful. Web unit tests: 12,534 passed, one timed out. The same test times out alone: pnpm --filter=@formbricks/web exec vitest run modules/ee/analysis/api/lib/cube-client.test.ts, test at line 88, "loads query with a per-request tenant scoped token and returns tablePivot result". No assertion or timeout changes were made.

Local Endform diagnostic run: https://endform.dev/app/org/2G1ZCj7X/suite-run?suite_run_id=3K5TKQ9K1fDI4G25eclkuERhgXl. All 134 tests were uploaded; the first attempted test (action.spec.ts:296) failed because Prisma could not reach 127.0.0.1:55432. The existing local maxFailures=1 policy stopped the remaining 133 tests. This run used the config's default proxy port 5432 rather than the isolated local port; use POSTGRES_PORT=55432 for that local database. That initial diagnostic preceded successful isolated-database migrations and app startup. Its 10-second elapsed time is an early-failure diagnostic, never a benchmark.

Endform PR workflow: https://github.com/endformdev/formbricks/actions/runs/36841831848. The entire PR workflow completed with failure. The E2E job failed at the existing license check; the Endform command was skipped. OIDC execution is therefore still unverified.

Endform commit: 720f8e8af504cb666c353cfc6ed841df5cfbf99d.

Stage	Native baseline	Endform PR E2E job
Dependency install	30 s	27 s
Build (native overlaps browser install)	264 s	192 s
Application startup	Not reached	Not reached
Test stage	Skipped	Skipped
E2E job elapsed	374 s	296 s

These are single failed setup jobs, with different workflow context and cache state. They cannot measure test-stage or whole-workflow speedup. No additional benchmark runs are justified until the secret is provided.

Frozen-lockfile policy verification passes for all 2232 entries. No dependency cooldown exceptions were added. No timing improvement or costs are claimed.

Final local attempt

Run: https://endform.dev/app/org/2G1ZCj7X/suite-run?suite_run_id=3K5UFPuJODGJt1XuyX9DMfVI45H

Command: POSTGRES_PORT=55432 npx endform@latest test --organization-id 2G1ZCj7X.

All 134 tests were uploaded. The first attempted test, action.spec.ts:242, timed out after 120 seconds in beforeEach while waiting for login navigation. The authorization worker reports authzed_unavailable on relationship writes and reconciliation. One test timed out; the existing local maximum-failure policy prevented the other 133 tests from running. No retry was added, and no tests were changed. The application health endpoint was 200, database migrations completed, and the worker published its readiness heartbeat, but authorization delivery did not work on this local Docker environment. The local enterprise license is also absent.

The CLI reported 2.3 minutes elapsed and 2.3 billable runner minutes for this incomplete diagnostic. No cost or speedup estimate is inferred.

Temporary application, outbox worker, and test containers are stopped after collecting evidence; volumes and ignored local build/report artifacts remain available. The working tree is clean.

Workflow elapsed context

The standalone native baseline workflow was created at 09:02:34 UTC and its completion update was 09:08:55 UTC: 381 seconds. Its sole E2E job started six seconds after workflow creation.

The Endform PR workflow was created at 09:18:09 UTC and its completion update was 09:34:48 UTC: 999 seconds. Its E2E job started 29 seconds after workflow creation. The full PR workflow also runs unit tests, lint, typecheck, API contract tests, and other gates; its elapsed time is not comparable to the standalone baseline workflow. Pure GitHub queue time was not separately measured.

Both browser stages were skipped. Test-stage speedup and whole-workflow improvement are unavailable. One failed setup run per configuration is not a performance sample. Runner hardware and Endform remote resources were not measured.

The evidence demonstrates uploading the existing suite and receiving an Endform failure report. It does not establish passing CI execution, successful CI OIDC authentication, a feedback-time improvement, or a stable speedup.