diff --git a/libsql-server/scripts/namespace-live-checks/README.md b/libsql-server/scripts/namespace-live-checks/README.md new file mode 100644 index 0000000000..8be6f87d5f --- /dev/null +++ b/libsql-server/scripts/namespace-live-checks/README.md @@ -0,0 +1,97 @@ +# Namespace live checks + +Scripts that run a built `sqld` locally and check how the namespace store +behaves under load, during shutdown, and across the admin API. They exist +because the interesting failures here are races: an entry emptied under a +request, a shutdown while requests are in flight, a destroy racing a lookup. +Unit tests pin each race deterministically; these scripts show the same +races against a real server, and let two binaries be compared. + +Nothing here touches a remote system. Everything listens on `127.0.0.1`. + +## Requirements + +`bash` 3.2 or newer, `curl`, `python3`, `lsof`. A debug build is fine and +widens the race windows, which is what you want. + +``` +cargo build -p libsql-server --bin sqld +cargo build -p libsql --example replica_sync_loop # for replica-sigterm.sh +``` + +## Scripts + +| Script | What it does | +|---|---| +| `load.py` | Keep-alive clients driving `POST /v2/pipeline`; tallies status codes per phase, with an optional SIGTERM or shell action at a given time. | +| `scenario.sh` | One focused scenario: `evict` (cache capacity far below the hot set) or `sigterm` (signal under load). Prints the tally, the namespace-store counters and the grouped store log lines. | +| `regress.sh` | A 65-check matrix across the admin and user APIs, one primary and one primary plus replica, optionally mixed versions. Writes `results.tsv`. | +| `compare.py` | Side-by-side diff of two `results.tsv` files. | +| `replica-sigterm.sh` | What embedded replicas (the `libsql` crate, `libsql/examples/replica_sync_loop.rs`) see when their primary is stopped and started again: every distinct `sync()` error with the time it was first seen, and the replicator's own retry log lines. | +| `window.sh` | How long the shutdown window (503s until the port closes) is for N loaded namespaces, each with a WAL. Compare with your clients' retry budgets. | + +## Comparing two binaries + +``` +S=libsql-server/scripts/namespace-live-checks +$S/scenario.sh target/debug/sqld evict # expect 0 non-200 answers +$S/scenario.sh target/debug/sqld sigterm # expect 503s after the signal, no 404s + +$S/regress.sh /path/to/old/sqld old /path/to/new/sqld +$S/regress.sh /path/to/new/sqld new /path/to/old/sqld +$S/compare.py namespace-live-checks-out/old/results.tsv namespace-live-checks-out/new/results.tsv old new + +$S/replica-sigterm.sh target/debug/sqld # expect no "doesn't exist"; see below +$S/window.sh target/debug/sqld 2000 # setup takes about 45 s per 1,000 namespaces +``` + +`regress.sh` writes server logs next to the results (`