Local LGTM observability stack for engine telemetry - #751
Open
tony wants to merge 26 commits into
Open
Conversation
tony
force-pushed
the
engine-ops-orchestration-bench
branch
from
August 23, 2026 01:21
45a28dc to
7d65eb8
Compare
tony
force-pushed
the
lgtm-observability
branch
from
August 23, 2026 01:21
a4d3724 to
eb4ede0
Compare
tony
force-pushed
the
engine-ops-orchestration-bench
branch
from
August 23, 2026 01:28
7d65eb8 to
a28598c
Compare
tony
force-pushed
the
lgtm-observability
branch
3 times, most recently
from
August 23, 2026 01:41
b07244b to
a1a468b
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## engine-ops-orchestration-bench #751 +/- ##
==================================================================
- Coverage 78.12% 77.73% -0.40%
==================================================================
Files 246 250 +4
Lines 24050 24594 +544
Branches 3170 3228 +58
==================================================================
+ Hits 18789 19118 +329
- Misses 4073 4282 +209
- Partials 1188 1194 +6 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
tony
force-pushed
the
lgtm-observability
branch
from
August 23, 2026 01:53
a1a468b to
d43441e
Compare
…size why: test_new_session_shell_env forwarded dict(os.environ) to new-session, which becomes one -e KEY=VAL argument per variable. tmux sums every argument and refuses past MAX_IMSGSIZE, so the test's outcome was a function of whoever ran it: green on a lean CI runner, "command too long" under a rich interactive shell, where the environment alone can exceed 16 KiB before libtmux appends its format string. The test asserts nothing about the environment's contents, so the size was incidental. what: - Pass a small explicit environment, so the assertion is about what the test claims: window_command survives alongside environment= - Cover the ceiling deliberately instead, with a payload built to exceed it and an assertion that the refusal is loud and creates no session
why: The capture-timeout tests read the child's pid from a file the stub writes on startup, and the timeout under test is 50ms. The stub was a Python script, so it had to start an interpreter before recording anything -- longer than 50ms on a busy machine. The child was then killed before it wrote the file and the test failed reading it, which reads as a reaping fault when reaping had in fact worked. Measured at roughly one failure in twelve runs before the change and none in thirty after, across both the sync and async cases. what: - Write the stub as a shell script, so recording the pid costs milliseconds rather than interpreter startup - Use exec for the blocking sleep, so the recorded pid stays the process the test later asserts was reaped, and no second process outlives it
why: kill-server exits 1 when no server is listening, which is the state teardown exists to reach. A server can finish exiting on its own between the ownership check and the kill landing, and the larger the topology the more routinely it does -- an 80x20x1 rung loses that race as a matter of course. The exit code was treated as authoritative, so a teardown that had demonstrably worked (processes absent, socket absent) recorded an error; complete cleanup requires an empty error list, so the rung failed. The escalating stress harness stops on a failed rung, which reported the pane ceiling as the point where cleanup was misread rather than where anything actually broke. what: - Judge teardown on the owning process's absence, which the code already established, rather than on the helper's exit status - Keep reporting any other non-zero status, which is the helper behaving unexpectedly rather than tmux reporting an absent server - Say so when a server outlives a kill that reported no server, once its survival is established - Pin both directions: the old behaviour fails the new test, and dropping the anomalous-exit report fails the existing one
why: --_test-stall-after parks a worker mid-run so the cancellation tests can prove the supervisor reaps it, and the wait was `while True`. That is correct only while the test is alive to do the reaping. A test that fails, times out, or is interrupted leaves the worker parked forever -- and because preflight refuses to start while any benchmark process is running, one orphan blocks every subsequent benchmark on the machine until reboot. Found by being bitten: a worker stranded by a load-flaked run was still parked 1h44m later, its pytest temporary directory long deleted, holding a tmux server and a tail alongside it, and it refused three unrelated matrix runs before anyone connected the two. The tests driving that path set the progress watchdog at 0.2 and 0.3 seconds. That is a no-progress timer, so a test parking a worker at a checkpoint has to let the run reach it first, and the slowest step before any stall point is starting a tmux server -- well under a second idle, seconds on a loaded machine. The threshold is not what either test asserts; they check what the run recorded. Measured at load 19, one passed 2 of 6 runs. what: - Bound the wait by a deadline and by the spawner going away, since reparenting is the portable signal that nobody is left to cancel - Take the bound as an argument so it is testable without patching a global - Cover both exits, plus that the shipped bound is finite - Raise both watchdogs to a shared named bound, with the constraint stated once, and widen the cleanup grace to match
why: Every rung inherited the benchmark's flat 120s no-progress watchdog, whatever topology it built. The cost per pass is superlinear on this ladder's own shapes -- one control/async pass measures 14.3s at 400 panes, 24.0s at 800, 52.5s at 1200 and 177.5s at 1600 -- so the allowance that is generous at the base is short at the top, and a slower rung is reported as a stuck one. That stops the axis before it reaches any real limit, which is the opposite of what an escalating harness is for. what: - Give each rung an allowance derived from the panes it builds, and pass it through to the benchmark - Cap it at the hard per-rung limit, so a genuinely stuck rung dies no later than it would have - Cover the scaling, the cap, and that the value actually reaches the child
why: The engine instrumentation seam could emit telemetry but had nowhere to send it, so nobody could see what a tmux workload actually costs. Standing up that stack by hand is unreproducible, and a dashboard nobody queries rots into panels that render an empty grid and look healthy doing it. The workload is short-lived, so every panel has to read the window the viewer selected rather than a counter at an instant: Prometheus marks a finished run's series stale within minutes, and an instant query then returns nothing beside a timeseries still drawing the same run. what: - Add scripts/lgtm/up.sh, pinning the otel-lgtm image and bind-mounting the datasource and dashboard provisioning, with a config label so changing a mount recreates the container instead of restarting it with stale state - Publish Grafana on 3900 and Prometheus on 9099: a host process already bound to their defaults still lets Docker publish, then answers first, so queries reach the wrong server and return plausible data - Add scripts/lgtm/telemetry.py, an OTelSink emitting spans and metrics together so the duration histogram records while its span is current and carries an exemplar, giving the metric-to-trace pivot - Add scripts/otel_smoke.py driving all four transports, with grouped commands so the inlining panels are not flat zero and rejected commands so the failure panels have real data - Generate the three dashboards from scripts/lgtm/generate_dashboards.py rather than hand-editing JSON, and commit the output - Query each panel's own window with increase(...[$__range]) rather than reading counters at a point, so a panel reports what happened in the range the viewer selected - Add scripts/otel_acceptance.py, which reads those dashboards, runs every panel's own query the way Grafana runs it -- range or instant -- and fails naming any that returned nothing; it re-checks until ingestion catches up, since "no data yet" and "no data ever" are indistinguishable at any single instant - Reject any Prometheus target with no range selector in a test, which catches a stale-series panel offline; a live check only sees it once the series has gone stale, which is precisely when nobody is watching - Add just otel-up/down/dashboards/smoke/acceptance/verify and an otel dependency group, kept out of dev so the ordinary gates stay lean
why: Telemetry that cannot say which branch, worktree, or experiment produced it can be observed but not compared, which is most of what a benchmark is for. The trap is treating that as one decision: copying every fact onto every signal makes each commit mint a fresh set of Prometheus series, and the cost lands on whoever runs the stack next month. what: - Add scripts/lgtm/identity.py resolving repository, ref, revision, and worktree from one git rev-parse at startup, using the OpenTelemetry vcs.* conventions, overridable by environment for CI's detached checkouts - Split the facts by signal: metrics carry only branch, run id, and spike, the dimensions worth grouping by; revision and worktree ride on traces and profiles where the drill-down happens. A SHA is never a comparison axis -- each run has one, so grouping by it is grouping by run - Copy baggage onto spans with a processor, so a value that changes mid-run reaches spans created inside the engines without becoming a parameter on calls that have no business knowing about telemetry - Give every board Transport and Branch selectors, and add a Compare board grouping the same measurements by run and by branch - Add tests pinning the metric label set closed and rejecting a panel that filters on an undefined variable
…rd guidance why: A fixed-worker loop cannot find saturation. As latency rises the workers slow with it, offered load falls, and the graph bends instead of breaking, so the steady workload could say how fast a transport is but never where it stops keeping up. Separately, five boards with no entry point is how dashboard sprawl starts: without directed browsing, finding the right one is guesswork and the fix people reach for is duplicating it. what: - Add scripts/lgtm/load_tmux.py and just otel-load, driving the engines under rampa's ramping-arrival-rate; the same control-mode engine measures p99 around 2 ms steady and around 16 ms at the top of the ramp - Keep telemetry on this project's sink rather than a rampa output backend: rampa's own backend would export under its service name and metric vocabulary, so a load-shaped run would arrive as a second account of the same work rather than a comparable one - Add a Home board naming what each board answers, and link Overview's panels down to the board that explains them - Reorganize Overview as rate, errors, and duration, since an engine is a service and RED is the frame its caller thinks in - Drop stacking and relax refresh to a minute: the data arrives in bursts from short runs, so a thirty-second poll buys nothing - Teach the dashboard tests that a text panel is documentation and is not expected to query anything
why: The README explained at length that a host process already bound to a port shadows the container's, so a query succeeds against the wrong backend and returns plausible data -- then shipped no way to detect it. Explaining a hazard without supplying the check is the worst of both: the reader knows to worry and has nothing to act on. what: - Add scripts/lgtm/verify.sh, comparing each service's build info as seen from inside the container against the same URL from the host, since liveness cannot tell the two apart but identity can - Run it from up.sh before reporting success, so a shadowed port fails the start rather than surfacing later as a confusing query result - Expose it as just otel-ports and document it - Cover loki, which the ad-hoc checks had been skipping - Drop the LANE alias in the dashboard generator; SCOPE was the only name it needed, and the regenerated JSON is byte-identical - Show just otel-load as a runnable command rather than only naming it in prose
why: A closed port is not always refused promptly -- under WSL and inside containers the packets are dropped and the socket waits out its full timeout. With one request per panel target, forgetting `just otel-up` produced a twenty-minute silence and then a report blaming empty panels, which points at the dashboards rather than at the stack. what: - Probe each backend once before checking any panel, and exit naming the fix; the same mistake now costs eighteen seconds and says what to do - Say so in the README, so the guard is discoverable before it fires
why: `scripts` is not in pytest's testpaths, so the doctests under scripts/lgtm were never executed. The examples there carry the load-bearing decision about which facts a metric may keep and which belong on a profile, so they were comments that looked like tests. what: - Run the identity and telemetry doctests from the test suite, skipping the latter when the otel dependency group is absent - Put scripts/lgtm on sys.path from telemetry.py itself, so importing it does not depend on the caller having arranged that first
why: The README's commands and board names are only correct until someone renames one, and the failure is silent -- the prose still reads well and the command simply does not work. Both sides live in this repo, so the check is cheap and there is no reason to rely on noticing. what: - Add tests asserting every `just` command the README demonstrates is a real recipe, and every generated board is named somewhere in it - Point the instrumentation page at the stack that consumes what it describes; it explained how to export to OTLP without mentioning that an exporter, a Grafana stack, and dashboards already ship here - Drop a fragile test count from that page, which was already stale - Remove a dead assignment in scope() and the unused parameter on the rampa teardown, which is called with none
why: The stack advertised that a log line links to the trace it came from, and the datasource was wired for it, but no log record carried a trace id. A record only picks up trace context while a span is current, and the lane summaries were logged between lanes, with no span open. The claim was false and nothing checked it. what: - Log each lane's result from inside a short span, so the record carries a trace id and the Loki-to-Tempo jump works; verified by taking a trace id from a Loki line and resolving it to its span in Tempo - Put the lane's totals on that span too, so the trace answers the same question the log line does
why: The resolver's edge cases carried the load and none were tested. A detached checkout answers the literal string HEAD, so used directly every detached run on every branch would share one dimension value and could not be told apart; CI needs its explicit ref to win over the checkout's; and a directory outside git has to yield nothing rather than raise. All three were verified by hand and nothing kept them true. what: - Test the ref override, the detached-HEAD fallback to tag then short revision, and the empty result outside a repository, each against a real git repository rather than a fixture that happens to exist - Correct the latency bucket comment, which claimed control mode costs tens of microseconds; the measured median is a couple of hundred. The buckets were already right and still resolve every lane's median into its own band
why: A single unknown lane name saturated the machine. Building the scenario's state costs a tmux server and three exporter threads, and it was built before the step that failed, so every retry left another set behind: 3,539 tmux servers, 10,619 threads, a load average near 2,000, and a test suite that then failed 24 tests for reasons that had nothing to do with the code. rampa is not at fault. Isolating a failing iteration and running the next one is what a load tool should do, and a minimal scenario that raises on every iteration holds a flat three threads. Reproducing it took twenty lines: build three threads per iteration, then raise, and a six-second run reaches a thousand threads. The defect is a scenario that rebuilds expensive state on a path that cannot succeed. what: - Latch a setup failure and re-raise it, so an impossible setup is attempted once however many iterations follow - Validate the lane at import, before a tmux server exists, so a typo costs three seconds and names the valid lanes - Test both: that fifty iterations produce one setup attempt, and that the lane check precedes anything that creates
why: Both Grafana and Tempo expose MCP servers, and neither was usable here. Tempo's was off, and the config the image hands out assumes Grafana's default port, which this stack deliberately moves. Following the shipped copy connects to nothing, and the failure is quiet in the worst way: the agent authenticates against nothing, finds no data, and reports an empty stack rather than a misconfigured one. what: - Enable Tempo's MCP server, which offers traceql-search, get-trace, and attribute discovery, so an agent can find the engine attributes without being told the schema; verified by driving the handshake and searching for this repo's own spans - Add just otel-mcp, printing a client config with this stack's real ports and the running container's token, which is never written to the repository - Test that the config and the stack agree on the Grafana port, since drift there fails silently
why: Only CPU sampling was ever collected, so "where did the time go" had an answer and "what did it allocate" did not. Pyroscope lists the other profile types it knows, which made the gap look like a wiring fault rather than a choice. what: - Add --memory-profile to the smoke workload, collecting alloc_space, alloc_objects, and inuse_space; left off by default because allocation profiling costs more than sampling - Record that these are per run rather than per transport: the lane tag scopes the CPU sampler and the allocation profiler does not consult it, so claiming otherwise would be wrong - Record that the goroutine, mutex, and block types Pyroscope advertises stay empty for a Python process, so an empty panel there is expected
why: The check reported that a published port reaches something other than this container, which is the hard half of the problem, and then stopped. The reader is left knowing they have a collision and not that the port is overridable, so the obvious next move is to start deleting containers. what: - Print the environment variable that republishes the affected service, and only for the two this stack actually moves; naming one for a service that keeps its upstream default would send the reader somewhere useless - Have up.sh show the same remedy when it refuses to report success
why: Notifications arrive out of band, so the instrumentation seam never sees them -- a sink wraps run(), and nothing routes a %output through run(). That left the streaming half of control mode unmeasured, including the engine's own count of notifications dropped when a subscriber falls behind, which is the number that says the stream is unhealthy. what: - Subscribe to notifications while commands keep flowing, recording those received and those dropped; 240 arrive in a three second lane with none dropped, and the commands are unaffected - Establish the subscription before generating output and pace the sends, because a tight command loop starves the consumer and it then waits for output that has already gone by - Give the lane its own shell window: the workload's panes run sleep, which ignores keystrokes and emits nothing - Read the counters with max_over_time rather than increase, since a value written once per run is flat and increase over it is zero
…inst why: The telemetry work had only ever run against one tmux, and control mode is the part most likely to drift between releases -- its notification set and client flags have both changed over the versions libtmux supports. A build that emitted nothing would leave the streaming panels blank with no clue why. Exercised on 3.2a, 3.3a, 3.4, 3.5, 3.6, 3.7, 3.7a and 3.7b: identical command and inlining counts on both transports, notifications delivered on every one, none dropped. what: - State the range in the README, since "works with tmux" is not a claim a reader can act on
why: A cold start failed with grafana UNREACHABLE, and the same check passed seconds later. The container reports healthy before every service inside it has finished binding, so verifying immediately catches Grafana mid-startup and refuses a stack that is fine. Shadowing is deliberately not retried: two different services answering the same port is a settled fact, not a timing question, and retrying it would only delay a report that will not change. what: - Retry a port that is not answering yet, up to LIBTMUX_LGTM_WAIT seconds; two consecutive cold starts now pass where they previously failed, and a healthy stack still verifies in under two seconds - Record what has been verified in scripts/lgtm/VERIFICATION.md: transport against signal, tmux 3.2a through 3.7b, load shapes, profile types, identity resolution, failure paths, and what is deliberately not covered - Recompute the panel total from the boards in a test, so the record cannot claim a number the dashboards do not define
why: Cleanup lived only in rampa's teardown hook. SIGINT reaches it, but SIGKILL runs no user code, so a killed run stranded its tmux server, that server's pane, and its scratch root, each holding a pty, with nothing left to reclaim them. Repeated interrupted runs walk toward pty exhaustion. Separately, the pane's holding command is a run-duration ceiling rather than a cleanup mechanism: when it exits the window closes, the last window closing ends the session, and the server goes with it. destroy-unattached off only survives detach. At sleep 600 any run past ten minutes lost its server mid-flight. what: - Reap roots whose owner is proven absent, at the start of the next run -- the one path that survives an exit running no user code - Stamp each root with its owner's pid and start time, so a reused pid cannot be mistaken for a live run - Leave a root alone while its owner runs, and leave a pre-owner-file root alone while a tmux still answers on its socket - Raise the holding command above any plausible --duration - Cover both directions, including the reused-pid case
why: The shipped scenarios run 10 and 16 seconds, so nothing in the recipes approaches a limit and a reader has no reason to suspect one. A long run used to climb in memory until it ended in swap rather than in a clean error, which presents as a CPU spike that never recovers. Bisecting it showed removing the OTel sink left the growth unchanged while removing rampa's per-iteration recording cut it by 70%: the cause was rampa's runner buffering every sample for outputs that were never configured, and it is fixed upstream. what: - State that duration is safe, and name the version the growth belongs to, so a climbing run has an obvious first thing to check - Explain why the pane's holding command outlasts any plausible duration
why: CI runs `mypy .`, which covers `scripts/`; the configured file list that plain `mypy` uses does not, so these went unchecked on developer machines and failed the gate the first time this branch had a pull request. Two of them were real: telemetry and the load scenario still reached command_count through control_mode, which re-exports it rather than defining it, so the import resolved at runtime but not under strict checking. The rest were missing type arguments and a literal inferred as list[object]. what: - Take command_count from libtmux.engines.base, its definition site - Give the acceptance script's dict annotations their arguments, and name the decoded JSON body rather than returning Any - Annotate the error thresholds so the generated panel keeps its shape - Narrow the git-dir probe explicitly, since bool() does not narrow Optional - Pass rampa a timedelta rather than a duration string: its field coerces the string through a validator, so the value is identical and the annotation stops lying - Name the async lane factory separately from the sync one, and mark the optional profiler import untyped
why: CI runs `mypy .`, which reaches `scripts/`, but installs only the default and dev groups -- the `otel` group is deliberately excluded so the ordinary gates stay lean. Every OpenTelemetry, pyroscope, and rampa import was therefore unresolvable in CI while resolving fine on a machine that had the group installed. what: - Allow those three imports to be missing, so the scripts type-check in both environments rather than only the one the developer happens to have - Relax subclassing-Any and untyped-decorator for the two modules that build on them, scoped per module: an inline ignore would itself be unused wherever the group is installed
why: Both tests guard on `opentelemetry`, but that namespace package arrives transitively, so the guard passes wherever libtmux is installed. What the modules under test actually import is `opentelemetry.sdk`, the OTLP exporters, and rampa, all of which ship only in the optional `otel` group -- which CI does not install. The guard therefore never fired and the tests failed on import instead of skipping, which was the stated intent. what: - Guard on `opentelemetry.sdk` rather than the namespace package - Guard the load scenario on rampa as well, since it imports both Verified in both directions: without the group the two skip, and with it installed they still run and pass rather than being silently disabled.
tony
force-pushed
the
lgtm-observability
branch
from
August 23, 2026 02:21
a0d0564 to
3db13bf
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
just otel-*, so the engine instrumentation seam has somewhere to send telemetry and a tmux workload's real cost becomes visible.run()-shaped seam cannot see, including the engine's own count of notifications dropped when a subscriber falls behind.Changes by area
The stack
scripts/lgtm/up.sh,verify.sh: pin the image, bind-mount provisioning, and refuse to report success when a published port answers from something other than this container. Liveness cannot tell a shadowed port from a healthy one; identity can.scripts/lgtm/telemetry.py: an OTel sink emitting spans and metrics together, so the duration histogram records while its span is current and carries an exemplar — that is the metric-to-trace pivot.scripts/lgtm/identity.py: resolves repository, ref, revision, and worktree from onegit rev-parseat startup, using OpenTelemetryvcs.*conventions, overridable by environment for detached CI checkouts.scripts/lgtm/generate_dashboards.pyanddashboards/*.json: Home, Overview, Commands, Transports, and Compare boards, with Overview organised as rate/errors/duration because an engine is a service and RED is the frame its caller thinks in.scripts/lgtm/mcp-config.sh: prints a client config carrying this stack's real ports and the running container's token, so an agent can query traces over MCP.Workloads
scripts/otel_smoke.py: drives all four transports, with grouped commands so the inlining panels are not flat zero and rejected commands so the failure panels have real data.--memory-profileadds allocation profiles, off by default because they cost more than sampling.scripts/lgtm/load_tmux.py: the rampa scenario, reaping roots whose owner is proven absent at the start of the next run — the one path that survives an exit running no user code.scripts/otel_acceptance.py: reads the committed dashboards, runs each panel's own query the way Grafana runs it, and probes the backends first so a forgottenjust otel-upcosts seconds rather than a socket timeout per panel.Benchmark-harness fixes
Teardown no longer reads tmux's "no server" exit as a failure, the stress ladder scales its watchdog with the rung it is building, a stalled test worker can no longer outlive the test that parked it, and two tests stop depending on machine load or the caller's environment size.
Design decisions
Load telemetry goes through this project's sink, not a rampa output backend. rampa's own backend would export under its service name and metric vocabulary, so a load-shaped run would arrive as a second account of the same work rather than a comparable one.
Identity is split by signal. Metrics carry only branch, run id, and spike — the dimensions worth grouping by. Revision and worktree ride on traces and profiles. A revision is never a comparison axis: each run has exactly one, so grouping by it is grouping by run, and copying every fact onto every signal mints a fresh set of series per commit.
Panels query the viewer's selected window, not a counter at an instant. The workload is short-lived, so Prometheus marks its series stale within minutes and an instant query then returns nothing beside a timeseries still drawing the same run. A test rejects any Prometheus target with no range selector, which catches this offline — a live check only sees it once the series has gone stale, which is precisely when nobody is watching.
Port collisions fail the start. Grafana and Prometheus are published on non-default ports because a host process already bound to their defaults still lets Docker publish, then answers first, so queries reach the wrong server and return plausible data.
Verification
Confirm every
justrecipe the README demonstrates actually exists:$ uv run pytest tests/test_lgtm_dashboards.py -k readmeConfirm no Prometheus panel target reads a counter at an instant:
Test plan
uv run ruff check .— lint cleanuv run ruff format .— formatting clean, tree unchangeduv run mypy— types cleanuv run pytest --reruns 0— suite clean apart from the load-sensitive stall family noted belowjust build-docs— docs build cleantest_lgtm_load.py— a stranded root is reclaimed, a live owner's root is left alone, and a reused pid is not mistaken for a live runtests/test_bench_orchestration_script.pycarries a stall/cancellation family that is load-sensitive above roughly load 20; those tests pass in isolation and their failing subset varies between runs on identical trees.