test(replay): verify Settings queries and bound scrolling

[thymikee]

Summary

Settings query fixtures prove native editor focus and exact values. iOS discovery requires General visible and Apps absent before scrolling reveals Apps; General must then be absent before upward scrolling reveals it.

Selectors target observed buttons: General also labels structural nodes; Apps labels a button beneath an unlabeled cell. The previous cell inference failed visibly and is removed.

The typing smoke waits for a visible software keyboard after coordinate focus before asserting AX-independent synthesis. This is a deliberate test precondition; no production routing or budget changes.

Four test files (+21/-11). Related to #3077.

Validation

Head 36aae30ba on main 295ccce70: the four commits rebased unchanged; every touched file is blob-identical to the previous head cbae29ea9 and to evidence head 746edc7.

pnpm check:affected --base origin/main --run: format, lint, typecheck, fallow and vitest-related pass. Replay suites are GitHub-authoritative; new-head CI pending.

Live review evidence, recorded for head 746edc7 on iPhone 17 Pro / iOS 26.2: ordinary iOS03 records one down and one up gesture plus all five assertions; iOS04 and Android04 focus/value assertions pass first attempt with zero healing. The iOS absent pre-checks assume that viewport size. The later iOS full-fixture app-switcher assertion fails separately and is not claimed green.

Fresh review: one medium (keyboard wait masks a product race) and three low/nit findings declined to keep this test-only scope and the evidence blobs; simulator model now named.

🤖 Generated with Claude Code

[github-actions]

Size Report

Metric	Base	Current	Diff
Installed (including dependencies)	5.20 MB	5.20 MB	-5.2 kB
Package (unpacked)	5.20 MB	5.20 MB	-5.2 kB
Package (download)	1.56 MB	1.56 MB	-1.6 kB

Startup median (7 runs, lower is better):

Scenario	Base	Current	Diff
CLI --version	26.4 ms	26.6 ms	+0.3 ms
CLI --help	76.1 ms	77.7 ms	+1.6 ms

[cubic-dev-ai]

All reported issues were addressed across 3 files

<sub>Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic</sub>

[thymikee]

The scroll fixture in 03-scroll-discovery.ad can pass without any scroll, so this is not ready to merge at 30942ef. CI is green (19 checks, 0 not passing), but the fixtures are manual-selected test material and not a CI route, so green CI does not show that the replays run. No conflicts.

Here is the problem. In 03-scroll-discovery.ad line 5, scroll down --until 'label=Accessibility' returns success with zero gestures when the first capture already shows the target. The PR run shows exactly that: 0 gestures. After relaunch, General is also on screen, so scroll up --until 'label=General' no-ops too. The two find ... exists lines then pass whether or not anything scrolled. The fixture never proves that the scroll moves or that the bound stops it, and the old fixed-count scroll at least exercised movement. Please pick a target that is off-screen after relaunch, such as a lower Settings row. Assert before the scroll that the target is not visible, so the scroll is proven necessary. Then assert arrival, and scroll back up to a row that has left the viewport. If you prefer, rename the fixture and state that it only proves no overshoot. To validate, please run it on the iOS simulator and show a run where the pre-assertion passes and the scroll takes at least one gesture before the target is found.

The open inline thread on 03-scroll-discovery.ad still stands: zero-gesture scroll.

Not blocking: the trailing find text "General" exists in 04-text-input-keyboard.ad can match the search field's own typed value, so it no longer shows that a result appeared, and you could drop it or use a result-row selector that excludes the editable field. I did not rerun the fixtures on a device or simulator, so whether Accessibility is in the first viewport on other simulator sizes is unknown, and the Android Search Settings selector is only confirmed by your run.

[thymikee]

The scroll fixture at 78213e2 now fixes the problem from the earlier review (#3560 (comment)): 03-scroll-discovery.ad asserts is absent 'label=Apps' before scroll down --until 'label=Apps', and General absent before the scroll up, so a no-op scroll now fails. What remains is the live proof. The PR body still says the run on this head is pending. I could not tell from the code whether Apps starts below the first viewport after relaunch, or whether General leaves the viewport after one downward scroll. If Apps is already visible, the first is absent fails. If General stays visible, the second one fails. These fixtures run manually and not in CI, so nothing else shows these assertions hold on a device. Please run agent-device replay on test/integration/replays/ios/simulator/03-scroll-discovery.ad on an iOS simulator at 78213e2 and paste the output (https://github.com/callstack/agent-device/blob/78213e2/test/integration/replays/ios/simulator/03-scroll-discovery.ad#L5). It must show the three is assertions passing before and after scrolling, one scroll down and one scroll up each reporting passes >= 1, and the final is visible 'label=General' passing. Please note the simulator device and OS version. A run that only reaches the old zero-gesture route does not count. Please also paste the runs of the two 04 text-input fixtures (Android and iOS) if they are not shown yet. The cubic-dev-ai P2 thread on the no-op scroll no longer applies, because the pre-scroll assertions now cover it, so you can resolve it: #3560 (comment). I did not run a simulator or device replay, and I did not rerun the CI gates. The Coverage failure in scripts/__tests__/eager-closure-budgets.test.ts:731, on the route packages/platform-android/src/mechanics.ts to app-lifecycle.ts to intent-arguments.ts, looks unrelated: this PR changes only three .ad fixtures, which no eager-closure budget reads. Before merge, the simulator run above must be posted.

[thymikee]

Thanks for the update. The cubic-dev-ai thread about a no-op scroll is fixed at 964e0a0: 03-scroll-discovery.ad now asserts is absent 'label=Apps' before the scroll down and is absent 'label=General' before the scroll up, so a scroll that does nothing now fails the fixture. I found no code problem in the changes, but the live evidence is still missing.

Those absent checks depend on how Settings lays out after relaunch and on one --until scroll. If Apps is already on screen at launch, or General stays visible after the scroll, the fixture fails on a real simulator. Nothing in this PR runs the fixture: it is manual-only (test:replay:ios), and the smoke lane runs replays/ios/fixture, not this directory. The same gap covers the focus and exact-value checks in both 04-text-input fixtures (editable=true focused=true, value="General" and value="Display"). Please run agent-device replay test/integration/replays/ios/simulator/03-scroll-discovery.ad on an iOS simulator at 964e0a0 and paste the output. It should show the pre-scroll is absent Apps passing, scroll down with passes >= 1, is visible Apps, is absent General, scroll up with passes >= 1, and the final is visible General. Please also paste the iOS and Android 04-text-input-keyboard runs, with the focus and value waits and is focused passing, and note the device and OS version. A run that takes a zero-gesture route does not count.

The cubic-dev-ai thread on the no-op scroll (#3560 (comment)) is fixed in the code, so you can resolve it. Its remaining on-device assumption is the evidence item above.

Smoke (iOS) failed in live-snapshot-depth-frontier.ts:78. The log shows a 2-node tree and an XCTest fallback after the simulator AX snapshot became unavailable. This PR changes no runtime code and no fixture that lane runs, so the failure looks unrelated. I did not check whether it is also red on main, so please rerun that job to confirm. Lint & Format, Compatibility & Provenance, Integration Tests and the second Smoke Tests are still queued. Before merge, the live replay output at 964e0a0 must be posted and the iOS Smoke job must pass on rerun.

[thymikee]

The earlier concern about 03-scroll-discovery.ad is fixed at 964e0a0. The fixture now asserts is absent Apps before the scroll down and is absent General before the scroll up, so the cubic-dev-ai thread at #3560 (comment) is addressed and can be resolved. One problem remains: no replay output for this head is posted, so nothing shows the new assertions hold on a device.

The pass or fail of this fixture depends on the Settings layout after relaunch. is absent 'label=Apps' fails if Apps is on screen at launch. is absent 'label=General' fails if General stays visible after one --until scroll. The same applies to editable=true focused=true and the exact value= waits in both 04-text-input fixtures, and to the Android text="Search Settings" selector. These fixtures run only by hand (test:replay:ios), so CI will not catch a bad assumption. They could fail on first use, or could still reach the zero-gesture route that 30942ef reached.

Please run agent-device replay test/integration/replays/ios/simulator/03-scroll-discovery.ad on an iOS simulator at 964e0a0 and paste the output, with the device and OS version. It should show the pre-scroll is absent Apps passing, the scroll down with passes >= 1, is visible Apps, is absent General, the scroll up with passes >= 1, and the final is visible General. Please also paste the iOS and Android 04-text-input-keyboard runs, showing the focus and value waits and is focused passing.

On CI, the Smoke Tests job 115251404043 failed at live-snapshot-depth-frontier.ts:78 with a 2-node tree on the XCTest fallback. This PR changes only three manual-only .ad fixtures that the smoke lane does not run, so the failure looks unrelated. I did not check whether it is also red on main. Please rerun that job. The other Smoke and Integration jobs were still running or queued when I looked. There are no conflicts.

I did not run any simulator or emulator replay, and I could not check the Settings layout after relaunch or the Android Search Settings selector. Before merge, the live replay output above must be posted and the failed Smoke job rerun.
This review covers 964e0a0. The branch has since moved to 169cee2, which rebases onto the updated base and does not change the three fixtures, so the request above still applies to the new head.

[thymikee]

Thanks for the update. The latest change (1ea2ab0) switches the four selectors to role=cell label=..., which is a good step. The main gap from the earlier review is still open, because no replay output has been posted for any head of this PR.

The fixture at 03-scroll-discovery.ad:5 depends on two device facts that no unit test or CI lane checks. First, is absent Apps on line 6 passes only if the Apps cell has zero nodes in the Settings snapshot at launch. A cell that is in the tree but off screen fails it. Second, is absent General on line 10 needs General to leave the snapshot after one --until scroll. This is the only replay fixture that uses is absent, so there is no precedent to lean on. Please run agent-device replay test/integration/replays/ios/simulator/03-scroll-discovery.ad on an iOS simulator at 1ea2ab0 and paste the output with the device and OS version. It should show these steps passing in order: is visible General and is absent Apps, scroll down with passes >= 1, is visible Apps, is absent General, scroll up with passes >= 1, then a final is visible General. A run that reaches the zero-gesture route does not count.

The two 04-text-input-keyboard fixtures have not been shown running either. On Android, text="Search Settings" at 04-text-input-keyboard.ad:7 and the role=text-field focused=true waits are unproven. On iOS, editable=true focused=true and value="General" are in the same state. Please paste both replay outputs in the same post, showing the focus waits, the value waits and is focused passing.

The zero-gesture scroll concern in this thread is now fixed in the code, since both is absent checks run before each scroll. It can be resolved once the live run above confirms they hold on a device.

I did not run a simulator, emulator or replay. I could also not read the earlier live run that hit AMBIGUOUS_MATCH on General, so the claim that role=cell disambiguates it is still unverified. CI is pending: the only checks not passing are two queued Smoke Tests jobs. The smoke lane exercises different routes than these three fixtures. The earlier smoke failure was on a runtime path this PR does not touch. Before merge, the live replay output above needs to be posted and match.

[thymikee]

The earlier findings on 1ea2ab0 are mostly addressed at 746edc7, and one gap remains: there is still no live proof for the changed iOS fixture. The absent checks before each scroll in 03-scroll-discovery.ad now make a zero-gesture scroll fail, so that earlier thread is resolved: #3560 (comment). CI is pending, with only two Smoke Tests jobs still queued or in progress.

The role=cell to role=button swap in 03-scroll-discovery.ad is still unproven on a device. The fixture needs is absent 'role=button label=Apps' to find nothing at launch, and is absent 'role=button label=General' to find nothing after one --until scroll. Each scroll must also take at least one gesture. No CI route checks this: ios.yml replays only 01-settings.ad, and test:replay:ios is manual. The only linked run, 38393783564, is at a different source commit and stopped before scroll up. No replay output has been posted for any head, so the fixture could fail on first use. Please run agent-device replay test/integration/replays/ios/simulator/03-scroll-discovery.ad at 746edc7 on an iOS simulator and paste the output with the device and OS version. It should show, in order: is visible General and is absent Apps passing, scroll down with passes >= 1, is visible Apps, is absent General, scroll up with passes >= 1, and a final is visible General. A zero-gesture run does not count. Please paste the iOS and Android 04-text-input-keyboard replays in the same post, showing the focus waits, value waits and is focused passing.

Not blocking: once both Smoke Tests jobs finish, the log should show the synthesis assertion (textEntryRoute === synthesized-first-responder) passing after the new keyboard wait in live-runner.ts, and you can take or leave that. This review was read-only, so I ran no simulator, emulator or replay.

[thymikee]

Posting the retained device evidence requested above for published head 746edc7. No new run or assertion waiver was needed.

iOS job: iPhone 17 Pro, iOS 26.2, runtime 4dd17b4. Its three replay files and live-runner.ts have identical Git blobs to this PR head. The six ordinary replays passed first attempt: 48 actions, zero healing.

For ordinary iOS03, replay.ad contains the five assertions below, and their replay_action_stop records all report ok:true. The actual ordinary session events independently record both gestures (literal lines, not counts borrowed from the separate probe):

{"version":1,"ts":"2026-10-13T14:11:11.672Z","session":"cwd:a87d3c22dc05c4d7:default:test:9e53321dd1ad888d:5-03-scroll-discovery:attempt-1","kind":"action.recorded","requestId":"9e53321dd1ad888d:test:5:03-scroll-discovery.ad:attempt:1","command":"scroll","summary":"Scrolled down by 568px","details":{"command":"scroll","positionals":["<arg>"],"flags":{"platform":"ios"},"direction":"down","passes":1,"amount":0.65,"pixels":568,"durationMs":400,"x1":201,"y1":721,"x2":201,"y2":153}}
{"version":1,"ts":"2026-10-13T14:11:15.139Z","session":"cwd:a87d3c22dc05c4d7:default:test:9e53321dd1ad888d:5-03-scroll-discovery:attempt-1","kind":"action.recorded","requestId":"9e53321dd1ad888d:test:5:03-scroll-discovery.ad:attempt:1","command":"scroll","summary":"Scrolled up by 568px","details":{"command":"scroll","positionals":["<arg>"],"flags":{"platform":"ios"},"direction":"up","passes":1,"amount":0.65,"pixels":568,"durationMs":400,"x1":201,"y1":153,"x2":201,"y2":721}}

Matching ordinary iOS03 predicate dispatch records (L5 General visible; L6 Apps absent; L8 Apps visible; L10 General absent; L12 General visible):

{"type":"replay_action_stop","ts":"2026-10-13T14:11:07.985Z","replayPath":"/Users/runner/work/agent-device/agent-device/test/integration/replays/ios/simulator/03-scroll-discovery.ad","line":5,"step":2,"command":"is","ok":true,"durationMs":2020}
{"type":"replay_action_stop","ts":"2026-10-13T14:11:08.991Z","replayPath":"/Users/runner/work/agent-device/agent-device/test/integration/replays/ios/simulator/03-scroll-discovery.ad","line":6,"step":3,"command":"is","ok":true,"durationMs":1002}
{"type":"replay_action_stop","ts":"2026-10-13T14:11:12.845Z","replayPath":"/Users/runner/work/agent-device/agent-device/test/integration/replays/ios/simulator/03-scroll-discovery.ad","line":8,"step":5,"command":"is","ok":true,"durationMs":1171}
{"type":"replay_action_stop","ts":"2026-10-13T14:11:13.298Z","replayPath":"/Users/runner/work/agent-device/agent-device/test/integration/replays/ios/simulator/03-scroll-discovery.ad","line":10,"step":6,"command":"is","ok":true,"durationMs":452}
{"type":"replay_action_stop","ts":"2026-10-13T14:11:16.270Z","replayPath":"/Users/runner/work/agent-device/agent-device/test/integration/replays/ios/simulator/03-scroll-discovery.ad","line":12,"step":8,"command":"is","ok":true,"durationMs":1128}

The additional public-CLI Settings probe retains the actual predicate response bodies. Below are compact JSON renderings of those retained stdout files, in execution order:

{"success":true,"data":{"predicate":"visible","pass":true,"selector":"role=button label=General"}}
{"success":true,"data":{"predicate":"absent","pass":true,"selector":"role=button label=Apps","matches":0}}
{"success":true,"data":{"predicate":"visible","pass":true,"selector":"role=button label=Apps"}}
{"success":true,"data":{"predicate":"absent","pass":true,"selector":"role=button label=General","matches":0}}
{"success":true,"data":{"predicate":"visible","pass":true,"selector":"role=button label=General"}}

iOS04 focus/value assertions in the retained executed replay.ad:

L6: wait 'editable=true focused=true'
L7: is focused 'editable=true'
L10: wait 'editable=true value="General"'
L11: is focused 'editable=true value="General"'

Matching literal replay-timing.ndjson output:

{"type":"replay_action_stop","ts":"2026-10-13T14:10:03.315Z","replayPath":"/Users/runner/work/agent-device/agent-device/test/integration/replays/ios/simulator/04-text-input-keyboard.ad","line":6,"step":3,"command":"wait","ok":true,"durationMs":8559}
{"type":"replay_action_stop","ts":"2026-10-13T14:10:03.348Z","replayPath":"/Users/runner/work/agent-device/agent-device/test/integration/replays/ios/simulator/04-text-input-keyboard.ad","line":7,"step":4,"command":"is","ok":true,"durationMs":18}
{"type":"replay_action_stop","ts":"2026-10-13T14:10:11.373Z","replayPath":"/Users/runner/work/agent-device/agent-device/test/integration/replays/ios/simulator/04-text-input-keyboard.ad","line":10,"step":6,"command":"wait","ok":true,"durationMs":5293}
{"type":"replay_action_stop","ts":"2026-10-13T14:10:11.619Z","replayPath":"/Users/runner/work/agent-device/agent-device/test/integration/replays/ios/simulator/04-text-input-keyboard.ad","line":11,"step":7,"command":"is","ok":true,"durationMs":235}

Result output (session/path omitted):

attempt: 1/3
status: passed
replayed: 9
healed: 0

Android job: Pixel 7 emulator, Android API 36, x86_64 google_apis_playstore, runtime 5a63e97. Android04 has the identical blob at this PR head. All six ordinary replays passed first attempt (87 actions, zero healing); the helper probe reports android-helper version 0.21.28-dev.

Android04 focus/value assertions in the retained executed replay.ad:

L8: wait 'role=text-field focused=true' 5000
L9: is focused 'role=text-field'
L16: wait 'role=text-field value="Display"' 5000
L17: is focused 'role=text-field value="Display"'

Matching literal replay-timing.ndjson output:

{"type":"replay_action_stop","ts":"2026-10-13T12:37:59.456Z","replayPath":"/home/runner/work/agent-device/agent-device/test/integration/replays/android/emulator/04-text-input-keyboard.ad","line":8,"step":4,"command":"wait","ok":true,"durationMs":983}
{"type":"replay_action_stop","ts":"2026-10-13T12:37:59.461Z","replayPath":"/home/runner/work/agent-device/agent-device/test/integration/replays/android/emulator/04-text-input-keyboard.ad","line":9,"step":5,"command":"is","ok":true,"durationMs":4}
{"type":"replay_action_stop","ts":"2026-10-13T12:38:02.018Z","replayPath":"/home/runner/work/agent-device/agent-device/test/integration/replays/android/emulator/04-text-input-keyboard.ad","line":16,"step":10,"command":"wait","ok":true,"durationMs":666}
{"type":"replay_action_stop","ts":"2026-10-13T12:38:02.021Z","replayPath":"/home/runner/work/agent-device/agent-device/test/integration/replays/android/emulator/04-text-input-keyboard.ad","line":17,"step":11,"command":"is","ok":true,"durationMs":3}

Result output (session/path omitted):

attempt: 1/3
status: passed
replayed: 20
healed: 0

Successful ordinary wire response bodies were not separately retained; the pasted dispatch records and executed scripts establish that both focused/value waits and both is focused assertions completed successfully. is fails its public dispatch when the predicate is false.

The iOS full fixture additionally passed the keyboard-visible precondition, native synthesized first-responder assertion, and .test suffix read-back. It later failed a separate app-switcher automation-press assertion; I am not claiming that whole workflow passed.

Original artifacts: iOS, Android. Their downloaded archives match GitHub SHA-256 digests (f8f5cde4… and dce40239…).

[thymikee]

Thanks for the device run. This PR is ready at 746edc7. The iOS job (https://github.com/callstack/agent-device/actions/runs/38405441715/job/115323744179) ran 03-scroll-discovery.ad and 04-text-input-keyboard.ad with the same file contents as this head. I checked that the blobs match. All five is steps in 03 passed, and the scroll down and the scroll up each took one gesture. That closes the last gap from the earlier review. All checks pass, and there are no conflicts.

[thymikee]

The earlier review of cbae29e was clean, and I found no code problems at 36aae30 either. The scroll discovery script now asserts 'is visible General' and 'is absent Apps' before 'scroll down --until', and 'is absent General' before 'scroll up --until', so the no-op scroll concern is fixed. The earlier iOS evidence still applies because the replay script blobs match the run you cited (job 115323744179 at 746edc7), but I did not re-run it or read its logs. I also did not run the Android 04 fixture changes on a device, and I could not confirm that the live-runner.ts keyboard wait was exercised in a live run.

The cubic-dev-ai thread about the no-op scroll is resolved and outdated, so you can close it: #3560 (comment)

Coverage fails on three unhandled vitest teardown errors that start in src/daemon/tests/request-router-open-claim.test.ts, while all 1758 test files pass. This PR touches no unit-test, platform-apple or daemon code, so I read the failure as unrelated. I did not check whether it also fails on main. Smoke Tests is still running. Please re-run Coverage and let Smoke Tests finish before merge. I know of no conflicts.

[github-actions]

PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-10-14 07:54 UTC