Repository navigation
Explain native inspection failure when the desktop or target window becomes unavailable #64
Description
Activity
The hosted connection and history checks are now unblocked using a second real, signed AppKit form: a pi/Anthropic run clicked/typed/pressed a key/submitted it, changed its persisted value, and replayed all results/images after Workers restart. PR #63 is merged.
The original Ace-dev availability problem is still open. Restarting only that disposable app produced a brief visible window with five chrome-only AX elements, followed by another offscreen/no-matching-Accessibility-window inventory. The second app remained available long enough to complete the hosted workflow. This narrows the validation failure but does not establish the cause or prove recovery for Ace-dev. No activation or alternate-window fallback was added.
Further #65 validation reproduced the same availability failure in a standalone, signed AppKit form, so this is not established as an Ace WKWebView-specific problem. Six real checks passed, including contextual selection and Unicode/multiline insertion. A subsequent focus click completed as confirmed-no-change, but its fresh observation and the next explicit inspection both failed with the incomplete Accessibility read.
Afterward, inventory still found the same process/window with
is_minimized: false,is_on_screen: false,pixels_only, andno_matching_accessibility_window. One bounded read-only retry failed too; no keyboard action was retried. Native keyboard validation is paused at that availability failure while checking the desktop state. The cause remains unproven.The draft keyboard PR is #67. A fresh disposable AppKit window with
canJoinAllSpacesandfullScreenAuxiliaryalso failed validation, this time because the captured target no longer matched the resolved generation/window/owner/bounds. The run stopped before shortcut input; no identity check was relaxed and no input was retried. That setup changes only test-window presentation, not Ace's production behavior, and does not establish the cause or recovery.For the earlier unavailable form, fresh inventory showed the app was not hidden. The console was unlocked and display sleep was prevented when checked. Those observations do not explain the offscreen/Accessibility transition. Remaining runtime checks for #67 are explicitly pending in #8 and #65.
Draft implementation: #71, stacked on #67.
Added explicit read-only
desktop_inspectpixels mode with exact process/window-generation and capture-digest verification. It returns no reusable action snapshot, element IDs, or fabricated Accessibility metadata. Native inspection failures preserve their original error and add a bounded later app/window inventory; those later readings are context, not proof of the failure's cause. Completed input retains its outcome when the following observation fails.Real signed-client validation against the current disposable AppKit fixture passed normal Accessibility inspection, pixel capture with no action authority, exact generation checks, invalid empty-mode refusal, and rejection of a window absent from the live inventory with its original error and complete inventory diagnostics preserved. Type checks, formatting/lint, signed desktop build, signature verification, and Ace diff review also passed.
This does not yet establish why living windows intermittently become offscreen or lose their Accessibility tree. The native passive capture path already retries a receipt mismatch once. Keep this issue open; the draft still needs the completed-input/failed-reinspection case and broader model flow. Explicit app/window recovery and canvas mutation support remain separate #8 work.
New dogfooding evidence from the development Ace app: a native inspection returned a screenshot of the full interface but only five Accessibility nodes (the window, group, and three window-chrome buttons). This was a sparse successful observation, distinct from the earlier incomplete-read errors.
The exact target remained PID
7944, window2533, process generation1791170764047349. An independent, read-only CUAgetApp(path)inspection then exposed the full WKWebView tree without sending input. The next native inspection of that same target returned 68 AX nodes plus nine OCR text labels;Messagewas a realAXTextArea, not an OCR-derived editable control. There was no rebuild or process/window restart between these observations.Local evidence:
- Before:
/tmp/ace-actions-validation-xaLuQq/keyboard-webview-Xlf1Xh/observation.jsonandscreenshot.jpg. - After:
/tmp/ace-actions-validation-xaLuQq/keyboard-webview-TnvO1M/observation.json.
This sequence does not prove that CUA was necessary or caused the recovery. WebKit's lazy Accessibility enablement or remote Accessibility token initialization is a candidate explanation for this subtree symptom. Compare repeated native-only reads on a fresh process before concluding causation; this also does not explain all of the earlier standalone AppKit failures. PR #71 provides diagnostics and explicit read-only pixel inspection, not a fix for this missing subtree.
Separately, a native AX click on
Messagetruthfully returneddispatched_unverifiedbut did not focus the field. CUA focus is explicit setup for the ongoing keyboard-only checks; it does not validate Ace's complete pointer-to-keyboard flow. That pointer work remains pending in #70 and should stay distinct from the subtree-availability investigation. Keep #64 open.- Before:
Controlled native-only follow-up for the sparse WKWebView subtree: after a fresh development GUI launch, the validation used only native application/window inventory and native inspection. The first inspection returned five AX nodes; the second native-only inspection returned 74 AX nodes plus nine OCR labels, including a real
Messagetext field. No CUA app read or other external AX inspection ran between the fresh launch and these two observations.Both observations identify PID
20701, window2679, process generation1791197308099437. Local evidence:- First:
/tmp/ace-actions-validation-xaLuQq/keyboard-webview-a1GJnR/observation.json. - Second:
/tmp/ace-actions-validation-xaLuQq/keyboard-webview-tNcEkQ/observation.json.
This reproduced recovery shows that an external CUA read was not necessary in this case. It does not establish why the first subtree was sparse, prove a general recovery strategy, or fix the earlier incomplete/offscreen AppKit failures. The build was the reviewed
d04c8c4keyboard parent plus the uncommitted #76 candidate, so this is runtime evidence rather than a clean release validation. No input was retried as part of these two observations. PR #71 remains diagnostics/read-only pixel support; keep #64 open.- First:
PR #71 is now a focused change on the merged application-state and selection/shortcut base, independent of unfinished literal insertion. It adds explicit pixel inspection without action authority and bounded later inventory diagnostics while preserving the original inspection error and any completed action outcome. Its diagnostic projection also retains
is_active_knownfrom #79.The current candidate passes type checks, formatting/lint, its own signed desktop build, deep/strict signature verification, and diff checks. A fresh read-only native helper is ready; current-candidate runtime follow-ups are still pending. This does not establish or fix the cause of sparse/incomplete Accessibility observations, so this issue remains open.
Merged #71 on
feat/github-cacheas61c5aa9.The independent signed build passed real Accessibility/pixel reads, exact target checks, invalid-mode rejection, and absent-window original-error diagnostics. A real AppKit button closed its target exactly once; the completed action and native receipt survived the failed follow-up inspection. Later inventory retained the closed window as offscreen/pixels-only, so the verifier was corrected against saved evidence without repeating input.
Pixel reads intentionally provide no action snapshot. This adds useful inspection feedback; it does not fix sparse webview Accessibility trees or complete the broader visual workflow. No new canary has been published.
Landed focused locked-desktop feedback in PR #87, merge
c89f87ba0450a6b05455da914f2909739ebeda3donfeat/github-cache.A known locked session now produces typed
target_unavailable, no-dispatch evidence, and guidance to unlock then refresh inventory before choosing another action. This fixes the observed focus failure that previously looked like an unsupported Accessibility operation. Request-shape checks still precede the lock preflight; false/unknown lock state follows the existing path.Static checks, independent review, isolated signed-client build/signature verification, and real locked-session activate/focus/restore checks passed. All three returned
dispatch_state: noneand kept the disposable receiver unchanged. The requests deliberately used stale process generations, preventing input even if the desktop unlocked between observation and execution. This verifies the real explicit-locked branch, not unlocked/unknown or mid-operation lock behavior. No desktop bundle was replaced or launched for this check.This broader issue remains open for its remaining scope; #8 tracks subsequent focused landings.
Observed while validating #109 in Ace Canary (2026-10-05, about 16:20Z).
desktop_inspectin accessibility mode on Google Chrome (pid 2175) window 474 failed twice in a row:DESKTOP_ERROR: The desktop observation provider returned inconsistent response evidence: window-context window title..Target availability, read after the failure, reported the window as present, on screen, key, not minimized, and
combined_eligible, with its app not active. Just before,desktop_windowslisted the window with a truncated active-tab title,Add JSDoc comments to all expor… #5 · iamnbutler/tasks-fixture. The tab had changed moments earlier: the window's title had moved through several tasks-fixture pages in the preceding minute. A title that changes, or is truncated differently between the window inventory and the Accessibility context, may explain the mismatch. That is unconfirmed. No input was dispatched and no snapshot was returned, so the agent couldn't verify the tab title before Cmd+W, and closing the test tabs needed a manual exception (see the Built in Ace section of #109). Not investigated further.- added a parent issue
on Oct 6, 2026
Status (Oct 6)
Shipped: pixels-mode inspection and target availability reporting (#71), and typed
target_unavailablerefusal for a known locked session with unlock-and-refresh guidance (#87, details). Remaining: a reliable distinction and recovery path for an offscreen/other-desktop target versus an unavailable desktop versus incomplete Accessibility, sparse initial WKWebView Accessibility trees (seen in #73), and the real recovery check below. The Chrome window-title mismatch is tracked separately in #113.Original report
During real desktop-action dogfooding for #8 / #63, native inspection became unavailable after previously passing against the same live Ace-dev process and window. The agent received an incomplete Accessibility-tree error without an actionable explanation of target availability.
Observed on the signed macOS development app:
desktop_inspectcalls failed withDESKTOP_ERROR: Warning: AX tree incomplete at incomplete accessibility read.is_minimized: false, butis_on_screen: false,observation_capability: pixels_only, andobservation_reason: no_matching_accessibility_window. Inventory itself reported complete.Expected: give the agent a bounded, useful explanation when native window/Accessibility availability changes, retain the exact target, and require a fresh observation when the target is available again. Do not silently switch to another window, steal focus, or replay uncertain input as recovery.
Investigate which native state reliably distinguishes an offscreen/other-desktop target, a locked/unavailable desktop, and incomplete Accessibility. Add a real recovery check under #8's visual targeting and longer-workflow slice. Keep the unavailable-desktop limitation explicit until that check passes.