Repository navigation
Focus and paste into WebKit composers with native desktop tools #106
Description
Activity
- added a parent issue
on Oct 5, 2026 - removed a parent issue
on Oct 5, 2026 - added a parent issue
on Oct 5, 2026 First focused candidate: draft #120, verified focus before
AXPressfor writable editable fields. Normal button and coordinate paths retain their behavior. It also moves the native focus write off MainActor while keeping its lane owned, and preserves ambiguous completion so it cannot fall through to a second input.Final source
4ec5802passed types/lint, Swift parsing, independent source review, patch application/reversal and a signed build with strict app/client signatures. Real native inventory found the fresh WebKit fixture, thenlistWindowsrefused withpermissionDenied. That operation requires Accessibility, so this establishes the isolated app's missing Accessibility grant; Screen Recording was not reached or tested. No focus/input was sent, no grants were requested or changed, and the fixture/runner are stopped.The reusable validation checkout is
desktop-app-state/ace2, app identitydev.ace.desktop.dev.4472239c. Reusing that checkout across future native candidates avoids adding another app identity for every PR. The bounded fixture is ready to verify actual focus, an already-focused no-op, a normal button, stale-bounds refusal and pre-dispatch cancellation. This PR remains draft until those live checks pass. Coordinate occlusion, type/paste and the window-focus timeout in this issue remain open.Merged #120 from reviewed/signed
9374d8after green CI. Single element clicks now verify focus on editable text fields before trying AXPress, with the write off the main actor so Ace can service its own request.Real native focus/no-op/button and pre-dispatch cancellation checks passed. A real OpenAI channel performed inspect → one focus → inspect; exact stored results/images and usage survived worker exit/reopen without another native call. The actual isolated Ace composer also changed from unfocused to focused with its default draft and empty history unchanged.
Remaining limits are explicit: the old stale-window classification conservatively reported unknown before the new focus path, first inspection of fresh WebKit still needed a later read (#64), and in-flight cancellation was not claimed. A separate one-shot self-target point-click baseline returned 'No pressable accessibility element' (no text change), so element focus does not establish coordinate clicking. #106 remains open. #122 is next for live clipboard validation; #131 is a draft containing-window resolver fix awaiting a matching reproduction and runtime proof.
Merged #131 as
39a17adafter green CI, independent review, and a matching real WebKit reproduction. The old signed bridge reported a visible Web count button as occluded and recorded zero clicks. The patched signed bridge resolved its native containing-window link and delivered one background Accessibility action; the actual button callback changed from 0 to 1. The editor text and unrelated input/counters stayed unchanged, and all owned runtimes/fixtures are stopped.Together with #120, this lands verified editable element focus and the external WebKit button’s coordinate-occlusion fix. It does not establish composer point-focus, background Cmd+V, or every pointer route; those #106 items remain open. #132 normal quit is now in finite live validation, and #122 plain-text clipboard resumes immediately afterward. Desktop use remains serialized.
Final combined-source dogfood check passed in the actual Ace composer. The signed
c64042bcandidate has exactly the same repository tree as merged main2c36c49. In one disposable channel, explicit activation followed by one native element focus and onedesktop_insertproduced the exact public Unicode/multiline draft and UTF-16 caret. Native consumption was observed, clipboard cleanup was restored, and ownership was released. Channel history stayed empty and unchanged after reopening; no message was sent.The original clipboard was privately restored. All six owned supervisor/host/launcher/GUI/worker/vault processes are gone, the port/socket and temporary project/channel are removed, and the existing paste gate is empty. No installed app or OS grant was changed.
This verifies the active-Ace element-focus → dedicated literal-insertion workflow. It does not claim general background
desktop_keyCmd+V, composer coordinate-focus, direct full-valuedesktop_type, or other remaining #106 cases are fixed. #8 remains open for the remaining slices.Merged #141 (
ad31506695d632174b3486910b73e187d6c9444c): ordinary point clicks now use verified focus when the first actionable target is an editable Accessibility field. Native hit-testing runs away from the GUI main actor, and the exact target/window is checked again before mutation.The signed
75f20cc96a253383cbae0647cea3bc29bb244919candidate passed an actual-Ace check: one coordinate click focused the Message field, theacedraft and caret stayed unchanged, and the empty channel history remained unchanged after a distinct worker exited and reopened it. No typing, paste, clipboard access or message submission was used. All owned runtimes, temporary channel/project and settings were cleaned up. Static checks, signed build and CI passed.The earlier unknown point result and separate validation-runner startup failure are preserved in the PR evidence; neither was hidden by an input retry. Background Cmd+V and direct AX value replacement remain open in #106. This is merged on main, not a claim about the currently installed Canary.
During native validation of #148, desktop_type on the Ace-dev ProseMirror Message AXTextArea returned: value write accepted but requested result could not be verified. A fresh pixel inspection confirmed the composer remained unchanged. No input replay or fallback injection was used; the effort menu itself was successfully exercised through AX clicks.
Another real
desktop_focustimeout refusal, from the #156 pixel-authority model validation (signed Ace-dev769438a, disposable AppKit pointer fixture). Before a planned drag, pi/Anthropic calleddesktop_windows, thendesktop_focuswith the window's unchanged target. The call returnedrefused(dispatch_state: none,refusal_reason: operation_unsupported,retry_safety: safe) with message "Timeout while waiting for condition" and hint "Use a window and host that support focus window". The model stopped without dragging or retrying. In an earlier run the drag was not preceded by focus: it was accepted (dispatched_unverified) but recorded no fixture events, consistent with the documented inactive-view limit. The focus cause is not established here.Observed again while native-verifying the queue UI for #24 in a signed source Ace-dev build (main 4d503d3 + queue changes).
desktop_inspectexposed the Message AXTextArea as value-settable, butdesktop_typereturned unknown: "The accessibility value write was accepted, but its requested result could not be verified." A fresh pixel inspection showed the composer still contained only the Ace mention. I did not replay the input; used Ace CLI admissions for the real queued-message visual fixture instead.
While validating PR #104 (#91) in Ace Canary 0.0.11, Ace's desktop tools couldn't place focus in the Ace-dev (Electrobun/WKWebView) composer or paste into it. This issue records the tested failures only. #103 keeps the Automation-prompt and agent-guidance problem.
October 5 progress: #120 fixes verified editable element focus; #131 fixes the reproduced external WebKit button’s false coordinate occlusion. A combined signed-main check also passed activation → element focus → one dedicated literal insertion in Ace’s actual composer, with exact Unicode/multiline draft/caret, unchanged chat history, and clipboard restoration. #141 additionally fixes actual-Ace editable coordinate focus: one normalized point click gave the Message field verified keyboard focus, left the existing draft unchanged, and preserved the empty history across worker exit/reopen. Background
desktop_keyCmd+V, direct AX value replacement, and the other remaining cases below are not claimed fixed.Targets were Ace-dev builds from the #104 branch:
The Message composer is a ProseMirror contenteditable, exposed as an AXTextArea "Message".
Tested steps and results
desktop_clickat (0.5, 0.32), on the composer, was refused: "Point (710, 363) is occluded for the pinned target window 4742: accessibility hit-testing resolved an element in a different window of the same app." A CGWindowList check at that point showed Ace-dev window 4742 as the topmost layer-0 window. Electrobun also creates invisible strip windows (2560x30 at y=0 and 1512x33 at y=1440); none of them contains that point.desktop_focus, then the same point click. Focus reportedconfirmed_change, but the click was refused again with the same occlusion error. On 56752, afterdesktop_activatereported the app active, a point click at (820, 804) on the composer was also refused as occluded.desktop_clickon the AXTextArea "Message" was accepted (accessibility_action,dispatched_unverified). The next inspect still reported the AXWindow as the focused element, and the textarea hadisFocused=false. This was seen on both PIDs, the second time right afterdesktop_activate.desktop_selectwithcursor_beforeon the textarea returnedconfirmed_no_change, and focus stayed on the AXWindow.desktop_typewriting "x" returned "The accessibility value write was accepted, but its requested result could not be verified." Nothing appeared in the composer.AXFocused=trueon the textarea outside the tools made the next inspect report the AXTextArea as focused.desktop_keyCmd+V, delivered as backgroundwindow_targeted_events, then changed nothing while Ace-dev wasn't frontmost. A plain-text clipboard didn't paste either, so this isn't specific to images.desktop_focusrefused. Once, while the user was working in another app,desktop_focuswas refused with "Timeout while waiting for condition",operation_unsupported.What worked: with Ace-dev frontmost after
desktop_activate, settingAXFocusedon the textarea and pressing the app's Edit > Paste menu item through Accessibility pasted reliably. Nativedesktop_keyCmd+V wasn't retried in the frontmost state because steps 2–4 never produced a focused control.Not tested (corrected assumption)
In Finder,
desktop_keyCmd+C was refused: "The snapshot has no exact focused control. Click a control, then inspect the window again." The agent didn't follow that hint (click a file row, inspect again) before switching toosascript. Whether the native tools can copy from Finder's file list is therefore unverified. #103 originally listed it as a gap; the review comment there corrected that.Expected
desktop_clickon a visible web control in Electrobun windows isn't refused as occluded.desktop_keydelivers Cmd+V to that control, or explains why it can't.This issue covers capability bugs; prompt guidance is in #103. Sub-issue of #8 (slices 3–5, tracked under #5). Related: #65, #68, #73. Found during #91 / PR #104.