Repository navigation
[Meta] Complete native computer use in Ace #8
Description
Activity
- added a parent issue
on Oct 4, 2026 The first inspection slice is ready for review in #47. Peekaboo is embedded in Ace, so macOS permissions belong to Ace and no separate Peekaboo installation is required. Pi remains the native harness; the collaborator-agent switch controls desktop tools along with all other tools.
Validated real application/window inventory, Accessibility text and screenshots, real Anthropic runs, local and hosted-channel routing, persisted image replay after local host and Workers restarts, and workspace cancellation/offline behavior. The hosted checks used the local Workers/Durable Objects runtime.
Dogfooding found that Ace's synchronous folder picker blocked the embedded bridge until the dialog closed. #47 also replaces that picker with an asynchronous one; inspecting the open dialog, canceling, selecting a comma-containing folder, and quitting while it is open now pass.
This issue remains open for the next slices, including native clicks and typing. The PR also documents the tailnet-only team/access model and that Ace is open source software operated by its users, with no Ace-operated hosted service.
PR #63 is merged. The first mutating slice passes real signed Ace and pi/Anthropic click/type/key/button workflows, plus delivered-input Stop and worker-restart checks with no replay. Exact local result/image history also survived restart.
Hosted validation passed through local Workers/Durable Objects and a signed disposable AppKit form: the collaborator switch, offline refusal, workspace loss after native launch with unknown delivery and cleanup, the full real-model native workflow, and exact replay of all ten result/error/image records after Workers restart without a workspace. This is local Workers validation, not a second physical machine. The earlier Ace-dev window-availability failure remains tracked in #64; the successful form run does not establish recovery from that condition.
The signed build, signature checks, React Doctor, type checks, formatting/lint, and workspace generation-correlation checks pass. Type checks and formatting/lint were repeated after integrating #46 before merge. No canary release was requested for this merge.
Next is #65: selection, insertion at the caret, and shortcuts. It begins the complete-input slice, which stays unchecked until pointer and clipboard work also land.
Native dogfooding exposed a separate composer persistence issue: #77 records the unsent synthetic draft
alpha duplicdisappearing across desktop quit/relaunch while channel messages remained present. It is tracked under #5 core/QOL and is distinct from #59's composer-mode behavior and #76's literal-input semantics. The before/after evidence and development-candidate qualification are in #77; this does not mark any native input slice complete.Literal insertion is now pushed in draft #67 at
1e9f5cee73d849ef0a368b711df06f224c1a8324. The GUI owns paste delivery and consumption-based cleanup, with process-wide coordination and bounded text verification. Types/lint, signed builds/signatures, and source review pass.The earlier standard WKWebView foreground case passed literal Unicode/multiline replacement without submission. The new inactive-window checks refused before input and restored the clipboard; exact chrome-admission diagnostics are being preserved rather than weakening the guard. An already-active exact editor will use the direct chord after repeated active/focus checks; that follow-up is source-reviewed and still needs runtime proof.
Actual Ace composer, affected model/hosted replay, and in-flight interruption validation remain pending. #78 tracks the separate unresolved clipboard-ownership gap between channels after uncertain delivery. #67 remains unmerged and the input slice remains unchecked.
The conditional literal-paste candidate has passed the signed build. The actual Ace composer check has not sent input yet: its preflight reports Ace inactive, and repeated native application inventories report no active application at all. The console is unlocked. Independent UI setup through Show Ace and opening the existing app through Finder did not change that native result. This is under investigation; it is not a composer pass or an insertion failure. The earlier
/tmpversus/private/tmphelper comparison was corrected before clipboard preparation.#78 is now part of the keyboard merge prerequisite: extend the existing clipboard gate with content-free unresolved ownership, so another channel cannot replace a payload while the first paste may still be queued. The proposed recovery uses observed consumption or termination of the exact receiver process generation; it will not persist private clipboard contents or automatically restore them after an uncertain result. #67 remains draft while this and the real runtime checks are completed.
New native dogfooding finding: #79.
desktop_appscopied activity/visibility defaults from mutation inventory rather than observing them, so repeatedactive: []output could not prove the desktop's actual activity. A focused client change will preserve exact PID/generation inventory and join real presentation metadata, reporting unknown when unavailable. No implicit activation or OS-lock conclusion; signed and live validation are pending.Landed two focused PRs on
feat/github-cache:- fix(desktop): report observed application activity and visibility #80 fixes inaccurate application activity/visibility. A real signed native read agreed with an independent AppKit check.
- Add native text selection and keyboard shortcuts #81 adds contextual text selection and complete keyboard shortcuts, including Do not reject editable text areas when AXEnabled is absent #66. Its selection/shortcut paths retain the real native and local/hosted model coverage; fresh static checks pass.
The insertion work stays in draft #67. #71 (inspection feedback) and #70 (pointer input) are now being validated independently so they can land without waiting for paste. No new canary has been published for these merges.
Merged #71 on
feat/github-cacheas61c5aa9.The independent signed build passed real Accessibility/pixel reads, exact target checks, invalid-mode rejection, and absent-window original-error diagnostics. A real AppKit button closed its target exactly once; the completed action and native receipt survived the failed follow-up inspection. Later inventory retained the closed window as offscreen/pixels-only, so the verifier was corrected against saved evidence without repeating input.
Pixel reads intentionally provide no action snapshot. This adds useful inspection feedback; it does not fix sparse webview Accessibility trees or complete the broader visual workflow. No new canary has been published.
Merged #82 on
feat/github-cachease5618c1: click variants and normalized screenshot-coordinate clicks. Six native click groups passed; the exact-PR82 pi/Anthropic run then made one coordinate click, verified the intended callback incremented once, and reopened identical tool results/images with unchanged usage and no repeated input. Evidence:/tmp/ace-pointer-fixture.TNcgBS/click-model-KfkLfz/result.json.#70 continues scrolling and atomic dragging. #68 remains open for that scope and its remaining interruption/hosted/coordinate validation. No new canary has been published.
Dogfooding found #83: a healthy native bridge's application list was trimmed to 107 apps, omitting the newly launched Ace process. #84 adds an optional name/bundle-ID search before bounding, preserving the text cap and native completeness plus explicit filter counts. Static checks and source review pass; a read-only check on the current signed runtime is prepared. This does not block #74's fixture validation.
Merged #84 as
eee1a39after type/lint checks, Ace diff review, independent source review, and the real read-only smoke on exact headee71fe0. The original omission reproduced (107 returned/42 omitted); name and bundle-ID queries found Ace's exact process generation, while no-match/invalid queries and the unchanged 32 KB cap passed. Native completeness and target metadata were preserved. Evidence:/tmp/ace-app-search-validation-LEAHpI/result.json. No model run or new native build was performed for this host-only behavior.Merged scroll-only #70 as
4431242from reviewed headf1682d0. Five real native scroll requests passed on the signedd9b0c86candidate; the real pi/Anthropic run made one upward scroll (2063 → 1834) and reopened exact saved text/images with unchanged usage and no repeated input. Evidence:/tmp/ace-pointer-remaining-fixture.5mF516/remaining-validation-D6zaIm(overall run stopped at the subsequent pre-input drag refusal) and/tmp/ace-pointer-remaining-fixture.5mF516/scroll-model-pKvFdN/result.json. The extracted scroll path is unchanged; final static/source checks passed without a new native build. Drag and the unverified broader cases remain in open #68. #74 and #84 progress is preserved.16 remaining items
Merged #143 (
b2a400b9), validated at exact head2aea91a9:desktop_openopens one existing path or explicit-scheme URL through an optional selected application or the macOS default handler. The signed result confirms accepted delivery to an exact process generation; it does not claim the item loaded.Four preflight/cancellation checks produced no receiver callbacks. An explicit-app Unicode document, default-handler custom URL, and one real pi/Anthropic model document each opened exactly once. Exact result, usage and execution counts survived distinct-worker reopen without a fourth callback. The first verifier exposed LaunchServices Unicode normalization; both URL spellings were independently verified to identify the same file, and that successful open was never repeated. All owned runtimes, fixture registration, channel/project/preferences, port and socket are cleaned. Details and limitations are in #143 and #73.
The final own-Ace menu check is next in #140; file-reference clipboard writes are progressing separately under #72. #8 remains open.
Merged #140 (
6bddf0f9), final headd1993fe1.desktop_menuinvokes one freshly resolved, unique, enabled external-app menu leaf through the existing native process lane. The real native refusal, literal/Unicode and modal checks plus a real Anthropic menu command/distinct-worker reopen passed. The final implementation preserves that tested external path.The serving Ace process’s own menu commands explicitly refuse before input; the final signed check verified
operation_unsupported,dispatch_state: none, the exact process receipt and no new visible window. An earlier experimental own-app press returned unknown without a subsequently observed picker. Its original result and read-only recovery remain preserved, with no repeat; the cause is not established. All owned runtimes, sockets/ports/settings and temporary data were cleaned. Own-Ace delivery and broader dialog/interruption/hosted validation stay open.Merged #147 as
f24257b7d6175fcaed97838a2c59ea28e49e04aa, from the exact validated head7c2262d330ca87327bdd5ec63b6eeca59f75cf11. This completes persistent file-reference clipboard writes alongside the landed text and image forms; broader temporary paste and lifecycle validation remain open in #72 / #75.Validation passed four signed native fidelity/boundary cases, nine unchanged-generation refusals, a real Claude Opus 5.5 write/read with distinct-worker durable reopen and no replay, and an actual delayed-paste reservation refusal followed by normal admission after receiver exit. Types, formatting/lint, signed build, strict signatures and CI passed.
One Finder Paste copied the public file and directory with matching contents. The subsequent exact-window close returned unknown; its original failed phase is preserved separately from copy success. Fresh inventory confirmed that exact window gone, and no input was repeated. The missing original close response and remaining management validation are recorded in #73.
Original clipboard contents stayed in GUI memory and were restored through the existing gate and generation checks. All owned fixture/native/host/worker processes, disposable channel/project and temporary settings are cleaned up; the user's Finder remains running. Main now contains this capability; it has not been promoted to an installed Canary by this merge.
The next bounded input slice is pixel-snapshot authority for existing coordinate actions. Source review confirms the existing signed canonical snapshot projection can be reused. Preserve automatic Accessibility refresh for the working click-to-edit path; unrestricted canvas single-click delivery remains a separate native transport gap.
Opened #156 (head
769438a, not merged). It adds pixel-snapshot authority for the existing coordinate actions.desktop_inspectwithmode: "pixels"now publishes a single-use snapshot through the existing producer-boundcreateSnapshotand signed canonicalgetDetectionResultprojection. There is no new bridge operation, store, or vendor patch. Pixel snapshots authorize only point clicks, point scrolls and drags in the exact observed window. Element, value, selection, key and insertion requests refuse naturally: there is no element or focused control. Post-action refresh stays in accessibility mode, so click-to-edit is unchanged.Signed native checks on a disposable AppKit fixture passed six pre-dispatch refusals, plus refusal of a consumed snapshot. A point click (count +1), a point scroll (0→687.5) and a drag (one down, one up, dropped) all took effect. A moved-window stale click returned signed
unknowninstead ofrefused, with no input received. An Accessibility-snapshot control on a fresh fixture showed this is pre-existing click classification, now tracked in #155. The original failed check is preserved.Two real pi/Anthropic runs used fresh pixel snapshots for every mutation, and clicks and scrolls took effect in both. Run 1's drag was accepted but had no effect on the inactive view. Run 2's prerequisite
desktop_focusrefused with the known #106 timeout. Neither run is a full three-effect model workflow. Distinct-worker reopen of both runs reproduced identical durable tool text/images, with unchanged usage, no new run and no input. All owned processes, profile data, channels and fixtures are cleaned.Remaining in slice 5: pixel-only targeting without native AX hit-test dependence (general canvas single-click transport), display/region observation, #155, #106, and multi-display/hosted/second-machine validation.
- added sub-issues
on Oct 6, 2026 - changed the title
[-]Inspect and operate native UI from an Ace channel[/-][+][Meta] Complete native computer use in Ace[/+]on Oct 7, 2026 Native tool observation during #184 cleanup: desktop_key Cmd+W on an observed Safari tab returned DESKTOP_ACTION_FAILED: background Cmd+W cannot be verified; recovery text asks for --foreground, which desktop_key does not expose. Fresh pixel inspection confirmed the tab remained open; no replay or workaround used. This is a tool capability/recovery-hint gap, not a sidebar issue.
Let an Ace channel inspect and operate the native desktop on its execution host, so Ace is developed and dogfooded through Ace. Tracked by #5.
Done when: in the signed app, real pi/provider runs open a project through Ace's own native picker. They also change a setting through menus and dialogs and complete a cross-app open/edit/save workflow. Each run recovers from a changed target without acting on the wrong window, and teammates can see and stop what the agent is doing. This holds across concurrent channels, interruption, multiple displays and a second machine. Shipping a primitive doesn't close the child issue that owns its broader validation.
This body tracks current state only. Failed outcomes and evidence stay in the earlier comments, PRs and child issues.
Shipped on
mainMerged isn't the same as released. An installed Canary or stable build contains a change only if its release includes it.
Current focus: #188, desktop tools extraction
The native desktop engine is now the public package githubnext/desktop-tools. Draft #213 makes Ace consume it at a pinned commit. Remaining: live capture and input checks, blocked on an unlocked session and the reference host's macOS grants. Details are in #188.
Paused task: #73, native Open Folder workflow
Paused at local
783daba: built signed and source-reviewed, but not live-tested or pushed. Acceptance below is unchanged.In a signed, isolated Ace built from current
main, open the native picker with the existing Cmd+O path. Delivering menus to Ace's own serving process comes later. Freshly identify the real picker by exact process and window, operate it with native tools, select an owned folder, and verify that it adds a project without creating a channel. Cancel must leave projects unchanged.First reproduce the recorded failure with the main window overlapping the picker (details). Fix only the first proven blocker. The recorded symptoms are a focus timeout, a key refusal with no focused control, and clicks returning
unknownwith a same-app occlusion message. They aren't assumed to share one cause.A real model run must complete the same workflow. Preserve refusal and
unknownevidence, and show no replay after a distinct-worker reopen.Backlog (priority order)
unknowneven though nothing was sent. Fixed by fix(desktop): report clicks on an already moved window as refused #187, merged as4d503d3.Supporting blockers (each keeps its existing parent, #5):
Constraints
Access: see architecture. The tailnet is the team boundary. Each channel has two separate settings that only its owner can change:
sharedgates new collaborator agent invocations.desktopgates every native desktop call in the channel.Neither cancels work already dispatched or sandboxes the shell. Don't add participant ACLs or per-action prompts.
No blind replay: input outcomes persist as completed, refused or unknown. Uncertain input, including an unresolved paste reservation, is reobserved and never retried automatically.
Targets: actions verify the existing process and window receipts, plus the observation authority each operation needs. Stale targets refuse. Peekaboo coordinates native work.
Cleanup: Stop or connection loss settles outstanding work and releases held input. A viewer disconnecting doesn't stop the channel.
Harness: pi stays the harness (Run and resume Codex and Claude Code in Ace channels #9 covers external harnesses). Peekaboo is embedded on macOS 15+, and macOS grants belong to Ace.
Observed content is data, not instructions.
Validation: focused PRs with signed native checks, real provider runs and a distinct-worker reopen.
Boundaries