Computer Use

Computer Use lets the agent read a native application through the operating system's accessibility layer and operate it: it takes a window as a numbered outline of its buttons, fields, and rows. From there it can press a control, type into a field, set a value, scroll, or drag. This reaches work that has no web UI — pulling a figure out of a spreadsheet, walking a desktop-only internal tool, or reading an error dialog and telling you what it says.

Rather than screenshotting and guessing at pixels, the agent asks the accessibility layer for a structured outline of a window — a numbered list of its controls — and then acts on a control by number. That keeps actions precise and lets the agent describe what it's looking at before it touches anything.

The agent can:

• List the windows and controls of an app it's allowed to see
• Press a button or menu item
• Type into a field or set a value directly
• Scroll and drag

Computer Use runs on macOS and Windows, each through its own native backend. Linux is not supported.

• macOS: reads and drives apps through the accessibility layer (AX).
• Windows: reads and drives apps through UI Automation.

The backends differ in two ways worth knowing. On macOS, an action can be delivered to the target application alone, so a background window is operated without moving your physical pointer. On Windows, a coordinate-based click or drag warps the real pointer to that point. macOS also enforces a per-call timeout on accessibility requests that Windows UI Automation does not expose.

On macOS, actions are delivered to the target application, not by moving your physical mouse. A background window can be operated without stealing your cursor or focus, so the agent working in one app doesn't interrupt what you're doing in another. One action can move your real cursor. The agent never selects it on its own and has to request it explicitly.

Computer Use is gated the same way the rest of Crew's tools are, with extra protections specific to reading a live desktop:

• Password fields are never read, and a window holding one is never captured.
• Destructive-command-shaped text is refused rather than typed.
• Every call is written to the audit log — whether it was allowed or refused.
• Calls run through the same approval and governance gates as other tools. See Security.

From the dashboard, open Settings → Computer Use and turn it on. On macOS, the first time you enable it the system asks you to grant accessibility permission to the app. Computer Use can't read or operate other apps until you do.

Two commands help you see what's addressable and debug tool behavior (macOS and Windows):

kirocrew computer apps        # list apps the computer-use tools can address
kirocrew computer call computer_list_apps   # run a single computer_* tool directly

kirocrew computer call is a human debug harness — it runs the computer_* tools through the same gated chokepoint the agent uses. It's for inspection, not for scripting the agent.

• Browser — operate a web browser instead of a native app
• Security — approval, denied commands, and the audit log
• CLI reference — the full kirocrew computer commands