Computer use AI: what it is, who builds it, when to use it

Strip the marketing and the mechanism is simple. The model is given a screenshot, a cursor position, and a goal. It decides on the next action: click at coordinates, type a string, scroll, take another screenshot, and a host runtime executes that action against a real (or virtual) computer. The loop repeats until the goal is met or the agent gets stuck.

This is a different category from tool use. Tool use means the model calls a structured function like read_file(path) or send_email(to, subject, body). Computer use means the model is driving the GUI itself: the same primitives a human uses, with all the messiness that comes with it. The advantage is reach: any application becomes addressable. The cost is reliability: screenshots are noisy, pixel coordinates drift, and the agent fails in ways tool-use agents do not.

Production agent systems mix both. A good desktop AI agent like LapuAI uses structured tool calls for everything it can: file read, shell, document parsing, and only falls back to computer use primitives (screenshot, click, type) when there is no API surface for the target app.

Anthropic introduced computer use on October 22, 2024, as a public beta with Claude 3.5 Sonnet. In Anthropic's own words, this capability lets Claude “interact with computers by looking at a screen, moving a cursor, clicking buttons, and typing text” (Anthropic, 2024). It launched simultaneously on the Anthropic API, Amazon Bedrock, and Google Cloud's Vertex AI.

What ships is not a product. It is a model capability plus a reference Docker image and example code. You, the developer, run the agent loop, capture screenshots, dispatch the click and type commands, and decide where it runs. Anthropic's guidance is explicit that “Claude's current ability to use computers is imperfect” and that “some actions that people perform effortlessly (scrolling, dragging, zooming) currently present challenges for Claude.”

The safety posture is also explicit. Anthropic's computer use documentation tells developers to run the agent in “a dedicated virtual machine or container with minimal privileges, ” restrict its internet access to an allowlist, and “ask a human to confirm decisions that may result in meaningful real-world consequences” (Anthropic docs). For a deeper look at why that matters, see the desktop AI permission model breakdown.

OpenAI launched Operator on January 23, 2025 as a research preview for ChatGPT Pro subscribers in the United States: at the time, the $200-per-month tier. Operator was powered by a model OpenAI called Computer-Using Agent (CUA), built on top of GPT-4o, and operated a remote virtual browser hosted by OpenAI rather than the user's own machine (MIT Technology Review, 2025).

On July 17, 2025, OpenAI introduced ChatGPT Agent and folded Operator's capabilities into it. ChatGPT Agent combines a visual browser, a text browser, a terminal, and direct API access inside ChatGPT, and is available to Pro, Plus, and Team users via the “agent mode” toggle (OpenAI, 2025). The standalone Operator preview at operator.chatgpt.com was deprecated and shut down on August 31, 2025.

The shape is fundamentally different from Anthropic's API capability. ChatGPT Agent is an end-user product. It runs in OpenAI's cloud, drives a browser on OpenAI's servers, and asks you for confirmation before logins, payments, or other high-risk steps. It does not touch your laptop's file system or run shell commands on your machine. For a feature-by-feature comparison with LapuAI, see the Operator vs LapuAI comparison.

There is a third path. Instead of a remote sandbox you build yourself (Anthropic) or a hosted browser you rent (OpenAI), a desktop AI agent runs natively on your computer. LapuAI is in this category. The agent is a macOS or Windows application; the tools it calls are local file read and write, real shell execution, and native desktop automation via macOS Accessibility and Windows UI Automation APIs. Reasoning calls route through LapuAI infrastructure to frontier models: no API key management, but the actions execute on your machine.

The trade-offs are honest. LapuAI cannot do everything a cloud-hosted browser agent can: it does not rent IP addresses, it does not silently solve captchas in someone else's data center, and it is not the right tool if you need an agent for a machine you do not own. What it does instead is touch your actual files, run your actual shell, and drive your actual installed apps, with a permission prompt before every risky action. For users whose work lives on their laptop rather than on the public web, that is a different tool for a different job. See the desktop AI agent hub for the full picture, or alternatives to Claude Desktop for a related buyer's view, or what a native AI desktop application looks like.

“Runs locally” describes where a desktop agent executes. It is not the interesting part. The interesting part is how it reaches the application, and that is where the third path stops being a hosting detail and becomes a different mechanism.

A screenshot-driven agent sees pixels. It decides the Save button is at (412, 283), then synthesizes a mouse move and a click at that point. This is the most general approach available, anything visible can be attempted, and it inherits every weakness of working from a picture. Move the window, change the resolution, or let a dialog open half a second late, and the click lands somewhere else.

LapuAI works one layer lower. macOS and Windows both expose a running application as a tree of named, typed elements: the same interface a screen reader uses to describe an app to someone who cannot see it. Through macOS Accessibility and Windows UI Automation, the Save button is addressable as the Save button, with its own role, label, and state. There is no coordinate to miss.

LapuAI keeps both modes and picks one per application. The driver layer works where an app exposes an automation surface, most macOS and Windows software does, including Microsoft 365 and much of the older line-of-business software that never got an API. LapuAI can also drive real keyboard and mouse input, and it does so where that is genuinely the better tool, not only where nothing else is left. SAP GUI is the clearest case: driving real input means a customer never has to ask IT to switch on sapgui/user_scripting, a setting many organizations deliberately keep off. In that mode the agent does occupy the mouse and keyboard while it runs, and the three properties above describe the driver-layer path.

So the contrast with a screenshot-only agent is not that LapuAI never sends input. It is that sending input is the second choice rather than the only one.

This also sets the honest boundary against traditional RPA. Tools like UiPath and Power Automate Desktop target UI elements too, so raw reliability is not the dividing line. The dividing line is what it costs to get there: RPA expects a developer to record and maintain explicit selectors, and the script breaks when the vendor ships a new version. LapuAI works from a description of the outcome and resolves the elements at run time, no recording step, no selector library to keep alive.

Computer use AI earns its keep on bounded GUI tasks that lack a clean API. Internal admin tools nobody will ever wrap in an SDK. Legacy desktop software with a dozen modal dialogs. Long forms scattered across systems that do not talk to each other. Repetitive cross-app workflows. See desktop automation use cases for concrete examples LapuAI handles.

If you are deciding where this technology belongs in a wider operating model, use the practical AI for business guide to choose a workflow, set boundaries, and evaluate a pilot.

It fails, predictably, in three places. First, anything that depends on precise pixel work or sub-second timing: drag-and-drop into a moving target, real-time scroll, video scrubbing. Second, tasks where the screen changes faster than the model can take a screenshot and reason about it; the agent loop is on the order of seconds per step. Third, tasks where a structured tool already exists. If a file can be edited with a text tool, you should never be asking the agent to click into a GUI editor. Good agent design picks the cheapest reliable primitive for each step.

Every serious computer use AI system has the same security question at its core: what is the worst thing this thing can do before anyone notices? The model itself is not the threat. The threat is the action surface: the set of operations the agent can execute without checking with a human first. Anthropic's own documentation acknowledges this and recommends a sandboxed VM plus human-in-the-loop confirmation on consequential steps.

ChatGPT Agent handles this by running in OpenAI's cloud and pausing before logins, payments, and explicit external actions. LapuAI handles it the desktop way: every file write, every shell command, and every accessibility action surfaces a per-action approval prompt with the exact plan. Low-risk reads can be auto-approved per session; destructive operations always require explicit confirmation, and the entire trail is logged.

What you should not do is run a raw computer use loop against an unsandboxed laptop with no permission gating. That is the configuration Anthropic explicitly warns against, and it is the configuration most often hyped in social media demos. The honest choice is between (a) a hosted product whose vendor controls the sandbox or (b) a desktop agent whose permission system you can inspect and tune.