What is computer use AI?

What is computer use AI?

Computer use AI is the pattern where an agent drives a real desktop or browser by reading rendered pixels and emitting clicks, keystrokes, and scrolls. The agent does not call an API. It looks at a screenshot, decides where to point, and acts like a human user would. Anthropic shipped this with Claude 3.5 Sonnet in October 2024. OpenAI followed in January 2025 with Operator, powered by its Computer-Using Agent (CUA) model. Browser Use is the OSS equivalent: any LLM plus a Playwright-controlled browser running an agent loop.

How it works

Each step is a model call. The runtime captures a screenshot, the model reasons over the pixels, and it returns coordinates plus an action (click at x=412 y=689, type "refund"). The host executes, snapshots again, loops. That screenshot-then-click pattern is why computer use runs roughly 10x slower and pricier than native tool calls: every iteration is a fresh vision-model inference.

When you'd encounter it

Reach for computer use when no API exists or the integration is too unstable to maintain (legacy ERP UIs, niche SaaS, internal admin panels). Skip it when an API or MCP server is available; tool use is faster, cheaper, and deterministic. Two recurring failure modes: anti-bot detection (Cloudflare, hCaptcha) defeats most computer use agents on protected pages, and DOM-driven stacks (Playwright plus accessibility tree, Stagehand) score 12-17 points higher than vision-only agents on common task benchmarks.