What is computer use?
How agents read screens, take actions, and work across applications.
Computer use lets an agent operate software through its interface. The agent reads a screenshot, chooses an action, and uses a mouse or keyboard to carry it out. It then checks the screen again.
An agent can use applications that have no API. Whether it can finish a particular task reliably still needs testing.
The computer use loop
A typical loop has four parts:
- Observe. Capture the current screen and relevant task context.
- Decide. Choose the next action, such as opening a menu or entering text.
- Act. Execute a click, keystroke, scroll, or another supported tool call.
- Check. Observe the result and decide whether to continue, retry, or stop.
Turning “click Save” into screen coordinates is called grounding. Some models return coordinates directly. Other systems pair a planning model with a separate grounding model.
The harness executes the model's actions and returns the results. It also handles errors, timeouts, and when to stop.
Choose the simplest interface for the task
Use an API when it provides the operation you need. Use browser automation for web tasks. Reach for the desktop when the work requires it.
| Interface | Good fit | Main consideration |
|---|---|---|
| Application API | Structured data and supported operations | The application must expose the operation you need. |
| Browser automation | Web tasks with identifiable page elements | Selectors, page state, and authentication still need handling. |
| Desktop computer use | Native apps, visual workflows, and tasks across applications | Screen interpretation can fail and adds model work. |
A useful agent can combine all three. It might use the desktop to open a report, a shell command to process a CSV, and an API to store the result.
Browser automation is not limited to Chromium, and traditional automation is not limited to APIs or selectors. Many systems also use images and accessibility information. Compare how each system observes state and chooses an action.
What makes a task reliable
Give the agent a clear starting state and a result it can check. “Create a file at this path containing these values” is more testable than “organize my work.”
Keep the screenshot and action coordinate systems aligned. If a screenshot is scaled down before reaching the model, a click may need to be scaled back up before execution.
Record enough of the run to diagnose mistakes: instructions, model and tool versions, actions, errors, and relevant screenshots. Set budgets for steps, elapsed time, and model usage. Distinguish completion from reaching a limit.
For consequential actions, decide where the agent should hand control back to a person. A separate computer helps contain the environment; it does not make every action inside a signed-in account appropriate.
How to evaluate models
Model IDs, supported tools, and provider availability change. Start with the provider's current integration documentation, then evaluate the task you actually need to run.
Benchmarks such as OSWorld help compare systems under defined conditions. Read the benchmark version, task set, step budget, and number of attempts alongside the score. A result from a different setup is not a direct comparison, and a benchmark score is not a production success rate.
For your application, measure completed tasks, time and cost per completion, and how often a person needs to intervene. Save unsuccessful runs as well as successful ones.
Start with a remote computer
Orgo supplies the desktop and an API for screenshots, input, and shell commands. Your model or harness decides how to use them.
The quickstart walks through creating a computer and controlling it over HTTP. The Claude guide includes a small Python example with an explicit model, step budget, and cleanup.
If the task stays inside a website, read cloud desktops vs. headless browsers before choosing the environment.