Running Agent S2 on a remote desktop

Agent S2 is an earlier generation of Simular's open-source computer use framework. It separates planning from grounding: one component chooses an action, and another finds the target on screen.

For a new project, start with the current Agent-S instructions. The upstream installation now targets Agent S3. The S2 example needs the compatibility checks below.

How the integration works

The original example captures a screenshot, asks Agent S2 for an action, executes the generated Python, and repeats until completion or a step limit.

In local mode, it uses PyAutoGUI. In remote mode, it uses Orgo to capture the desktop and execute the action on a Linux computer. The planner can run on your machine while the task runs remotely.

Keep the S2 dependencies separate

The example imports gui_agents.s2, but its requirements do not pin a version. An unconstrained install can pull in a different generation of the framework.

Use a dedicated environment and constrain the S2 package line when evaluating the old example:

git clone https://github.com/spencerkinney/agent-s2-example.git
cd agent-s2-example
python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt "gui-agents>=0.2,<0.3"
cp .env.example .env

These are macOS and Linux shell commands. You also need Tesseract for OCR and keys for the model providers you select. Pin the full dependency set after validating your environment; constraining one package does not freeze its dependencies.

Review the model configuration

The example's default grounding model is claude-3-7-sonnet-20250219, which Anthropic has retired. Do not copy that default into a new setup. Check Anthropic's model lifecycle documentation and the framework's model adapter before selecting a replacement.

The planner and grounding model are configured separately. Confirm that both model IDs are available to your accounts and that the installed adapter supports them. A currently available model is not automatically compatible with an older agent framework.

The example also hardcodes a grounding coordinate space of 1366 × 768 for its Anthropic adapter. Validate the adapter's coordinate handling against the screenshots you supply. Changing your desktop resolution alone is not a reliable fix for misplaced clicks.

Connect the remote computer

After configuring and checking the models, these entries in .env select Orgo's remote environment:

ORGO_API_KEY=your_orgo_api_key
USE_CLOUD_ENVIRONMENT=true
MAX_STEPS=10
STEP_DELAY=0.5

Run a small task from the example repository:

python agent_s2.py "Open the browser"

The script imports PyAutoGUI even in remote mode. Run it from an environment with the display dependencies it expects; the remote flag alone does not make the controller suitable for a headless server.

Watch the first run and inspect the generated actions. The script executes model-generated Python, and its step limit is not proof that a task completed. Stop or delete the remote computer in the dashboard when you finish; this example does not implement automatic cleanup.

For a maintained starting point

Use the upstream Agent-S setup for the current framework, or the Claude and Orgo guide for a smaller integration with an explicit SDK version and cleanup.

The original S2 video remains useful for understanding the flow, but its package versions and model defaults are historical.