A Simple Harness Outperforms VLA and World Action Models
Project Page · Quick Start · Documentation · Citation · 中文
RoboHarness overview: visual keypoints are tracked persistently, converted into geometric constraints, executed, and verified in a closed loop. View original PDF
News
Overview
Without training models, RoboHarness combines visual clicks, keypoint tracking,
and geometric constraints to provide an embodied control panel for an LLM agent.
The LLM marks points of interest on a 2D image and continuously receives their
positions via optical-flow tracking and depth back-projection; with this
information, it composes accurate primitive actions to perform complex
manipulation.
The reported BEHAVIOR results use Claude Code 2.1.259 with
Qwen3.8-Flash-Next-FP8 and the R1 Pro robot. A Codex harness is also available.
RoboHarness V2 replaces the tool plugin with a robot workspace that the agent
works inside: robot model, joint manuals, readable Python tools and skills.
It runs on the same tasks and evaluator with --harness v2; see
harness/v2. The commands below use V1 and reproduce
the paper results.
Repository structure
RoboHarness/
├── run.sh # entry point: ./run.sh --task task01 [--harness claude_code|codex|v2]
├── harness/ # agent side: how the agent sees and controls the robot
│ ├── v1/ # paper version: Claude Code and Codex plugins
│ └── v2/ # workspace harness: R1 Pro and ARX X5 workspaces
├── eval/ # evaluation side: runner, BEHAVIOR interface, RoboDojo bridge, viewers
├── BEHAVIOR/ # official simulator and evaluator (submodule)
├── tasks/ prompt/ # task definitions and archived prompts
├── configs/ # example configurations
└── scripts/ docs/ tests/ website/
| Path | Purpose |
|---|
harness/v1/claude_code/ | Claude Code harness used for the paper results |
harness/v1/codex/ | Codex harness |
harness/v2/ | V2 workspace harness (R1 Pro on BEHAVIOR, ARX X5 on RoboDojo); --harness v2 |
eval/runner/ | Task runner behind ./run.sh: starts the evaluator stack and agent, collects official scores |
eval/behavior/ | BEHAVIOR interface: RGB-D observations, robot control and the official evaluator wrapper |
eval/robodojo/ | RoboDojo bridge for V2 |
BEHAVIOR/ | BEHAVIOR simulator and evaluator, pinned to v3.9.1 |
tasks/ | Task definitions, evaluation instances and step budgets |
prompt/ | Task-specific instructions for household manipulation |
reference_results/ | Reference evaluator scores (hosted on Hugging Face; setup.sh fetches them) |
validation_results/ | Evaluation reports and scores (hosted on Hugging Face) |
See eval/README.md for the evaluation side in more detail.
Installation
Use Linux x86-64, an NVIDIA RTX GPU, Python 3.11 and the CUDA 12.4 toolkit.
The validation host has 48 GB VRAM per GPU; this is the tested capacity, not a
measured minimum. You also need BEHAVIOR scene data, local cache space and a
separately configured model endpoint.
git clone --recurse-submodules https://github.com/BinceQu/RoboHarness.git
cd RoboHarness
./scripts/setup.sh
cp configs/example.json configs/local.json
Videos, reference results and validation reports are hosted in the
RoboHarness-assets
dataset, pinned by assets.lock.json, so cloning stays small. setup.sh
fetches the reference results the runner needs; python3 scripts/fetch_assets.py --group all
fetches everything and verifies each SHA-256. The paper and project page correspond to
the v1.0-paper release.
Follow the installation guide to obtain the licensed datasets,
install Claude Code 2.1.259, and configure the model endpoint and authentication.
Edit configs/local.json for this machine's data and Python paths.
Run a task
The model server and the evaluation runner are separate processes. Keep your
model server running throughout evaluation. setup.sh installs the simulation
and agent environments; it does not download LLM weights or start a model server.
1. Connect to your model service.
The default Claude Code harness uses an Anthropic-compatible /v1/messages
endpoint with image input, tool calls and streaming. The paper configuration
uses Qwen3.8-Flash-Next-FP8. If your server only provides OpenAI Chat
Completions, follow the bridge setup
first. Codex uses a different Responses configuration.
On the evaluation machine, open a terminal in this repository and replace
the host, served model ID and key below with your service's values:
This is the beginning of the README. Read the rest at the original. Read the full README ↗
