RPent (Recursive Physical Agent) is an open framework for building embodied agents that continuously evolve through recursive interaction with the physical world. Rather than prescribing a single foundation model, RPent provides a recursive agent framework that harnesses heterogeneous intelligence, including perception, reasoning, memory, execution, and self-evolution, into a unified physical agent. Through continuous interaction, reflection, and adaptation, RPent enables physical agents to acquire new capabilities and evolve beyond their initial design. We build RPent upon a foundation of service-oriented, standardized, and composable design principles, ensuring the framework remains highly extensible.
Leaderboard
Compare RPent with reference methods on LIBERO, LIBERO-PRO, RoboCasa365 Target50, and RoboTwin C2R. Rankings apply to the methods and evaluation coverage shown; see the Leaderboard for suite results and model configurations.
Codex / GPT-6 Astra / low / reasoning: 92.63% Overall (741/800) across all eight LIBERO-PRO suites. See the suite results and memory-batch explanation, including the separately frozen Long and Spatial/Object/Goal memory batches.
Who Should Consider Using RPent?
RPent is built for four kinds of users:
- Embodied intelligence researchers targeting high success rates on embodied tasks and benchmarks — especially long-horizon manipulation. RPent's memory-guided, agentic composition consistently lifts task success beyond what a frozen VLA delivers alone.
- Online-learning and reinforcement-learning researchers studying self-evolving embodied agents. RPent's recursive interaction, reflection, and memory-distillation loops provide a ready substrate for continual and reinforcement learning in the physical world.
- Robotics application developers deploying embodied solutions on real robot hardware. RPent's service-oriented, standardized architecture and customizable agentic control logic improve real-world success rates and shorten the path from prototype to production.
- End users of deployed embodied agents — the customers of the developers above. Install RPent together with the relevant real-robot extensions and run the predefined tasks out of the box, with no ML expertise required.
What's NEW!
Feature Matrix
Quick Start
1. Choose an environment and install RPent.
git clone https://github.com/RLinf/RPent rpent && cd rpent
pip install -e ".[libero-pro]" # Recommended default (LIBERO-PRO)
# Other environment configurations:
pip install -e ".[robocasa]" # RoboCasa
pip install -e ".[robotwin]" # RoboTwin
pip install -e ".[franka]" # Franka and dual Franka
.[libero-pro] is the recommended default. See the installation docs for other environments.
For RoboCasa setup, task memory, and the Target50 protocol, see the
RoboCasa guide.
RoboCasa uses current-task files in robocasa/task-specific/ together with
robocasa/global/GLOBAL_MEMORY.md from
RPent-memory.
The CLI and Dashboard provide both layers for on-demand reading, with no
full-read requirement before actions. HF memory follows the current branch
without a fixed data revision.
The example below continues with LIBERO-PRO.
2. Download the LIBERO-PRO simulator assets.
liberopro-download-assets --skip-existing
💡 Slow connection to Hugging Face? Download through the mirror: HF_ENDPOINT=https://hf-mirror.com liberopro-download-assets --skip-existing.
See the installation docs for other simulators.
3. Configure keys and checkpoints, then run.
# Anthropic key; no need to export the base url if you use the official endpoint.
export ANTHROPIC_API_KEY=sk-xxx
# VLA checkpoint — download from
# https://huggingface.co/RLinf/RLinf-Pi05-LIBERO-130-fullshot-SFT
hf download RLinf/RLinf-Pi05-LIBERO-130-fullshot-SFT \
--exclude optimizer.pt \
--local-dir ./checkpoints/RLinf-Pi05-LIBERO-130-fullshot-SFT
export PI05_CHECKPOINT_PATH=$PWD/checkpoints/RLinf-Pi05-LIBERO-130-fullshot-SFT
# SAM 3.0 checkpoint — download from
# https://modelscope.cn/models/facebook/sam3
pip install -U modelscope
modelscope download facebook/sam3 \
--local-dir ./checkpoints/sam3
export SAM3_CHECKPOINT_PATH=$PWD/checkpoints/sam3/sam3.pt
export LIBERO_TYPE=pro
# Run one task: libero_object_swap, task 2, seed 0, using Claude Code
# with Claude Opus 4.8.
rpent --robot libero --suite libero_object_swap --task 2 --seed 0 \
--cuda-device 0 --planner claude_code --model claude-opus-4-8
See the planner docs to configure other planners (api, codex) and model providers.
For the exploration workflow and local-memory evaluation, see LIBERO exploration mode.
Interactive CLI mode
With claude_code or codex, add --tui (-t) to steer the agent live from your terminal. The built-in task is pre-filled at the you> prompt. Press Enter to submit it as-is. To change or replace the task, edit the input before pressing Enter. While the agent runs, type follow-up messages to steer it (/help lists commands; /quit or Ctrl-D ends). Requires a TTY. The api planner uses a native CLI that runs the preset task first and accepts follow-ups between runs; use /exit to close it.
rpent --robot libero --suite libero_object_swap --task 2 --seed 0 \
--planner claude_code --model claude-opus-4-8 --tui
Live Dashboard
Add --dashboard to start a local Dashboard and print its URL in the terminal. Session settings come from the CLI, and the page opens directly in the live monitor. Once the services are ready, start a task with /rpent-task <suite> <task> <seed>. The page streams agent reasoning, camera views, and the action timeline, and you can submit another task after the current one finishes. Use --dashboard-language zh-cn for the Chinese UI.
rpent --robot libero --dashboard --dashboard-language zh-cn \
--planner claude_code --model claude-opus-4-8
For a complete list of CLI options, see the Command-Line Reference. RoboCasa and RoboTwin provide their own task parameters — see the RoboCasa and RoboTwin docs.
For more detailed documentation, see the RPent documentation.
Contributing
Contributions are welcome. See CONTRIBUTING.md for the
development setup, required checks, testing policy, and integration checklists.
Citation and Acknowledgement
If you find RPent or Harness VLA helpful, please cite the paper:
@article{zhang2026harnessvla,
title={Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents},
author={Zhang, Yixian and Zhang, Huanming and Gao, Feng and Li, Xiao and Liu, Zhihao and Zhu, Chunyang and Qiu, Jiaxing and Yan, Yuchen and Liu, Jiyuan and Tang, Wenhao and Fang, Zhengru and Nie, Yi and Wei, Changxu and Wang, Yu and Ding, Wenbo and Yu, Chao},
journal={arXiv preprint arXiv:2607.08448},
year={2026},
url={https://arxiv.org/abs/2607.08448}
}
RPent builds on the simulators, VLA models, and training infrastructure of RLinf, and on the agent SDKs of the broader open-source community — pydantic-ai, the Claude Agent SDK, and the OpenAI Codex SDK. Thanks to the teams behind LIBERO, RoboCasa, robosuite, MuJoCo, and openpi.
