Video to Data (V2D)
An end-to-end pipeline that converts human demonstration videos into simulation-ready assets and physics-grounded robot training data.
Documentation · Datasets (Hugging Face) · Robotic Grounding Project Page · Robotic Grounding Tech Report
Contents
Overview
Video to Data (V2D) turns raw human demonstrations into robot-ready training data through three composable stages and a repository-root E2E workflow. Each stage runs independently and writes its artifacts to disk, so you can stop, inspect, cache, and recompose the pipeline at any boundary.
- Video Ingestion Agent — a LangGraph-driven agentic workflow that segments demonstration videos into temporally-bounded action clips, extracts an entity-relation scene graph, and stores per-frame SigLIP-2 embeddings. The result is a queryable action database (
graph.db + vector.db) that lets downstream stages select which clips to process via natural-language retrieval, instead of brute-forcing the full video.
- Reconstruction — containerized vision modules turn the selected RGB (or stereo) clips into per-frame depth, object masks, textured meshes, 6-DoF object poses, and SMPL human body parameters. Multi-view pipelines (
run_mv_hoi_reconstruction, run_mv_calibration) orchestrate the full reconstruction from a rosbag.
- Robotic Grounding — human motion (e.g. Arctic) is retargeted onto the target robot embodiment (Sharpa), then the reconstructed scene and retargeted motion drive Isaac Lab environments trained with RSL-RL PPO to produce deployable policies.
E2E workflow
An end-to-end workflow serves as a reference for connecting the repository's components into a
complete reconstruction-to-policy pipeline. The included example starts with a raw egocentric
human demonstration captured on an iPhone and continues through reconstruction, retargeting,
source-expert training, demonstration collection, GR00T N1.7 fine-tuning, and closed-loop
evaluation. It has been verified with both floating Sharpa and Vega Sharpa embodiments.
run_e2e.sh provides a reproducible launcher that orchestrates the stages and records their state
for inspection and resumption. A companion agentic skill uses the launcher under the hood to
guide setup and execution. The main path trains its own source expert; the bundled tissue-box ONNX
policy is an optional shortcut for users who want to start directly with GR00T fine-tuning.
The complete workflow can run on a single NVIDIA RTX A6000 GPU with 48 GB of VRAM and has also
been verified on NVIDIA L40 and L40S GPUs. Follow the E2E workflow guide
for the runnable tissue-box demo, setup requirements, acceptance gates, and resume behavior.
Packages
| Package | Role | Runtime |
|---|
video_ingestion_agent/ | Video → action segments + entity scene graph + frame embeddings. LangGraph pipeline (segment → verify/refine → entity graph → embeddings) plus an EGAgent-style natural-language retrieval agent and an optional Gradio UI. | Python venv + vLLM server |
reconstruction/ | Video → depth, masks, meshes, 6D poses, human body. 18 containerized modules + multi-view pipelines. | Docker (per-module images) |
robotic_grounding/ | RL training on NVIDIA Isaac Lab 2.3.1 with RSL-RL (PPO); motion retargeting utilities. | Docker (nvcr.io/your-org/your-team) |
robotic_grounding/flash_chord/ | Accelerated reference-tracking training with PPO and FlashSAC; setup and complete example workflow. | Separate Newton / Warp / JAX container |
groot_finetune/ | Expert demonstration collection, LeRobot dataset conversion, GR00T N1.7 fine-tuning, and closed-loop evaluation. | Python 3.10 + external Isaac-GR00T checkout |
Prerequisites
The separate v0.2 multi-view HOI object-reconstruction pipeline is validated on RTX A6000 (SM 86)
and L40S (SM 89). Blackwell GPUs with compute capability 12.0 (sm_120),
including RTX PRO 6000 Blackwell, are not supported by the TensorRT and cuVSLAM
versions in v2d_cusfm. The reconstruction container build and pipeline
launcher check this before starting; see the
HOI GPU compatibility notes.
This is the beginning of the README. Read the rest at the original. Read the full README ↗
