Video to Data (V2D)

An end-to-end pipeline that converts human demonstration videos into simulation-ready assets and physics-grounded robot training data.

Documentation · Datasets (Hugging Face) · Robotic Grounding Project Page · Robotic Grounding Tech Report

---

Contents

• Overview
• E2E workflow
• Packages
• Prerequisites
• Quickstart
  • Video Ingestion Agent
  • Reconstruction
  • Robotic Grounding
  • FlashCHORD accelerated training
• Design philosophy
• Contributing

---

Overview

Video to Data (V2D) turns raw human demonstrations into robot-ready training data through three composable stages and a repository-root E2E workflow. Each stage runs independently and writes its artifacts to disk, so you can stop, inspect, cache, and recompose the pipeline at any boundary.

1. Video Ingestion Agent — a LangGraph-driven agentic workflow that segments demonstration videos into temporally-bounded action clips, extracts an entity-relation scene graph, and stores per-frame SigLIP-2 embeddings. The result is a queryable action database (graph.db + vector.db) that lets downstream stages select which clips to process via natural-language retrieval, instead of brute-forcing the full video.
2. Reconstruction — containerized vision modules turn the selected RGB (or stereo) clips into per-frame depth, object masks, textured meshes, 6-DoF object poses, and SMPL human body parameters. Multi-view pipelines (run_mv_hoi_reconstruction, run_mv_calibration) orchestrate the full reconstruction from a rosbag.
3. Robotic Grounding — human motion (e.g. Arctic) is retargeted onto the target robot embodiment (Sharpa), then the reconstructed scene and retargeted motion drive Isaac Lab environments trained with RSL-RL PPO to produce deployable policies.

E2E workflow

An end-to-end workflow serves as a reference for connecting the repository's components into a complete reconstruction-to-policy pipeline. The included example starts with a raw egocentric human demonstration captured on an iPhone and continues through reconstruction, retargeting, source-expert training, demonstration collection, GR00T N1.7 fine-tuning, and closed-loop evaluation. It has been verified with both floating Sharpa and Vega Sharpa embodiments.

run_e2e.sh provides a reproducible launcher that orchestrates the stages and records their state for inspection and resumption. A companion agentic skill uses the launcher under the hood to guide setup and execution. The main path trains its own source expert; the bundled tissue-box ONNX policy is an optional shortcut for users who want to start directly with GR00T fine-tuning.

The complete workflow can run on a single NVIDIA RTX A6000 GPU with 48 GB of VRAM and has also been verified on NVIDIA L40 and L40S GPUs. Follow the E2E workflow guide for the runnable tissue-box demo, setup requirements, acceptance gates, and resume behavior.

Packages

Package	Role	Runtime
video_ingestion_agent/	Video → action segments + entity scene graph + frame embeddings. LangGraph pipeline (segment → verify/refine → entity graph → embeddings) plus an EGAgent-style natural-language retrieval agent and an optional Gradio UI.	Python venv + vLLM server
reconstruction/	Video → depth, masks, meshes, 6D poses, human body. 18 containerized modules + multi-view pipelines.	Docker (per-module images)
robotic_grounding/	RL training on NVIDIA Isaac Lab 2.3.1 with RSL-RL (PPO); motion retargeting utilities.	Docker (nvcr.io/your-org/your-team)
robotic_grounding/flash_chord/	Accelerated reference-tracking training with PPO and FlashSAC; setup and complete example workflow.	Separate Newton / Warp / JAX container
groot_finetune/	Expert demonstration collection, LeRobot dataset conversion, GR00T N1.7 fine-tuning, and closed-loop evaluation.	Python 3.10 + external Isaac-GR00T checkout

Prerequisites

• Docker with GPU support (install)
• NVIDIA Container Toolkit
• Python 3.10+
• NVIDIA driver 580.126.09 / CUDA 13.0 recommended (for robotic_grounding)

The separate v0.2 multi-view HOI object-reconstruction pipeline is validated on RTX A6000 (SM 86) and L40S (SM 89). Blackwell GPUs with compute capability 12.0 (sm_120), including RTX PRO 6000 Blackwell, are not supported by the TensorRT and cuVSLAM versions in v2d_cusfm. The reconstruction container build and pipeline launcher check this before starting; see the HOI GPU compatibility notes.

Quickstart

Video Ingestion Agent (video → queryable action database)

cd video_ingestion_agent

uv venv .venv && source .venv/bin/activate
uv pip install -e ".[all]"     # vLLM, webapp, benchmark, dev tools

# 1. Start the vLLM server (loads the VLM, ~1 minute)
python scripts/serve.py -c configs/ingestion.yaml

# 2. Ingest a video — segmentation → entity graph → report
python scripts/run_ingestion.py path/to/video.mp4 \
  -c configs/ingestion.yaml --no-verify -o runs/my_run

# 3. Retrieve clips with natural language
python scripts/run_retrieval.py "Find clips where someone picks up a mug" \
  -d outputs/ -c configs/retrieval.yaml

# 4. Or browse interactively in the web UI
python scripts/run_webapp.py

See video_ingestion_agent/README.md for hardware requirements, the full extras list, the verify/refine loop, and batch-ingestion across multiple GPUs. Pre-publication TODOs are tracked in video_ingestion_agent/docs/release_readiness.md.

Reconstruction (video → 3D data)

cd reconstruction

# Install host-side orchestration wrappers (lightweight, no model dependencies)
./scripts/install_pacakages.sh

# Build per-module Docker images
./scripts/build_containers.sh

# Run a minimal video→depth example (MoGe)
python -m v2d.moge.docker.run_download_weights --output_dir data/weights/moge
python -m v2d.moge.docker.run_video_to_depth \
  --video_path modules/v2d_moge/assets/test_video.mp4 \
  --depth_folder data/outputs/moge/depth \
  --intrinsics_folder data/outputs/moge/intrinsics \
  --weights_path data/weights/moge

Full multi-view HOI pipeline (rosbag → textured object mesh + SMPL body):

python -m v2d.pipelines.run_mv_hoi_reconstruction \
  --rosbag_path /data/rosbags/session1 \
  --output_dir  /data/datasets/session1 \
  --calibration_camera_params_path /data/datasets/calibration/extrinsics/edex \
  --obj_mesh_path /data/meshes/object.glb

See reconstruction/README.md for the complete module reference, including Grounding DINO, SAM2, FoundationPose, SAM3D-Body, and others.

Robotic Grounding (data → RL policy)

Accelerated training: use FlashCHORD's setup and workflow for Newton/Warp/JAX training on the included Sharpa and G1 examples. FlashCHORD has its own container and dependencies.

Quick start: the from-scratch setup & run guide is robotic_grounding/docs/SETUP.md — it covers the two Docker images, downloading each dataset from its original public source, the directory layout, and how to run the full hand→robot retargeting pipeline.

Throughout, <HMD> (human-motion-data root) is a directory you choose — e.g. ~/datasets/human_motion_data — that holds mano/ and one subdirectory per dataset (taco/, hot3d/, …); see docs/SETUP.md §4.

cd robotic_grounding

# One-time host setup (git-lfs, pre-commit) + robot assets (LFS)
bash workflow/setup_deps.sh
git lfs pull

# Build both pipeline images (loader + robotic-grounding) in one shot
python scripts/run_pipeline_docker.py --build-only

# Run the full pipeline on a dataset (download it first per docs/SETUP.md §6).
# <HMD> is the data root holding mano/ and each <dataset>/.
python scripts/run_pipeline_docker.py taco \
    --hmd <HMD> --mano-dir <HMD>/mano --max-sequences 2     # small smoke test

Reproduce the sequences end-to-end (arctic / hot3d / taco) in a self-contained workspace — see robotic_grounding/docs/EXAMPLE_SEQUENCES.md for the sequence list and prerequisites:

HMD=<HMD> ./run_example_sequences.sh        # → RL-ready parquets under <HMD>/example_sequences/<ds>/<ds>_processed/

Then enter the Isaac Lab container and train a policy on the retargeted motion:

./workflow/run.sh build
./workflow/run.sh start [version] [gpu_id]              # build + enter the container
python scripts/rsl_rl/train.py --task Sharpa-V2D-v0    # inside the container

See robotic_grounding/README.md for retargeting, debug environments, and task definitions.

Visualizer (retargeting gallery)

Browse retargeted sequences as interactive 3D animations in your browser.

See robotic_grounding/README.md#visualizer for setup instructions.

Design philosophy

• Host orchestration, containerized inference. The host runs thin Python wrappers that docker run each module; all model dependencies live inside their respective images. No CUDA or PyTorch is ever installed on the host.
• Typed contracts between packages. Modules communicate through strongly-typed dataclasses in v2d_common (DepthImage, CameraIntrinsics, Transform3d, BoundingBox, Mask) — never raw arrays across package boundaries.
• File-based dataflow. Modules write intermediate artifacts to disk (depth PNGs, pose JSONs, mask PNGs, etc.), enabling independent execution, caching, and pipeline composition via v2d_pipelines.

Contributing

See the contributing guide in reconstruction/README.md for adding new reconstruction modules. Each new module must expose a Docker image, a run_download_weights entry point (if weights are required), a run_shell entry point, and a typed API surface consistent with v2d_common.