MomentBench

momentbench.com: corpus-scale video moment retrieval benchmark.

Design target: 75 hours of rights-cleared video, 500 queries across 4 difficulty tiers, evaluated against frontier models and retrieval pipelines.

Current release (v0.2.5): 27 hash-verified assets in three domains: 7 Blender open movies (about 72 minutes), 13 NASA aerospace videos (about 32 minutes, 2022 to 2026) and 7 NASA robotics videos (about 13 minutes, 2019 to 2023), with machine transcripts in data/transcripts. The queries, transcripts, runs and the 25 videos with confirmed licences are also on Hugging Face at Mixpeek/momentbench. Queries:

• data/queries/film_v0.2.json: 184 public film queries across all four tiers; 82 more are held out.
• data/queries/aerospace_v0.2.json: 202 NASA queries across all four tiers, 164 public and 38 rolling (footage published after current model training cutoffs); 65 more are held out.
• data/queries/robotics_v0.2.json: 70 public NASA robotics queries across all four tiers; 16 more are held out.
• data/queries/sample.json: the 4-query v0.1 seed, kept for comparison with earlier runs.

In total 619 queries across film, aerospace and robotics, 456 in this repository and 163 held out (committed by hash in data/heldout_commitments.json). Claude drafted, verified and reviewed them; then two model families other than Claude judged every published query, and the 333 both accepted form the scored set in data/queries/scored/ that results use (docs/labels.md, the label log). The labels are model-made and model-adjudicated, with no human review yet. The technical report is docs/report.md and the leaderboard is docs/leaderboard.json, ranked only within compatibility groups. docs/protocol.md defines the task, metrics and what a reported result must state. docs/input-spec.md lists exactly what a model sees, for reimplementing the track in another harness. docs/scope.md states what a score measures and what it does not. PRs adding rights-cleared footage or queries are welcome (see SUBMISSION.md).

Maintained by Mixpeek. Mixpeek's own pipeline is one entrant on the leaderboard and follows the same rules as every other. See GOVERNANCE.md.

Start here: a validated result in four steps

docs/bring-your-system.md is the shortest path, and the first two steps need no API key and no model. Clone main here: it carries the v0.2.5 queries, manifests, transcripts and scorer unchanged, plus the prediction-file tools added after the tag (scripts/bm25_predictions.py, scripts/convert_predictions.py). Results still cite v0.2.5.

git clone https://github.com/mixpeek/momentbench.git && cd momentbench
pip install -e .
momentbench corpus-download -m data/diagnostic/manifest.json --corpus-dir diag/
momentbench diagnose --corpus-dir diag/          # 5 checks, all should read PASS

1. Check your setup with diagnose (above, a few minutes).
2. Score a retrieval system on the prediction-file track: write ranked (asset_id, t_in, t_out) per query, then momentbench score-predictions -p preds.json -q data/queries/scored/film_v0.2_scored.json. scripts/bm25_predictions.py is a complete example that writes such a file, and docs/prediction-files.md converts QVHighlights-style or CSV output you already have.
3. Get a citable result: momentbench bundle -r <run>.json writes JSON, a model-card block, a LaTeX table and a citation.
4. Submit with momentbench submit and a pull request: a check validates it and a valid run is merged and published to momentbench.com without a maintainer.

Quick Start: the agentic track

Evaluate a model on the v0.2 agentic track from an empty directory. MomentBench is not on PyPI: the repository carries the queries, manifest and transcripts the commands read, so clone a release tag and work inside it. ffmpeg and ffprobe must be on the PATH (brew install ffmpeg, apt install ffmpeg).

git clone --branch v0.2.5 https://github.com/mixpeek/momentbench.git
cd momentbench
pip install -e ".[openai]"   # Python 3.10+; or .[gemini], .[claude-code], .[inspect]
momentbench corpus-download -m data/corpus_manifests/sample_manifest.json --corpus-dir corpus/   # 2.6 GB
momentbench agentic <model> --provider openai-compatible --base-url http://localhost:8000/v1 \
  -q data/queries/scored/film_v0.2_scored.json -c data/corpus_manifests/sample_manifest.json \
  --corpus-dir corpus/ --anonymize-titles

--provider is one of openai-compatible (vLLM, SGLang, Ollama and other /v1/chat/completions servers), openai, gemini, anthropic, claude-code or baseline. docs/integrating.md covers each route, including a runnable template for your own agent (examples/custom_agent.py) and the Inspect AI task. Current scores are in docs/report.md and on momentbench.com; for a comparable row use the scored query files and --anonymize-titles, as the record runs do.

For world-model and physical-AI teams

MomentBench asks whether a system can find one specific moment in a large corpus of video. These parts of it target physical understanding directly:

• Robotics domain (data/queries/robotics_v0.2.json): NASA rovers on slopes and ramps, free-flying robots and a Mars helicopter, from test labs and from flight.
• Physical-event subset: queries tagged metadata.physical_event, in which a machine visibly moves or makes contact (a wheel onto a ramp, a rocket leaving the pad, a capsule under drogue parachutes, a splashdown). Run only these with --physical-only, or -T physical_only=true under Inspect.
• Visual-only ablation: --no-transcripts removes speech, so a score reflects what the model saw rather than what was said.
• Title control: --anonymize-titles hides titles and descriptive asset ids, so a model cannot pick a video by its name.
• Temporal precision: Recall@1 is reported at tIoU 0.3, 0.5 and 0.7, and tier 4 asks for ordered cause-and-effect across consecutive shots.

What it does not have yet: robot manipulation (grasping and placing), egocentric footage or driving. Openly licensed robot-learning datasets are candidates for v0.3, each pending its own licence review. PRs with rights-cleared footage are welcome.

Query Tiers

Difficulty targets were pre-registered before any baseline evaluations were run.

Tier	Name	Description	Pre-registered difficulty target
1	Single-condition visual	Identify a single visual event	>80% recall for frontier models
2	Cross-modal joint	Joint reasoning across modalities	50-70% recall
3	Corpus-scale hard negatives	Discriminate near-duplicates at corpus scale	<35% recall
4	Temporal reasoning	Temporal ordering and sequence reasoning	<25% recall

Verticals

Target scope per vertical, with the v0.2 asset count:

• Film (target ~20h, v0.2: 7 assets): Blender open movies; Prelinger Archives and National Archives planned
• Aerospace (target ~25h, v0.2: 13 assets): NASA mission recaps, explainers and NASA Minute episodes
• Robotics (v0.2: 7 assets): NASA rover, free-flyer and helicopter footage, from flight and from test labs. Queries target physical events: contact, wheels on slopes, lift-off and landing, cause then effect
• Civic (target ~30h, v0.2: none): congressional hearings and agency briefings are planned. The v0.1 Senate asset was dropped because its source link no longer resolves

Scoring

• Hit: correct asset AND temporal IoU ≥ 0.5 vs gold moment
• Metrics: Recall@1, Recall@5, mean temporal IoU, per-tier breakdowns
• No-answer queries: a query with an empty gold_moments list has no match in the corpus. It is a hit only when the system returns no moments. On the agentic track that takes an explicit empty submission; a refusal or a run that hits its limit is a miss. Tier 3 in v0.2 is made of these: real shots with one concrete detail changed.

Getting the Corpus

momentbench corpus-download -m data/corpus_manifests/sample_manifest.json --corpus-dir corpus/
momentbench corpus-verify   -m data/corpus_manifests/sample_manifest.json --corpus-dir corpus/

corpus-download fetches every asset whose manifest entry has a file block (all 27 in v0.2.5) and checks its sha256. corpus-verify checks presence, sha256 and duration for every asset and exits non-zero if any fails.

Agentic Track

The agentic track measures a tool-using model. The model cannot watch video. It gets five tools (list_assets, search_transcript, read_transcript, sample_frames, submit_answer) and a per-query budget of frames, tool calls and turns, and it must submit a video and a time span.

momentbench agentic claude-opus-5-5 \
  -q data/queries/scored/film_v0.2_scored.json -c data/corpus_manifests/sample_manifest.json \
  --corpus-dir /path/to/videos --anonymize-titles --effort high --trials 5

• Corpus on disk: --corpus-dir holds <asset_id>.mp4 (or .mov, .mkv). Transcripts are read from --transcripts-dir, default data/transcripts. Queries whose gold video is missing are skipped, and the run says how many.
• Repeated trials: each query runs --trials times. Recall@1 is the mean over trials, with a 95% bootstrap interval resampled over queries.
• Cost: token usage is recorded per trial and priced from configs/pricing.json, which carries an as-of date. Report Recall@1 against cost per query at each effort level.
• Failures count: a trial that refuses, hits the turn limit or never submits scores zero and stays in the denominator.
• No retrieval index: transcript search is plain keyword overlap, so no entrant depends on a vendor's embeddings.

The same track runs under Inspect AI, with the same tools, budget and scoring rule:

pip install -e ".[inspect]"
inspect eval momentbench/agentic/inspect_task.py --model anthropic/claude-opus-5-5 \
  -T corpus_dir=/path/to/videos --epochs 1
# defaults to the film, aerospace and robotics v0.2 files; -T queries=a.json,b.json to change

To run through a local Claude Code install instead of an API key, add --provider claude-code (and pip install -e ".[claude-code]"). The Claude Agent SDK runs the tool loop with Claude Code's built-in tools, settings and other MCP servers switched off and the benchmark's system prompt in place of Claude Code's. The corpus tools charge the same budget. Each run records harness: "claude-code" or harness: "api", so results from the two are never mixed silently.

GPT and Gemini run with --provider openai (needs OPENAI_API_KEY, pip install -e ".[openai]") and --provider gemini (needs GEMINI_API_KEY, pip install -e ".[gemini]"). Every provider gets the same tools and budget, and frames go back inside the tool result at the same resolution. OpenAI frames are sent at detail: high so they are not downscaled.

Requires ffmpeg on the PATH. Other providers implement momentbench.agentic.model.AgentModel. Results are in docs/results.md, generated by scripts/results_doc.py from the run files it cites.

Contamination Control

Every query carries a canary string, and queries are assigned to a public, held_out or rolling split. Held-out queries are kept outside this repository and scored by the maintainers. Rolling queries use footage published after 2026-06-30. See docs/contamination.md.

Corpus Licensing

Footage is US government work in the public domain (17 U.S.C. § 105) or Blender open movies under CC BY. Two exceptions are recorded per asset in the manifest (license_status, is_sellable): Spring and Agent 327 are pending licence confirmation from Blender Studio, and the Tears of Steel soundtrack is CC BY-ND. Check those fields before redistributing or selling. See ATTRIBUTION.md for full provenance.

Project Structure

momentbench/
├── momentbench/           # Core library
│   ├── eval/              # Scoring engine
│   ├── models/            # Legacy single-call adapters (see input_kind)
│   ├── corpus/            # Corpus management & sourcing
│   ├── taxonomy/          # Query tier definitions & generation
│   └── mixpeek_integration/  # Mixpeek pipeline integration
├── site/                  # Leaderboard (momentbench.com)
├── scripts/               # Automation scripts
├── configs/               # Benchmark configuration
├── data/                  # Queries, manifests, results
└── tests/                 # Test suite

Legacy adapters (momentbench run)

For multimodal evaluation use the agentic track (momentbench agentic): the model samples frames and reads transcripts through tools, and every reported number on this page comes from it. The adapters below are the older single-call track. Each declares an input_kind, which momentbench run prints and records in the run's config:

• metadata-only: the model sees asset titles and durations, no frames, audio or transcript. It measures guessing from titles.
• whole-video-control: ranks videos by title/description text and returns the whole video as the span. A retrieval floor, with no localization.
• indexed-platform: queries a provider's index built from the media; segment times come from the provider.

Adapter	Model	Input kind
gemini	Gemini 2.5 Pro	metadata-only
openai	GPT-4.1	metadata-only
claude	Claude Sonnet 4	metadata-only
qwen-vl	Qwen3-VL / Qwen2.5-VL (vLLM, SGLang)	metadata-only
hf-vlm	Any OpenAI-compatible VLM server	metadata-only
clip	OpenAI CLIP (ViT-L/14)	whole-video-control
jina-clip	Jina CLIP v2	whole-video-control
mixedbread	mxbai-embed-large	whole-video-control
e5-omni	E5-Omni (Microsoft)	whole-video-control
twelvelabs	TwelveLabs Marengo	indexed-platform
mixpeek	Mixpeek pipeline	indexed-platform

# Run API-based models
TWELVELABS_API_KEY=... momentbench run twelvelabs -q data/queries/sample.json -c data/corpus_manifests/sample_manifest.json
MIXEDBREAD_API_KEY=... momentbench run mixedbread -q data/queries/sample.json -c data/corpus_manifests/sample_manifest.json

# Run OSS models (requires a local inference server or GPU)
QWEN_API_BASE=http://localhost:8000/v1 momentbench run qwen-vl -q data/queries/sample.json -c data/corpus_manifests/sample_manifest.json
momentbench run clip -q data/queries/sample.json -c data/corpus_manifests/sample_manifest.json

# Run only OSS baselines
OSS_ONLY=1 ./scripts/run_all_baselines.sh

# Run only closed-source baselines
CLOSED_ONLY=1 ./scripts/run_all_baselines.sh

Install OSS dependencies: pip install -e ".[oss]"

Submit Results

We welcome external submissions. Run the agentic track on a whole query file, package the run, and open a PR adding the directory it writes under submissions/. SUBMISSION.md has the steps and what is checked.

momentbench submit \
  --results data/results/agentic/agentic-<model>-<id>.json \
  --model-name "Your Model" \
  --contact [email protected]

Run Your Own Footage

# Ingest your video corpus into Mixpeek
momentbench ingest -m your_manifest.json

# Run decomposition
momentbench decompose -m your_manifest.json

# Evaluate
momentbench run mixpeek -q your_queries.json -c your_manifest.json

Citation

@misc{momentbench2026,
  title     = {MomentBench: Corpus-Scale Video Moment Retrieval Benchmark},
  author    = {Mixpeek},
  year      = {2026},
  version   = {0.2.5},
  url       = {https://github.com/mixpeek/momentbench},
  license   = {Apache-2.0}
}

License

Apache-2.0