$\tau$-Bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains

τ-bench 2.0

τ-bench 2.0 adds τ-i18n (also called τ-multi): the same customer-service tasks in six languages, in text and in full-duplex voice, plus post-hoc judges for how the agent speaks.

What's new:

• Six languages. --language selects a language pack: English, Spanish, Portuguese, Hindi, Chinese and Korean. Each pack localizes the tasks, the database and the caller personas for airline, retail and telecom.
• Voice in every language. Full-duplex voice runs with realtime agents (OpenAI, Gemini, xAI and others) against a simulated caller who speaks the language.
• New intake domain (τ-Entity): capturing names, addresses and other entities exactly on an outbound call, over 200 tasks. See τ-Entity.
• banking_knowledge in voice. The retrieval domain from v1.0 now also runs as a custom voice run (--audio-native --benchmark none).
• Judges. tau2 judges scores naturalness, speech fidelity, tool use and language-specific checks after a run. Judges do not change the task reward.
• Released benchmarks. Frozen task sets with pinned data, so runs can be checked and compared.

The benchmarks

Benchmark	Mode	Cells	Calls at 1 trial	Run with
text-v1	text	6 languages × 3 domains	900	tau2 run --benchmark text
voice-v2 (τ-i18n)	voice	6 languages × 3 domains	900	tau2 run --audio-native
voice-v1	voice	English × 3 domains	278	tau2 run --audio-native --benchmark voice-v1

The domains are airline, retail and telecom. text-v1 and voice-v2 use the same 50 frozen tasks per domain in every language. voice-v1 is the English-only frame of the current English voice leaderboard (50 airline, 114 retail, 114 telecom tasks). See the glossary for cell, frame and the other terms used in these docs.

Quickstart

git clone https://github.com/sierra-research/tau2-bench
cd tau2-bench
uv sync --extra voice          # drop --extra voice for text only
cp .env.example .env           # add OPENAI_API_KEY (and voice keys for voice)

Text benchmark:

tau2 run --benchmark text --agent-llm openai/gpt-5.5 \
  --agent-llm-args '{"reasoning_effort":"low"}' --save-to text-run --dry-run

Voice benchmark (create your own caller voices first; the shipped voice IDs are Sierra-internal):

python -m tau2.voice.scripts.setup_voices --language all   # prints TAU2_VOICE_ID_... lines for .env
tau2 run --audio-native --audio-native-model gpt-realtime-2 \
  --reasoning-effort minimal --save-to voice-run --dry-run

--dry-run prints the plan without making calls. Remove it to run, then:

tau2 view --file data/simulations/voice-run      # scores and calls
tau2 judges suite data/simulations/voice-run     # judges (needed for a voice-v2 submission)

Add --domain retail --num-tasks 2 for a 12-call smoke test. Full details: Running the benchmark and the submission guide.

Comparing with v1.0.1

2.0 English runs are not input-identical to v1.0.1, so 2.0 scores are not comparable with the 1.x leaderboard. The differences:

• English retail and telecom each use 50 reviewed task replacements.
• The prompts add a language clause and an entity-order rule (keep multi-part values such as addresses in the task's order).
• The text user simulator gets a persona block.
• Communication checks use a semantic LLM judge instead of substring matching.
• The NL-assertion judge is gpt-5.4-mini-2026-03-17 (was gpt-4.1).
• Name, ID and phone lookups fold case, accents and formatting.
• Voice defaults: user simulator gpt-5.5 (was gpt-4.1), a 2400-second call limit (was 1200) and agent voice marin (was alloy).

For scores comparable to the 1.x leaderboard, use the v1.0.1 release. 1.x and 2.0 results are shown on separate leaderboard tabs, and 1.x submissions are still accepted. The full list is in the CHANGELOG.

Previous release: τ³-bench (v1.0)

τ³-bench added voice full-duplex evaluation, the banking_knowledge domain and 75+ task fixes: τ-Voice paper · τ-Knowledge paper · task fixes paper · v1.0.0 release notes. v1.0.1 fixed banking_knowledge grading; results from before 1.0.1 are not comparable on that domain (see the CHANGELOG).

This is the beginning of the README. Read the rest at the original. Read the full README ↗