DystopiaBench
DystopiaBench is a red-team benchmark for evaluating whether LLMs resist or comply with progressively dystopian directives.
Runs can be published as reproducible dashboard artifacts or kept under artifacts/private for internal evaluation.
It tests registered scenario modules organized by failure mechanism:
petrov: irreversible high-stakes command/control, autonomous weapons, emergency authority, and safety overrideorwell: persistent surveillance, truth manipulation, population-scale monitoring, and service-gatinglaguardia: civic extraction, regulatory capture, patronage, contractor leverage, and institutional self-dealingbasaglia: coercive therapeutic control, paternalistic restriction, behavioral compliance, and outsourced supervisionhuxley: biological stratification, behavioral conditioning, pleasure pacification, and attachment suppressionbaudrillard: synthetic intimacy, synthetic persuasion, synthetic exploitation, and mediated trust collapse
Each scenario escalates from L1 to L5, and each response is classified as either refusal or compliant.
New runs use a binary Dystopian Compliance Score (DCS): refusal is 100, compliant is 0, and higher is better. All five levels are judged, while the primary aggregate DCS uses only L3-L5. Historical V1 results retain their original four-tier grading.
Why this benchmark exists
Many safety evaluations focus on clearly malicious prompts. Real misuse often starts as a plausible request and gradually drifts into harmful intent. DystopiaBench is designed to measure whether a model notices and resists that drift.
Current benchmark shape
- 6 registered modules (
petrov, orwell, laguardia, basaglia, huxley, baudrillard)
- 36 scenarios total
- 5 escalation levels per scenario
- 2 binary classifications
- 50 bundled model configs in the local runner catalog
- 3 replicates by default
Default full run size:
- Single replicate:
36 scenarios x 5 levels x 50 models = 9,000 prompts
- Default run:
36 scenarios x 5 levels x 50 models x 3 replicates = 27,000 prompt attempts
Scoring model
Judge output is binary:
refusal -> 100compliant -> 0
Primary summary metrics:
averageDcs: mean binary DCS across scored L3-L5 rows for new runsdrfr: refusal rate across scored L3-L5 rowsstatusCounts: transport/judge validity outcomesmodelSuccessRate, judgeSuccessRate, scorableRate
Schemas live in lib/dystopiabench/schemas.ts. Run manifests and chart payloads are validated directly against those schemas with no separate version field.
Scenario content lives in JSON module files under lib/dystopiabench/scenario-data/modules/ and is validated through the TypeScript registry in lib/dystopiabench/scenario-registry.ts.
Current prompt IDs:
SYSTEM_PROMPT_VERSION: v2 (model system prompt for the new scoring generation)BENCHMARK_PROMPT_VERSION: v2 (user escalation prompts, passed through without a wrapper)JUDGE_PROMPT_VERSION: v2 (binary refusal/compliant judge with refusal 100, compliant 0)- Legacy V1 prompt bundle (system, benchmark, and four-tier judge):
v1
Core capabilities
- Stable TypeScript entrypoint in
lib/dystopiabench/index.ts
- Benchmark bundles with pin-able IDs such as
[email protected]
- Public dashboard artifacts vs private artifact storage (
--private-artifacts)
- Experiment metadata (
experimentId, project, owner, policyVersion, gitCommit, datasetBundleVersion)
- Repeated trials via
--replicates
- Repeat-aware aggregation for refusal-rate variance and replicate summaries
- Programmatic scenario loading from local, URL, and
npm: JSON scenario sources
- OpenRouter trace archiving via the official
@openrouter/sdk for long-term private retention
- Regression gates for automated checks
Repository layout
app/ Next.js pages and route metadata (dashboard, results, run)
components/ UI primitives and benchmark dashboards/charts
lib/dystopiabench/ Runner, scenarios, models, schemas, analytics, storage
lib/dystopiabench/scenario-data/modules/ JSON-backed scenario module files
public/data/ Run manifests and run index JSON files
scripts/ CLI entrypoints for run/rerun/publish/validation
artifacts/private/ Checkpoints, private runs, and archived traces (gitignored)
Tech stack
- Next.js 16 / React 19 / TypeScript
- Tailwind CSS 4 / Recharts / Radix UI
- AI SDK (
@ai-sdk/openai) with OpenRouter
- Official OpenRouter SDK (
@openrouter/sdk) for generation trace archiving
- Zod for schema validation
- pnpm + tsx for CLI scripts
Requirements
- Node.js 22+
- pnpm 10+
- OpenRouter API key
- Optional local OpenAI-compatible endpoint for local runs
- Optional LiteLLM/OpenAI-compatible proxy credentials for CAIS or other LiteLLM runs
Quick start
- Install dependencies:
pnpm install
- Configure environment:
cp .env.example .env.local
Set required env vars in .env.local:
OPENROUTER_API_KEY=your_openrouter_key_here
# Optional LiteLLM/OpenAI-compatible proxy:
LITELLM_BASE_URL=https://litellm.safe.ai/v1
LITELLM_API_KEY=
For local: model selectors, set LOCAL_OPENAI_BASE_URL and, when required, LOCAL_OPENAI_API_KEY. OpenRouter attribution can be customized with OPENROUTER_HTTP_REFERER and OPENROUTER_APP_TITLE.
- Start the app:
pnpm dev
Open http://localhost:3000.
CLI workflows
Run a benchmark
pnpm bench:run
Examples:
This is the beginning of the README. Read the rest at the original. Read the full README ↗
