Ghost Data Still Exists – deletion audit agent

A deletion workflow says a customer is gone. This tool checks whether that is actually true.

For each of the 518 deletion requests it reads the request, works out every identifier the person ever had, reads all 94 evidence files (database dumps, logs, JSON exports, a handover note, PDFs, screenshots) and the activity stream, applies the company's deletion rules to every record it finds, and writes a verdict with an ordered list of findings, fixes and checks to output.csv.

Everything that decides a case is ordinary, repeatable code. A vision model is used only to read the 11 screenshots; each one is read once and cached by file hash.

Results

Requests audited	518: 153 COMPLETE, 355 INCOMPLETE, 10 NOT_APPLICABLE (not a deletion request)
Verdict accuracy on the 32 public samples	32 / 32
Exact step-list match	23 / 24 development samples, 5 / 8 held-out samples
State and evidence citation on every matched step	1.000 / 1.000
Final run	one cold run, 83 s, 11 model calls, 54,033 tokens, $0.21; a warm re-run takes 2 s and makes no calls
Determinism	two runs of the same code produce byte-identical output

Details, the held-out analysis, the custom test cases and the list of failures we found and fixed are in code/evaluation/evaluation_report.md. The token and cost ledger of the final run is in code/evaluation/usage_report.md.

Quick start

Requirements: Python 3.11 or newer; Windows, macOS or Linux. The challenge data is not in this repository: copy the official dataset/ folder from https://github.com/interviewstreet/hackerrank-orchestrate-october26 into the project root so that dataset/deletion_requests.csv exists.

Windows (PowerShell):

python -m venv .venv; .\.venv\Scripts\Activate.ps1; pip install -r code/requirements.txt; Copy-Item .env.example .env

macOS / Linux:

python3 -m venv .venv && source .venv/bin/activate && pip install -r code/requirements.txt && cp .env.example .env

Put your key in .env (ANTHROPIC_API_KEY=sk-ant-...), then run everything:

python code/main.py

That reads dataset/, writes output.csv in the project root and prints the verdict counts, model calls and cost. Without a key the run still completes; only the 11 screenshots are skipped and the run says so.

Other commands:

Command	What it does
python code/main.py --clear-cache	cold run: wipe .cache/, re-read every file, re-call the model
python code/main.py --offline	never call the model (cached screenshot readings only)
python code/main.py --cases sample	the 32 public samples, written to .cache/runs/sample_output.csv
python code/main.py --cases dev / --cases holdout	the 24 development samples / the 8 held back
python code/evaluation/main.py --predictions .cache/runs/sample_output.csv --truth dataset/sample_output.csv	score a run against the public answers
python code/evaluation/main.py --validate output.csv --requests dataset/deletion_requests.csv	contract check: 518 rows, order, allowed operations, evidence rules
python code/evaluation/custom_cases.py	our nine extra test cases
python code/tools/inspect_case.py case_26	everything the engine can see for one case
python -m pytest code/tests -q	120 unit tests

How it works

dataset/ ──► loaders ──► identity (who is this person, under every name they ever used)
                    ├──► intent   (is this really a deletion request?)
                    ├──► evidence (94 files → normalized "observations")   ← vision model for screenshots only
                    └──► activity (sync / restore / block events as claims, never as proof)
                                 │
                                 ▼
             state (what is true for each record at audit_at) ──► policy (what the rules require)
                                 │
                                 ▼
             planner (findings → remediation → verification, fixed order) ──► writer ──► output.csv

Path	Purpose
code/main.py	entry point and pipeline
code/ghost/	the engine: identity, intent, activity, evidence/, state, policy, planner, writer
code/ghost/llm.py	the only place that talks to the model: cache + usage ledger
code/prompts/	the three prompts (request intent, screenshot extraction, PDF extraction)
code/config.json	model names, prices, batch size, held-out list (no secrets)
code/evaluation/	scorer, custom cases, evaluation_report.md, usage_report.md, notes
code/tools/	packaging, viewer, demo recorder, case inspector
.cache/	model answers and the usage ledger (git-ignored, not in the submission)

This is the beginning of the README. Read the rest at the original. Read the full README ↗