Derek

A plain-language compliance system for Australian Public Service content, built on the Australian Government Style Manual.

Derek finds text that breaks a style rule, flags it with a calibrated confidence, and lets the user keep the original or accept a change. It is the successor to Octavius.

Read docs/00-postmortem-octavius.md first. Octavius shipped 801 rules that, fired at the Style Manual's own professionally-edited prose, raised 1,438 findings per 1,000 words — roughly 1.4 violations per word on text that is by definition correct. Every rule passed its own test suite. Most of Derek's design is a direct response to a specific, documented failure there, and the postmortem says which.

---

What is different this time

	Octavius	Derek
Rule inventory	LLM proposed 3,114 "rules" from 181 pages	Read deterministically off document structure
Heading fidelity	trafilatura flattened levels; structure was guessed back	Levels taken from the source DOM
Reproducibility	Re-running produced a different rulebook	Byte-identical across runs; CI-enforced
Polarity	No field for it; 62% of rules detected the compliant pattern	violation_condition + separate compliant/violating examples, guarded three ways
Test data	Model-generated alongside the rule it tested	2,390 gold example lines harvested from the manual's own Write this / Not this blocks
Quality gate	None; product-level suppression hid the problem	Dogfood gate on raw output, in CI
Human review	3,114 rows, reviewed after the fact	Nothing loads until a human accepts it
Rules	Model-generated Python, exec()d with full builtins	Declarative data from a closed matcher vocabulary
Change detection	Trusted the sitemap's lastmod; never noticed removals	Content hashes, weekly full re-hash, explicit add/alter/remove changesets

---

Status

Foundations built; no rules reviewed yet — by design.

Offline Style Manual snapshot	186 pages
Eligible rule-source pages	128
Deterministic rule candidates	736
With at least one polarised gold example	170
With a paired compliant + violating example	169
Gold example sentences	812
Accepted into the runtime	0

Built and tested: the snapshot layer (content-addressed, with add/alter/remove changesets), deterministic extraction, the rule ledger, the dogfood gate, the triage UI, and the span annotator that is now the main way rules are identified (ADR-023). The corpus is frozen while the golden set is drawn (ADR-024).

Not yet done: the snapshot has not been run against the live site from this repository, and no rule has been triaged. The first triage pass is the next task — see docs/04-roadmap.md.

---

Quick start

pip install -r requirements.txt

# Rebuild the rule ledger from the corpus (deterministic, no model call at extraction time)
python -m derek.extract.build

# Prove reproducibility — fails if rebuilding would change the ledger
python -m derek.extract.build --check

# Fire accepted rules at the Style Manual's own prose
python -m derek.eval.dogfood

# Triage rules — keyboard-first: A accept, R reject, D defer, N not-text
python tools/review/server.py

# The same app as files, for a machine that can't run any of the above:
#   https://thomas-amann-ipaustralia.github.io/Derek/
python tools/review/build_static.py                     # → site/
python tools/review/apply_decisions.py export.jsonl     # decisions made there → ledger

# Refresh the offline snapshot (needs requirements-pipeline.txt)
python -m derek.corpus.snapshot --sweep-slice 0

# Reproduce the Octavius postmortem figures
python -m derek.eval.dogfood --octavius reference/octavius-v1/rules_working_draft.v1.jsonl

pytest tests/ -v

---

How it works

  stylemanual.gov.au
        │  Layer 0 — SNAPSHOT        content-addressed; add/alter/remove changesets
        ▼
  corpus/pages/**.md
        │  Layer 1 — EXTRACTION      pure function of the corpus; no model runs
        ▼
  736 candidates (stable uid, statement, heading path, gold examples)
        │  Layer 2 — INTERPRETATION  model proposes, human decides
        ▼
  ledger/rules.jsonl
        │  Layer 3 — DETECTION       accepted rules only
        ▼
  Tier 0 deterministic · Tier 1 classifier · Tier 2 generator (deferred)
        │  Layer 4 — INTERFACES
        ▼
  HTTP · MCP · CLI · LLM middleware · review UI (local + published)

Each layer is reproducible from the one below it. Full detail in docs/01-architecture.md.

Rules are read off the document, not invented

The Style Manual states its rules as headings, and publishes them with a clean outline (h1 page title → h2 section → h3 rule → h4 example block):

### Place a comma after adverbs and other introductory words
### Use a comma after phrases and clauses that change the whole sentence
### Avoid beginning a sentence with a string of numbers and dates

So the rule inventory is a pure function of the corpus. A model may later classify or formalise a candidate; it may never decide that one exists. That is what makes "two runs extract the same rules" true by construction rather than by hope.

This only works if the conversion preserves those levels. Octavius used trafilatura, which flattens them, so 71% of its structure had to be guessed back by a heuristic. Derek reads the levels from the DOM instead (ADR-020).

This is the beginning of the README. Read the rest at the original. Read the full README ↗