§pecs

From the new specification to a certified release — with AI that shows its work.

An AI review assistant for ITSG's system review of German payroll software, built at Futury Build Days #3 (26–27 Oct 2026) · challenge sponsor ITSG · tech sponsor hessian.AI

▶ Try the demo in your browser

Guided demo · Pitch deck · Live app (our hackathon server, real AI)

Quick start · Architecture · How it works · Trust model · Docs

---

📖 Contents

• About
• Highlights
• Screenshots
• Quick start
• Architecture
  • System context
  • Components
  • An AI run, end to end
  • Life of a spec section
  • Concurrency model
  • Deployment
• How it works
• Trust model
• Roles and accounts

---

💡 About

Every payroll program in Germany (SAP, DATEV, Lexware, ~100 in total) must pass a system review by ITSG before it may send reports to the social insurance institutions (§95b SGB IV). Each year the government publishes a new version of the specification — hundreds of pages — and today experts read it by hand, write test cases by hand, and vendor engineers type them into their software by hand.

§pecs automates that loop for income replacement benefits (Entgeltersatzleistungen, DTA EEL):

1. Import the new specification PDF next to last year's version.
2. Detect changes — the official change log, reconciled with a section-by-section diff (it catches changes the log forgot).
3. Understand each change with an LLM that quotes the spec word for word; an examiner approves, edits or declines it.
4. Generate test cases whose expected results are computed by tested code, not by the AI.
5. Hand over a package with an AI recap to the vendor's developers.
6. Compare the vendor's DTA output file field by field, explain every difference in words, and issue a sealed certificate.

[!NOTE] The background — who ITSG is, the challenge, the glossary and the data format — is in docs/. Start with the problem and our solution.

✨ Highlights

🔍 Change detection that doesn't trust the log	The change log called §3.12 "editorial"; the diff found that the 6-month rule flipped at its boundary.
🧾 Every claim is quoted	The AI must quote the spec verbatim; quotes are checked against the PDF text before anything is shown.
🧮 Expected values from tested code	A small, safe formula language and an oracle compute every expected result; the AI's own number must match it.
👩‍⚖️ Human in the loop	Nothing reaches a developer before an examiner has approved the reading of the change and verified the test case.
🔒 No answers leak to vendors	The server strips expected values — even from the AI's chain of thought — before a developer sees a case.
📜 Audit trail	Every action lands in a hash-chained audit log; reports and certificates are sealed with SHA-256.
⚡ Built for a room full of users	A fair scheduler shares the AI workers across everyone's runs, with automatic back-off when the gateway is busy.
📦 Tiny footprint	Python standard library + pypdf, two single-file HTML apps, one Docker container.

📸 Screenshots

Changes and review	Test-case review
The change log says editorial; the AI shows the rule flipped, with before/now and the quote.	One case per rule. The expected result is computed by the oracle; checks G1–G5 are shown per case.

Developer checklist
What the vendor's developers receive: an AI recap of what changed and what to do, and one row per test case.

---

🚀 Quick start

Open demo/index.html in a browser. It is a single file with the slides and the guided demo; press D to start the demo, → to advance, P for the narrator window. The last slide shows the QR code to the live app. Details: demo/README.md.

Needs Python 3.10+ and a hessian.AI key.

cp pipeline/.env.example pipeline/.env     # then put the key after HAI_API_KEY=

python -m pip install -r pipeline/requirements.txt

python pipeline/server.py

Open http://localhost:8787/ for the live app, or http://localhost:8787/demo/ for the guided demo on recorded data. On Windows you can also double-click pipeline/start.cmd; on macOS/Linux run sh pipeline/start.sh. Sign in as istg (ITSG examiner), sap or lexware (vendors).

Address	What
/ (also /app)	The live app: sign-in, real PDF upload, AI runs, review, packages, vendor results, messages, certificates
/demo/	The same page on recorded data: slides and the 5-minute guided demo (/demo/#demo)
/pipeline	Redirects to / (old links and QR codes keep working)

The live app and the guided demo are one file, demo/index.html: the server adds window.SPECS_MODE="live" at / and "demo" at /demo/. Without the flag (opened as a file, on GitHub Pages) it is the recorded demo.

Put HAI_API_KEY and PIPELINE_PASSWORD in pipeline/.env on the host, then:

bash deploy/docker.sh 80

This builds the image, starts the container eel with --restart unless-stopped, keeps all data in the eel-data volume and waits for the health check. Logs: docker logs -f eel. Without Docker: deploy/setup.sh.

deploy/qr.png is a QR code to the hackathon instance, for phones in the room.

Update the running server (130.83.85.181:34080) after a push to main:

ssh -p 34022 <user>@130.83.85.181
cd ~/eel && git pull origin main        # or copy the repo there again if ~/eel is not a git clone
bash deploy/docker.sh 80                # rebuilds and restarts; data stays in the eel-data volume

Leave NAME and DATA_DIR unset so the same container (eel) and volume (eel-data) are reused. The script prints page HTTP 200, API without sign-in HTTP 401 when it worked.

cd web && npm install && npm run judges

Starts the Python pipeline and a landing page on port 3000 that proxies to it, and prints the LAN address with a QR code for phones on the same Wi-Fi. Details: web/README.md.

---

🏗 Architecture

One Python process serves one single-page app in two modes (live at /, recorded at /demo) and a JSON API. AI work runs on a pool of worker threads behind a fair scheduler; every LLM answer passes deterministic guards before a person sees it. State lives in memory and is written to disk by a background thread.

System context

flowchart LR
    examiner(["👩‍⚖️ ITSG examiner"])
    vendor(["👨‍💻 Vendor developer<br/>SAP, Lexware …"])

    subgraph system ["§pecs"]
        app["Web app + API<br/>one container"]
    end

    spec[("📄 Specification PDFs<br/>new + archive")]
    llm["🧠 hessian.AI gateway<br/>OpenAI-compatible · Kolibri-1"]
    smtp["✉️ SMTP server<br/>optional"]

    examiner -- "uploads spec, approves changes,<br/>verifies and sends test cases,<br/>certifies" --> app
    vendor -- "reads checklist, uploads DTA output,<br/>asks questions" --> app
    spec --> app
    app -- "streamed chat completions<br/>+ tool calls" --> llm
    app -- "certificate e-mails" --> smtp

Components

flowchart TB
    subgraph browser ["Browser"]
        demo["<b>demo/index.html at /demo</b><br/>slides · guided demo<br/>recorded data · oracle in JS"]
        live["<b>demo/index.html at /</b><br/>the same screens, live mode<br/>sign-in · runs · review · checklist · messages · certificates"]
    end

    subgraph server ["pipeline/server.py — one Python process"]
        direction TB
        http["HTTP router<br/>static pages · JSON API · /v1 proxy"]
        auth["Auth<br/>accounts · session tokens · roles"]

        subgraph core ["Domain"]
            docs["documents.py<br/>PDF → sections · change log · diff"]
            runs["Runs<br/>plan · items · events"]
            cases["Test cases<br/>review · verify · send · redact"]
            exch["Exchange<br/>messages · result files"]
            certs["Expected files · reports<br/>certificates · audit trail"]
        end

        subgraph ai ["AI execution"]
            sched["scheduler.py<br/>fair queue · per-run cap"]
            agent["agent.py<br/>LLM loop · tools · guards G1–G5"]
            sandbox["sandbox.py<br/>isolated Python terminal"]
            formula["formula.py<br/>safe formula language · oracle"]
            recap["recap.py<br/>developer briefing"]
        end

        persist["Persister<br/>dirty flags · atomic writes · 1 s"]
    end

    llm["hessian.AI gateway"]
    disk[("/data<br/>store.json · runs/ · uploads/<br/>cache/ · exchange/")]

    demo & live -- "fetch / poll" --> http
    http --> auth --> core
    runs -- "enqueue section" --> sched --> agent
    agent -- "run_python" --> sandbox
    agent -- "evaluate" --> formula
    cases --> formula
    cases --> recap
    agent & recap --> llm
    http -- "/v1 proxy" --> llm
    core --> persist --> disk
    docs -- "parse cache" --> disk

Component	Responsibility	Key design choice
server.py	HTTP API, auth, runs, test cases, exchange, reports, persistence	Standard library only; one process, thread per request
scheduler.py	Fair queue for AI jobs	Round-robin across runs, cap per run, dedupe, cancel on Stop
agent.py	Prompt, streamed LLM loop, tool calls, guards	Model output is never trusted: guards feed failures back to the model
formula.py	Safe formula language and evaluator (the oracle)	Exact decimals and dates, fixed function set, no eval
sandbox.py	The AI's calculation terminal	Separate process, import allow-list, 10 s timeout, empty temp dir
documents.py	PDF → numbered sections, change log, word diff	Structure-based retrieval, not fuzzy search; parsed once and cached
recap.py	Developer briefing for a package	Written from redacted cases; leaks are replaced by a plain summary

An AI run, end to end

sequenceDiagram
    autonumber
    actor E as Examiner
    participant UI as index.html (live)
    participant API as server.py
    participant Q as Scheduler
    participant A as Agent
    participant LLM as hessian.AI
    participant T as Sandbox + oracle
    actor D as Developer

    E->>UI: drop new spec PDF
    UI->>API: POST /api/upload
    API->>API: parse sections, change log (cached by SHA-256)
    E->>UI: Generate test cases
    UI->>API: POST /api/runs
    API->>API: diff old ↔ new, one item per changed section
    API->>Q: enqueue(run, section, "extract")
    API-->>UI: run id (returns at once)

    loop every changed section, fairly across all runs
        Q->>A: process_item
        A->>LLM: section text + change log (streamed)
        LLM-->>A: reasoning + tool calls
        A->>T: run_python / evaluate
        T-->>A: result
        A->>A: guards G1–G5, failures go back to the model
        A-->>API: spec-change JSON + chain of thought
    end

    UI->>API: GET /api/runs/{id}?after=seq (polling)
    E->>UI: approve / edit / decline each change
    UI->>API: POST /api/runs/{id}/cases
    API->>Q: enqueue(run, section, "cases")
    Q->>A: write test cases for the approved change
    A->>LLM: approved change + quotes
    A->>T: formula must match the oracle
    E->>UI: verify, Send to developers
    UI->>API: POST /api/send
    API->>API: redact expected values · write recap · expected DTA files
    API-->>D: "ITSG sent you new test cases"

Life of a spec section

stateDiagram-v2
    direction LR
    [*] --> candidate: found by diff,<br/>not in the top set
    [*] --> queued: changed section
    candidate --> queued: Generate
    queued --> running: worker free
    queued --> skipped: Stop
    running --> review: spec change extracted
    running --> error: LLM / guard failure
    error --> queued: Generate again
    review --> queued: examiner approves → write cases
    review --> declined: examiner declines
    running --> done: test cases written
    done --> [*]

Concurrency model

flowchart LR
    subgraph runs ["Runs (one per person)"]
        r1["Run A · 100 sections"]
        r2["Run B · 5 sections"]
        r3["Run C · 12 sections"]
    end
    subgraph sched ["Scheduler"]
        rr{{"round-robin<br/>≤ AI_PER_RUN per run"}}
    end
    subgraph pool ["AI_CONCURRENCY worker threads"]
        w1["worker"]
        w2["worker"]
        w3["…"]
    end
    gw["hessian.AI"]

    r1 & r2 & r3 --> rr --> w1 & w2 & w3
    w1 & w2 & w3 -- "429 / 502 / 503 / 504 →<br/>wait 2 → 30 s, retry" --> gw

• Fair: free workers take sections round-robin across runs, so a run of 100 sections never blocks the person who starts after it. A queued section shows how many runs go first.
• Bounded: AI_CONCURRENCY (default 16) calls in flight against the gateway, at most AI_PER_RUN (default 6) from one run.
• Resilient: a busy gateway is waited out with exponential back-off (honouring Retry-After) instead of failing the section. Stop drops a run's queued sections at once.
• Cheap on the hot path: saving only marks state dirty; a persister thread writes store.json and run files atomically once a second. SIGTERM (docker stop) flushes before exit. Idle finished runs leave memory and reload from disk.
• Threads, not asyncio: LLM calls are I/O-bound streams, so a few dozen threads cost little and the GIL is not a factor. The ceiling is the gateway, not the server.

Measured with a stub gateway (1.5 s per call): 30 people starting runs at the same moment all got their first section running within 3.4 s, and every run finished.

Deployment

flowchart LR
    phone(["📱 Phones and laptops<br/>scan deploy/qr.png"])
    subgraph net ["Hackathon network · NAT"]
        nat["130.83.85.181<br/>:34080 → :80<br/>:34022 → :22"]
    end
    subgraph vm ["Debian VM · 2 vCPU · 4 GB"]
        subgraph docker ["Docker"]
            c["container eel<br/>python:3.12-slim<br/>server.py :8787<br/>health check · --init"]
            v[("volume eel-data<br/>/data")]
        end
        env["pipeline/.env<br/>chmod 600"]
    end
    gw["hessian.AI gateway"]

    phone -- "HTTP" --> nat -- ":80 → :8787" --> c
    c --- v
    env -. "--env-file" .-> c
    c -- "HTTPS" --> gw

The API key and passwords are passed at run time with --env-file; they are never baked into the image. Rebuilding or restarting the container keeps every run, test case and upload in the volume.

---

🔄 How it works

#	Stage	What happens	Where
1	Import	The new spec PDF is stored next to earlier versions in docs/law_file/.	Inputs & archive
2	Index	pypdf text → numbered sections (longest increasing heading sequence) and the change log.	documents.py
3	Detect changes	Section-by-section word diff, reconciled with the change log; changes the log missed are flagged.	documents.py
4	Understand & approve	The AI writes a spec-change JSON with verbatim quotes; the examiner approves, edits or declines.	Review changes
5	Generate test cases	One case per rule: preconditions, steps, formula, variables; the oracle computes the expected result.	agent.py, formula.py
6	Hand over	Verified cases go to the developers as a checklist with an AI recap; expected DTA files are written.	/api/send, recap.py
7	Compare	The vendor's DTA file is compared field by field; every difference is explained with its field and §.	Compare output
8	Report & certify	A sealed test report per file; a sealed certificate once everything passes, delivered and e-mailed.	Reports & certificates

What is compared with what: a dropped PDF is diffed against the best-matching version in docs/law_file/ (heading overlap ≥ 20 %); without a match it is used on its own, from its change log or the sections with the most rule signals. The pairing can be overridden under Options.

🛡 Trust model

The AI explains each step and does its arithmetic in a terminal — and its answers are still checked:

Guard	Rule
G1	Every quote appears verbatim in the specification (whitespace- and hyphenation-insensitive).
G2	The formula parses in the safe formula language — exact decimals, dates, a fixed set of functions.
G3	The oracle evaluates the formula, and the result equals the AI's own terminal value.
G4	Every legal parameter (e.g. 42 days) appears in its quote — "sechs Wochen" counts as 42.
G5	The formula was actually run in the terminal.

Failed guards go back to the model, which corrects itself. Verifying a case with a failed guard needs a written reason, and the expected value a developer receives always comes from the oracle. Developers never see ITSG's expected values before validation — the server strips them, including from the chain of thought.

[!IMPORTANT] The sandbox is a guard rail for a prototype, not a security boundary for hostile input. A reachable server must set PIPELINE_PASSWORD: the server holds the API key and proxies /v1.

👥 Roles and accounts

Sign in as	Role	Sees
istg (or itsg)	ITSG examiner	Everything: inputs, changes, AI pipeline, review, messages, reports, certificates
sap, lexware	Vendor	Their checklist with AI recap, their results rounds, conversations, files and certificate; never ITSG's expected values
dev, vendor	API only	A developer account and a generic vendor that picks its company (not on the app's sign-in page)

Without a password (local use) anyone can pick a role. With PIPELINE_PASSWORD set, the server checks it, issues a session token (valid PIPELINE_SESSION_DAYS, default 7) and takes user and role from the token, never from the browser.

🔧 Configuration

All settings go in pipeline/.env (copy pipeline/.env.example; never commit .env).

Variable	Default	Purpose
HAI_API_KEY	—	hessian.AI key (required for live AI)
HAI_BASE_URL	https://hip-gw.hessian.ai/v1	OpenAI-compatible gateway
HAI_MODEL	vllm/Kolibri-1	Default model
PIPELINE_PASSWORD	(none)	Turns sign-in on; per account: PIPELINE_PASSWORD_ISTG, _DEV, _SAP, _LEXWARE
PIPELINE_SESSION_DAYS	7	How long a sign-in lasts
AI_CONCURRENCY	16	AI calls in flight, shared by everyone
AI_PER_RUN	6	AI calls in flight for one run
SMTP_HOST, SMTP_PORT, SMTP_USER, SMTP_PASSWORD, SMTP_FROM	(none)	E-mail certificates from the server
PIPELINE_HOST, PIPELINE_PORT	127.0.0.1, 8787	Bind address (the container uses 0.0.0.0)
PIPELINE_DATA, LAW_DIR	pipeline/data, docs/law_file	Where state and archived specs live

📡 API reference

All endpoints return JSON. With a password set, send the session token from POST /api/login as X-Token.

Method	Path
GET	/api/status	Model, key configured, setup problems, queue stats
POST	/api/login · /api/logout	Sign in / out
POST	/api/session/vendor	Pick a vendor company
GET	/api/users	Known users
GET	/api/audit	Hash-chained audit trail

Method	Path
GET	/api/documents	Archive and uploads
GET	/api/documents/{name}?kind=	One document's sections as paragraphs (the Specification screen)
POST	/api/upload?name=&target=	Store a PDF (raw bytes)
DELETE	/api/upload/{name}	Remove an upload
POST	/api/plan	What would be compared with what
POST	/api/runs	Start a run
GET	/api/runs · /api/runs/{id}?after={seq}	Runs; one run with new events and queue position
POST	/api/runs/{id}/items/{sec}/generate	(Re)process one section
POST	/api/runs/{id}/decide · /cases · /stop	Approve/decline changes, write test cases, stop
PATCH	/api/runs/{id}/items/{sec}/change	Edit the AI's reading of a change
POST	/api/chat	Ask the AI about one change (run, sec: grounded in its old/new text, change log, extracted change and reasoning) or about data the page sends

Method	Path
GET / PATCH / DELETE	/api/testcases[/{id}]	List, edit, delete
POST	/api/testcases/{id}/verify · reject · reopen · ack	Review actions
GET	/api/testcases?run={id}&full=1	A run's full test cases (redacted for developers and vendors)
GET	/api/testcases/{id}/export	JSON export
POST	/api/evaluate	Evaluate a formula with the oracle
POST	/api/send	Send cases to developers; to sap/lexware it also posts the package in the vendor's certification thread
GET	/api/inbox · /api/packages/{id} · /api/checklists	Developer side
POST	/api/checklists/{id}/coverage · recap	Tick coverage, rewrite the recap
POST	/api/checklists/{id}/submit · results?name=	A results round: values, or a CSV (id;value) / JSON file. Compared on the server, sealed as a report, verdict posted to the vendor's thread; the next round is the retest
GET / POST	/api/submissions · /api/results · /api/tracking	Validation flow and progress

Method	Path
GET / PATCH	/api/exchange	Shared ITSG ↔ vendor state
POST	/api/exchange/threads[/{id}/messages · /read]	Conversations
POST / GET	/api/exchange/files	Result files (≤ 10 MB, original bytes)
GET / POST	/api/expected[/{id}]	Expected DTA files
GET / POST	/api/reports[/{id}]	Sealed test reports
GET / POST	/api/certificates[/{id}/html · /email]	Certificates (from report ids or a package's newest round; announced in the vendor's thread)
*	/v1/...	Authenticated pass-through to the AI gateway

📁 Project structure

.
├── demo/
│   ├── index.html          # the app: recorded demo offline and at /demo, live app at / (single file)
│   ├── embed.py            # embeds the full spec text and images into index.html
│   ├── proxy.py            # minimal /v1 proxy for the demo's live chat
│   └── test_demo.cjs       # UI tests for both modes (node)
├── pipeline/
│   ├── server.py           # HTTP API, auth, storage, static pages
│   ├── scheduler.py        # fair queue for the AI workers
│   ├── agent.py            # LLM loop, tools, guards G1–G5
│   ├── documents.py        # PDF → sections, change log, diff
│   ├── formula.py          # safe formula language (the oracle)
│   ├── sandbox.py          # the AI's isolated Python terminal
│   ├── recap.py            # developer briefing
│   └── test_pipeline.py    # unit tests
├── deploy/
│   ├── docker.sh           # build + (re)start the container
│   ├── setup.sh            # host without Docker
│   └── qr.png · qr.svg     # QR code to the hackathon instance
├── web/                    # judge portal (Next.js): landing page + proxy to the pipeline
├── pitch/                  # pitch deck (.pptx, .pdf) and its generator build_deck.py
├── docs/                   # challenge, glossary, data format, concept, meeting notes
├── hackathon-files/        # ITSG source material (specs, sample DTA file)
├── images/                 # logo and slide images
└── Dockerfile

Deep dives
pipeline/README.md	The live pipeline: every screen, the developer loop, the exchange, reports and certificates
web/README.md	The judge portal: one command for the pipeline, the landing page and a LAN QR code
demo/README.md	Presenting the guided demo: keys, narrator window, the 23 beats, what is real and what is simulated

🧪 Testing

python -m unittest pipeline/test_pipeline.py

node demo/test_demo.cjs

The unit tests cover the formula language, the sandbox, PDF sectioning and diff, the guards, the agent loop with a scripted model, the HTTP API with passwords on, the scheduler (fairness, per-run cap, cancel), gateway back-off and persistence. The demo's core library has its own self-test (32 checks), run from demo/README.md.

📚 Documentation

Document	What's in it
Problem and solution	Start here — the problem and our solution, with diagrams
Challenge overview	Who ITSG is, what a system review is, judging
Glossary	German ↔ English terms, submission-reason codes, record names
Source documents	What each provided file is and when to use it
Reference test case	ITSG's fully worked prior-illness request (DBVO)
Data format	The fixed-width DTA EEL format, with an annotated sample
Solution concept	Architecture, MVP scope, open questions
Demo design	Inputs, workflow, outputs, trust architecture, traceability
hessian.AI API	Endpoint, models and auth (without the key)
ITSG meeting notes ·

🤝 Team

Built in 48 hours at Futury Build Days #3 by Achraf, Amine Cheikhrouhou and Yassine.