Daily
News that know you
A native iOS news app built on a ten-stage LLM pipeline,
measured by offline replay and tested in a public sandbox.
Open Daily Lab · Evaluation evidence · Recorded editions · Architecture · Docs
What Daily is
Daily turns a short onboarding conversation into a reader profile, finds the sources that matter to that reader, and assembles a personal edition. Readers search, save stories, discuss any story with an assistant that cites its sources, and retune their profile in plain language through Tune, which proposes a change and applies it only when the reader confirms.
Behind the app is a FastAPI service on PostgreSQL and pgvector that runs the pipeline as ten stages, from source discovery to learning, each with its own contract and activation gate. An offline harness replays the production feed code against frozen corpora and recorded model responses, so every change is measured for $0 and every lost story is traced to the stage that dropped it. Daily Lab, a Next.js site on Vercel, lets anyone run code against the pipeline's verdict-to-article contract inside a Firecracker microVM.
The repository holds about 99,000 lines of first-party Python, TypeScript, Swift and SQL, built across more than 250 commits and 90 merged pull requests since November 2025.
Highlights
- Public sandboxed lab. Visitors submit a Python parser and run it against 42 recorded model responses and 22 fault injections. Each run boots a fresh microVM in Vercel Sandbox with networking denied, and a TypeScript evaluator outside the VM grades it under a hash-pinned specification.
- Pre-registered fault injection. Nine faults and their expected effects were pinned by SHA-256 before the matrix was recorded. All 6 targeted faults moved their declared metrics and all 3 blind-spot predictions held. Shifting each verdict onto its neighbor's article cuts judge precision from 0.6405 to 0.3278 (sign test p = 0.0001).
- Offline replay at zero cost. The production feed code runs unchanged against SHA-256-verified corpora and 2,169 recorded model responses. Replaying the whole fault matrix costs $0.00 instead of about $5.40 at API prices, and CI fails on any cache miss, including one the application swallowed.
- Agent benchmark. 31 budgeted agent investigations by GPT-4.1, GPT-5 mini and Kimi K2 worked on the parser through the Vercel AI Gateway, for $0.78 in total. Their 8 proposals faced the same scope gate, microVM and evaluator as a human submission, and 2 were accepted for review.
- Correctness under concurrency. The server uses PostgreSQL leases with version tokens, compare-and-set publication and per-reader monotonic sequencing. The app fences feed and reader work by operation and account, so a late response cannot replace a newer edition or cross an account boundary.
- Hardened fetching. Every live fetch of an untrusted URL goes through
safe_http, which requires every DNS answer to be public, connects to the validated address, revalidates each redirect and caps the response at 2 MB.
- Tested in depth. More than 2,900 backend tests, 330 web unit tests, 90 Playwright tests in three browser projects and 150 iOS tests. CI provisions PostgreSQL 16 with pgvector 0.8.0 and fails when a required database suite is skipped.
Daily Lab
Language models return relevance verdicts in batches, and a parser has to attach each verdict to the article it describes. Daily Lab puts that contract in front of anyone with a browser: edit a parser, run it in a microVM, and see the exact case that breaks it.
- Scope gate. The submission may write exactly one file,
candidate.py, up to 64 KB.
- Fresh microVM. Vercel Sandbox boots a Firecracker microVM (Python 3.13, 2 vCPUs, 120 s limit) under a platform-applied
deny-all network policy. It receives the harness, the 64 cases with their answers removed, and the candidate.
- Harness.
parse() runs on every case. Prediction records leave the VM inside a random-UUID frame, and the record schema has no field for a grade.
- Isolation probes. Four probes run in the same microVM after the candidate: DNS must fail, HTTPS must fail, no evaluator code may exist on disk, and no credential may exist in the environment. All 36 probes in the published runs held.
- Independent grading. The evaluator grades the records outside the VM against six criteria, each with a threshold of 1.0, and names the first misattributed article in reading order. Every run is graded under both generations of the specification.
| Candidate | Verdict |
|---|
keyed-v2, the ID-keyed contract | Accepted for review: 60 of 60 applicable cases, and the same association under all 720 orderings of a batch |
count-guard-v1, a count-mismatch guard | Rejected at 48 of 52: an equal-length reordered response passes a count check |
positional-v0, positional association | Rejected at 0 of 52 |
keyed-fallback-v1, keyed with a positional fallback | 60 of 60, then rejected by generation 2's protocol-exclusivity criterion |
| Three seeded controls, defective by design | Rejected, each by the criterion its defect violates |
| 31 agent investigations | 8 proposals reached a microVM, 2 accepted for review |
The public runner admits each run through one atomic Redis transaction against four shared limits, and refunds every refusal. Orchestration is a durable Vercel Workflow whose sandbox step is never retried, so a failure costs one microVM. The Lab README covers the specification, the isolation boundary, admission control and the agent benchmark in full.
Evaluation
The harness answers two questions about any change: did the edition get better or worse, and which stage lost a story the reader needed? ProductionRunner calls the production feed service itself. It replaces only the database, with an in-memory connection that serves production SQL from a frozen snapshot and raises on any query it does not recognize, and it routes model calls through a cache keyed by the SHA-256 of each request. Every article carries a stage trace, so a missed story is attributed to the stage that dropped it.
| Input | |
|---|
| Corpora | 3 content-hashed snapshots of 1,328, 1,234 and 1,358 articles from about 50 sources |
| Reader fixtures | 10, each built to stress one axis: cold start, exclusions, transliterated entities, follow-ups, hyper-local news |
| Ground truth | 10,431 relevance grades from a two-pass model grader (gpt-4.1-mini, then gpt-4.1 on contested cases) and an agent editorial pass, plus 19 planted must-see articles and 19 lookalikes per snapshot |
| Model responses | 2,169, recorded and keyed by request hash |
The fault matrix below was committed at 9cf217e on 2026-09-30, one hour after its declarations were pinned. A fresh replay at HEAD matches it, and CI replays it on every pull request.
| Fault injected into the replay | Declared effect | Recorded, pooled across readers |
|---|
| Each verdict shifts onto its neighbor's article | Judge precision falls | 0.6405 to 0.3278, 17 of 24 pairs worse |
| The ranked feed is reversed | Needle recall falls | 0.6333 to 0.2833 |
| The scorer marks every article relevant | Judge precision falls | 0.6405 to 0.1160 |
| Each reader is graded against another reader's labels | Needle recall falls | 0.6333 to 0.0000 |
| Prototype: verdict bodies shift while their IDs stay | Judge precision falls | 0.8483 to 0.2470, 27 of 28 pairs worse |
| Prototype: forced world-critical slots are ignored | Event delivery falls | 0.775 to 0.375 |
| No diversity pass, top 12 reversed, ID-keyed list rotated | No metric moves | Largest change 0.0167 |
A tenth control changes the scorer's requests upstream. It makes no network call and raises 90 cache misses that the application catches and hides; the gate counts them and fails the run. Matrix · Methodology · Gate tests
Architecture
| Stage | Responsibility | Mechanism |
|---|
| S1 Sources | 38 global feeds and per-reader discovery from an 82-source seed registry | SSRF-resistant fetching, sources validated by a live fetch |
| S2 Content | Article extraction into versioned artifacts | FOR UPDATE SKIP LOCKED leases with version compare-and-set; native text only under an explicit source policy |
| S3 Understanding | Entities, topics and structured meaning | Transactional outbox and inbox between stages |
| S4 Events | Clustering stories into events and assessing their gravity | Six-dimension assessment into world-critical, major and routine tiers |
| S5 Reader model | The durable profile behind onboarding and Tune | Optimistic concurrency on generation and revision, idempotent operations |
| S6 Retrieval | Candidates for the ranker | Reciprocal rank fusion (k = 60) over lexical, dense pgvector and identity legs |
| S7 Ranking | LLM judgments against the reader | Exact-ID contract, verbatim evidence quotes, budget reserved before every provider call |
| S8 Assembly | Turning a ranking into an edition | Deterministic fairness and diversity constraints, a recorded disposition for every candidate |
| S9 Delivery | Versioned editions to the app | Monotonic per-reader sequence, delivery receipts |
| S10 Learning | Folding reading behavior back into the profile | One shared reward definition, attributed feedback |
| Layer | Stack |
|---|
| iOS app | Swift, SwiftUI, Swift concurrency with main-actor default isolation, iOS 26, Google Sign-In, Keychain |
| API and workers | Python 3.12, FastAPI, six background loops with leader election through PostgreSQL advisory locks, Fly.io |
| Data | PostgreSQL and pgvector: 80 tables, 86 CHECK constraints, 1,536-dimension embeddings behind an HNSW index |
| Models | OpenAI gpt-4.1-mini, gpt-4o-mini and text-embedding-3-small |
| Web and Lab | Next.js 16, React 19, strict TypeScript, Tailwind CSS 4 |
| Lab infrastructure | Vercel Sandbox, Vercel Workflow, Vercel Blob, Vercel AI Gateway, Upstash Redis |
Design decisions
Verdicts are keyed by article ID and checked as an exact set. A response can parse cleanly and still attach the right verdict to the wrong article, and a count check cannot see an equal-length reorder. The ranking contract requires the exact set of requested IDs, a matching evidence hash for each, and a verbatim quote from the frozen evidence for every positive grade.
# backend/app/services/ranking_contract.py
ids = [v.article_id for v in values]
if len(ids) != len(set(ids)) or set(ids) != set(packs):
raise ValueError('ranker must return exact article ID set')
Every measurement runs on frozen inputs. News, model responses and reader state change independently, so a before-and-after comparison holds all three still. Snapshots are content-hashed, responses are keyed by request hash, and a changed prompt is a new recording. Any regression reproduces on a laptop with no provider key.
Authority is rechecked at publication. An article can change, a reader can edit their profile, or a worker can lose its lease while a model call is in flight. Database claims decide who may do work, and a transactional compare-and-set at completion decides whether the result may still publish. Budget is reserved under a single control-row lock before each provider call, against global and per-account daily caps.
Untrusted code never shares a boundary with a secret. Lab candidates run in a different process, language and machine from their grader, with no credential in reach. CI follows the same rule: the workflow holding the Blob publishing token runs only after a successful push to main, checks that the commit is an ancestor of origin/main, re-validates every artifact it downloads, and uploads the manifest last.
Getting started
Without setup. Run a parser in Daily Lab, compare runs in the evidence explorer, or read a recorded edition.
iOS app. Xcode 26 or later and an iOS 26 simulator.
git clone https://github.com/mmarufov/Daily.git && cd Daily
open Daily.xcodeproj
Run the Daily scheme. AppConfig.swift points at the hosted backend, and the DAILY_BACKEND_URL Info.plist key overrides it. A device build needs your signing team.
Evaluation and Lab contract. Python 3.12, with no provider key and no database.
cd backend
python3.12 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt -r requirements-dev.txt
# Replay the pre-registered fault matrix: 31 passed
EVAL_OFFLINE=1 python -m pytest tests/test_eval_degradation.py -q
# The Lab contract, including all 720 orderings of a batch: 741 passed
python -m pytest tests/test_lab_contract.py -q
Web and Lab. Node.js 24.
cd web
npm ci
npm run dev # http://localhost:3000
Backend deployment settings live in fly.toml and .env.example. Web configuration and every command are in the web README.
Testing
| Suite | Tests | Runs in |
|---|
| Backend, pytest | 2,900+, including an exhaustive 720-ordering permutation suite | CI on every pull request, offline; 180 database tests against PostgreSQL 16 and pgvector 0.8.0 |
| Web unit, Vitest | 330+ | CI |
| Web end-to-end, Playwright | 90+ tests in 3 projects: desktop Chrome, Pixel 7, iPhone 14 WebKit | Against marufov.com, with every Lab API call intercepted |
| iOS, XCTest | 150 | Xcode, Daily test action |
- Backend covers extraction policy, stale leases, ranking publication, budget admission, delivery receipts, adversarial edition assembly and concurrent claims against real PostgreSQL.
- iOS covers late responses, account switches, monotonic cache writes, body revocation, offline grants and reader recovery.
- Web covers evaluator rejection cases, patch scope, run recovery, shared limits, artifact provenance, byte-for-byte export determinism, touch targets, forced-colors mode and reduced motion.
Repository layout
| Path | Contents |
|---|
| Daily/ | SwiftUI app: edition, reader, Tune, search, saved stories, account-scoped storage |
| DailyTests/ | XCTest suites for delivery, storage, auth and the reader |
| backend/app/ | FastAPI routes, the ten pipeline stages, PostgreSQL schema and repositories |
| backend/evals/ | Frozen corpora, response cache, metrics, fault registry and results |
| backend/lab/ | Parser contract, recorded and synthetic cases, candidates and controls |
| web/ | Next.js site: Daily Lab, evidence explorer and recorded editions |
| docs/ | Stage audits, plans and status records, architecture, design system |
| .github/workflows/ | Backend tests and evaluation gate, artifact validation, trusted publication |
Every pipeline stage has a written audit, and S3 through S10 add an implementation plan and a status record: 27 documents and about 71,000 words in docs/stages, indexed in docs/README.md.
License
GNU Affero General Public License v3.0. Copyright 2026 Muhammadjon Marufov.
