AI SOTA Feed Bot (Prototype)
Git-backed AI platform engineering news intelligence.
What it does
- Collects fresh items from high-signal RSS feeds
- Normalizes and de-duplicates (URL + near-title similarity)
- Mechanical (no-LLM) enrichment (
pipeline/enrich.py):
- Release titles get a repo prefix (
0.139.0 → codex 0.139.0)
- Release summaries use extracted changelog bullets (
release_highlights)
why_it_matters is only set from real signal (LLM text or matched
matched_topics keywords) — no generic placeholder- Cross-source duplicates surface as
also_covered ("Also covered by …")
instead of being dropped silently
- Scores/ranks items with the deterministic ranking core plus a Jev semantic gate
- Categorizes judged candidates for the daily/weekly/wiki/playbook/foundations routines
- Applies diversity-aware ranking (strict minimum mix + caps for paper/news/release)
- Tracks source reliability/health and incorporates it into ranking
- Applies source circuit breaker on repeated failures with cooldown auto-recovery
- Records low-noise degradation alerts for operational inspection
- Builds a Markdown digest
- Publishes the feed and versioned digest artifacts through the website
- Publishes a weekly "What happened in AI this week" recap at
/weekly (see below)
- Publishes Agent Builder Foundations at
/foundations: durable,
evidence-tiered explanations of the mechanisms behind reliable agents
Pipeline
A single edition build runs every two hours:
collect
-> editorial policy (source roles, quarantine, story clusters, seen-memory)
-> Stage A prefilter
-> Jev gate + category (cached, durable judgment store, fail-open)
-> ranking (judge score as a bounded weight + headroom floor)
-> edition
pipeline/editorial.py handles the deterministic editorial policy.pipeline/editorial_judge.py fetches paragraph excerpts when the feed teaser
is too short, asks Jev for keep, platform_relevant, release_lane,
corporate_marketing, and a primary category, and reuses prior judgments
from data/processed/judgments.json.pipeline/ranking.py keeps control of slots, quotas, freshness, and the final
edition. Release items are exempt from the judge's headroom floor so the
Releases tab and model radar still fill.- Outputs:
data/processed/latest.json (the feed),
data/processed/classified.json (compact judge artifact for editor decks),
and data/digest/<date>.md. pipeline/deck_quality.py joins the judge
verdict/category onto the daily, weekly, Playbook, wiki, and Foundations
input bundles.
- The old Tier-1 snapshot and API fresh blend are gone. The API reads
data/processed/** only.
The Jev gate needs OPENROUTER_API_KEY; without it the build logs
judge_skipped reason=not_configured and runs deterministically. The gate is
enabled in config/judge.yaml.
Weekly recap (/weekly)
A reader-facing weekly summary page rendered from data/weekly/<week>.json.
The recap content is produced by an agent (a Claude Code routine) using the
weekly-summary skill in .agents/skills/weekly-summary/ — committing the JSON
to the repo is how it gets "posted".
Pipeline:
S=.agents/skills/weekly-summary/scripts
python $S/build_weekly_input.py # 1. bundle the week's unique news items -> data/weekly/input/
# (news-only by default; --types all to include papers/releases)
# 2. agent reads the bundle, writes data/weekly/<week>.json
python $S/build_weekly_index.py # 3. validate + rebuild data/weekly/{index,latest}.json
# (also re-renders static /weekly pages + sitemap, see below)
# 4. git add data/weekly/ web/weekly/ web/sitemap.xml && git commit && git push
- Page:
/weekly (latest), /weekly/<week> (archive). API: /api/weekly[?week=|?list=1].
- Smoke-test the UI with a placeholder recap:
bash $S/run_weekly.sh --seed
- Agent contract + schema:
.agents/skills/weekly-summary/SKILL.md
Storylines (/storylines)
Cross-day threads of a developing story ("what happened next with that thing
from Tuesday") — e.g. a model launch followed by hands-on posts and reactions.
pipeline/build_storylines.py clusters the durable story store mechanically
(no LLM): stories join a thread only when their titles share a rare anchor
token pair, and a thread only ships when it has ≥3 items across ≥2 days from
≥2 sources (precision over recall — a junk thread costs reader trust).
- Runs in the external Claude Code storyline routine every 5 hours; the two-hour
run_full.sh only syncs the durable story store. The routine writes
data/storylines/index.json + data/storylines/<slug>.json, scouts missed
links, refreshes narratives, validates, and publishes.
- Pages:
/storylines (active threads), /storyline/<slug> (day-by-day
timeline with a Follow button). API: /api/storylines[?slug=].
- Feed cards that belong to a thread get a
📈 Developing · day N badge;
followed threads (localStorage) surface a "your storylines moved" notice on
the feed when they gain items. Slugs are carried over between runs by member
overlap so follows and shared links survive recluster jitter, and detail
JSONs are never pruned so old /storyline/<slug> links keep working.
Agent Builder Foundations (/foundations)
Durable concept explanations for serious agent builders. Foundation pages live
under data/foundations/concepts/*.md, compile through
pipeline/build_foundations.py into data/foundations/index.json, and render
to /foundations plus /foundations/<slug>.
- Page:
/foundations. Detail: /foundations/<slug>. API:
/api/foundations[?slug=].
- Schema:
config/foundations_schema.md.
- Agent routine:
.agents/skills/foundations-curator/ with scheduler config in
.agents/routines/foundations-curator-weekly/.
- First concept:
prompt-reliability, answering "What makes a prompt reliable?"
Quick start
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python collectors/collect.py
python pipeline/build_digest.py
To enable the in-page email subscription form, set EMAIL_API_KEY in the
Vercel environment. Optionally set DIGEST_EMAIL_SIGNUP_URL to use an external
signup page instead. The form stays hidden when email is unconfigured; RSS
remains available through /rss.xml and autodiscovery.
Current web app behavior:
- Subscribe is a first-class action: a labeled email menu in the feed header
offers inline signup.
All user-facing subscription links route to the canonical
/subscribe page.
Returning visitors get a one-time dismissible subscribe nudge
after the 3rd story (first visits get topic onboarding instead). Events:
subscribe_menu_open, subscribe_click (channel + placement),
subscribe_nudge_dismiss. All pages carry RSS autodiscovery for /rss.xml.
- Sends batched
impression events to PostHog on feed render
- Sends
click events to PostHog when opening an item link
- One-tap reader feedback on every feed card (
👍 Useful / 👎 Not relevant / 🫧 Hype),
persisted in localStorage and sent to PostHog as item_feedback
- Feedback is visibly closed-loop: the feed API exposes the auto-tune output
(
reader_tuning block + per-item reader_adjustment from
data/feedback/source_adjustments.json); clearly boosted sources
(adjustment ≥ 0.05) get a 👍 Reader-boosted badge on their cards, and the
footer states how many sources reader feedback boosted/downweighted and when
- Trending badges on feed cards:
🔥 N sources (cross-source coverage from
also_covered) and 📈 Climbing (rank improved ≥2 positions between the two
most recent runs, via rank_at_last_seen vs rank_prev_seen)
- Pinned "my topics": readers can pin the current label selection as their
default (localStorage); it auto-applies on visits without
?label= in the
URL, with a one-tap re-apply chip after clearing filters (labels_pin event)
- Share permalinks (growth loop): the 📤 button shares
/s?u=<source url>
(api/share.js) instead of the source article, so shares land readers on
this site. When the story has a durable /story/<sid> page (see below) the
endpoint redirects there, so old share links never go dead. Otherwise it
serves item-specific OG/Twitter tags to unfurlers and redirects humans to
/?item=<url>&utm_source=share, where the feed scrolls to and highlights
the story (widening the window once to 30 days if needed; pinned topics are
suspended so they can't hide it). Events: item_share, share_landing
- Uses per-item batch/run context for telemetry (
ingest_batch_id preferred, fallback to run timestamp)
- PostHog tracking for dashboarding (
$pageview, feed_view, impression_batch, click, )
PostHog env vars (optional):
POSTHOG_ENABLED=1POSTHOG_PROJECT_API_KEY=<project key>POSTHOG_HOST=https://assets.llm-digest.com (default; proxied through the
infra/llm-digest-proxy-worker Cloudflare Worker — see
docs/how-to/posthog-setup-and-dashboard.md)POSTHOG_UI_HOST=https://us.posthog.com (default; PostHog's real domain,
used for in-app links like the toolbar — never the proxy)
Reader feedback loop (v1.3)
Reader taps land in data/feedback/events.jsonl via a daily PostHog sync
(.github/workflows/feedback-sync.yml); future tuning consumes the aggregates.
python pipeline/feedback.py add --url <item_url> --signal useful|irrelevant|hype # manual entry
python pipeline/feedback.py summary [--days N] # net counts by signal/source
python pipeline/feedback.py sync-posthog # pull web events (needs env below)
Sync env vars: POSTHOG_PERSONAL_API_KEY, POSTHOG_PROJECT_ID, optional
POSTHOG_API_HOST (query API host, default https://us.posthog.com).
Spec: docs/product-specs/feedback-loop.md.
Source weight auto-tuning (v1.3) consumes the feedback plus click-through
signal and writes capped per-source score adjustments the ranking pipeline
applies next to source_bias (knobs: config/ranking.yaml → auto_tune:):
python pipeline/auto_tune.py sync-ctr # pull per-source clicks from PostHog
python pipeline/auto_tune.py report # dry-run: proposed adjustments + inputs
python pipeline/auto_tune.py apply # write data/feedback/source_adjustments.json
Feed API (v1):
- The API reads
data/processed/** only; there is no separate Tier-1 blend.
- Per-item rank trajectory:
rank_at_last_seen (newest run) and rank_prev_seen (previous run that included the item)
SEO / static recap pages (growth loop)
The /daily and /weekly pages are client-rendered shells, invisible to
crawlers and link unfurlers. pipeline/render_static_pages.py therefore
pre-renders every published recap into fully static, indexable pages:
web/daily/<date>.html → served at /daily/<date>web/weekly/<week>.html → served at /weekly/<week>web/story/<sid>.html → served at /story/<sid> (story permalinks, below)web/sitemap.xml → /sitemap.xml, web/robots.txt → /robots.txt
Each page carries a real title, meta description, canonical URL, Open
Graph/Twitter cards, JSON-LD, and RSS autodiscovery, and mirrors the dynamic
pages' design. The index builders (build_daily_index.py /
build_weekly_index.py) invoke the renderer automatically, so publishing a
recap always refreshes the static pages — commit web/daily/, web/weekly/,
web/story/, and web/sitemap.xml alongside data/. Base URL override:
--base-url or SITE_BASE_URL (default https://www.llm-digest.com).
Vercel also runs scripts/vercel_build.py as its buildCommand; it invokes the
renderer and stages web/ into the configured public/ output directory.
PR previews and production deploys therefore reflect renderer code changes
even when generated web/ artifacts are omitted from a code-only commit.
Story permalink pages (growth loop)
Every story that makes a published feed snapshot gets a durable, indexable
page at /story/<sid> (sid = sha256(normalized url)[:16], derivable from
the URL alone). Feed snapshots are pruned after ~45 days, so
pipeline/story_store.py sync first captures each story into an append-only
store (data/stories/<YYYY-MM>.json + a compact index.json the share
endpoint bundles), and the static renderer turns the store into pages: title
(linking out to the source), why-it-matters, summary or release highlights,
topics, "also covered by", related stories (internal links), and links into
the live feed and that day's recap. Recap pages link each article's Details
badge to its story page, and /s share links redirect there, so shares and
search hits land on pages that never expire. Both steps run in
run_full.sh every cycle:
python pipeline/story_store.py sync # upsert stories from processed runs
python pipeline/render_static_pages.py # render recap + story pages + sitemap
LLM discovery endpoints:
/llms.txt (LLM-oriented site map + API usage notes)/llm-guide.txt (deeper instructions for feed navigation/debugging)
Single edition builder:
build_digest.py reads raw, applies the editorial policy and the optional Jev gate, then ranks.- Incremental diagnostics toggle:
TIER0_INCREMENTAL=1 (default on)
- Optional no-delta short-circuit:
TIER0_INCREMENTAL_SKIP_NO_DELTA=1 (default off)
- Jev gate: set
OPENROUTER_API_KEY (and optionally OPENROUTER_JEV_MODEL); without it the build logs judge_skipped reason=not_configured and continues deterministically.
- The compact judge artifact for editor decks is
data/processed/classified.json.
Collector crawl cooldown controls:
COLLECT_DEFAULT_POLL_MINUTES (default for sources without explicit poll_interval_minutes)COLLECT_BYPASS_COOLDOWN=1 to force fetch- Cooldown-only cycles no longer overwrite raw items; they reuse previous snapshot.
Runtime snapshot retention controls:
PROCESSED_RUN_RETENTION_DAYS (default 45)WEEKLY_ARCHIVE_AFTER_DAYS (default 365; older snapshots compact to weekly)- Prune utility:
python pipeline/prune_runtime_data.py [--processed-days N] [--weekly-archive-after-days N]
LLM integration status
The legacy ranking LLM is disabled (config/llm.yaml -> enabled: false).
The pipeline's semantic gate is a separate Jev decision-model call
(config/judge.yaml), used only for keep/drop and category signals. It
requires OPENROUTER_API_KEY and fails open to the deterministic policy.
Source health + circuit breaker + alerts (v1.4/v1.5/v1.6)
python pipeline/source_health.py update
python pipeline/source_health.py report
python pipeline/source_alerts.py
# state files: data/health/circuit_breaker.json, data/health/alerts_state.json
Architecture notes (why this design for now)
- We use a deterministic + configurable ranking core first, with a narrow Jev decision-model gate for semantic keep/drop and category.
- Reason: reliability and cost control. Jev is a typed decision model, not a free-form LLM ranker; the deterministic core keeps slots, quotas, freshness, and the final edition.
- Source/slot/category/provider constraints are explicit in config so we can tune behavior quickly without rewriting pipeline logic.
- The Jev gate is cached by a durable judgment store, so only new or changed candidates call the API.
- Current production flow diagram and knob guide:
docs/ranking-v2-flow.md
- Current operational snapshot (latest behavior/tuning):
docs/status/current-system-state.md
- Tuning governance playbook:
docs/status/tuning-governance.md
- Git hygiene guide:
docs/status/git-hygiene.md
- Source onboarding + filtering debug guide:
docs/how-to/sources-and-filter-debugging.md
- PostHog setup + dashboard runbook:
docs/how-to/posthog-setup-and-dashboard.md
- Jev quality-judge benchmark:
docs/how-to/jev-quality-benchmark.md
- Deck selector replay and grading:
docs/how-to/deck-selector-benchmark.md
- Model prices and Pareto comparison: contract and operations
- Confirmed product idea one-pagers:
docs/ideas/
- Doc images (UI audit screenshots, diagrams):
docs/assets/<topic>/
Config
config/sources.yaml: feed list + source weightsconfig/profile.yaml: platform relevance weights and keywordsconfig/ranking.yaml: ranking slots, caps, time decay, top-band, judge weight/headroomconfig/editorial.yaml: source roles, quarantine, story clusters, seen-memoryconfig/llm.yaml: legacy ranking LLM config (currently disabled/no-op)config/judge.yaml: Jev gate/category config (OpenRouter System One, thresholds, content fetch, judgment store)config/user_preferences.yaml: preference profile used by prompts when LLM mode is enabledconfig/prompts/label_system.txt, config/prompts/rerank_system.txt: legacy ranking prompt templates (inactive)scripts/oauth_login.sh: legacy OAuth helper for future re-enable
GitHub Actions
feed-full-publish (every two hours): full pipeline via
skills/ai-feed-digest-local/scripts/run_full.sh - collect, single edition
build (editorial + Jev gate + ranking), durable story sync/static pages,
prune, data commit + push. It intentionally does not build storylines.
Set the OPENROUTER_API_KEY repository secret to enable the Jev gate;
without it the build is deterministic and logs judge_skipped.- Storylines: external Claude Code routine every 5 hours; no GitHub Actions
workflow generates them.
feed-ops-summary (daily): operational health snapshotfeedback-sync (daily): reader feedback + CTR sync from PostHog and source
auto-tune apply (no-op if secrets are missing)ci (pushes and pull requests): Python and JavaScript regression suites on a
clean hosted runner- These scheduled workflows run on GitHub-hosted standard
ubuntu-latest
runners. The repository is public, so this standard runner usage is free and
unlimited; publishing no longer depends on the owner's Mac. The retired
self-hosted runner and rollback notes remain in
infra/actions-runner/README.md.
Repository secrets (optional)
POSTHOG_PERSONAL_API_KEY (feedback sync)POSTHOG_PROJECT_ID (feedback sync)
