Latent SpaceMail: real AI research, explained plainly

What's different: one spotlight research paper and the week's top AI news, picked, summarized and sent by a team of AI agents that checks its own work every run.

• Agentic orchestration: six agents on Google Cloud, coordinated by Pub/Sub, plus a LangGraph review loop inside one of them.
• Live monitoring: health checks, drift detection, cost tracking and an LLM judge report on every run.
• Measured, not claimed: evaluation results are generated from scripts, with confidence intervals.
• Real cloud operations: Cloud Run, Cloud Build, GitHub Actions CI, secrets management and no-redeploy rollback switches.

⚙️ How it works

Stage-by-stage detail: docs/architecture.md

%%{init: {"themeVariables": {"fontSize": "16px"}, "flowchart": {"nodeSpacing": 26, "rankSpacing": 40, "padding": 10, "curve": "basis"}}}%%
flowchart LR
    ORC["☁️ Sun 12 PM<br/><b>Orchestrator</b>"] --> A1A["<b>Agent 1a</b><br/>Pick paper"]
    ORC --> A1B["<b>Agent 1b</b><br/>Fetch news"]
    A1A --> A2A["<b>Agent 2a</b><br/>Summarize"]
    A1B --> A2B["<b>Agent 2b</b><br/>Summarize"]
    A2A --> FAN{"<b>Fan-in</b><br/>2 of 2"}
    A2B --> FAN
    FAN --> A3["<b>Agent 3</b><br/>Compose"]
    A3 --> DB[("Firestore<br/>run state")]
    DB --> A4["☁️ Mon 7 AM<br/><b>Agent 4</b> · Send"]
    DB --> HC1["☁️ Sun 1:15 PM<br/><b>Draft check</b>"]
    DB --> HC2["☁️ Mon 7:10 AM<br/><b>Send check</b>"]

    classDef agent fill:#4f46e5,stroke:#312e81,color:#fff
    classDef fan fill:#7c3aed,stroke:#4c1d95,color:#fff
    classDef store fill:#d97706,stroke:#92400e,color:#fff
    classDef health fill:#059669,stroke:#065f46,color:#fff
    class ORC,A1A,A1B,A2A,A2B,A3,A4 agent
    class FAN fan
    class DB store
    class HC1,HC2 health

Agent	What it does
1a Papers	Picks the week's spotlight paper from Hugging Face's trending list and scores it with Claude.
1b News	Gathers Hacker News and NewsAPI stories, filters by language and sorts them into 7 categories.
2a / 2b Summaries	Write a short summary of the paper (2a) and of each article (2b), in parallel.
3 Compose	Picks the best articles per category, writes the editor's note and builds the HTML email.
4 Send	Sends each subscriber their personalized version through SendGrid on Monday morning.
Health checks	A draft check on Sunday and a send check on Monday email a report on every run.

Agents hand work to each other through Pub/Sub; Firestore holds each run's state. Cloud Scheduler triggers the pipeline, the send and both health checks.

🕸️ Agentic orchestration

• Between agents: Pub/Sub events plus a Firestore counter that joins agents 2a and 2b before agent 3 starts.
• Inside an agent: agent 1b is a LangGraph graph that sends low-confidence articles to a small tool-using review loop, and agent 3 runs a duplicate-removal graph on each section's picks.
• Failure handling: every agent records its errors to Firestore, a watchdog kills hung runs, and the crash-prone agent 2b retries in an isolated process.
• Rollback: AGENT1B_MODE=single_pass restores the original linear code without a redeploy.

%%{init: {"themeVariables": {"fontSize": "16px"}, "flowchart": {"nodeSpacing": 28, "rankSpacing": 44, "padding": 10, "curve": "basis"}}}%%
flowchart LR
    COL["<b>Collect</b><br/>HN + NewsAPI<br/>drop paywalls"] --> LANG["<b>Language</b><br/>English or<br/>French only"]
    LANG --> CAT["<b>Categorize</b><br/>7 topics +<br/>confidence 1-5"]
    CAT -->|"confident"| FIN["<b>Finalize</b><br/>write results<br/>+ audit log"]
    CAT -->|"unsure"| LLM["<b>Review: think</b><br/>Claude decides"]
    LLM -->|"needs text"| TOOL["<b>Review: act</b><br/>fetch the article"]
    TOOL --> LLM
    LLM -->|"answer"| FIN

    classDef step fill:#4f46e5,stroke:#312e81,color:#fff
    classDef loop fill:#7c3aed,stroke:#4c1d95,color:#fff
    classDef out fill:#059669,stroke:#065f46,color:#fff
    class COL,LANG,CAT step
    class LLM,TOOL loop
    class FIN out

Agent 3 has a second, smaller graph that removes duplicate stories from each section's picks, once per section and selection pass. One Claude call checks all the picks, the highest-scoring article of each duplicate group is kept, and replacements from the ranked runners-up are checked one at a time until the section is back to its size. Any error falls back to the original picks, and AGENT3_DEDUP_MODE=off disables it without a redeploy.

%%{init: {"themeVariables": {"fontSize": "16px"}, "flowchart": {"nodeSpacing": 28, "rankSpacing": 44, "padding": 10, "curve": "basis"}}}%%
flowchart LR
    SEL["<b>Select</b><br/>picks + ranked<br/>runners-up"] --> CHK["<b>Check</b><br/>one call finds<br/>duplicate groups"]
    CHK -->|"none found"| FIN["<b>Finalize</b><br/>section + audit log"]
    CHK -->|"duplicates"| RES["<b>Resolve</b><br/>keep top HN score,<br/>drop the rest"]
    RES -->|"section full"| FIN
    RES -->|"gap to fill"| REF["<b>Refill</b><br/>check next runner-up<br/>against all seen"]
    REF -->|"still a gap"| REF
    REF -->|"full, or out of<br/>runners-up or tries"| FIN

    classDef step fill:#4f46e5,stroke:#312e81,color:#fff
    classDef loop fill:#7c3aed,stroke:#4c1d95,color:#fff
    classDef out fill:#059669,stroke:#065f46,color:#fff
    class SEL,CHK step
    class RES,REF loop
    class FIN out

Why LangGraph runs inside agents but not between them: ADR 0001 and ADR 0002.

📡 Live monitoring

Every weekly run ends in an email report to the maintainer, whether or not anything is wrong, so a missing email is itself a signal.

Signal	What it watches	Response
🟢 Health checks	Every pipeline stage, the composed newsletter and delivery	Email report: all clear or problem detected
🟢 Failure recording and watchdog	Agent errors and hung or crashed runs	Written to Firestore; hung runs are killed
🔵 Drift	Agent 1b's confidence, category mix and review rate	Flagged only for large, significant shifts
🔵 Token and cost	LangSmith token and cost totals per run	Flagged on big jumps versus prior weeks
⚪ Summary judge	A weekly sample of summaries checked against their sources	Report only (see Results)
⚪ Click signal	Aggregate link clicks, with no subscriber data	Informational section in the report

🟢 acts on failures   🔵 flags large shifts   ⚪ informational only

Details: docs/monitoring.md

☁️ Cloud, CI and operations

Area	What's in place
Cloud	10 Cloud Run services from one Docker image, Pub/Sub, Firestore, Cloud Scheduler, Secret Manager, Firebase Hosting and Auth
Build and deploy	Cloud Build builds every image in parallel; deploys are separate, deliberate steps; API keys are mounted from Secret Manager
CI	GitHub Actions runs the full test suite (stubbed Claude client, no keys) and a Docker build check on every push and pull request
Evaluation	Paid evals run on demand behind a cost estimate and a hard cap; results are JSON files with confidence intervals
Observability	LangSmith tracing, the weekly health check report, structured logs
Reliability	Idempotent fan-in, isolated retries, a 24-hour stale-newsletter guard, rollback switches
Stack	Python 3.11, Anthropic Claude, LangGraph, FastAPI, SendGrid, Firebase Auth

Deployment and local setup: docs/development.md

💸 What one issue costs

Each weekly issue now costs about $0.55 in model calls, down from about $2.06 at the start of the cost work (a 73% cut, measured on one production run on 2026-10-28). Picking the paper (agent 1a) and summarizing the news (agent 2b) are the two big lines: agent 1a reads the full text of only its top 8 papers after scoring all 35 abstracts, and agent 2b summarizes only a shortlist of about 70 articles instead of all 500. Sorting the news (agent 1b, the language filter and categorizer) moved from Claude Haiku to DeepSeek V4.1 Flash, which cut its line by about 90% with no measurable change in what gets through; the paper scoring and summaries stay on Haiku because DeepSeek scored noisier on papers and was easier to prompt-inject on summaries.

Costs are list-price estimates per run (LangSmith's for Claude, the provider's list for DeepSeek), archived by the health check. Generated by python -m evals.fetch_pipeline_cost and python -m evals.make_charts. More: docs/monitoring.md and the full step-by-step record in docs/cost-reduction-roadmap.md.

📊 Results

All charts below are generated from evals/results/*.json by python -m evals.make_readme_table, never drawn or typed by hand. Each has its table in docs/evaluation.md.

Does the review loop help?

Agent 1b's review loop was compared with the original single-pass code on the same hand-labeled articles. It did not measurably help: the differences sit inside the noise, and the model's own confidence did not predict its mistakes. Reported plainly.

Prompt-injection tests

Hand-written attacks (forced categories, tag breakouts, prompt leaks, planted URLs) were run through the real agent code paths, once with the injection and once without. The attacks that worked before the fixes were stopped by input sanitising, guard text on every Claude call and a fetch guard. The checks catch canary-style compliance only, and small samples mean "none observed" is not "safe".

Summary judge

A second Claude model checks a weekly sample of summaries against the text they were written from. Calibrated against 40 summaries I labeled by hand, it agreed with me no better than chance (kappa in the chart), so it runs report-only and can never trigger an alert.

Duplicate removal

Agent 3 checks each section's picks for the same story told by several outlets and swaps duplicates for the next-best article. On 57 hand-labeled article sets (33 duplicate groups), both Haiku and Sonnet found roughly nine in ten duplicates and removed a unique article in under 1% of cases; the intervals overlap, so the cheaper Haiku stays. Small sample, and only the first check is scored, not the replacement path.

Full results and caveats: docs/evaluation.md

📚 Documentation

• Architecture: stage-by-stage detail, orchestration and the subscription system
• Design decisions: why the system is built this way
• Monitoring: health checks, drift, cost, judge and click signal
• Evaluation: full review-loop, prompt-injection and duplicate-removal results
• Configuration: environment variables
• Local development and deployment
• Repository structure
• Evaluation harness conventions

License

MIT.