New Scientist Story Ideas

Automated pipeline that searches PubMed for recent mental health and brain science research, drops anything the press already has, uses Claude to write a pitchable story-ideas digest, fact-checks its own output against the source abstracts, and publishes every run to a browsable dashboard — https://new-scientist-story-ideas.vercel.app

Built on the same methodology as senior-research-digest: one sequential pipeline per day, a rotating subject focus, markdown digests that accumulate in outputs/ and are never overwritten, a self-fact-check pass, and a per-topic memory file that gives the trends section a longer horizon than the last run. The differences from that repo are noted in Where this differs below.

How it works

.github/workflows/daily-digest.yml runs scripts/main.py on a daily cron (10:00 UTC / 5:00 AM CDT) via GitHub Actions. Each run:

1. Searches PubMed (pubmed.py) across 325 curated mental health, psychiatry and brain science journals (journals.py) for articles from the last 30 days, optionally filtered to a subject focus.
2. Picks today's focus — either a manual override, a fixed topic from config/digest_config.json, or the next topic in the daily rotation (main.py's DEFAULT_FOCUS_ROTATION / config["focus_rotation"]).
3. Screens cheaply (screening.py) — drops studies already covered by a previous digest, plus editorials, errata and animal-only work, then ranks the rest by how strongly the title signals novelty.
4. Drops what's already been reported (media_filter.py) — checks each candidate against Google News via SerpAPI and discards anything with 3+ hits.
5. Checks newscientist.com specifically (ns_check.py) — a general media check isn't enough here: a study can clear it and still be something New Scientist itself already ran, which is more disqualifying than a stranger outlet covering it. check_ns_overlap runs a site:newscientist.com SerpAPI search per surviving candidate and drops anything with 2+ hits (a lower bar than the general filter's 3, for the same reason). Separately, fetch_ns_style_examples pulls a handful of real, currently-published New Scientist headlines in the run's subject area, once per run, for the digest prompt to use as a style reference.
6. Generates the digest (digest_generator.py) — Claude selects the most pitchable studies from up to 40 abstracts and writes a structured entry for each (headline, novelty type, NS fit score, study summary, why it matters, pitch angle, wider angle, caveats), matching the real NS.com headlines supplied as a style reference where available.
7. Fact-checks itself (fact_checker.py) — a second Claude pass compares every entry against the original abstract and flags inaccuracies with a ✅/⚠️/❌ verdict per study. It also challenges the NS fit score.
8. Compares against history (trends.py) — Claude compares the new digest to the most recent prior digest on the same topic and to a running per-topic memory file (topic_memory/<topic>.md), producing a "Research Trends & Continuity" section plus a "Bigger Picture: Feature Pitch" if the batch suggests a larger story, with 3-4 real outlets that specific angle could go to.
9. Records what it covered — selected PMIDs go into seen_pmids.json so the same study is never pitched twice.
10. Rebuilds the dashboard (build_dashboard_data.py) — parses every digest + fact-check in outputs/ into docs/data/.
11. Commits everything back — outputs/, topic_memory/, docs/ and seen_pmids.json are committed and pushed by the workflow, so history accumulates in the repo and Vercel redeploys automatically.

Nothing is ever overwritten: if two digests would land on the same filename (e.g. two runs on the same topic in one month), main.py appends "(Part N)".

Where this differs from senior-research-digest

Five deliberate changes, each because this repo generates pitches rather than reader-facing summaries:

• A media filter. A digest written for readers is still useful when the study made the news; a pitch is not — an editor won't commission a story the Guardian ran last week. media_filter.py sits between the title screen and the abstract fetch. It degrades gracefully: with no SERPAPI_KEY set, every candidate passes and is labelled "Not verified" rather than being silently presented as a fresh find. The same holds when the key is set but SerpAPI is not answering — a circuit breaker stops the screen after SERPAPI_FAILURE_THRESHOLD consecutive failed queries (default 3) instead of paying the full timeout on all 60 lookups. On 2026-09-05 every lookup timed out and the un-broken version spent ~16 minutes doing it, taking the 30-minute job down with it; the surviving candidates are now marked "SerpAPI unavailable this run" so a run that could not screen never reads as a run that found everything clean.
• A newscientist.com-specific check, on top of the general media filter. ns_check.py runs after media_filter.py and asks a narrower, more exacting question: not "has anyone covered this?" but "has New Scientist itself already run it?" — which sinks a pitch even when the general filter passes it, so check_ns_overlap uses a lower hit threshold (2, vs. the general filter's 3). The same module's fetch_ns_style_examples pulls real, current New Scientist headlines in the run's subject area once per run and hands them to digest_generator.py as a style reference, so entries are framed the way New Scientist actually writes rather than in a generic science-news voice. Both degrade the same way the media filter does: no key, no block — entries are labelled "Not verified" and the style reference is simply omitted.
• A PMID ledger. seen_pmids.json records every study ever written up, and those are dropped before screening. A study that has been pitched once is not a new story idea. The senior digest has no equivalent, because re-covering a finding for a new audience is fine there.
• Different entry fields. Its two "story angles" (to/about an audience) are replaced by a Pitch angle (how to sell it to New Scientist) and a Wider angle (a different, larger story elsewhere), plus Novelty, NS fit (1-10, honest use of the range), Media check, and NS.com check per entry.
• Vercel with Git integration left connected. The senior repo deploys via an explicit vercel deploy --prod from inside docs/, with Git integration deliberately disconnected, because a repo-root connection there builds an empty site and steals the production alias. Here the project's , so the connected-Git build finds and every push to is a deploy — no deploy step in the workflow and no secret. Don't clear that Root Directory setting; that is the whole reason this arrangement is safe.

Two portability fixes were also needed at this repo's scale, both worth knowing if you sync changes back the other way:

• ISSN batches are sampled round-robin. The senior pipeline searches ~146 journals over 90 days and rarely fills its PMID quota, so taking batches in order was harmless. Here, 325 journals over 30 days returns 5,000+ articles on a broad run, and an in-order fill would exhaust the 300-PMID quota on the first two batches — leaving ~275 journals unread every day. search_by_issns now draws from every batch newest-first and interleaves them.
• Text is read from all content blocks. response.content[0].text returns "" on Claude Opus 5, because adaptive thinking is on by default and the first block is a thinking block. That fails silently — an empty digest, no exception. llm.py's response_text joins the blocks that actually carry text; all three Claude passes go through it.

Dashboard

docs/index.html is a static, no-build dashboard that reads docs/data/ and lets you browse every digest ever generated — filter by topic, search by headline/PMID/journal, see each study's NS fit score and fact-check verdict inline, and export any run to .docx. When a run includes a feature pitch, a "Jump to Feature Pitch" link appears in the run header.

PMIDs cited in the trends and feature-pitch prose link to PubMed, the same as the study cards. All links take var(--accent) from a base a rule, never browser blue (#0000EE), which is hard to read in dark mode. The model cites PMIDs three ways, and linkPmids handles each: "PMID 42635035", a list ("PMIDs 42660579, 42635370"), and a bare "(42447742)". A bare number only counts inside parentheses and at 7+ digits, so long figures in the prose are not mistaken for PMIDs.

The type filter in the sidebar picks out one Novelty type — Counterintuitive, First-in-class, New mechanism, Overturns prior research, Failed replication, or Included for relevance. It is the same "Counterintuitive finding" tag the topic digests carry in research-digest-dashboard, where this repo's studies used to be filterable by it. It narrows the run list to runs with at least one study of that type, badges each run with its count, and shows only those studies in the open run (and in its .docx export). The options and their study totals come from novelty_types in data/index.json; per-run counts are each run's novelty_counts.

The data is split so first load stays flat as runs accumulate: data/index.json holds only what the sidebar and search box need, and each run's full body lives in data/runs/<id>.json, fetched on demand when that run is opened.

Archive and delete

Every study card has Archive and Delete buttons, plus a status filter (Active only / Archived / Deleted / All) in the run header. Status is stored per-PMID in the browser's localStorage, not written back to the digest files or docs/data/ — it's a per-viewer preference, the same mechanism already used here for study sort order and the minimum-score filter, and this dashboard has exactly one viewer, so there was no case for a shared backend (unlike research-digest-dashboard's move to Neon Postgres, which exists because that dashboard has more than one). Archiving or deleting a study never changes its underlying data — a study hidden by "Active only" is still in docs/data/, restorable at any time with the Restore button. Clearing the browser's site data resets every study back to active.

This status is additive to the digest's own fields (Novelty, NS fit, Media check, NS.com check) — it's a dashboard-only view state layered on top, not a replacement for anything the pipeline writes.

Vercel serves the /docs folder of main (set as the project's Root Directory), so a push is a deploy. To view it locally without pushing:

cd docs && python3 -m http.server 8000
# open http://localhost:8000

To rebuild the dashboard data by hand (e.g. after editing a past digest):

python3 scripts/build_dashboard_data.py

This parser expects the exact markdown structure Claude is instructed to produce in digest_generator.py's SYSTEM_PROMPT (### N. Headline, **Journal:**, **PMID:**, **NS fit:**, **Story angles:** with Pitch angle/Wider angle bullets, etc.) and fact_checker.py's per-study verdict line (**PMID:** ... | **Verdict:** ...). If either prompt's output format changes, update the corresponding regexes in build_dashboard_data.py too.

No longer feeds research-digest-dashboard

Until 2026-09-14 this repo also emitted a flat data/results.json feed that research-digest-dashboard pulled in as its "New Scientist — Mental Health" source. That source was retired: the story ideas now live only on this repo's own dashboard. The feed builder (scripts/build_aggregator_feed.py), the feed file and the workflow's dispatch to that repo were removed in the same change; all three are in git history if the feed is ever wanted back. Nothing was lost with them, since the feed was regenerated from outputs/ on every run.

The archive

This repo previously ran a different pipeline that accumulated every screened study into a single data/results.json — 904 studies. That file is preserved verbatim at archive/legacy-results.json.

scripts/migrate_legacy_results.py back-converted the 778 of those published within 60 days of the cutover into 46 markdown digests in outputs/, one per original run date, so the dashboard has history from day one. They are stamped **Source:** Converted from the previous JSON pipeline and carry no trends section and no per-study fact-check verdicts, because the old pipeline never produced those — those fields render as "—" rather than being invented. The old pipeline's nested ns_pitch object is flattened into the Pitch angle field; Wider angle reads "Not generated by the previous pipeline."

All 904 PMIDs — not just the converted ones — were written into seen_pmids.json, so nothing the old pipeline covered can be pitched again.

The migration is a one-time script that has already been run. It skips files that already exist, so re-running it is safe but pointless.

Running locally

pip install -r requirements.txt
export ANTHROPIC_API_KEY=...
export SERPAPI_KEY=...       # optional — without it the media filter is skipped
export NCBI_API_KEY=...      # optional — raises PubMed rate limit from 3 to 10 req/sec
cd scripts
python3 main.py

Set DIGEST_FOCUS to override today's rotation (e.g. DIGEST_FOCUS="sleep" python3 main.py). An empty value is treated as "no override" and falls through to the rotation, so the broad digest cannot be forced this way — it only runs when the rotation lands on it.

A run takes roughly 15-25 minutes, most of it the SerpAPI lookups — the general media filter (1 req/sec, up to 60) plus the newscientist.com-specific check (1 req/sec, up to 60, but only over whatever survives the general filter, so usually well under that) — and the three Claude passes.

The Claude passes go through the Message Batches API (scripts/llm.py), which halves their cost but answers asynchronously: usually a few minutes per pass, occasionally much longer. A pass waits up to 45 minutes, then cancels its batch and runs live at full price, so a run can take an hour or more on a slow day. The job cap is 240 minutes to allow for that.

Choosing topic wording

A focus is ANDed into the PubMed query as an exact phrase match ("<focus>"[Title/Abstract]) with no synonym or MeSH expansion, so the exact string decides how many articles a topic can draw from. Measured across all 325 journals over one 30-day window:

Wording	Articles		Wording	Articles
(broad — no focus)	5347		artificial intelligence	117
depression	685		ADHD	112
anxiety	602		psychotherapy	97
stress	509		neuroimaging	96
memory	364		addiction	89
sleep	360		social media	88
autism	214		exercise	87
schizophrenia	214		dementia	77
trauma	193		loneliness	56
decision-making	176		cannabis	55
bipolar	124		obsessive-compulsive	50
emotion regulation	121		consciousness	41

Aim for roughly 75+ articles; the pipeline reads up to 40 abstracts. The 20 topics in the rotation are the ones that clear that bar. Measured-but-excluded topics sit below it and would produce a thin digest on their own — psychedelic (36), brain stimulation (30), ketamine (22), adolescent mental health (20), gut microbiome (8) — they are better reached through the broad run than as their own rotation slot.

Note that closely related phrasings pull largely separate literature — a topic is only as broad as its literal string.

Renaming a rotation topic breaks two continuity links, both keyed to the focus string: topic_memory/<slug>.md (rename the file to match) and the prior-digest lookup in trends.py, which matches the Focus field exactly and so will not see digests filed under the old name.

Configuration

Edit config/digest_config.json and commit — the next run picks it up:

• publication / section — who the pitch angles are written for.
• reader_profile — the readership the digest is told to write toward.
• subject_focus — leave empty to rotate through focus_rotation daily, or set a fixed topic to always search that one.
• focus_rotation — the list of topics rotated through by day-of-year.
• days_back — PubMed lookback window (days).
• media_threshold — news hits at or above which a study is dropped as already covered.
• ns_site_threshold — newscientist.com hits at or above which a study is dropped as already run there. Lower than media_threshold by design: NS.com covering it is more disqualifying than a general outlet doing so.
• ns_style_examples — how many real, current New Scientist headlines to pull as a style reference for the digest prompt.
• max_candidates — how many screened studies go to the media filter.
• max_abstracts — how many survivors get full abstracts fetched and sent to Claude.

The journal list

data/Mental Health - Brain Mental Health.csv is the editable source of truth. Edit it, then regenerate the module the pipeline imports:

python3 scripts/build_journals.py

Rows with neither a print nor an electronic ISSN are dropped, since the PubMed query is built entirely from ISSNs.

Required secrets (GitHub Actions)

Set these under repo Settings → Secrets and variables → Actions:

• ANTHROPIC_API_KEY — required, used for digest generation, fact-checking and trends synthesis.
• SERPAPI_KEY — optional but strongly recommended; without it the media filter and the newscientist.com-specific check (ns_check.py) are both skipped, entries are labelled "Not verified", and no style reference is fetched.
• NCBI_API_KEY — optional, raises the PubMed rate limit.

Optional tuning env vars (all have working defaults, none need setting):

• DIGEST_USE_BATCH_API — set to 0 to send the Claude passes live at full price instead of through the Batch API, for a manual run that has to finish now (default on).

• SERPAPI_TIMEOUT_SECONDS — per-lookup read timeout (default 15).

• SERPAPI_FAILURE_THRESHOLD — consecutive failed queries before a SerpAPI engine is given up on for the rest of the run (default 3). Queries, not attempts: one slow lookup is not an outage.

• SERPAPI_ENGINE_TIME_BUDGET_SECONDS — wall-clock ceiling per engine (default 240), covering the slow-but-succeeding case a failure count never sees.

• SERPAPI_TIME_BUDGET_SECONDS — ceiling across every SerpAPI call in the run (default 600). Outranks the per-engine budgets: past this, everything stops, because the risk being managed is the job ceiling rather than the upstream.

The breaker (scripts/serp_breaker.py) is shared by both SerpAPI screens and counts per engine, so the media filter losing Google News does not also cancel the newscientist.com check — they fail independently. Sharing one budget across both matters too: run back to back with a per-screen budget, the two of them could spend twice the job ceiling while each looked well-behaved alone. With a dead upstream both screens together now cost ~90s instead of ~32 minutes.

Repo layout

scripts/
  main.py                    entry point — orchestrates the full pipeline
  pubmed.py                  PubMed E-utilities client
  journals.py                generated list of (journal, ISSN, categories)
  build_journals.py          regenerates journals.py from the CSV
  screening.py               pre-Claude title screen and novelty ranking
  media_filter.py            SerpAPI "already covered?" check
  ns_check.py                newscientist.com-specific overlap check + style reference
  llm.py                     shared Claude call + text extraction helpers
  digest_generator.py        Claude prompt + call that writes the digest
  fact_checker.py            Claude prompt + call that fact-checks the digest
  trends.py                  Claude prompt + call for trends/feature pitch + topic memory
  build_dashboard_data.py    parses outputs/*.md into docs/data/
  migrate_legacy_results.py  one-time back-conversion of the old JSON archive
outputs/                     every digest + fact-check ever generated (.md)
topic_memory/                per-topic running memory used by trends.py
docs/                        static dashboard, served by Vercel (Root Directory)
archive/legacy-results.json  the previous pipeline's full 904-study archive
data/
  Mental Health ... .csv      the curated journal list
config/digest_config.json    publication, rotation and threshold settings
seen_pmids.json              every PMID ever written up — never pitched twice
.github/workflows/           daily cron (daily-digest.yml)