Give Me Some Good News

A catalog, coverage map, and combined feed of the local newsrooms being built to last — nonprofits, worker-owned cooperatives, family and community papers, Native and Black-owned and Spanish-language outlets, and the small university and nonprofit magazines that make up an arts desk. Chains and hedge-fund papers are out of scope, and so are the metro dailies; the point is local newsrooms. They're surviving; this site is for reading them.

Live at https://givemesomegood.news.

The output is plain HTML with one small inline stylesheet. The only JavaScript is two small enhancements — infinite scroll on the feed, and tap-a-dateline-for-your-own-timezone — and the site works fully without either. IBM Plex is served from here rather than a font CDN, for the same reason images are cached locally: no third party needs to see who is reading. Everything links back to the newsrooms.

What's here

data/orgs.yaml          the hand-curated catalog (slug, url, place, model,
                        features, feed hints). Add a newsroom here.
data/orgs_literary.yaml the arts desk: literary, ideas and arts magazines
data/orgs_directory.yaml  bulk import, generated — never hand-edit
data/sources/           the directories the import came from, as fetched
data/us_places.tsv      Census gazetteer, for offline geocoding
data/institutions.yaml  funders, networks and associations
data/about_overrides/   hand-collected About text for sites whose firewalls
                        block scripted fetches (with source URLs)
data/catalog.json       machine-readable export, regenerated by the build
data/us_states.geojson  state outlines for the map (US Census-derived)
assets/fonts/           IBM Plex woff2, copied into site/fonts/ at build
assets/share.png        the social share card, drawn by share_card.py and
                        copied into site/ at build — not redrawn by it
schema.sql              Postgres schema; requires pgvector
givemesomegoodnews/     the pipeline (Python, five small dependencies):
  fetch_*.py            the crawl — feeds, about pages, support links
  images.py             the picture cache; migrate_images.py lays it out
  classify.py subjects.py language.py  what a story is about, and in what
  embed.py embedder.py  the vectors, and where to swap the embedder
  trending.py           what newsrooms are covering more than usual
  build_site.py         the build: what is written, and where
    shell.py            the page around the writing
    cards.py            one story as a card, and its rail
    pages.py            the pages that are not the feed
    topics.py           trending topics, as tags
    prose.py links.py dedupe.py  shaping, addresses, one story vs two copies
  share_card.py         the picture a link to the site unfurls into
  searchd.py admin.py   the two small services behind nginx
site/                   the generated site — output, not source, and not in
                        git: every page is rebuilt from the database, and the
                        database is what the backups hold

How it works

1. seed upserts all three catalog files into Postgres; the hand-curated orgs.yaml always wins on slug.
2. fetch_about pulls each org's About page and stores its main text — the catalog describes every newsroom in its own words, quoted and attributed, never paraphrased. Overrides in data/about_overrides/ win over live fetches (for WAF-blocked sites).
3. fetch_support crawls each org's homepage for the page where a reader can pay them — a Donate page, a membership page, a subscription. Every feed item links to it. Links are scored (payment processors rank high, newsletter signups are excluded); a hand-set support_url in orgs.yaml always wins.
4. fetch_feeds finds each org's RSS/Atom feed (configured hint → <link rel="alternate"> autodiscovery → common paths, including the Arc and TownNews patterns), then stores new articles with their headline, summary, author, the publisher's own categories, and an image. Syndicated wire copy (/wires/ URLs) is skipped. Every article gets an embedding at insert time.
5. classify fills in subjects the publisher didn't declare (below).
6. build_site renders plain HTML from the database: the catalog grouped by state, an Albers-projected SVG coverage map, the combined feed newest-first, one page per subject, per-org pages, a self-contained onepage.html, and the cross-region connections page.

Where the catalog comes from

orgs.yaml is hand-curated. orgs_directory.yaml is generated by import_directory from the Media and Democracy Project directory, kept in data/sources/. The importer drops chain-owned papers, drops national wire services (The Center Square, Public News Service) that the directory files as nonprofits, drops rows whose only website is a Facebook page, and keeps one entry per site. Re-running it never touches hand-written entries.

Coordinates come from the Census gazetteer via geocode.py — offline, no API key, deterministic. Directory city fields are free text, so most resolve to a place, some only to a county, and the rest to a state centroid; orgs.geo_precision records which.

Tags

Each newsroom carries tags for what it is rather than a paragraph about itself: Nonprofit, Co-op, Worker-owned, Family-owned, Journalist-owned, University-affiliated, Public media, Independent, and community tags like Black-owned, Spanish, Native-owned, LGBTQ+, Street paper, INN member. Ownership tags are matched from each newsroom's free-text model; community tags come from the directory. Every tag gets its own page.

The archive

Stories are never deleted. A piece that has aged out of its publisher's own feed may not be easy to find anywhere else, so prune only garbage-collects what nothing points at any more: cached image files no article references, and — nightly, with --stale-pages 7 — generated pages no build has written in a week, which are the pages of newsrooms that have left the catalog and the tail of a feed that has since grown shorter.

At the observed rate — around 1,500 stories a day across roughly a thousand live feeds, about two thirds carrying a picture — the database grows about 3GB a year and the image cache about 11GB. On a 50GB disk that is a few years of headroom. If it ever gets tight, prune --images-older-than N releases the local copy of old pictures while keeping the story, its link, its text and its image_url; a block-storage volume is the other answer.

Images

Feed images are downloaded, downscaled to 480px wide, and served from this site, named by SHA-1 of the source URL. Nothing is hotlinked: a publisher's server is hit once per image rather than once per reader, and the page keeps working when they reshuffle their CDN. image_url keeps the provenance. A broken image is a missing image, never a failed crawl.

Files are spread over 256 subdirectories by the first two characters of the name — site/img/a3/a3f2….webp. One directory holding every picture works until it doesn't: at a thousand new pictures a day it passes a million entries inside three years, and long before that, listing it, pruning it or backing it up means reading one enormous directory. The shard is derived from the name, so nothing records where a file went, and make migrate-images moves anything still at the top level down into place (the deploy runs it). nginx serves either spelling while the two overlap.

The pictures average 19 KB (480px WebP at quality 78; 66,000 files, 1.3 GB), growing about 7 GB a year. AVIF at quality 60 would take 18% off that at the same measured quality, which was judged not worth losing the browsers that cannot read it.

Subjects

Every article is tagged with one of eleven subjects — News, Politics, Education, Health, Environment, Housing, Business, Arts, Sports, Food, Opinion — in four layers, most trustworthy first. The first three involve no model; the fourth asks one.

1. declared — the publisher's own RSS categories, mapped through a whitelist in givemesomegoodnews/subjects.py. About four in five entries carry categories. The whitelist is what separates real subjects (education, sports) from place names (jefferson county) and CMS furniture (home: lead, uncategorized): an unmapped term contributes nothing rather than guessing.
2. url — whole path segments like /sports/ or /opinion/, for the newsrooms whose feeds declare nothing.
3. vector — classify averages the embeddings of everything labeled by layers 1 and 2 into one centroid per subject, then gives each remaining article its nearest centroid, if it is both close enough and clearly closer than the runner-up. Ambiguous stories stay untagged.

Layer 3 works with the hashing embedder because a centroid of hundreds of sports stories is dense in the vocabulary sports stories actually use. Matching against the bare word "Sports" would not work at all — it is one hashed bucket that most sports articles never contain. Thresholds are MIN_SUBJECT_SIM, MIN_SUBJECT_MARGIN, and MIN_SUBJECT_SEEDS. 4. jev — what the vectors leave blank (about a quarter of the archive) goes to Jev, TypeSafe's System One model, as one typed question per story: which of the eleven subjects, or none of them. It returns a probability for each option and writes nothing; the subject is taken only at MIN_JEV_P (0.6) or more, and obituaries, calendars and the like come back as none. Each story is asked about once (subject_judged_at), newest first, at most JEV_LIMIT (600) a build; a story costs about 750 input tokens (measured: 736). Without TYPESAFE_API_KEY the layer is skipped and the story stays untagged.

Search

/search is the one page that isn't a static file: it runs givemesomegoodnews.searchd, a small local HTTP service behind nginx that queries Postgres full-text search (websearch_to_tsquery, so quotes, OR and -word all work). Titles are weighted above summaries. It is a plain GET form, so it works without JavaScript, and the rest of the site keeps serving if the service is down.

Not a client-side index on purpose: shipping every headline and summary to every reader would cost more bandwidth than the whole feed does.

Trending

trending.html is what local newsrooms are covering more than usual. Once an hour, givemesomegoodnews.trending sets the last 24 hours against the 14 days before and stores a snapshot; the build renders the newest one.

Finding the topics is counting. Stories are read the way the front page reads them, reprints fold into one story, and each headline's words and two-word phrases are counted by the newsrooms that used them, so one busy feed or a wire piece in eleven papers cannot make a trend. A term has to reach three newsrooms in two states and several times its usual rate (the usual rate scaled to how many newsrooms published today). Terms sharing their stories merge into topics, and then topics whose stories sit close by embedding merge too: on September 11th, "25th anniversary", "remembers" and "first responders" are one event in different words. Topics are compared by the stories they do not share, so one headline that mentions two things cannot join them.

Naming the topics is the one place on the site a language model is used, and the page says so. The headlines of the strongest candidates — headlines only — go to DeepSeek, which labels each topic, merges the ones that are the same story, and rejects coincidences of vocabulary. It cannot add a story or change a count; anything it returns that does not refer to what it was sent is ignored. Without DEEPSEEK_API_KEY, or when the call fails, topics are named by the phrase their headlines share, and only phrase-built topics are shown. Snapshots are kept, one per hour; an hour whose candidates have not changed reuses the last hour's names without a call.

A trending topic is a tag like the others. The current topics run as a row of tags under the masthead on every page (search included — searchd looks them up again every five minutes), the front page opens with a section of them, and a story in a topic carries its tag in the card's rail. Each topic has a page of full cards at topics/<slug>.html, and keeps it for 48 hours after it drops off the list, marked as no longer trending, so a shared link still works that evening.

The vector part

Articles live in an articles table with an embedding vector(384) column and an HNSW cosine index (pgvector). That powers:

• Connections across regions — for recent articles, the nearest neighbor from a different state; the strongest pairs become connections.html. This is where you see five Western outlets carrying the same wildfire investigation, or Mississippi Today and Verite News circling the same police story from two states.
• Related links in the feed — cross-state echoes under feed items.
• Search — python3 -m givemesomegoodnews.search "housing eviction" -k 10.

The default embedder (givemesomegoodnews/embedder.py) is deterministic local feature hashing: unigrams+bigrams hashed into 384 signed buckets, L2 normalized. No model downloads, no API keys, and good enough to catch shared-vocabulary stories. It is built to be swapped: any embedder with .dim, .name, and .embed(texts) can be registered in get_embedder() (384 dims drop in cleanly, e.g. MiniLM; for other dimensions, ALTER TABLE articles ALTER COLUMN embedding TYPE vector(N) then python3 -m givemesomegoodnews.embed --all).

Running it

Postgres 16 with pgvector, or use the included compose file:

docker compose up -d
export DATABASE_URL=postgresql://givemesomegoodnews:givemesomegoodnews@localhost:5433/givemesomegoodnews

pip install -r requirements.txt
make all        # db + seed + about + feeds + build
make serve      # http://localhost:8000

A fresh checkout has no site/ until something builds one; make all or make build writes it, and the VMs rebuild every fifteen minutes.

make refresh (feeds + build) is the recurring job; it's idempotent — articles dedupe on canonical URL. A crontab line like 17 */3 * * * cd ~/givemesomegoodnews && make refresh keeps the feed steady. DATABASE_URL defaults to postgresql:///givemesomegoodnews.

Useful invocations:

python3 -m givemesomegoodnews.fetch_about --force            # refetch all about pages
python3 -m givemesomegoodnews.fetch_about hell-gate racket   # just these slugs
python3 -m givemesomegoodnews.fetch_feeds the-city           # crawl one org
python3 -m givemesomegoodnews.fetch_support --force          # re-find support links
python3 -m givemesomegoodnews.classify --dry-run             # tag report, no writes
python3 -m givemesomegoodnews.search "school board recall"   # vector search
python3 -m givemesomegoodnews.trending --dry-run --no-label  # trending topics, no writes

Backups

ops/backup/pull-backup.sh runs on a machine at home, from cron, and pulls: one directory per night holding the database, and the picture cache beside them. It only ever copies down and only ever adds — no --delete, no overwriting, nothing written to the server.

A night costs about 20 MB, plus that day's new pictures. Two things are deliberately left out, which is most of the weight:

• The embedding column, 27 MB of a 45 MB dump. It is computed from the headline and summary, which are backed up, and make embed puts it back — verified identical to what production stores, to six decimals. This holds only while EMBEDDER=hashing, which is deterministic and local.
• Indexes. No pg_dump ever holds index data, only the statements that rebuild them, and schema.sql is already those statements.

Restoring is ops/backup/RESTORE.md, which the pull also copies beside the backups: schema, load, re-embed, rebuild the index, then --backfill the days since from the feeds. A full restore of 101,280 articles takes about ten seconds plus the re-embedding; it was done for real when the VM was erased on 23 October 2026 (ops/exe/README.md has the fresh-VM recipe that goes with it). The pull checks first that the VM can be reached, and says why not when it cannot — after a rebuild, the VM's ssh host key has changed, and the machine that pulls has to forget the old one.

Nightly full dumps rather than incremental ones, and that is measured, not assumed: this database writes about 989 MB of WAL a day (the vector index churns), while a whole compressed dump is 45 MB. WAL archiving would move twenty times more data for no more safety.

Adding a newsroom

Add an entry to data/orgs.yaml (slug, name, url; lat/lon for the map; about_url/feed_url/support_url only if autodiscovery needs help), then:

make seed && python3 -m givemesomegoodnews.fetch_about <slug> && python3 -m givemesomegoodnews.fetch_feeds <slug> && make build

Provenance and politeness

• About text is copied from each org's About page deliberately — the point is their self-description, in their voice — always quoted, dated, and linked to the source. If any newsroom would rather not appear, delete its entry and reseed.
• The crawler fetches one About page and one feed per org per run, with a clear bot User-Agent. Feeds are the interface publishers offer for exactly this; headlines and summaries link to the original stories.
• Known gaps, current as of the first crawl: about text pending for a few sites behind bot walls (Gothamist meters bots via Tollbit; Verite, Mississippi Today, and Documented block non-browser TLS fingerprints — their feeds still work); no feeds found for the print-first Daily Sitka Sentinel and The News Reporter, for Invisible Institute and Wick Communications, or for The Beacon (its TLS is currently broken).

The share card

A link to this site posted anywhere unfurls into assets/share.png: the masthead over the same Albers map, with a red dot on every town that has a newsroom in the catalog. Every page points at it through Open Graph and twitter:card, and config.SITE_DESCRIPTION is the sentence beside it — the same one the pages give as their <meta name="description"> when they have nothing more particular to say.

It is a committed asset, because the alternative is a browser in the fifteen-minute build. Redraw it when the catalog has grown enough to change the map, and commit the PNG:

python3 -m givemesomegoodnews.share_card   # needs a headless Chrome; $CHROME works

Map

givemesomegoodnews/albers.py implements the standard Albers equal-area conic for the lower 48 (parallels 29.5°/45.5°, central meridian 96°W) and renders state outlines plus one dot per newsroom, clustered rings when several share a city. Alaska/Hawaii and no-fixed-geography orgs are listed below the map. State outlines derive from US Census data (data/us_states.geojson).