Daily Reader
One page for the day's reading: the top posts from 11 subreddits, Bangladesh news, the
Ritholtz AM Reads + SatPost newsletters, and GitHub's trending top 10. Read/favorite state syncs across devices.
Live: https://reddit-scraper-lyart.vercel.app · Health: /api/status
Working on this repo? Read AGENTS.md first. It is the source of truth:
every scraper, the API, the hard rules, tests, monitoring and a runbook. This page is the
short version.
How it works
Mac mini (07:35 + 19:35) GitHub Actions (nightly + mornings + every 6 h)
headless browser → Reddit News, AM Reads, SatPost, Reddit backup, GitHub Trending
\ /
CSV files in data/, pushed to main
│ Vercel redeploys on every push
▼
server.py (FastAPI) → /api/data → templates/index.html
│
read + favorites ↔ localStorage ↔ Firebase
- Reddit comes from a real headless browser on the owner's Mac mini
(
core/scrape_reddit_browser.py, run by launchd via mac/reddit-browser.sh), because Reddit
blocks GitHub's servers. It saves real upvotes. Each list has backups on the Mac: the page,
then Reddit's JSON through the same browser, then its RSS feed, then a fresh browser profile.
Every run ends with a METHOD CHECK line saying whether each backup still works. If the Mac
misses a list for 36 hours, GitHub's RSS backup (core/scrape_top.py, every 3 hours) refills
it, showing each post's rank instead of upvotes. No Reddit API key anywhere.
- Tabs: Monthly and Yearly = the top 50 of all tracked subs, mixed by a tier score so one
big sub can't take over. That mix reshuffles on every scrape, so a post you haven't read yet
stays at the bottom ("Still unread from earlier lists") until you read it, or 30 days after it left.
News = every story from the last 7 days. AM Reads = today's list only. GitHub = the top 10
of github.com/trending (today, GitHub's own order), refreshed every 6 hours
(
core/scrape_github_trending.py).
- On the phone: the tabs sit in a bottom bar, the header slides away while you scroll
down, and pulling down at the top refreshes. Under the date: how many items you cleared
today and your streak of days with at least one cleared (counted across your devices). Tap
"N cleared today" to see those items (untick one to bring it back).
- Order: Monthly and Yearly have a Mixed / Most upvotes switch. Most upvotes re-sorts the
same 50 posts high to low; which posts make the list never changes. Chips under it show
one subreddit at a time (TIL, ELI5, ...); tap the chosen chip again for All.
- The 11 subs live in
core/reddit_common.py: dataisbeautiful, todayilearned, bestof,
getmotivated, UnethicalLifeProTips, LifeProTips, UpliftingNews, lifehacks,
Productivity, PersonalFinance, explainlikeimfive.
Commands
.venv/bin/uvicorn server:app --reload # local site: http://localhost:8000
.venv/bin/python -m pytest tests -q # unit + robustness tests (~10 s)
python3 tests/e2e_site.py http://localhost:8000 # real-browser checks (needs Playwright)
Deploy = push to main (Vercel). Tests also run on GitHub for every code push.
Rules that break production if ignored
- Keep
requirements.txt minimal (Vercel's 250 MB limit; no pandas/numpy since 10 Oct 2026,
see AGENTS.md §3). Scraper libraries go in requirements-scraper.txt.
vercel.json uses the legacy builds API; bundled folders go in config.includeFiles.server.py builds every path from BASE_DIR, and calls TemplateResponse with keyword args.- Files under
data/ must be added with git add -f (the folder is gitignored).
- Never show an invented upvote number, and don't change the tier-score math without the
owner's OK.
Details and the reasons behind each: AGENTS.md §5.
