RSS Generator

---

Documentation

Doc	Read it when
docs/agent-guide.md	Start here. Complete tour of what the project does, how every pipeline stage works internally, and the invariants that silently break output.
PROJECT.md	Adding a site: selector discovery, SPA/Cloudflare strategies, testing protocol.
docs/development.md	Local dev setup, Docker image, workflow.
docs/research.md	Background on RSSHub / rss-proxy and the ideas borrowed from them.
AGENTS.md	Conventions for AI agents working in this repo.

---

Published feeds

All feeds are listed on the index page, grouped by category and language — that page and the OPML below are generated from config/sites.yaml, so they are always current. The index page and the reader are behind a password; the feeds themselves and feeds.opml are public, because an RSS reader fetches them with nowhere to type one (see Security).

 📥 Download OPML  — import into Inoreader or any RSS reader, pre‑sorted into folders.

Feeds are enriched with TMDb posters, years, IMDb links, and YouTube trailer links. Each run regenerates in place: a source that fails transiently keeps its last published feed, and only a source the engine declares gone is dropped, after one retry pass.

Recent additions (migrated from PolitePaul): xfilme.ro (Filme + Episodes), XCinema.ro (Filme) and PortalulTauTV.info — all RO feeds, scraped directly with the http method.

---

Quick start

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# Install Playwright browser
python -m playwright install chromium

# Full pipeline
PYTHONPATH=. python scripts/generate_feeds.py
PYTHONPATH=. python scripts/enrich_feeds.py
PYTHONPATH=. python scripts/fix_feeds.py
PYTHONPATH=. python scripts/generate_index.py

# Just one site (matches `name` or the feed_file, with or without .xml;
# repeatable). Handy when iterating on one scraper.
PYTHONPATH=. python -m core.cli generate --site showrss

# One feed through the WHOLE flow (get -> enrich -> process). Order matters:
# generating rewrites the feed from the site's own RSS, which discards the
# article bodies enrichment added. Ends with a per-item body-length report.
./start.sh one gabriel-ursan
./start.sh one ghacks thehackernews

# Serve locally (static files only — the generated index/feeds, no UI)
./scripts/serve.sh

# Local Inoreader-style feed review UI (reads feeds/*.xml directly)
./scripts/start_reader.sh          # opens http://localhost:8080/reader

Generated feeds are committed only by GitHub Actions: the update.yml workflow rebuilds feeds/ in place (generate_feeds.py → enrich_feeds.py → fix_feeds.py → generate_index.py) and pushes feeds/*.xml + feeds.opml back to main so raw.githubusercontent.com serves the latest copy. Each run updates the committed feeds rather than replacing the whole set, so a source that fails transiently keeps its last published feed. Never commit a feed by hand — local feeds/, index.html and feeds.opml are gitignored, and a local run writes them to the working tree only.

Local feed reader

scripts/local_reader.py serves an Inoreader-like review UI on http://localhost:8080/reader. It reads the generated feeds/*.xml directly — no TMDb key, no network scraping, no CI dependency — and groups feeds into categories (Movies, Episodes, Torrents, …).

./scripts/start_reader.sh                     # colored control menu
./scripts/start_reader.sh status              # one-shot: start|stop|restart|status|logs|open
./scripts/start_reader.sh regen               # regenerate all feeds locally (long)
PORT=9000 ./scripts/start_reader.sh start     # pick a different port

Menu item 7) (or regen) runs the same pipeline GitHub Actions uses — generate_feeds.py → enrich_feeds.py → fix_feeds.py → generate_index.py — so the reader can show freshly-generated feeds without CI. Drop a gitignored .env in the repo root (TMDB_API_KEY=…) and the enrich step opens as well (the script sources it automatically; without it that step is skipped).

The menu (or the one-shot actions) replaces the manual pkill …; setsid nohup python3 …local_reader.py … incantation — it starts the reader detached, verifies it is serving, and keeps a live log at /tmp/rss-reader.log.

Features: folder tree with unread counters, article list, read/unread state (localStorage), search, and a Refresh button that re-parses the local feed files. Use it to sanity-check feeds after generation before pushing.

---

Features

• Streaming‑site feeds — scraped via HTTP, cloudscraper, and Playwright
• Category‑based filtering — block unwanted content (e.g. erotic) by WordPress category class
• Multi‑page scraping — paginate through multiple pages for larger item pools
• Poster, IMDb & trailer enrichment — TMDb API with rate‑limiting; falls back to title‑based search; auto‑injects IMDb and YouTube trailer search links
• Standalone player page — every item that resolves to a TMDb or IMDb id carries a ▶ Play link to the published player/ page, which passes tmdb/imdb/type, the season+episode parsed from the title, the resolved epguides slug and a year‑folded title. The page loads the embed servers, EpGuides (for TV) and TMDB in one iframe — an episode goes straight to its TMDb episode page — while IMDb and Trailer sit beside them as ↗ new‑tab links, because neither site will render inside a third‑party frame. The eight embed servers that read a TMDb id come first; VidSrc and VidAPI read an IMDb id, so they sit at the end of the row and are left out altogether when a title has no IMDb id to give them. The page also has a manual lookup row — TMDb id, IMDb id, Movie/TV, and Season/episode once TV is chosen — prefilled from the current URL and submitting new parameters, so any title can be played by id without going through a feed. When the frame is showing a TMDb page, that row gains a TMDb page box and a Get the TMDb id button: paste in (or copy, and leave the box empty) the address of the TMDb page you were reading, and the button clears the lookup and fills TMDb id, Movie/TV, season and episode from it — the player cannot read the frame's own address, so the page URL has to come from you.
• Torrent download links — an item whose own link is a magnet URI (e.g. a showRSS personal feed) ships a Download button in its description, so a reader hands the magnet straight to your torrent client
• Full catalog — includes future‑dated (unreleased) movies; no removal pass
• Live status page at / — per‑feed health, last update, item count
• One‑click OPML — bulk import into Inoreader with folder structure
• Transient‑failure carry‑forward — a source that fails a run keeps its last published feed; only one the engine declares gone is dropped (after one retry pass)
• Config‑driven — all sites declared in config/sites.yaml; no code changes needed

---

Architecture

GitHub Actions (cron @:19 hourly)
  └─ GenerationEngine
       ├─ Fetcher (http → cloudscraper → Playwright fallback)
       ├─ Parser (elementpath XPath 2.0 → lxml XPath 1.0)
       │   └─ category_selector — extract page‑level category classes
       ├─ DedupStore (URL‑based dedup, 500 URLs/site cap)
       ├─ Feed writer (feedgen — RSS 2.0)
       ├─ _fetch_extra_pages — multi‑page pagination
       └─ _filter_items — blocked_categories + title_filter_patterns

• config/sites.yaml — master site list (XPath selectors, fetch method, category, kind, blocked_categories, pages, enabled)
• core/config.py — SiteConfig dataclass: category_selector, blocked_categories, title_filter_patterns, pages, enabled
• core/engine.py — orchestration: shuffle, stagger, timeout (240s), concurrent (6 sites), multi‑page fetch
• scraper/fetcher.py — 3‑strategy fetch chain with anti‑detection and browser challenge retry
• scraper/parser.py — XPath 2.0 parsing via elementpath, falls back to lxml XPath 1.0; category extraction
• core/feed.py — RSS 2.0 generation with WebSub hub, syndication tags, TMDb poster sizing
• core/tmdb.py — rate‑limited TMDb API client with in‑memory cache; search_movie(title) / search_tv(title) fallbacks
• scripts/enrich_feeds.py — enrichment orchestrator; routes each feed by its site category to a mode (streaming / article / none), overridable per site with enhance_mode
• scripts/enrichers/streaming_enricher.py — streaming mode: TMDb ID lookup, title‑based year search, poster enrichment; cleans torrent‑style titles for TMDB search (TV episodes via /search/tv); injects IMDb + YouTube trailer search links, EpGuides series links and the ▶ Play link to the player/ page
• scripts/enrichers/article_enricher.py + scripts/enrichers/ad_remover.py — article mode: fetch the full article body, strip ads/boilerplate, keep a featured image
• scripts/fix_feeds.py — post‑processing: year formatting, poster style, per‑site chrome removal, magnet Download links
• scripts/refresh_feed.py — runs get → enrich → process for named feeds only (./start.sh one <site>), then reports each item's body-text length so a truncated article is obvious

---

Site Configuration

Field	Required	Description
name	Yes	Unique site key (YAML key)
url	Yes	Full page URL to scrape
method	Yes	Fetch method: http, cloudscraper, playwright, or rss
item_selector	Yes*	XPath 2.0 expression to select each listing item
title_selector	Yes*	XPath for item title (relative to item)
link_selector	Yes*	XPath for item link (relative to item)
feed_file	No	Output filename (default: {name}.xml)
enabled	No	false skips the site during generation while keeping its config (default: true). Use it to park duplicate or fallback feeds. Disabled sites are excluded from index.html/feeds.opml and from enrichment, and --site <name> reports them as unmatched.
category	No	Feed category (movies, episodes, torrents, cinema, releases, blogs, news, cyber, tech, education, economy, local). Determines enrichment mode.
kind	No	Content type: movie or series. When series, items route to TMDb TV search and receive EpGuides links. If unset, title-based heuristics (SxxEyy markers) are used.
language	No	Language code (ro or en, default: ro)
max_items	No	Maximum items per feed
display_name	No	Human-readable feed title

*Required for http, cloudscraper, and playwright methods. Native RSS/Atom feeds (method: rss) don't need XPath selectors.

Enrichment Modes

Source Category	Enrichment Mode	Description
movies, episodes, cinema, torrents, releases	streaming	TMDb posters, years, IMDb links, trailer links, EpGuides
blogs, news, cyber, tech, education, economy, local	article	Full article content extraction with ad removal
any	none	No enrichment (pass through)

Use enhance_mode: none to skip enrichment for individual sites.

Adding or removing a site

Edit config/sites.yaml, then run PYTHONPATH=. python scripts/generate_index.py to rebuild index.html and feeds.opml automatically.

For new sites, use the interactive onboarding:

PYTHONPATH=. python scripts/onboard_site.py

---

GitHub Actions

.github/workflows/update.yml — runs hourly + on push + manual dispatch:

1. Install Python + Playwright (cached)
2. Generate all feeds
3. Enrich with TMDb posters, years, IMDb links, and trailer links
4. Post‑process (Next.js image URLs, poster normalization, per‑site chrome removal, magnet Download links)
5. Rebuild index.html + feeds.opml
6. Commit & push changes (rebase on conflict, --theirs for feeds)
7. Deploy to GitHub Pages
8. Ping WebSub hub for real‑time updates

Secrets needed: TMDB_API_KEY, RSS_GENERATOR_PROXY_URL (optional: TMDB_API_KEY2 for rate-limit failover, OMDB_API_KEY for the IMDb/OMDb fallback when TMDb search misses)

Enable Settings → Actions → General → Workflow permissions: Read and write and set Settings → Pages → Source to GitHub Actions.

Linting

.github/workflows/lint.yml runs ruff on every PR and push to main. Rules are configured in pyproject.toml ([tool.ruff]); run locally with:

pip install ruff
ruff check .

---

Security

Pages password

index.html and reader.html are gated by a client-side password check (scripts/pages_gate.py). The password lives in .pages_pass.txt, which is gitignored and local only; what the repository and the deployed page carry is a PBKDF2-HMAC-SHA256 verifier (random salt + 600 000 iterations) in config/pages-gate.json. Rotate with python scripts/pages_gate.py --set-password, and confirm the two halves agree with --check.

This is a speed bump, not access control. GitHub Pages serves static bytes: the page's own markup is public, and anyone can defeat the prompt from a console. It keeps casual visitors out of the two UI pages. feeds/*.xml and feeds.opml stay public on purpose — they are what an RSS reader fetches. Real access control would need a host that authenticates the request (Cloudflare Access in front of Pages, or a private deployment).

Automated secret scanning runs on every push, PR, manual dispatch, and daily schedule via .github/workflows/secret-scan.yml. See SECURITY.md for reporting.

---

Detailed architecture and development reference: PROJECT.md