PullMD

Anything in, clean Markdown out. Self-hosted conversion of web pages, documents, images, audio and YouTube videos into token-lean Markdown, for you and for your AI agents.

Quick start · Use with AI agents · API · Configuration · Changelog

---

Why PullMD

Agents are bad at reading the web. Raw HTML burns context on navigation, cookie banners and scripts, and built-in fetch tools often stumble on JavaScript-heavy pages, PDFs and comment threads. PullMD sits in between: one URL or file in, clean Markdown out, with source and metadata in YAML frontmatter instead of the body.

curl "https://pullmd.example.com/api?url=https://example.com/article"

What that saves, measured on three public pages (tokens counted with the cl100k_base tokenizer, 2026-10-16):

Page	Plain curl	PullMD	PullMD with ?query=
Encyclopedia article (long-form)	99,458	12,549	662
Developer documentation page	48,611	6,591	454
Reddit thread with comments	blocked (bot check), no thread content	6,381	644

The curl column is what a naive fetch tool hands an agent. The last column is query-scoped extraction: only the sections relevant to a question.

Highlights

Reads almost anything

• Web pages through a fallback chain: Cloudflare's native Markdown, then Readability + Trafilatura (best result wins), then headless Chromium for JavaScript-heavy sites
• Reddit and Hacker News threads with their comment trees (depth and count configurable)
• Documents: PDF, Word, PowerPoint, Excel, EPUB, by URL or upload, plus an optional OCR tier for table-grade PDFs
• YouTube title, description and transcript with clickable timecodes, no API key needed
• Images and audio via any OpenAI-compatible or local model (captions, transcripts)

Built for agents

• MCP server for Claude Code and other MCP clients, with OAuth 2.1 for claude.ai and Claude Desktop
• Query-scoped extraction: ?query= returns only the sections relevant to a question, typically 70-95% fewer tokens on long pages
• Lean output: clean body, metadata in frontmatter, unpadded tables, configurable field allowlist
• Live share links: /s/:id re-fetches when the cached copy is older than an hour and serves the last good snapshot if the source dies

Comfortable to self-host

• One docker compose up, zero config, multi-arch images (amd64/arm64) on Docker Hub and GHCR
• Every sidecar is optional, and PullMD degrades gracefully without it
• Auth modes from open homelab to multi-user, SSRF protection, rate limits, a health endpoint for Uptime Kuma
• Per-site extraction recipes as declarative JSON

Four ways in: PWA · REST API · MCP server · Claude Code skill

---

Quick start

mkdir pullmd && cd pullmd
curl -O https://raw.githubusercontent.com/AeternaLabsHQ/pullmd/main/docker-compose.yml
docker compose up -d
# → http://localhost:3000

No .env needed: every variable has a sensible default. Add a .env next to the compose file to override anything (see Configuration). Then connect your agent: see AI-agent integration.

Upgrading? v3's only breaking change is the clean-body output format (one-line opt-out in MIGRATION.md). Running behind a reverse proxy? Since v3.13 you need PULLMD_TRUST_PROXY. Full history in the changelog.

docker-compose.yml (zero-config, abridged)

services:
  pullmd:
    image: aeternalabshq/pullmd:latest
    container_name: pullmd
    restart: unless-stopped
    ports:
      - "${PORT:-3000}:3000"
    environment:
      - PUBLIC_URL=${PUBLIC_URL:-http://localhost:${PORT:-3000}}
      - TRAFILATURA_URL=http://trafilatura:8001/extract
      - PLAYWRIGHT_URL=http://playwright:8002/render
      - MARKITDOWN_URL=http://markitdown:8003/convert
      - CACHE_DB=/data/cache.db
    volumes:
      - ./data:/data
    networks:
      - pullmd-internal
    depends_on:
      - trafilatura
      - playwright
      - markitdown

  trafilatura:
    image: aeternalabshq/pullmd-trafilatura:latest
    container_name: pullmd-trafilatura
    restart: unless-stopped
    networks:
      - pullmd-internal

  playwright:
    image: aeternalabshq/pullmd-playwright:latest
    container_name: pullmd-playwright
    restart: unless-stopped
    networks:
      - pullmd-internal

  markitdown:
    image: aeternalabshq/pullmd-markitdown:latest
    container_name: pullmd-markitdown
    restart: unless-stopped
    mem_limit: ${MARKITDOWN_MEM_LIMIT:-1g}
    networks:
      - pullmd-internal

networks:
  pullmd-internal:
    driver: bridge

Abridged for readability — the docker-compose.yml in the repo additionally passes every optional .env variable through to the containers (Reddit credentials, auth, media/OCR keys, YouTube options, output shaping). Use the curl -O command above rather than copying this block, or .env overrides beyond the basics won't reach the containers.

Note: the Playwright sidecar adds ~3.7 GB to your image cache (Chromium + Firefox + WebKit binaries from the official Playwright base image). It's optional — leave PLAYWRIGHT_URL unset and the playwright service block off, and PullMD silently degrades to static extraction with a fallback note in the metadata.

Note: the MarkItDown sidecar is optional. Leave MARKITDOWN_URL unset and remove the markitdown service block to disable document conversion. Web-page URLs always work without it.

Mirror on GHCR: ghcr.io/aeternalabshq/{pullmd,pullmd-trafilatura,pullmd-playwright,pullmd-markitdown}. Replace the image: lines if you prefer GitHub's registry.

Behind Traefik

For deployments behind Traefik with TLS, use docker-compose.traefik.yml instead. Same images, but with Traefik labels and the proxy external network. Set HOST_DOMAIN in .env:

curl -O https://raw.githubusercontent.com/AeternaLabsHQ/pullmd/main/docker-compose.traefik.yml
echo "HOST_DOMAIN=pullmd.example.com" > .env
docker compose -f docker-compose.traefik.yml up -d

This file sets PULLMD_TRUST_PROXY=1: Traefik is the only peer (no port is published), so PullMD takes the client address and the https scheme from Traefik's forwarding headers.

Behind another reverse proxy

PullMD ignores X-Forwarded-For, X-Forwarded-Proto and X-Forwarded-Host unless PULLMD_TRUST_PROXY says the connecting peer is a proxy. Behind nginx, Caddy, a tunnel or a load balancer, set:

PULLMD_TRUST_PROXY=1                          # one proxy hop, port not reachable directly
# or: PULLMD_TRUST_PROXY=172.18.0.0/16       # the proxy's address/network
PUBLIC_URL=https://pullmd.example.com

A hop count trusts whatever connects, so use it only when the container port is not published to anyone but the proxy; otherwise list the proxy's address. Without the setting, all clients share one rate-limit bucket (login, signup, share-link misses, OAuth, media budget) and PullMD logs a one-time warning. PUBLIC_URL keeps the session cookie Secure and is accepted as this site's host by the login form check. Do not send Referrer-Policy: no-referrer for PullMD's pages: browsers then post forms with Origin: null, which is refused. See MIGRATION.md.

Local development (no Docker)

git clone https://github.com/AeternaLabsHQ/pullmd.git
cd pullmd
npm install
npm start             # http://localhost:3000
npm test              # node --test

---

Configuration

All variables go in .env (copy from .env.example):

v3.0.0 output format change: the markdown body is clean by default - just # Title followed by content. The source URL, fetch date, and all extraction metadata remain in the YAML frontmatter unchanged - the body no longer duplicates them. Set PULLMD_SOURCE_HEADER=true to restore the old inline header. Use PULLMD_FRONTMATTER_FIELDS to pick which frontmatter fields are emitted (handy for trimming tokens in agent pipelines).

Variable	Required	Purpose
HOST_DOMAIN	Traefik variant only	Public hostname without scheme. Used by Traefik routing and as fallback for PUBLIC_URL. Unused by the default compose.
PUBLIC_URL	no	Full public origin embedded in /help and the skill zip. Defaults to https://${HOST_DOMAIN}. An https:// value also marks the session cookie Secure, and its host is accepted by the same-origin check on login and other form posts.
PULLMD_TRUST_PROXY	behind a reverse proxy	When to honour X-Forwarded-*. Off by default. 1, 2, ... = hop count; true = every hop (the client address is then the first X-Forwarded-For entry, which the client can set; prefer a hop count); or a comma-separated list of addresses, CIDRs or loopback / linklocal / uniquelocal. An invalid list warns and stays off. docker-compose.traefik.yml sets 1. See Behind another reverse proxy.
TRAFILATURA_URL	no	URL of the Trafilatura sidecar's /extract endpoint. Unset → skip Trafilatura, Readability only.
PLAYWRIGHT_URL	no	URL of the Playwright sidecar's /render endpoint. Unset → skip Playwright fallback for JS pages.
MARKITDOWN_URL	no	URL of the MarkItDown sidecar's /convert endpoint. Unset → document-conversion path disabled; POST /api/file returns 502.
PULLMD_VISION_API_KEY / …_BASE_URL / …_MODEL	no	Image captioning via an OpenAI-compatible vision endpoint. Enabled when the key is set. _MODEL defaults to gpt-4o-mini.
PULLMD_EMBEDDED_IMAGES	no	Describe and transcribe the images of Reddit posts with the vision model above. off = never, a request cannot switch it on; (default) = only when a request passes ; = on unless a request passes . Needs a vision provider. Invalid value: startup warning, falls back to . See .

PUBLIC_URL matters for self-hosting: the help page and downloadable skill embed it as the canonical endpoint. Set it correctly and your users get a copy-paste setup that points at your instance.

PullMD rotates its outbound User-Agent for the web fetch path from a pool of current desktop browsers, refreshed every 48 hours from a live feed of real-world UAs maintained by @WinFuture23. A built-in seed pool ensures rotation works even when the feed is unreachable. Set PULLMD_USER_AGENT to pin a single UA, or PULLMD_UA_FEED_URL to point at your own feed. The Reddit path keeps its dedicated REDDIT_USER_AGENT because Reddit's API expects a stable, identifying UA.

DISABLE_PUBLIC_HISTORY=true is the privacy switch for shared instances (multi-tenant VPS, office deployments). Conversions still get cached and assigned share IDs; users just can't see what other users have fetched (the MCP list_recent tool follows the same rule, and /api/stats leaves out its per-domain lists). Anyone with a known /s/:id link still gets their markdown back. Share ids are 128 random bits, so a link cannot be guessed, and /s/:id throttles unknown-id lookups to 120 per minute and IP (valid links are never throttled). With sign-in enabled (PULLMD_AUTH_MODE=single-admin or multi-user) history is scoped per user anyway; the switch matters for instances without sign-in.

---

Authentication (v2.0+)

Version pinning: :latest tracks the newest release (v3). v3's only breaking change is the clean-body output format — to stay on the v2.x output format instead, pin the explicit major tag:

services:
  pullmd:
    image: aeternalabshq/pullmd:2

PullMD ships with three auth modes. Pick one with PULLMD_AUTH_MODE:

Mode	Behavior
disabled	Default. No auth, everything open. Existing v1.x behavior.
single-admin	One user, credentials from env vars. No self-signup. For homelab.
multi-user	Self-signup at /signup (unless PULLMD_ALLOW_SIGNUP is off), login at /login, per-user data isolation.

disabled is meant for a private host or LAN. If a provider key is set (PULLMD_VISION_API_KEY, PULLMD_STT_API_KEY, PULLMD_LLM_API_KEY or PULLMD_PDF_OCR_API_KEY), every client that can reach the port can spend it, bounded only by PULLMD_LLM_RATE_LIMIT; the server prints an [auth] warning at startup in that case. Turn on an auth mode, or bind the port to loopback ("127.0.0.1:${PORT:-3000}:3000", shown as a comment in docker-compose.yml), when others can reach the instance.

In single-admin and multi-user modes, PULLMD_ADMIN_EMAIL + PULLMD_ADMIN_PASSWORD bootstrap the first admin user on first startup. After that, changing these env vars does not change the password — use the admin CLI:

docker compose exec pullmd node scripts/admin.js reset-password [email protected]

Create an account without opening self-registration (useful when PULLMD_ALLOW_SIGNUP is off):

docker compose exec pullmd node scripts/admin.js create-user [email protected]

Both commands read the password from stdin, so they need it attached: docker compose exec, docker exec -it, or a pipe (echo "…" | docker exec -i <container> node scripts/admin.js …). A plain docker exec without -i aborts with an error and exit code 2 instead of doing nothing.

Auth boundary

Endpoint	Auth required (when mode != disabled)
/, /help, static assets, /pullmd.zip	no
/login, /signup, /api/me (auth surface)	no
/s/:id (share links)	no
/api, /api/stream	yes
POST /api/html, POST /api/file	yes
/mcp	yes
/api/history, /api/archive	yes
DELETE /api/cache/:id, DELETE /api/cache	yes
/api/stats, /api/storage, /api/config (aggregate)	no
/api/status, /api/recipes/status (health)	no

/api/stats returns its per-domain lists (lowQualityDomains, fallbackByDomain) only to the admin, or to everyone when auth is off and public history is on; the aggregate counts are public. Signed-in callers of /api/history and MCP list_recent get their own conversions.

Cache deletes are scoped to the caller. An admin (and every caller in disabled mode) removes the shared, URL-deduped cache row, which affects every user's history. A regular user only unlinks the entry from their own history - the shared row and its /s/:id share link keep working. The response says which happened via "scope": "user" | "global".

Authentication paths

1. Session cookies — POST /login sets pullmd_session (HttpOnly, SameSite=Lax, 90-day TTL with sliding expiry). The PWA uses this automatically. The cookie is Secure when the request arrived over HTTPS (directly, or through a proxy covered by PULLMD_TRUST_PROXY) or when PUBLIC_URL starts with https://. POST /login allows 10 attempts per minute and POST /signup 5 per hour per client address (429 with Retry-After beyond that). Login, signup, logout and OAuth consent form posts are accepted only when Origin (or Referer) names this host or the host of PUBLIC_URL; posts without either header are accepted.
2. API keys — generate at /settings, send via Authorization: Bearer pmd_<32-char-base62>. Stored as SHA-256 hashes; only shown once at creation.
3. Legacy PULLMD_AUTH_TOKEN — deprecated. single-admin mode only. Maps to admin user. Kept for migration compatibility; slated for removal in a future major release.

Migration from v1.x

See MIGRATION.md for the full upgrade checklist. The TL;DR: leave PULLMD_AUTH_MODE unset and v2.0 behaves exactly like v1.x.

OAuth 2.1 (claude.ai Web Connector)

PullMD ships with a full OAuth 2.1 Authorization Code flow so the claude.ai web app's Custom Connector feature can authenticate users against your PullMD instance. All endpoints needed by the spec are implemented: Dynamic Client Registration (RFC 7591), PKCE-S256 (RFC 7636), Authorization Server Metadata (RFC 8414), Protected Resource Metadata (RFC 9728), and Token Revocation (RFC 7009).

Setup:

1. Set PULLMD_AUTH_MODE to single-admin or multi-user (OAuth requires Phase-1 auth).
2. Set OAUTH_JWT_SECRET to a 32+ character random string (openssl rand -hex 32).
3. Set PUBLIC_URL to your instance's public origin (e.g. https://pullmd.example.com).
4. In claude.ai → Settings → Connectors → Add custom connector, point it at https://pullmd.example.com/mcp — claude.ai discovers everything else automatically via the well-known endpoints.
5. The first time the user clicks the connector, they'll be redirected to PullMD's /login, then to a consent screen, then back to claude.ai.

Tokens:

• Access tokens are JWTs (HS256), TTL 1 hour, audience-bound to your /mcp URL, and accepted only on /mcp. Other routes, /api/me included, treat them as unknown bearers; use an API key there.
• Refresh tokens are opaque (pmd_rt_…), TTL 30 days, rotated on every refresh, with reuse-detection that invalidates the entire refresh chain on replay.
• Revoke a token via POST /oauth/revoke (RFC 7009).

Scope: Currently a single mcp:full scope (URL conversion + history read). Granular scopes are tracked for a future minor release.

Shipped in v2.3.0 (issues #6 and #10).

---

AI-agent integration

Three install paths. Once your instance is running, ${PULLMD_URL}/help shows the same boxes with your URL pre-filled. Replace ${PULLMD_URL} below with your hostname (e.g. https://pullmd.example.com).

1. Universal prompt

Drop into any chat agent (ChatGPT, Claude, Gemini, …):

When you need to read a web page, fetch it via PullMD instead of your
built-in fetch/browse tool - not just when that one fails:

  GET ${PULLMD_URL}/api?url=<URL>

Returns clean Markdown (text/markdown). Optional query params:

  comments=false        skip Reddit comments
  comment_depth=N       comment nesting depth (default 3)
  frontmatter=true      prepend YAML metadata block
  format=text           strip Markdown, return plain text
  nocache=true          bypass the 1h cache and refetch
  render=force|skip     override the auto Playwright fallback
  pdf=ocr               high-quality PDF conversion (tables)
  query=<text>          got a specific question? pass it and get back only
                        the matching sections (70-95% fewer tokens)
  max_tokens=N          budget for query= (default 600, 64-20000)
  lang=de|en            language for the comments section header

Response headers worth checking:
  X-Source       reddit | hackernews | cloudflare | readability | trafilatura |
                 playwright | markitdown | youtube | pdf-ocr | ...
  X-Quality      0.0-1.0 extraction confidence
  X-Share-Id     32-hex permalink, openable as /s/<id>
  X-Transcript-Status  youtube only: ok | none | blocked | error
                 (blocked/error = transient, not cached — retry later)

Reddit URLs are auto-detected (incl. redd.it short links and /s/ shares).
Hacker News URLs are auto-detected too — items, comment permalinks, and the
front/newest/ask/show/jobs listings.
Use this whenever you would otherwise fetch raw HTML — the markdown is
much cleaner and saves significant context window space. When you only
need specific information rather than the whole document, pass your
question as query= and get back just the relevant sections.

2. Claude Code skill

pullmd.zip is auto-built with your URL embedded:

curl -O ${PULLMD_URL}/pullmd.zip
mkdir -p ~/.claude/skills
unzip pullmd.zip -d ~/.claude/skills/
# Restart Claude Code; the skill activates on web-reading requests.

Upgrading from pre-v3? The skill was renamed from web-reader to pullmd in v3.0.0. Installing the new zip does not replace an existing install — remove the old one first, or both skills will be active side by side:

rm -rf ~/.claude/skills/web-reader

(The old download URL /web-reader.zip keeps working as a redirect.)

3. MCP server

Remote MCP server at ${PULLMD_URL}/mcp (Streamable-HTTP transport, stateless). Three tools: read_url, get_share, list_recent. Server-side updates reach every client automatically — no local install needed. With auth enabled, list_recent lists the caller's own conversions.

Claude Code (CLI):

claude mcp add --transport http pullmd ${PULLMD_URL}/mcp

Claude Desktop / Cursor / other MCP hosts — JSON config:

{
  "mcpServers": {
    "pullmd": {
      "type": "http",
      "url": "${PULLMD_URL}/mcp"
    }
  }
}

Once registered, the three tools surface natively in the agent — no prompt instructions needed, the LLM picks them up via their schema descriptions.

MCP client compatibility

Client	Bearer (Authorization: Bearer pmd_...)	OAuth (v2.3+)	Notes
Claude Code CLI	✅	✅	Recommended. Generate a key at /settings.
Cursor	✅	✅	Same as CLI.
Claude Desktop	❌	✅	Connector UI lacks a header field — use OAuth.
claude.ai (web)	❌	✅	Requires OAuth.

OAuth shipped in v2.3.0 — enable it via OAUTH_JWT_SECRET (see above) and Claude Desktop / claude.ai connect natively. The reverse-proxy workaround below is only needed on instances that keep OAuth disabled.

Claude Desktop limitation (without OAuth)

The Claude Desktop "Add custom connector" UI accepts URL + OAuth Client ID/Secret but no custom-header field. Additionally, claude_desktop_config.json entries with "type": "http" are silently rewritten to {} after Desktop launches (current Desktop only honors stdio servers in that file).

If OAuth is not enabled on your instance, the practical workaround for Claude Desktop users is a reverse proxy that accepts the auth token as either a bearer header (for CLI) or as a URL path prefix (for Desktop, which has no header field).

Caddy workaround for Claude Desktop

Contributed by @WinFuture23:

@bearer header Authorization "Bearer {$AUTH_TOKEN}"
handle @bearer { reverse_proxy pullmd:3000 }

@token_path path /{$AUTH_TOKEN}/* /{$AUTH_TOKEN}
handle @token_path {
    uri strip_prefix /{$AUTH_TOKEN}
    reverse_proxy pullmd:3000
}

Then in Claude Desktop's connector dialog, use the URL with the token path prefix: https://your-instance.com/<TOKEN>/mcp. CLI clients keep using the Authorization header as normal.

This is a stopgap pattern; enabling the built-in OAuth removes the need for it.

---

API

Endpoint	Returns
GET /api?url=…	Markdown (or JSON / plain text via format=). Also handles direct links to documents (PDF, Office, EPUB, …) when the markitdown sidecar is configured.
GET /api/stream?url=…	Server-Sent Events stream of extraction-stage status, ending in a result event. Used by the PWA.
POST /api/html	Convert a local/raw HTML document (body = HTML, max 10 MB). Never cached — no history entry, no share link.
POST /api/file	Convert an uploaded document (raw file bytes in body; set Content-Type to the file's MIME type; filename via X-Filename header or ?filename=; max 25 MB). Returns Markdown. Requires the markitdown sidecar (MARKITDOWN_URL) for document types (PDF, Office, EPUB, ...); image and audio uploads instead use the PULLMD_VISION_* / PULLMD_STT_* tier (no markitdown container needed).
GET /s/:id	Cached Markdown by share id; refreshes from source if > 1 h old.
GET /api/history	Recent conversions (JSON). ?limit= 1-100, default 20. Signed-in callers see their own.
GET /api/archive	Paginated full archive. ?limit= 1-200, default 50; ?offset=.
GET /api/storage	Cache size / hit-rate stats.
GET /api/stats	Extraction telemetry (sources, quality, latency). ?window=-7 days. Per-domain lists only for global-history viewers (see Auth boundary).
GET /api/config	Which optional tiers this instance has enabled (auth mode, markitdown, vision, STT, PDF OCR, YouTube). The PWA reads it to show or hide controls.
GET /api/recipes/status	Which site recipes loaded, which were rejected, and any frontmatter fields dropped by the allowlist.

/api parameters

Param	Default	Notes
url	—	Required.
comments	true	Include Reddit / Hacker News comments. Ignored for other URLs.
comment_depth	3	Max nesting depth (1–10). Applies to Reddit and Hacker News.
comment_limit	none	Max top-level comments (uncapped by default).
frontmatter	false	Prepend YAML metadata.
format	md	text strips Markdown; json returns structured response.
nocache	false	Bypass the 1-hour cache.
render	auto	force → always render via Playwright. skip → never render. Bypasses cache.
extractor	auto	Force readability / trafilatura / playwright and skip the quality pick. Bypasses cache.
pdf	—	ocr → route PDFs through the OCR tier. Bypasses cache.
yt_timecodes / yt_chunk	see YouTube	Transcript format overrides. Bypass cache when set.
lang	de	Comments-section header language (de or en).
images	instance default	Reddit posts only: on / off (also true/1, false/0) to describe and transcribe the post images with the server's vision model. Other values return for Reddit URLs; non-Reddit URLs ignore the parameter. Needs a vision provider and not ; see . Does not bypass the cache.

Both query and max_tokens, and images ("on" / "off"), are also available on the MCP read_url tool.

Response headers

• X-Source — reddit · cloudflare · readability · readability-fallback · trafilatura · playwright · recipe-content · coverage-guard · markitdown · youtube · image-caption · audio-transcript · pdf-ocr
• X-Quality — 0.0–1.0 extraction confidence
• X-Share-Id — the 32-hex permalink id
• X-Suggested-Filename — a download filename for this conversion, e.g. YT-some-talk-dQw4w9WgXcQ.md. YouTube gets title plus video id, images/audio/documents keep the original file name from the URL, everything else uses the title; PULLMD_FILENAME_DATE_PREFIX prepends a date. Also on /s/:id, POST /api/html and POST /api/file; on /api/stream the same value arrives as suggestedFilename in the result event, since SSE has no headers to carry it.
• X-Transcript-Status — YouTube only: ok · none · blocked · error. blocked (YouTube rate-limited the transcript fetch, HTTP 429) and error are transient and not cached — retry later; none means the video genuinely has no transcript
• X-Extracted — true / false. Present only when query is active.
• X-Extract-Confidence — high · medium · low. Present whenever X-Extracted is present, except when the page was already small enough that extraction was skipped.
• X-Extract-Sections — number of sections/regions included in the returned markdown.
• X-Extract-Original-Tokens / X-Extract-Returned-Tokens — estimated token counts (chars / 4) of the full page and of the returned markdown.

Bot-protection challenges

PullMD does not solve bot-protection interstitials, and the Playwright sidecar is rejected by them as well. When the upstream answers with a Cloudflare managed challenge (cf-mitigated: challenge) or a Bunny Shield proof-of-work page (cdn-challenge: true, or the same page served with HTTP 200), /api answers 502 with a machine-readable body instead of a bare 403 or the interstitial rendered as Markdown:

{"error":"Site is behind a bot-protection challenge (Cloudflare); PullMD does not solve these. Workaround: save the page from your browser and upload the HTML via POST /api/html.","code":"bot_challenge","vendor":"Cloudflare"}

Nothing is cached for such a response. The workaround is the one in the body: save the page from a real browser and convert it via POST /api/html or drag-and-drop in the PWA.

Query-scoped extraction

When you have a specific question about a long page, you rarely need the whole document. Pass the question as ?query=<text> on GET /api (or the query param on the MCP read_url tool) to get back only the sections relevant to it instead of the full Markdown body - typically 70-95% fewer tokens on long pages:

GET /api?url=https://example.com/long-article&query=how+does+caching+work

It's a BM25 ranking over the page's heading-based sections (falling back to paragraph-level blocks for pages with fewer than two headings) — no network calls, no LLM involved.

• Budget — max_tokens caps the returned size (default 600, range 64–20000, estimated as chars / 4).
• Whole-page fallback — the full page is returned unchanged when either the page is already small enough that extraction wouldn't help (estimated tokens ≤ max(800, 1.25 × max_tokens)), or nothing in the page scores against the query's terms. In both cases extracted is false; the no-match case reports confidence: low, the small-page case omits confidence.
• Atomic blocks — code fences, tables, lists, and blockquotes are never split; if any part of one is relevant, the whole block is kept. Non-contiguous output regions are joined by an elision marker (<!-- … -->).
• ?frontmatter=true adds extracted, extract_confidence, sections_selected, original_tokens, returned_tokens to the YAML block. format=json adds a top-level extract object with the same data (extracted, confidence, sectionsSelected, originalTokens, returnedTokens).
• Extraction runs on the already-cached full page — the cache always stores the complete Markdown, so several different query values against the same URL cost one fetch (within the normal 1-hour cache TTL), not one per query.
• Omitting query (or passing an empty/whitespace-only string) leaves /api behavior completely unchanged — no extra headers, no extra JSON or frontmatter fields.
• MCP read_url has no response headers; when nothing matches, it prepends an in-band comment (<!-- query-extract: no match; returning full page -->) to the returned markdown instead.

---

SSRF protection

A URL-fetching service will happily be pointed at its own network if you let it. Since v3.3.0 PullMD does not: before any fetch, the target host is resolved and the resulting addresses are checked. Anything in a private, loopback, link-local, CGNAT or cloud-metadata range is refused with 403 — including 169.254.169.254 (AWS/GCP/Azure) and 100.100.100.200 (Alibaba).

• GET /api and the MCP read_url tool check the URL up front and answer 403 before doing any work. GET /api/stream is covered by the same guard one layer down, in the Reddit / web / Playwright fetch paths, and surfaces the refusal as an SSE error event.
• Every redirect hop is re-checked before it is followed, so a public URL that redirects inward is blocked at the hop, not after it.
• IPv6 is covered too, including blocked v4 ranges reached through transition addressing (IPv4-mapped, NAT64, 6to4, IPv4-compatible).
• The rejection happens before the cache is touched — a blocked URL never produces a cache row or a share id.
• The Playwright sidecar enforces the same policy inside the browser (since 3.15.2). Chromium's own navigations never pass through Node, so the sidecar checks every request it makes: redirects, <meta refresh>, location changes, iframes, subresources, WebSockets. A page that sends the browser to an internal host fails the render with 403; the allowlist below travels with each render request.

To reach an internal host on purpose (an intranet wiki, a document server on the same network), allowlist it:

# CIDRs, exact hostnames, or both
PULLMD_ALLOWED_HOSTS=10.0.5.0/24,wiki.internal

Empty (the default) means every internal target stays blocked.

Known residual: the guard checks resolved IPs at request time but does not pin the outbound socket to the address it checked, so a DNS-rebinding attack (where the name resolves differently between check and connect) is not fully closed. When PullMD runs behind an outbound HTTP proxy, the proxy resolves names itself — its egress filtering, not this guard, is the authoritative layer for traffic it forwards.

Reported as #41, shipped in v3.3.0.

---

Cache & TTLs

• /api?url=… re-fetches from source if the cache row is older than 1 hour.
• /s/:id does the same on-demand refresh, so share links double as live endpoints.
• Cache rows are pruned PULLMD_CACHE_RETENTION_DAYS days (default 90; 0 keeps them forever) after the last write. /s/:id hits keep the row alive (since they trigger refresh + write); read-only access does not extend the TTL.
• If the source is unreachable on refresh, the last good snapshot is served — share links keep working even when the original URL dies.

---

Monitoring

GET /api/status reports whether the extraction sidecars are reachable:

{
  "ok": false,
  "version": "3.9.0",
  "services": {
    "trafilatura": { "status": "ok", "latencyMs": 3 },
    "playwright":  { "status": "down", "error": "unreachable" },
    "markitdown":  { "status": "not-configured" }
  }
}

• status is ok, down, or not-configured. A sidecar you never configured is not a failure — self-hosting without Playwright or markitdown is a supported setup.
• error is one of unreachable, timeout, unhealthy (answered 200 but reports itself broken), http <code>, or misconfigured. Deliberately a closed vocabulary: the endpoint is public and must not echo internal hostnames.
• HTTP 200 when every configured sidecar answers, 503 when at least one is down, so a plain HTTP check catches it without keyword matching.
• Results are cached for 5 seconds and concurrent callers share one round of probes, so polling it cannot amplify into internal traffic.

Why it exists: a broken sidecar does not take PullMD down, it silently degrades extraction quality — pages that need a browser render come back near-empty while everything still answers 200. Without an explicit check that can go unnoticed for a long time.

Uptime Kuma: monitor type HTTP(s), URL https://<your-host>/api/status, accepted status codes 200-299. On an instance with authentication enabled the endpoint stays reachable without credentials.

---

Document conversion

When the markitdown sidecar is running (set MARKITDOWN_URL=http://markitdown:8003/convert), PullMD can convert document files to Markdown in addition to web pages.

Supported formats: PDF, DOCX, PPTX, XLSX/XLS, EPUB, ZIP (contents listed), CSV, JSON, XML.

Two ways to convert documents:

• By URL — pass a direct document link to the regular API:

  GET /api?url=https://example.com/report.pdf
  

  PullMD detects the document content type and routes it through the sidecar automatically.

• By upload — POST the raw file bytes as the request body (max 25 MB):

  POST /api/file
  Content-Type: application/pdf        # the file's MIME type
  X-Filename: report.pdf               # URI-encoded; or use ?filename=
  # body: raw file bytes
  

  The PWA supports both drag-and-drop and a file picker (desktop and mobile) for the same path.

If MARKITDOWN_URL is unset, document conversion is unavailable: a document URL (GET /api?url=…) and POST /api/file both return 502. Regular HTML pages are unaffected, and the PWA hides the document-upload affordance (it reads the markitdown flag from /api/config).

The default docker-compose.yml includes the markitdown sidecar with MARKITDOWN_URL pre-wired. To opt out, remove the markitdown service block and unset the env var.

Media tier (image captions + audio transcription)

Image captioning and audio transcription run inside pullmd itself - the markitdown sidecar is not required for media features (it is only needed for document conversion). By default pullmd processes images with EXIF metadata only and audio files with track metadata only - no model calls, no external traffic. Set the relevant PULLMD_VISION_* or PULLMD_STT_* credentials to unlock richer extraction:

• Images - a vision-capable model generates a text caption for each image, embedded in the Markdown output. This applies to image uploads and URLs that point straight at an image; images embedded in a page are not sent to the model, with one exception: the images of Reddit posts, see Embedded images. Enabled when PULLMD_VISION_API_KEY (or the shared fallback PULLMD_LLM_API_KEY) is set.
• Audio - a speech-to-text model transcribes the audio; the transcript replaces the bare metadata block. Enabled when PULLMD_STT_API_KEY (or the shared fallback PULLMD_LLM_API_KEY) is set.

Both modalities are independently configurable via PULLMD_* env vars on the pullmd service:

Scope	Variables
Vision	PULLMD_VISION_API_KEY, PULLMD_VISION_BASE_URL, PULLMD_VISION_MODEL (default gpt-4o-mini)
STT	PULLMD_STT_API_KEY, PULLMD_STT_BASE_URL, PULLMD_STT_MODEL (default whisper-1)
Shared fallback	PULLMD_LLM_API_KEY, PULLMD_LLM_BASE_URL (used when per-modality key is unset)

Per-modality vars override the shared LLM_* fallback when set. The fallback deliberately stops at the key and the base URL: a single model name cannot serve captioning, transcription and OCR, so PULLMD_LLM_MODEL does not exist. Set PULLMD_VISION_MODEL / PULLMD_STT_MODEL / PULLMD_PDF_OCR_MODEL instead - if a PULLMD_LLM_MODEL is set anyway, the server says so at startup and the modalities stay on their defaults. Any OpenAI-compatible endpoint works - point *_BASE_URL at a local server (e.g. Ollama, LM Studio) to keep everything on-host and avoid per-call cloud costs.

Note: cloud endpoints send image and audio content to a third-party API. Use a local model server if data-residency matters.

Embedded images (Reddit posts)

On a Reddit image post the image is often the actual content, and the comments make little sense without it. With a vision provider configured, PullMD can describe each image of a Reddit post (a single image, a gallery, or an image referenced inline in a text post; images in comments are not handled) and transcribe its legible text. The result is placed directly under the image as a blockquote, a short description (one or two sentences) first, then the transcript:

![](https://i.redd.it/example.jpeg)

> **[Image description]** Screenshot of a chat with an AI assistant answering a question.
>
> **[Image transcript]**
> ...

The labels are Bildbeschreibung / Bildtranskript for lang=de (the default) and Image description / Image transcript for lang=en. This is opt-in because it is the first time PullMD sends images found inside a page to a provider:

PULLMD_EMBEDDED_IMAGES	Behaviour
off	Hard off. images=on has no effect, no image is fetched, no marker is added.
on-request (default)	No provider calls unless a request passes images=on (MCP read_url: images: "on").
always	Every Reddit post request transcribes its images, unless it passes images=off.

• Marker: on an instance with a vision provider and a mode other than off, an eligible image without a transcript gets a one-line > **[Image not transcribed]** marker (Bild nicht transkribiert for lang=de), so a reader of the output can tell context is missing. Without a vision provider, or with off, the body is byte-identical to the output before this feature (the one exception: transcripts stored earlier, while a provider was configured, are still shown unless the instance is set to off). The images_transcribed: "<done>/<total>" frontmatter field is set whenever a post has eligible images, whatever the configuration.
• Limits: Reddit only. Images are fetched only from i.redd.it and preview.redd.it (on every redirect hop, in addition to the SSRF guard), at most 20 MB each, and only if the download is served as image/*. A single-image post must be a jpg/jpeg/png/webp file; gallery and inline images count when Reddit's metadata marks them as non-animated images. GIFs and videos are skipped. At most PULLMD_EMBEDDED_IMAGES_MAX images per post (default 4) are transcribed, and all images of one post share a 90-second deadline.
• Cost: each image that is transcribed for the first time uses one unit of the hourly PULLMD_LLM_RATE_LIMIT budget and one vision call. Transcripts are stored per image: the same image is transcribed once, whichever URL form of the post is requested (old.reddit.com, a redd.it short link, a /s/ share link, tracking parameters), for as long as cache retention keeps it. Any extraction (a cache miss, nocache, a share refresh) reuses stored transcripts without a download, a model call or budget, also for a request that does not ask for images (an instance set to off ignores stored transcripts). Token usage appears as the llm_* frontmatter fields, summed over the images. With always, refreshing a share link (/s/:id, which needs no sign-in) can transcribe images that have no transcript yet, charged to the viewer's budget; use on-request if share links on a sign-in instance should never cause provider calls.
• Failures: the post is always returned. An image that cannot work (HTTP 4xx other than 408/429, not an image, over 20 MB, host or SSRF rejection, or a provider answer of 400, 413, 415 or 422) counts as failed for that cached result only: it is not retried while that row is fresh, but it is retried after the row expires or with another URL form of the post. Failures are never stored per image, so one post cannot mark another post's image as failed. Clearing the whole cache forgets the stored transcripts. When an image misses for a transient reason (budget used up, the 90-second deadline, an HTTP 5xx, a DNS failure, any other provider error), the result is served but not cached, and a share refresh keeps the old snapshot and restarts its one-hour refresh clock. Images that did succeed in that request are stored anyway, so a gallery with one failing image keeps the others. A cached row with fewer resolved images than the instance would now produce counts as a miss for an request, so is not needed.

Choosing a vision model. A small comparison from October 2026: nine images from current Reddit posts (chat screenshots, a social media post, two charts, a meme, a photographed menu, a German phone ad, a plain photo), each sent once per model with the prompt PullMD uses (lang=de). Cost is computed from the providers' list prices at the time; nine images are a spot check, not a benchmark.

Model	Prompt tokens per image	Avg. time	Cost per 1,000 images	Observations
claude-haiku-5-5	~3,100	2.1 s	~$0.48	Complete transcripts; descriptions name what the image is (e.g. "chat with an AI assistant, question on top, answer below").
gpt-4.1-mini	~3,200	2.3 s	~$1.52	Complete transcripts; shorter, vaguer descriptions.
gpt-4o-mini (the default)	~29,400	3.4 s	~$4.50	Left out the key figures of a chart, altered source text (fixed a typo, completed a cut-off price), misdescribed one screenshot.
claude-sonnet-5-5	~3,100	5.7 s	~$9.12	Reference; descriptions slightly more detailed than Haiku's, transcripts the same.

None of the models said what made the plain photo notable; they do well on screenshots, charts and text, less so on the point of a photo. Claude models work through Anthropic's OpenAI-compatible endpoint without further setup:

PULLMD_VISION_BASE_URL=https://api.anthropic.com/v1
PULLMD_VISION_API_KEY=sk-ant-...
PULLMD_VISION_MODEL=claude-haiku-5-5

The default stays gpt-4o-mini so that existing instances do not change behaviour; set PULLMD_VISION_MODEL to pick another model. The choice applies to all vision captioning, including image uploads.

Provider calls are budgeted: image captioning, audio transcription and PDF OCR share PULLMD_LLM_RATE_LIMIT requests per hour (default 30, 0 = off), counted per signed-in user or, without sign-in, per client address. Over budget the request gets what an instance without the provider returns (metadata only, or the markitdown PDF conversion), and that result is not cached. A tier without a configured provider never consumes budget.

YouTube transcripts

When MARKITDOWN_YOUTUBE=true is set on the pullmd service, YouTube video URLs are routed through the markitdown sidecar instead of the regular web-extraction pipeline. The sidecar fetches the video's title, description, and auto-generated or community transcript, and returns them as clean Markdown. No API key is required.

GET /api?url=https://www.youtube.com/watch?v=dQw4w9WgXcQ

The transcript format is configurable via environment defaults on the sidecar, and can also be overridden per request:

Query param	Values	Default	Effect
?yt_timecodes	links / plain / none	links	links = clickable YouTube timestamp URLs; plain = [MM:SS] labels; none = transcript text only
?yt_chunk	integer seconds	30	Groups transcript snippets into blocks of this many seconds. 0 keeps the original per-snippet granularity.

Both params are also available on the MCP read_url tool.

Language preference: set MARKITDOWN_YT_LANGS=de,en on the sidecar to prefer German transcripts, falling back to English if no German track exists. If none of the preferred languages are available, the sidecar falls back to the first available track.

Rate-limit caveat: YouTube's transcript API aggressively rate-limits requests from datacenter IP ranges. If you run PullMD on a cloud VPS and see frequent failures, set MARKITDOWN_YT_PROXY on the sidecar to route requests through a residential or ISP proxy.

Note: YouTube extraction requires the markitdown sidecar (MARKITDOWN_URL) to be configured. If MARKITDOWN_URL is unset, YouTube URLs are processed by the regular web pipeline (HTML extraction via Readability/Trafilatura), which does not include a transcript.

High-quality PDF (OCR)

By default, PullMD converts PDFs through the free markitdown path, which works well for text-heavy files but loses complex tables. For table-grade output, enable the opt-in OCR tier:

1. Set PULLMD_PDF_OCR_API_KEY to your OCR provider key (reference provider: Mistral OCR, ~$0.002/page).
2. Request ?pdf=ocr on any PDF URL (also on /api/stream and POST /api/file uploads), or set fetch.pdf: ocr as a recipe default.

PULLMD_PDF_OCR_API_KEY (and optionally PULLMD_PDF_OCR_BASE_URL) must be set explicitly - unlike vision and STT, PDF-OCR does NOT fall back to the shared PULLMD_LLM_* key, because the OCR endpoint is typically a different provider (Mistral OCR vs. a chat LLM).

On success, source becomes pdf-ocr and the frontmatter gains a pdf_pages field with the page count. If the OCR call fails or no key is configured, pullmd falls back to the standard markitdown conversion automatically - no error is returned to the caller.

Note: the OCR provider charges per page. Check your provider's pricing before enabling this on high-volume instances. OCR requests count against the hourly PULLMD_LLM_RATE_LIMIT budget described under Media tier.

Self-hosted / heavy local engines (advanced)

PULLMD_PDF_OCR_BASE_URL can point at any endpoint that speaks the Mistral OCR API: pullmd sends POST {BASE_URL}/ocr with {"model": ..., "document": {"type": "document_url", "document_url": ...}} and reads the Markdown from pages[].markdown in the response. Local engines such as Docling have their own API and do not work here directly; wiring one in needs a small adapter in between that translates the request and response. Be aware that such engines are also heavy - a multi-GB container image, and model cold-start times of tens of seconds on first request are typical. This is deliberate: pullmd does not bundle any heavy OCR engine. The reference Mistral OCR path uses a lightweight cloud API call; running a local engine behind an adapter is an advanced operator choice. The OCR tier only handles PDFs; other document types always go through markitdown.

Source-specific frontmatter fields

When you request ?frontmatter=true, Reddit, media, and YouTube results include extra fields in the YAML block - post metadata for Reddit, usage fields that let agents compute cost or log telemetry for media. These fields appear only in the frontmatter and are never inserted into the body — without ?frontmatter=true the body is pure content.

Field	Present when	Description
subreddit	Reddit post	Subreddit, e.g. r/selfhosted.
author	Reddit post (also web pages with author metadata)	Post author, e.g. u/someone.
published	Reddit post (also web pages with publish date)	Post creation date (ISO 8601).
upvotes	Reddit post	Score at time of extraction.
images_transcribed	Reddit post with eligible images	"<done>/<total>": images with a transcript out of the eligible post images, e.g. 1/2. Also present, with 0/N, when no transcript was produced.
duration	YouTube	Video duration (humanized, e.g. 12:34 or H:MM:SS).
views	YouTube	View count at time of extraction.
image_size	Image with caption	Dimensions of the source image: WxH pixels.
audio_seconds	Audio transcription	Length of the audio file in seconds.
llm_model	Caption or transcription	Model name returned by the upstream API (e.g. gpt-4o, whisper-1).
llm_tokens	Caption or transcription	Total tokens consumed (prompt + completion).
llm_prompt_tokens	Vision caption	Prompt token count (vision calls only).
llm_completion_tokens	Vision caption	Completion token count (vision calls only).
pdf_pages	PDF with OCR (source: pdf-ocr)	Number of pages processed by the OCR provider.

A few things worth noting:

• PullMD reports no computed cost. Token counts and the model name are what pullmd receives from the upstream API; multiply by the model's per-token rate on your end.
• Image captioning and audio transcription are direct API calls made by pullmd.
• Frontmatter fields for ordinary web pages (title, url, source, quality, …) are always present when ?frontmatter=true; the LLM fields above are additive.

---

Site recipes

Site recipes are small declarative JSON entries that adjust extraction for specific hosts — force a browser render, strip site-specific noise, unwrap wrappers that confuse the reader, rebuild CSS-grid comparison tables as real tables, or promote structured data (JSON-LD / CSS selectors) into the frontmatter. The built-ins ship in site-recipes.default.json; self-hosters can add their own via data/site-recipes.json or the PULLMD_SITE_RECIPES env var (see .env.example). GET /api/recipes/status reports what loaded.

To write or contribute a recipe, see the site-recipe contributor guide.

---

Architecture

• server.js — Express app factory (createApp) with dependency injection for tests. Exposes /api and /api/stream (SSE).
• lib/reddit.js — Reddit URL normalization, redirect resolution, post + comment extraction.
• lib/web.js — Orchestrator: Cloudflare-Markdown short-circuit, then static Readability + Trafilatura with pickBest, then optional Playwright re-render + re-extract on body-soup / low-quality output.
• lib/render-decision.js — Predicate that decides when to fall back to Playwright (readability-fellback + thin, body-soup signature, or quality < 0.5; plus force / skip overrides).
• lib/playwright-client.js — HTTP client for the Playwright sidecar with AbortSignal propagation for SSE-disconnect cancellation.
• lib/scoring.js — Quality scoring used to pick between extractors and as a render-trigger heuristic.
• lib/cache.js — SQLite cache (better-sqlite3) with configurable retention (default 90 days) and 32-hex (128-bit) share ids.
• lib/mcp.js — Stateless MCP server registering the three tools.
• lib/distrib.js — Public-URL substitution in /help and /pullmd.zip.
• trafilatura-sidecar/ — Python sidecar (FastAPI) wrapping Trafilatura.
• playwright-sidecar/ — Python sidecar (FastAPI + Playwright + Chromium) for JS-rendered pages.
• markitdown-sidecar/ — Python sidecar (FastAPI) wrapping MarkItDown for document conversion (PDF, Office, EPUB, …) plus the keyless /youtube transcript endpoint.
• lib/markitdown-client.js — HTTP client for the markitdown sidecar (documents + YouTube).
• lib/youtube.js — YouTube URL detection and normalization.
• lib/llm/ — provider resolver + adapters for image captioning (vision), audio transcription (STT), and PDF OCR against OpenAI-compatible endpoints.
• lib/frontmatter.js — YAML frontmatter builder with the PULLMD_FRONTMATTER_FIELDS allowlist.
• lib/coverage-guard.js — detects extractions that kept only a sliver of the page and recovers the container holding the body.
• lib/query-extract.js — Opt-in query-scoped extraction (?query=): BM25 over heading-based sections of the converted markdown, returning only the relevant sections/blocks. Composes (block/section parser) and (tokenizer + scorer).

---

License

GNU AGPL v3 — Copyright © 2026 Aeterna Labs.

PullMD is free software: you can redistribute it and modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, version 3 or later. If you run a modified version as a network service, you must make your modifications available to its users.