into-md

Turn any URL into clean, structured markdown built to feed LLMs. Unlike most url→markdown tools, into-md is tuned for context quality: it auto-detects whether a page needs a static fetch or a headless browser, runs Readability to keep the real article and drop nav/ads/chrome, and serializes tables to JSON so models parse them reliably instead of choking on pipe-delimited rows.

Installation

# Global install with bun from npm
bun add -g into-md
# or
npm install -g into-md
# or
yarn global add into-md

Usage

into-md <url>

By default, into-md auto-detects whether a page needs a headless browser. It fetches with a static HTTP request first, inspects the result for SPA signals, and falls back to Playwright if needed.

The static request sends Accept: text/markdown, text/html. If the server answers with text/markdown, that markdown is used as-is (no extraction or conversion), any frontmatter it carries is kept, and the strategy is reported as markdown. With --verbose, the server's x-markdown-tokens count is printed when present.

Examples

# Auto-detect (default) — static fetch, falls back to headless if needed
into-md https://example.com/article

# Force headless browser rendering (skip auto-detect probe)
into-md https://spa-site.com/page --js

# Force static HTTP fetch (never launch a browser)
into-md https://example.com/page --no-js

# Skip content extraction, convert full page
into-md https://example.com --raw

# With authentication cookies
into-md https://private-site.com/page --cookies cookies.txt

# Verbose output (includes auto-detect decisions)
into-md https://example.com/article -v

Options

Flag	Description	Default
-o, --output <file>	Write output to file instead of stdout	stdout
--js	Force headless browser rendering (skip auto-detect)	auto-detect
--no-js	Force static HTTP fetch (never launch a browser)	auto-detect
--raw	Skip content extraction, convert entire HTML	disabled
--cookies <file>	Path to cookies file for authenticated requests	none
--user-agent <string>	Custom User-Agent header	browser-like UA
--encoding <encoding>	Force character encoding (auto-detected by default)	auto
--strip-links	Remove hyperlinks, keep only anchor text	disabled
--exclude <selectors>	CSS selectors to exclude (comma-separated)	none
--timeout <ms>	Request timeout in milliseconds	30000
--no-cache	Bypass response cache	cache enabled
-v, --verbose	Show detailed progress information	minimal
-h, --help	Show help	-
-V, --version	Show version	-

--js and --no-js are mutually exclusive — passing both is an error.

Auto-Detect

When no rendering flag is passed, into-md runs a two-stage heuristic to decide whether the page needs a headless browser:

Stage 1 — Raw HTML inspection (before Readability extraction):

• Checks for empty SPA root divs (#root, #app, #__next, #__nuxt, #__svelte)
• Checks for <noscript> tags mentioning "javascript" combined with a near-empty body (< 100 chars of body text)

Stage 2 — Post-extraction content threshold (after Readability):

• Falls back to headless if extracted text is < 200 characters and contains no structural HTML tags (<article>, <p>, <pre>, <li>, <h1>–<h6>)
• Skipped when --raw is passed

Non-HTML responses (PDFs, images, etc.) skip auto-detect entirely and use static processing.

Output Format

Strategy Line

A strategy summary is printed to stderr on every run:

Strategy: static
Strategy: headless
Strategy: markdown            # server returned markdown directly
Strategy: auto > static       # auto-detect chose static
Strategy: auto > headless     # auto-detect fell back to headless
Strategy: auto > markdown     # auto-detect got markdown from the server

Frontmatter

Page metadata is included as YAML frontmatter. title, description, and author come from the page's <title>/meta tags (falling back to Readability's title and byline) and are omitted when absent:

---
title: "Article Title"
description: "Meta description from the page"
author: "Author Name"
strategy: "auto>headless"
source: "https://example.com/article"
---

The strategy field records how the page was fetched: static, headless, markdown, or the auto> form of each. When the server returns markdown with its own frontmatter, those fields (for example date) are carried over.

Tables

Tables are converted to fenced JSON blocks for reliable LLM parsing:

{
  "caption": "Quarterly Revenue",
  "headers": ["Quarter", "Revenue", "Growth"],
  "rows": [
    { "Quarter": "Q1", "Revenue": "$1.2M", "Growth": "12%" },
    { "Quarter": "Q2", "Revenue": "$1.5M", "Growth": "25%" }
  ]
}

Caching

Converted markdown is cached in ~/.cache/into-md/ with a 1-hour TTL. Each entry stores the markdown, page metadata, final URL, and the strategy that produced it (static, headless, or markdown), so a cache hit skips fetching and conversion entirely and reports the original strategy in the Strategy: line and frontmatter.

Entries are keyed by URL plus the options that change the output (--raw, --exclude, --strip-links, --encoding). Auto-detect reuses any fresh entry. A forced flag reuses only an entry its strategy could have produced, and otherwise re-fetches and overwrites it:

Flag	Reuses cached	Re-fetches on cached
(none)	any strategy	—
--js	headless	static, markdown
--no-js	static, markdown	headless

--no-cache skips both reading and writing the cache.

Playwright & Browser Binaries

Playwright is a required dependency but browser binaries are downloaded on first use. If binaries are missing when needed:

• Interactive terminal: asks whether to install Chromium now and, if you agree, runs Playwright's installer with the current runtime (Bun or Node)
• Non-interactive / CI (no TTY, or CI set): exits with an error telling you to run bunx playwright install chromium

Development

bun install              # Install dependencies
bun run start -- <url>   # Run the CLI from source
bun run build            # Build the CLI
bun run build:watch      # Build with watch mode
bun run test             # Run tests
bun run check            # Check for lint errors
bun run fix              # Auto-fix lint errors
bun run typecheck        # Type check

Tech Stack

• Runtime: Bun
• Language: TypeScript
• HTML Parsing / DOM: linkedom, with parse5 normalizing documents linkedom would build differently from a browser (missing <html>/<body>, misplaced table content)
• Markdown Conversion: turndown
• Content Extraction: @mozilla/readability
• Headless Browser: playwright
• CLI Framework: commander

License

MIT