MTScraper

Free website crawler for AI coding agents, by MazTech.

Ask your agent to "scrape example.com". MTScraper crawls the site, follows its internal links, and saves each page's clean main text (no menus or footers) to JSONL or CSV. Linked PDFs are read too, text and tables. Then your agent works with the data for you.

Free, open source, no credits, no page caps, no API key, runs on your PC.

• Clean text: main content only, extracted with Mozilla Readability
• Linked PDFs: text and tables extracted by default (--no-pdf to skip)
• JavaScript sites: optional headless Chromium rendering (--js)
• One-page apps: clicks through tabs and buttons and saves every view (--click)
• Resumable: stop a big crawl and pick it up later (--resume)
• Polite: obeys robots.txt and stays on the target domain
• One record format for everything: url, title, view, type, text, tables
• Node.js only, nothing to install: runs through npx

Why MTScraper

• Runs on your machine. Pages and PDFs are never uploaded anywhere.
• No account, no API key, no per-page credits. A crawl of 5 pages or 5,000 costs the same: nothing.
• One record format for HTML pages and PDFs, so the data is ready for a database, a spreadsheet or an embedding index without cleanup.
• Works inside the agents people already use. Claude Code, Codex, Gemini CLI, Cursor and Copilot pick it up as a skill or an MCP server.
• Polite by default: robots.txt obeyed, four requests at a time, stays on the target domain.
• A few hundred lines of plain JavaScript you can read in one sitting.

How it works

1. The start URLs go into a Crawlee crawler: robots.txt is checked, links on the same domain are followed, failed requests are retried, and at most four requests run at once.
2. Each HTML page goes through Mozilla Readability, which keeps the main content and drops menus, footers and ads.
3. Each linked PDF is downloaded and read with pdf.js. Lines are rebuilt from the text positions, and lines whose cells line up become tables.
4. Every page becomes one record with the same six fields, appended to a JSONL or CSV file as it arrives.
5. With --js, pages are rendered in headless Chromium first. With --click, each matching tab or button is clicked and every new view is saved as its own record.
6. With --mcp, the same crawler is exposed as one MCP tool, crawl, over stdio.

Requirements

• Node.js 20 or newer
• An AI agent that can run skills or MCP servers (see Install), or just a terminal
• Chromium, once, only for --js, --click and buttons: npx -y github:mazprotech/mtscraper setup

Install / where it works

MTScraper reaches your agent in two ways, and both run the same tool through npx -y github:mazprotech/mtscraper:

• Skill: a folder (skills/mtscraper) with instructions the agent reads. The agent runs the CLI itself.
• MCP server: your app starts npx -y github:mazprotech/mtscraper --mcp and gets one tool, crawl.

To copy the skill folder, clone the repo once and copy skills/mtscraper into the skills folder listed for your tool below (~ is your home folder; in PowerShell use $HOME).

git clone https://github.com/mazprotech/mtscraper.git
mkdir -p ~/.claude/skills && cp -r mtscraper/skills/mtscraper ~/.claude/skills/    # example: Claude Code

git clone https://github.com/mazprotech/mtscraper.git
New-Item -ItemType Directory -Force "$HOME\.claude\skills" | Out-Null              # example: Claude Code
Copy-Item -Recurse -Force mtscraper\skills\mtscraper "$HOME\.claude\skills\"

Every MCP client needs the same values: command npx, arguments -y github:mazprotech/mtscraper --mcp. In the clients that use an mcpServers JSON file it looks like this:

{
  "mcpServers": {
    "mtscraper": {
      "command": "npx",
      "args": ["-y", "github:mazprotech/mtscraper", "--mcp"]
    }
  }
}

Claude Code

Plugin (recommended). Inside Claude Code, run:

/plugin marketplace add mazprotech/mtscraper
/plugin install mtscraper@maztech

Or copy the skill folder to ~/.claude/skills/ (all projects) or .claude/skills/ (one project). Restart Claude Code, or run /reload-plugins after a plugin install.

Claude Desktop

MCP only. Open Settings, Developer, Edit Config, or edit the file directly:

• macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
• Windows: %APPDATA%\Claude\claude_desktop_config.json

Add the mcpServers block above (merge it with any servers you already have), then fully quit and restart Claude Desktop.

OpenAI Codex (the "ChatGPT" option)

"Works with ChatGPT" here means Codex: the Codex CLI, the Codex IDE extension and the ChatGPT desktop app. The ChatGPT chat app in the browser or on your phone cannot run local tools, so it cannot use MTScraper.

• Skill: copy the folder to ~/.agents/skills/ (all projects) or .agents/skills/ (one repo).
• MCP: codex mcp add mtscraper -- npx -y github:mazprotech/mtscraper --mcp. This writes ~/.codex/config.toml, which the CLI, IDE extension and desktop app share. Or add it by hand:

  [mcp_servers.mtscraper]
  command = "npx"
  args = ["-y", "github:mazprotech/mtscraper", "--mcp"]
  

Gemini CLI

• Skill: copy the folder to ~/.gemini/skills/ (or ~/.agents/skills/), or to .gemini/skills/ (or .agents/skills/) in a project.
• MCP: add the mcpServers block above to ~/.gemini/settings.json (all projects) or .gemini/settings.json (one project).

以上为 README 的开头部分。其余内容请阅读原文。 查看完整 README ↗