ByteCrawl
Léelo en español.
Give your AI agent focused web crawling. ByteCrawl is an MCP server (and a
small Python library) that doesn't just scrape a page — it crawls a whole site
and returns the pages most relevant to your topic first, using Shark-Search
and OPIC in pure Python.
Why focused crawling?
Most crawlers visit pages in whatever order they find them. Under a request
budget, order is everything. From en.wikipedia.org/wiki/Silicon_Valley with
query san francisco, 20 pages each:
| Strategy | Pages over 0.1 relevance | Best page |
|---|
shark | 20 | 0.7774 |
opic | 4 | 0.2799 |
bfs | 1 | 0.1068 |
Same twenty requests; only the order changed. Reproduce it with
result.relevant(0.1).
That is a token argument as much as a quality one. Those pages end up in a
context window. BFS spends nineteen of its twenty on whatever happened to be
linked first, and you pay for all nineteen. And each page arrives as Markdown
rather than raw HTML — around 6.5× fewer tokens for the same content.
fetch_markdown reports both counts, so you can state the saving rather than
assume it.
How it works. Every link gets a score before it is visited. Shark-Search
takes how much the anchor text and the parent page look like your query, decayed
with depth so a branch that stops matching fades out on its own; OPIC scores by
importance instead, cash flowing along links. The frontier is a priority queue,
so the crawler always spends its next request on the best link it knows about —
which is why the budget lands on relevant pages instead of nearby ones.
from bytecrawl import SharkSearch
result = SharkSearch(query="vector databases").crawl(url, max_pages=100)
result.top(10) # [{url, title, relevance, depth, order}, ...]
- Shark-Search (Hersovici et al., 1998) — topical best-first; links inherit
their parent's relevance with decay, so barren branches die out on their own.
- OPIC (Abiteboul et al., 2003) — live PageRank via "cash" flowing along
links, no full graph needed.
pagerank() is included to compare against.
- BFS — level by level. The honest baseline.
All three share one loop — pop, fetch, score links, push — so an equal page
budget is a fair comparison, and every surface takes the same three names.
Scrapy is a framework you wire up yourself and Firecrawl is a paid SaaS.
ByteCrawl is a library with a three-package core and these strategies built in.
Quick start
One command, nothing installed:
claude mcp add --scope user --transport http bytecrawl https://bytecrawl.vercel.app/mcp
Your agent now has six tools — the same six the playground gives a human:
| Tool | What it does |
|---|
focused_crawl | Crawl a site, rank pages by relevance to a query (Shark-Search / OPIC / BFS) |
compare_strategies | Run all three on the same budget and see which one wins |
fetch_markdown | One page → clean Markdown, around 6.5× fewer tokens than raw HTML |
extract | Structured records via CSS selectors — or call it with just a URL and it tells you what the page offers |
list_links | Every outbound link, absolute and deduplicated |
fetch_json_api | Hit a hidden JSON API |
Then ask in plain language — you never call the tools yourself:
Use bytecrawl to find everything on python.org about the packaging ecosystem
Say the name. Most agents ship their own single-page fetcher and reach for it by
default; bytecrawl is what gets you a crawl ranked by your query instead of
one page read in isolation.
The hosted server is static-only, rate-limited, capped at 10 pages per crawl,
and refuses non-public URLs. For JS-rendered pages or bigger crawls, run it
yourself:
pipx install "bytecrawl[mcp]" # a CLI app, hence pipx
claude mcp add --scope user bytecrawl -- bytecrawl-mcp
/bytecrawl — the agent skill
The server gives your agent the tools; the skill gives it the judgment about
using them, and you a slash command:
mkdir -p ~/.claude/skills/bytecrawl && \
curl -so ~/.claude/skills/bytecrawl/SKILL.md \
https://bytecrawl.vercel.app/agent-onboarding/SKILL.md
The same command updates it. Installing copies the file, so later fixes never
reach your copy — the skill compares its hash against
/agent-onboarding/skill.json
once a session and tells you when yours is behind. It never overwrites anything
on its own.
No skills directory? Hand the file to any agent instead — it works for that
conversation, installs nothing:
Read and follow https://bytecrawl.vercel.app/agent-onboarding/SKILL.md
Library API
from bytecrawl import Scraper
bot = Scraper()
page = bot.fetch("https://books.toscrape.com") # auto: static, browser fallback
books = page.extract("article.product_pod",
{"title": "h3 a::attr(title)", "price": "p.price_color::text"})
page.markdown() # clean Markdown for LLMs · page.tokens() # token estimate
bot.static(url) # plain HTML
bot.api(url, params={...}) # hidden JSON API
bot.browser(url, wait="div.results") # JS via Playwright
bot.crawl(url, item="article", fields={...},
next_page="li.next a::attr(href)") # pagination
bot.session().login(url, data, csrf_field="csrf_token") # authenticated
Install
pip install bytecrawl # slim core (requests + beautifulsoup4 + lxml)
pip install bytecrawl[llm] # + Markdown for LLMs
pip install bytecrawl[browser] # + Playwright (then: playwright install chromium)
pip install bytecrawl[mcp] # + local MCP server (Python 3.10+)
pip install bytecrawl[all]
A missing extra never fails silently — each one raises an ImportError naming the
exact command to run.
HTTP API (no install)
The same engine behind one hosted GET endpoint. Everything except url is
optional; no key, no account.
curl "https://bytecrawl.vercel.app/api?url=books.toscrape.com"
curl "https://bytecrawl.vercel.app/api?url=en.wikipedia.org/wiki/Silicon_Valley&method=crawl&query=san+francisco&strategy=shark"
method is one of markdown (default), text, html, links, json,
extract, crawl. Static HTML only, crawls capped at 10 pages, rate-limited,
responses cached 10 minutes, private and loopback addresses refused. Full
reference: https://bytecrawl.vercel.app/docs
Learn each scraping technique
A guided walkthrough (in Spanish) with a runnable example against a practice
site:
static HTML ·
dynamic JS ·
hidden APIs ·
pagination ·
login ·
graph crawling ·
Markdown for LLMs ·
ethics
Contributing
pip install -e ".[llm,dev,mcp]"
pytest # 120 tests, no network required
pytest -m live # + live browser tests (needs the browser extra)
ruff check bytecrawl tests
Scrape responsibly: respect robots.txt, terms of service and rate limits.
ByteCrawl ships with a configurable delay between requests.
License
MIT — see LICENSE.
