crawlcheck-core

The dependency-free modules of CrawlCheck, the verification layer for the agentic web, extracted from the production service. These are the readers behind its signed answer about what a site permits and serves to AI crawlers and agents. No build step, no dependencies, ESM, Node 20+.

The hosted service fetches a site as a set of named crawler request profiles plus comparison controls, and measures it section by section across three stages (the current counts are published, with the date they were read, at crawlcheck.io/spec/canonical.json; this README does not repeat them, so it cannot fall behind) — reach (can the crawler get in), read (what it actually receives), quote (can an answer engine lift it). This repository is the open core: the readers and checks that need no network, no store and no key, published so the measurements can be reproduced.

What is here today: the robots.txt policy reader, the crawler identifier / log-line forgery verifier, the quotable-content reader, the image-signal reader, and the markdown-negotiation and agent-surface probes. What stays in the service: the scoring — the thresholds each reading is judged against, the weight every section carries in the grade, and the corpus calibration those numbers were set from — plus fetching as those crawler identities, the record over time, monitoring, the dataset, and billing. The rule is simple: a reading of your page is a fact you are entitled to check, and the calibration is the product.

Also in this repository: the SDKs

• sdk/ — @crawlcheck/sdk (npm): the typed client for the CrawlCheck API and the zero-dependency offline verifier for signed answers, evidence bundles and receipts. npm test runs its suite against production.
• sdk-python/ — crawlcheck (PyPI): the same client and verifier for Python 3.9+, no dependencies.

Both are published from the private service repository by a workflow that refuses to publish unless the directory here is byte-identical to what it ships, so this copy is the source the package pages point at, not a mirror that can drift.

Install

npm install crawlcheck-core

Published on the npm registry as crawlcheck-core; npm install github:emmanuelorta/crawlcheck-core still works if you want the repository head. Every subpath below resolves from either install.

robots — which crawlers your robots.txt actually admits

import { crawlerPolicy } from "crawlcheck-core/robots";

const policy = crawlerPolicy(robotsTxt, /* unreadable */ false, "/wp-admin/");
policy.agents.find(a => a.ua === "GPTBot");
// { ua:"GPTBot", label:"OpenAI training corpus", role:"train",
//   allowed:true, allowed_root:true, named:true, rules_shadowed:true }
policy.shadowed;
// [{ uas:["gptbot"], missing:["/wp-admin/","/feed/"], missing_count:2 }]

For each of 116 named agents (answer engines, search indexes, training crawlers, tools, social unfurlers, regional engines, and a few retired names still seen in robots.txt files) it reports: which user-agent group applies (named), whether the given path and the root are allowed under that group, and whether the group shadows the * rules.

The shadowing defect

A crawler obeys only its most specific group. The moment you write

User-agent: *
Disallow: /wp-admin/

User-agent: GPTBot
Allow: /

GPTBot no longer sees Disallow: /wp-admin/ — the * group stops applying to it entirely. Sites that add an allowlist for AI agents routinely void every rule they thought still held. shadowedGroups() lists each named group and the * disallows it lost. This was found on our own sites first: how to block AI crawlers and the glossary entry for user-agent group.

API

Export	What it does
crawlerPolicy(txt, unreadable, path)	Full per-agent policy for one file. measured:false with a reason when the file is empty or unreadable — an absence is never reported as a policy
parseGroups(txt)	RFC 9309 groups: consecutive User-agent lines share one group; Allow/Disallow rules in order
groupFor(groups, ua)	The group that applies to an agent, per RFC 9309: the most specific matching token wins (version suffix ignored), every group naming that token is combined, a group for a more specific token (Googlebot-News) never applies to the general crawler, Google's documented fallbacks (Google-InspectionTool obeys Googlebot) apply, and * is the last resort
pathAllowed(group, path) / rootAllowed(group)	RFC 9309 matching (* anywhere, trailing $ anchors the end); longest rule wins; Allow beats Disallow at equal length
ruleMatches(rule, path) / decide(rules, path)	The matcher itself, and the decision with the rule that made it
shadowedGroups(groups)	Named groups for which a * disallow is lost (not covered by one of the group's own rules, e.g. its own Disallow: /)
AGENTS	The 116-agent table: [token, label, role]

This is the beginning of the README. Read the rest at the original. Read the full README ↗