robotstxtr

Faithful robots.txt parsing and URL-allowed decisions for R, powered by Google’s own matcher.

robotstxtr evaluates whether a URL may be crawled according to a site’s robots.txt rules. It wraps a vendored, upstream-test-validated snapshot of Google’s open-source C++ robots.txt parser and matcher, so allow and disallow decisions match Googlebot’s behavior — including the quirky parts (longest-match wins, Allow-beats-Disallow ties, */$ wildcards, case-sensitive paths, percent-escape canonicalization). It can also fetch robots.txt for you under a deterministic, conservative policy, through the ssrfr guard against server-side request forgery, construct fetch origins with rurl, and return richly typed, vectorized decisions with per-rule match metadata.

This is a faithful, upstream-test-validated replica of the pinned matcher, not a claim of universal behavioral identity with Googlebot.

Installation

Install the released version from CRAN:

install.packages("robotstxtr")

Or the development version from r-universe:

install.packages(
  "robotstxtr",
  repos = c("https://bart-turczynski.r-universe.dev", "https://cloud.r-project.org")
)

The development version also lives on GitLab. Its dependencies, including rurl, are all on CRAN, so it installs with no extra repository configuration:

# with pak
pak::pak("gitlab::bart-turczynski/robotstxtr")

# or with remotes
remotes::install_gitlab("bart-turczynski/robotstxtr")

Installing from source compiles the vendored C++ matcher and requires a C++17 toolchain. Once dependencies have been obtained, installing, testing, and checking the package does not make live network requests.

Usage

Match one or more URLs against a robots.txt body you already have, fully offline, with allowed_by_robots_text():

library(robotstxtr)

robots <- "user-agent: *
disallow: /private
allow: /private/public
"

x <- allowed_by_robots_text(
  robots,
  c(
    "https://example.com/index.html",     # not disallowed  -> TRUE
    "https://example.com/private/report", # under /private  -> FALSE
    "https://example.com/private/public"  # re-allowed      -> TRUE
  ),
  "mybot"
)

x$results[c("url", "allowed", "decision_source")]
#>                                  url allowed decision_source
#> 1     https://example.com/index.html    TRUE   default_allow
#> 2 https://example.com/private/report   FALSE   rule_disallow
#> 3 https://example.com/private/public    TRUE      rule_allow

allowed is the TRUE/FALSE/NA answer; decision_source explains why.

If you do not already have the robots.txt text, allowed_by_robots_url() derives the governing origin, fetches it once per origin, and matches:

allowed_by_robots_url("https://example.com/some/page", "mybot")

See the “Introduction to robotstxtr” vignette for the full walkthrough, including match metadata, the fetch policy, and body inspection with robots_body().

Validate a robots.txt document

Document validation is separate from URL allow/disallow evaluation. Validate supplied text (or raw acquisition bytes) with robots_validate_text():

validation <- robots_validate_text(
  "user-agent: *\ndisallow: /private\nunknown-field: value\n"
)

validation$documents
validation$diagnostics

Use robots_validate_url() when the document must first be fetched. It calls the same deterministic robots_fetch() path and validates each distinct stored raw body once; acquisition limits and failures remain explicit evidence.

The profile is deliberately named google-parser-compatible. It describes the pinned parser plus conservative structural checks, not universal crawler or RFC 9309 validity. Validation reports file syntax and structure only: it does not answer whether a particular URL is allowed, infer whether rules match the webmaster’s intent, or fetch and check remote sitemap resources.

Engine-aware contract

Multi-engine integrations should use the versioned facade, robots_evaluate_text_v1() and robots_evaluate_url_v1(), which selects the engine policy and the matcher (Google, Yandex or Bing) explicitly and returns explicit non-decisions instead of falling back to Google. The “Engine-aware contract” vignette walks through it.

Citation

If you use robotstxtr in your work, please cite it. Run citation("robotstxtr") for the current citation, or see CITATION.cff.

Each release is archived on Zenodo. Cite the concept DOI 10.5281/zenodo.23018995 to refer to robotstxtr in general; it always resolves to the latest release. Use the version-specific DOI shown on the Zenodo record for a particular release.