sitemapr

sitemapr is a deterministic toolkit for reading and validating XML, text, and index sitemaps against the Sitemap Protocol 0.9 and the related W3C and RFC standards. It parses sitemap sources into tidy tibbles, expands sitemap indexes, discovers a site’s sitemaps from its root URL, and reports schema and protocol findings with stable, reproducible output.

Status: experimental. The package works and is covered by an extensive test suite, but the API may still change without deprecation and it is not yet on CRAN. Pin a commit or tag if you depend on it.

Installation

sitemapr is not on CRAN yet. Install it from r-universe:

install.packages(
  "sitemapr",
  repos = c("https://bart-turczynski.r-universe.dev", "https://cloud.r-project.org")
)

Or install the development version from GitLab:

# install.packages("pak")
pak::pak("gitlab::bart-turczynski/sitemapr")

Its URL-parsing dependency, rurl, is installed automatically from CRAN.

Usage

Read a sitemap into a tidy tibble

read_sitemap() accepts a sitemap URL or a local file (.xml, .txt, .gz, or .tar.gz) and returns one row per URL, with lastmod, changefreq, priority, and the image/video/news/alternate extension columns.

library(sitemapr)

sitemap <- '<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2024-01-01</lastmod>
    <priority>1.0</priority>
  </url>
  <url>
    <loc>https://example.com/?page=about</loc>
    <changefreq>monthly</changefreq>
  </url>
</urlset>'

path <- tempfile(fileext = ".xml")
writeLines(sitemap, path)

read_sitemap(path)
#> # A tibble: 2 × 9
#>   loc    lastmod             changefreq priority images video  news   alternates
#>   <chr>  <dttm>              <chr>         <dbl> <list> <list> <list> <list>    
#> 1 https… 2024-01-01 00:00:00 <NA>              1 <NULL> <NULL> <NULL> <NULL>    
#> 2 https… NA                  monthly          NA <NULL> <NULL> <NULL> <NULL>    
#> # ℹ 1 more variable: source_sitemap <chr>

A top-level sitemap index is expanded recursively (cycle-safe, depth- and count-capped) so every reachable child sitemap’s rows carry provenance.

Validate against the schema and protocol

validate_sitemap() returns a stable findings tibble — one row per issue, with a code, severity, and the layer (schema, protocol, classification, or index) that produced it. The same source and mode always yield a row-for-row identical result.

invalid <- '<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url><loc>https://example.com/</loc><priority>2.0</priority></url>
  <url><loc>/relative/path</loc></url>
</urlset>'

path <- tempfile(fileext = ".xml")
writeLines(invalid, path)

findings <- validate_sitemap(path)
findings[, c("code", "severity", "layer")]
#> # A tibble: 3 × 3
#>   code                           severity layer   
#>   <chr>                          <chr>    <chr>   
#> 1 SCHEMA_INVALID                 error    schema  
#> 2 PROTOCOL_PRIORITY_OUT_OF_RANGE error    protocol
#> 3 PROTOCOL_URL_NOT_ABSOLUTE      error    protocol

Pass mode = "non-strict" to downgrade schema violations to warnings and drop strict-only findings.

Discover a site’s sitemaps

sitemap_tree() takes a site-root URL, tries a catalog of common sitemap paths (generic, then WordPress/Shopify), and returns a discovery tree: one row per candidate, marked accepted or rejected. It needs network access, so it is not run here.

sitemap_tree("https://example.com")

Learn more

• Full reference and articles: https://bart-turczynski.gitlab.io/sitemapr/
• Getting-started vignette: vignette("introduction", package = "sitemapr")