rurl

rurl is a lightweight, vectorized toolkit for URL parsing, normalization, extraction, and matching in R.

Current package capabilities include:

• Robust parsing via safe_parse_url() and safe_parse_urls()
• URL normalization with fine-grained controls for protocol, www, case, trailing slashes, index pages, path normalization, scheme-relative URLs, host encoding, and path encoding
• Opt-in query-string handling: drop trackers, keep contentful params, and audit params across a URL set with query_param_summary()
• URL component extractors (get_* helpers)
• URL joins on identity with the url_*_join() family (url_inner_join(), url_left_join() and four more)
• Built-in memoization caches with introspection and configuration (rurl_cache_info(), rurl_cache_config(), rurl_clear_caches())

Installation

# From CRAN
install.packages("rurl")

# From r-universe
install.packages(
  "rurl",
  repos = c("https://bart-turczynski.r-universe.dev", "https://cloud.r-project.org")
)

# Development version from GitLab
# install.packages("remotes")
remotes::install_gitlab("bart-turczynski/rurl")

Quick Start

safe_parse_url() is the core workhorse. It returns parsed components and a normalized clean_url.

library(rurl)

parsed <- safe_parse_url(
  "HTTP://www.Example.com/a//b/../index.html?x=1#frag",
  protocol_handling = "https",
  www_handling = "strip",
  case_handling = "lower_host",
  trailing_slash_handling = "strip",
  index_page_handling = "strip",
  path_normalization = "both",
  host_encoding = "idna",
  path_encoding = "encode"
)

parsed$clean_url
#> [1] "https://example.com/a"
parsed$parse_status
#> [1] "ok"

clean_url is a normalized canonical key built from scheme, host, and path. By default the query is dropped and port, fragment, and userinfo are always excluded — read them from the dedicated components (get_port(), get_query(), get_fragment(), get_userinfo()) instead. The query can be selectively kept on clean_url via query_handling on either safe_parse_url() or get_clean_url() (see Query handling). With path_encoding = "decode" the path shows its percent-encoded UTF-8 and unreserved characters decoded, so clean_url is human-readable rather than guaranteed URL-safe; reserved triplets such as %2F stay encoded.

Scheme-relative URL handling is configurable:

safe_parse_url("//example.com/path", scheme_relative_handling = "keep")$parse_status
#> [1] "ok-scheme-relative"
safe_parse_url("//example.com/path", scheme_relative_handling = "https")$clean_url
#> [1] "https://example.com/path"

For vectors, use safe_parse_urls():

safe_parse_urls(c("example.com", "https://www.example.com/path"))[, c("original_url", "clean_url", "parse_status")]

Normalization Controls

safe_parse_url() and get_clean_url() support these controls:

• protocol_handling: keep, none, strip, http, https
• www_handling: none, strip, keep, if_no_subdomain
• case_handling: lower_host (default), keep, lower, upper
• trailing_slash_handling: none, keep, strip
• index_page_handling: keep, strip
• path_normalization: none, collapse_slashes, dot_segments, both
• scheme_relative_handling: keep, http, https, error
• host_encoding: keep, idna, unicode
• path_encoding: keep, encode, decode
• subdomain_levels_to_keep: NULL, 0, or N > 0
• query_handling: drop (default), filter, allow, keep (see Query handling)

Subdomain retention is applied after www_handling:

get_host("http://www.three.two.one.example.com", www_handling = "strip", subdomain_levels_to_keep = 1)
#> [1] "one.example.com"
get_clean_url("http://www.deep.sub.example.com/path", subdomain_levels_to_keep = 0)
#> [1] "http://www.example.com/path"

Host and path encoding controls:

get_clean_url("http://münich.com/a%20b",
              host_encoding = "idna",
              path_encoding = "encode",
              case_handling = "lower_host")
#> [1] "http://xn--mnich-kva.com/a%20b"

get_clean_url("http://xn--mnich-kva.com/caf%C3%A9%2Fx",
              host_encoding = "unicode",
              path_encoding = "decode",
              case_handling = "keep")
#> [1] "http://münich.com/café%2Fx"

Accessors

u <- "https://user:[email protected]/path/to/page?a=1&b=2#frag"

get_scheme(u)
get_host(u)
get_subdomain(u)
get_domain(u)
get_tld(u)
get_path(u)
get_query(u)
get_query(u, format = "list")
get_fragment(u)
get_port(u)
get_user(u)
get_password(u)
get_userinfo(u)
get_parse_status(c(u, "mailto:[email protected]"))

Query handling

By default clean_url is query-free (query_handling = "drop"), so a tracker like ?utm_source= never changes the canonical key. Opt in to keep contentful params while stripping known trackers (utm_*, fbclid, gclid, …) via a built-in denylist:

# "filter": keep contentful params, drop known trackers
get_clean_url("https://youtube.com/watch?v=dQw4w9&utm_source=nl",
              query_handling = "filter")
#> [1] "https://youtube.com/watch?v=dQw4w9"

# "allow": keep ONLY the names you list (params_keep is the allowlist)
get_clean_url("https://shop.com/p?id=42&ref=abc&fbclid=xyz",
              query_handling = "allow", params_keep = "id")
#> [1] "https://shop.com/p?id=42"

# "keep": keep every param, canonicalized (optionally sorted)
get_clean_url("https://ex.com/?b=2&a=1",
              query_handling = "keep", sort_params = TRUE)
#> [1] "https://ex.com/?a=1&b=2"

The same engine arguments flow through safe_parse_urls(), so filtering also shapes a key built from clean_url — ?v=1 and ?v=2 stay distinct while utm-only differences still collapse under "filter":

safe_parse_urls(c("https://ex.com/?v=1&utm_source=x", "https://ex.com/?v=2"),
                query_handling = "filter")[, c("original_url", "clean_url")]
#>                       original_url           clean_url
#> 1 https://ex.com/?v=1&utm_source=x https://ex.com/?v=1
#> 2              https://ex.com/?v=2 https://ex.com/?v=2

get_query() takes the same arguments (defaulting to query_handling = "keep") to pull a cleaned query directly:

get_query("https://youtube.com/watch?v=dQw4w9&utm_source=nl",
          query_handling = "filter")
#> [1] "v=dQw4w9"

Audit a URL set before choosing a policy with query_param_summary(), whose would_drop column previews what "filter" would remove:

urls <- c(
  "https://ex.com/?utm_source=nl&id=42",
  "https://ex.com/watch?v=abc&utm_source=x",
  "https://ex.com/?id=99"
)
query_param_summary(urls)
#>        param n n_urls example_value                             example_url would_drop
#> 1 utm_source 2      2            nl     https://ex.com/?utm_source=nl&id=42       TRUE
#> 2         id 2      2            42     https://ex.com/?utm_source=nl&id=42      FALSE
#> 3          v 1      1           abc https://ex.com/watch?v=abc&utm_source=x      FALSE

URL Joins

The url_*_join() family (url_inner_join(), url_left_join(), url_right_join(), url_full_join(), url_semi_join(), url_anti_join()) matches rows on URL identity, the comparison key from get_url_key(), never on a cleaned display string:

A <- data.frame(URL = c("https://Example.com:443/page", "https://example.com/other"),
                ValA = 1:2, stringsAsFactors = FALSE)
B <- data.frame(URL = c("https://example.com/page", "https://example.com/page?utm_source=nl"),
                ValB = c("x", "y"), stringsAsFactors = FALSE)

# Host case and the default port do not change identity, so the first row of A
# matches the first row of B. The query does, so B's second row stays apart.
url_left_join(A, B, by = "URL")

To match on cleaned URLs instead, build the key with get_clean_url() and merge on it. Cleaning drops the query here, so the first row of A now matches both rows of B. incomparables = NA keeps URLs that have no clean form from matching each other. This is an inner join; add all.x = TRUE to keep every row of A:

A$k <- get_clean_url(A$URL)
B$k <- get_clean_url(B$URL)
merge(A, B, by = "k", incomparables = NA)

canonical_join(), which joined on the cleaned URL, is deprecated and will be removed in rurl 4.0.0.

Documentation

The reference index lists every exported function. The Getting Started article walks through parsing, extraction and the parse caches, and Standard-conformant parsing explains the url_standard profiles.

Public Suffix List

Domain and TLD extraction is delegated to the pslr package, which owns the Public Suffix List and its refresh cycle. rurl ships no embedded copy of the list. To update the PSL, call pslr::psl_refresh() (see the pslr documentation for details).

Acknowledgments

These packages build on data, libraries, and prior work from many others. See ACKNOWLEDGMENTS.md for the full list of thanks.

Related packages

rurl is part of a small ecosystem of R packages by the same author:

• pslr — the Public Suffix List engine that powers domain and TLD extraction in rurl. Use it directly when you need raw eTLD / registrable-domain queries without full URL parsing.
• punycoder — the Punycode and IDNA codec that rurl uses for internationalized host handling. Useful on its own for host normalization and Unicode ↔ ACE round-trips.

Citation

If you use rurl in your work, please cite it. Run citation("rurl") for the current citation, or see CITATION.cff.

Each release is archived on Zenodo. Cite the concept DOI 10.5281/zenodo.20972584 to refer to the software in general (it always resolves to the latest version), or the version-specific DOI shown on the Zenodo record for a particular release.

Code of Conduct

Please note that this package is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

License

MIT © 2026 Bart Turczynski