WebScraping.AI Ruby Client

Official Ruby client for the WebScraping.AI API — web scraping with Chromium JavaScript rendering, rotating datacenter/residential/stealth proxies, and AI-powered question answering and structured field extraction on any page. See the API documentation for the full parameter reference.

Installation

# Gemfile
gem "webscraping_ai", "~> 4.1"

Or:

gem install webscraping_ai

Requires Ruby 3.1+.

Quick start

Sign up to get an API key — a free trial, no credit card required. Your key lives in the dashboard.

require "webscraping_ai"

client = WebScrapingAI::Client.new(api_key: ENV.fetch("WEBSCRAPING_AI_API_KEY"))

# Page HTML
html = client.html("https://example.com", js: true)

# Visible text
text = client.text("https://example.com")

# CSS-selected fragment
title = client.selected("https://example.com", selector: "h1")

# Multiple selectors at once
fragments = client.selected_multiple("https://example.com", selectors: ["h1", ".price"])

# Ask the LLM a question about the page
answer = client.question("https://example.com", question: "What is the main product?")

# Extract structured fields with the LLM
data = client.fields(
  "https://example.com",
  fields: {
    title: "Main product title",
    price: "Current product price",
    description: "Full product description"
  }
)

# Google search results (SERP) for a query
results = client.serp(q: "coffee machines", gl: "us", hl: "en", page: 1)
results["organic_results"].first["link"]

# Two pages of results searched from Austin, as one de-duplicated list
results = client.serp(q: "coffee machines", location: "Austin, TX", pages: 2)

# Location names for serp's location: (free)
client.serp_locations("austin").first["canonical_name"] # => "Austin,Texas,United States"

# Structured data for any public web page
page = client.data("https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html")
page["data"]["title"]

# Check your account quota
info = client.account
# => { "remaining_api_calls" => 200_000, "resets_at" => 1_617_073_667, "remaining_concurrency" => 100 }

Configuration

Configure globally once, then create clients without arguments:

WebScrapingAI.configure do |config|
  config.api_key      = ENV.fetch("WEBSCRAPING_AI_API_KEY")
  config.timeout      = 60   # seconds, total request timeout
  config.open_timeout = 10   # seconds, connection timeout
end

client = WebScrapingAI::Client.new

The gem also reads WEBSCRAPING_AI_API_KEY from the environment automatically.

Per-instance overrides:

client = WebScrapingAI::Client.new(
  api_key: "...",
  timeout: 90,
  base_url: "https://api.webscraping.ai"
)

Endpoints and options

All page-fetching endpoints accept these common options (passed as keyword arguments):

Option	Type	Default	Description
headers	Hash	—	HTTP headers to send to the target page (e.g. { "Cookie" => "session=..." })
timeout	Integer	10000	Page retrieval timeout in ms (1–30000)
js	Boolean	true	Execute on-page JavaScript via headless Chromium
js_timeout	Integer	2000	JS rendering timeout in ms (1–20000)
wait_for	String	—	CSS selector to wait for before returning (overrides js_timeout)
proxy	String	"datacenter"	One of datacenter, residential, stealth
country	String	"us"	Proxy country: us, gb, de, it, fr, ca, es, ru, jp, kr, in, hk, tr
custom_proxy	String	—	Your own proxy in http://user:pass@host:port form
device	String	"desktop"	One of desktop, mobile, tablet
error_on_404	Boolean	false	Raise an error if the target page returns 404
error_on_redirect	Boolean	false	Raise an error if the target page redirects

Endpoint-specific options:

• #html — return_script_result (Boolean), format ("json"/"text")
• #text — text_format ("plain"/"xml"/"json"), return_links (Boolean, only with text_format: "json")
• #selected — selector (String), format ("json"/"text")
• #selected_multiple — selectors (Array<String> or single String)
• #question — question (String, required), format ("json"/"text")
• #fields — fields (Hash<String, String>, required) — keys are field names, values are descriptions

Returns: String for HTML/text responses, Hash/Array for JSON responses.

SERP (#serp)

#serp is query-shaped rather than URL-shaped, so none of the page-fetch options above apply. It returns the parsed search results as a Hash. Billed per results page returned (see pricing); failed searches are not charged.

Every search runs from one fixed location: location, lat + lon, or uule, otherwise the capital of the gl country (for the few countries outside the location list, Google places the search by the exit IP), so repeats and later pages are consistent. gl defaults to the location's country. pages (1–5) fetches consecutive pages, starting at page, as one de-duplicated list, billed per page returned.

This is the beginning of the README. Read the rest at the original. Read the full README ↗