ScrapeBadger

The official Node.js/TypeScript client library for the ScrapeBadger API — Twitter, Google, Vinted, and general web scraping.

Features

• Full TypeScript Support - Complete type definitions for all API endpoints
• Modern ESM & CommonJS - Works with both module systems
• Async Iterators - Automatic pagination with for await...of syntax
• Smart Rate Limiting - Reads API headers and throttles pagination automatically
• Resilient Retries - Exponential backoff with colored console warnings
• Typed Exceptions - Distinct error classes for every failure scenario
• 37+ Twitter endpoints - Tweets, users, lists, communities, trends, geo, real-time streams
• 20 Google product APIs - Search (with optional deferred AI Overview follow-up), Maps, News, Hotels, Trends (incl. topic autocomplete), Jobs, Shopping (+ merchant URL enrichment, barcode offers), Patents, Scholar (search + profiles + author + author citation + cite formats), Images, Videos, Finance, AI Mode, Lens, Local Pack, Shorts, Flights, Products, Ads Transparency Center
• Vinted scraping - Search items, item details, user profiles, brands, colors, markets
• Web scraping - Anti-bot bypass, JS rendering, screenshots, and selector/AI data extraction

Installation

npm install scrapebadger

yarn add scrapebadger

pnpm add scrapebadger

Quick Start

import { ScrapeBadger } from "scrapebadger";

const client = new ScrapeBadger({ apiKey: "your-api-key" });

// Get a tweet
const tweet = await client.twitter.tweets.getById("1234567890");
console.log(`@${tweet.username}: ${tweet.text}`);

// Scrape a website
const result = await client.web.scrape("https://scrapebadger.com", { format: "markdown" });
console.log(result.content);

// Get a user profile
const user = await client.twitter.users.getByUsername("elonmusk");
console.log(`${user.name} has ${user.followers_count.toLocaleString()} followers`);

Authentication

// Pass API key directly
const client = new ScrapeBadger({ apiKey: "sb_live_xxxxxxxxxxxxx" });

// Or use environment variable SCRAPEBADGER_API_KEY
const client = new ScrapeBadger();

Dedicated ecommerce APIs

The etsy, aliexpress, target, asos, and bestbuy clients expose 44 dedicated ecommerce endpoints. These generated methods return response objects; consult the endpoint documentation for response fields, supported filters, pagination and current pricing.

const client = new ScrapeBadger({ apiKey: "your-api-key" });
const etsy = await client.etsy.searchListings({ query: "ceramic mug" });
const aliexpress = await client.aliexpress.searchProducts({ query: "wireless mouse" });
const target = await client.target.searchProducts({ query: "coffee" });
const asos = await client.asos.searchProducts({ query: "shoes", market: "uk" });
const bestbuy = await client.bestbuy.searchProducts({ query: "laptop" });

Client	Endpoints	Documentation
client.etsy	8 — listing search/detail/reviews, shop profile/listings/reviews, categories, autocomplete	Etsy
client.aliexpress	8 — product search/detail/reviews, store profile/products/categories, category navigation, autocomplete	AliExpress
client.target	11 — search, category browse/navigation, autocomplete, product detail/reviews/questions/availability, nearby stores/store detail, markets	Target
client.asos	7 — product search/detail, stock/prices, category navigation, search filters, autocomplete, markets	ASOS
client.bestbuy	10 — search, category browse/navigation, deals, autocomplete, product detail/reviews/questions/availability, markets	Best Buy

Etsy and AliExpress category navigation is separate from search: their search methods do not offer a category filter. ASOS search follows its upstream semantic matching. Target and Best Buy serve the US market. The linked endpoint references describe further measured limits.

Available APIs

API	Description	Documentation
Web Scraping	Scrape any website with JS rendering, anti-bot bypass, screenshots, and selector/AI extraction	Web Scraping Guide
Twitter	37+ endpoints for tweets, users, lists, communities, trends, and real-time streams	Twitter Guide
Google	20 products — Search, Maps, News, Hotels, Trends, Jobs, Shopping, Patents, Scholar, Images, Videos, Finance, AI Mode, Lens, Autocomplete, Local, Shorts, Flights, Products, Ads Transparency	Google Guide
Vinted	Search items, get details, user profiles, and reference data across all Vinted markets	Vinted Guide
Wallapop	10 endpoints across 3 markets (es, it, pt) — listing search with Wallapop's own filters (incl. cars: year, km, gearbox, horsepower), listing detail, seller profile/listings/reviews, autocomplete, categories, search filters and filter options, markets	Wallapop Docs
Reddit	Search posts/subreddits/users, subreddit details, post comments, user profiles, wiki pages, trophies	Reddit Guide
Instagram	User profile/about/related/posts/videos/reels/tagged/pinned/followers/following/stories/highlights, media detail/comments/replies/likers/oEmbed, search (users/hashtags/places/top/reels/music/autocomplete), hashtag/location/audio feeds	Instagram Guide

Quotes, feedback and free re-reads

These calls are free and never billed.

import { ScrapeBadger, type AgentTrial } from "scrapebadger";

const client = new ScrapeBadger({ apiKey: "your-api-key" });

// What would a call cost, are its params valid, how long does it take?
const quote = await client.agent.quote("GET", "/v1/google/search", { params: { q: "coffee" } });
console.log(quote.valid, quote.errors, quote.credits.max, quote.usd, quote.latency);

// Re-read a billed result for ~15 minutes, reshaped. The ID is the call's
// X-Result-ID header (raw HTTP, the CLI) or AsyncRequest.resultId.
const titles = await client.results.get<string[]>(resultId, { jq: "[.organic_results[].title]" });
const outline = await client.results.get(resultId, { summary: true });
const first5 = await client.results.get(resultId, { fields: ["title", "link"], maxItems: 5 });

// Poll a call made with `Prefer: respond-async` (202 + request_id).
const state = await client.requests.get(requestId);
if (!state.running) console.log(state.status, state.creditsUsed, state.data);

// Report a bug or send feedback (pass the failing call's X-Request-ID).
await client.agent.reportBug("Empty results for a valid query", { requestId });
await client.agent.feedback("Need a Google Maps photos endpoint");

// Agent trial keys: account info carries an `agent_trial` block.
const info = await client.account.getAccountInfo();
const trial = info.agent_trial as AgentTrial | undefined;

results.get returns the stored body as-is (any JSON value) and throws NotFoundError once it has expired. requests.get returns running: true while the call is in progress — wait retryAfterSeconds and poll again; never re-submit the call. Once finished it returns the original body, or throws the original call's error. The SDK's own methods never send Prefer: respond-async; requests.get polls a request ID obtained elsewhere (raw HTTP, the CLI or MCP server).

Error Handling

import {
  ScrapeBadger,
  AuthenticationError,
  RateLimitError,
  NotFoundError,
  InsufficientCreditsError,
} from "scrapebadger";

const client = new ScrapeBadger({ apiKey: "your-api-key" });

try {
  const tweet = await client.twitter.tweets.getById("1234567890");
} catch (error) {
  if (error instanceof AuthenticationError) {
    console.error("Invalid API key");
  } else if (error instanceof RateLimitError) {
    console.error(`Rate limited. Retry after: ${error.retryAfter}`);
  } else if (error instanceof NotFoundError) {
    console.error("Tweet not found");
  } else if (error instanceof InsufficientCreditsError) {
    console.error("Out of credits");
  } else {
    throw error;
  }
}

Configuration

const client = new ScrapeBadger({
  // Required: Your API key (or use SCRAPEBADGER_API_KEY env var)
  apiKey: "your-api-key",

  // Optional: Custom base URL (default: https://scrapebadger.com)
  baseUrl: "https://scrapebadger.com",

  // Optional: Request timeout in milliseconds (default: 30000)
  timeout: 30000,

  // Optional: Maximum retry attempts (default: 10)
  maxRetries: 10,

  // Optional: Initial retry delay in milliseconds (default: 1000)
  retryDelay: 1000,
});

Retry Behavior

The SDK automatically retries requests that fail with server errors (5xx) or rate limits (429) using exponential backoff (1s, 2s, 4s, 8s, ...). Each retry prints a colored warning:

⚠ ScrapeBadger: 503 Service Unavailable — retrying in 4s (attempt 3/10)

Rate Limit Aware Pagination

When using *All pagination methods (e.g. searchAll, getFollowersAll), the SDK reads X-RateLimit-Remaining and X-RateLimit-Reset headers from each response. When remaining requests drop below 20% of your tier's limit, pagination automatically slows down to spread requests across the remaining window — preventing 429 errors:

⚠ ScrapeBadger: Rate limit: 25/300 remaining (resets in 42s), throttling pagination

This works transparently with all tier levels (Free: 60/min, Basic: 300/min, Pro: 1000/min, Enterprise: 5000/min).

Exceptions

• ScrapeBadgerError - Base exception class
• AuthenticationError - Invalid or missing API key
• RateLimitError - Rate limit exceeded
• NotFoundError - Resource not found
• ValidationError - Invalid request
• ServerError - Server error
• TimeoutError - Request timeout
• InsufficientCreditsError - Out of credits
• AccountRestrictedError - Account restricted
• WebSocketStreamError - WebSocket stream failure (auth, limit, or network)

Requirements

• Node.js 18+ (for native fetch support)
• TypeScript 5.0+ (for best type inference)

License

MIT License - see LICENSE for details.

Links

• Documentation
• Discord
• GitHub
• npm