The official Node.js/TypeScript client library for the ScrapeBadger API — Twitter, Google, Vinted, and general web scraping.

Features

• Full TypeScript Support - Complete type definitions for all API endpoints
• Modern ESM & CommonJS - Works with both module systems
• Async Iterators - Automatic pagination with for await...of syntax
• Smart Rate Limiting - Reads API headers and throttles pagination automatically
• Resilient Retries - Exponential backoff with colored console warnings
• Typed Exceptions - Distinct error classes for every failure scenario
• 37+ Twitter endpoints - Tweets, users, lists, communities, trends, geo, real-time streams
• 20 Google product APIs - Search (with optional deferred AI Overview follow-up), Maps, News, Hotels, Trends (incl. topic autocomplete), Jobs, Shopping (+ merchant URL enrichment, barcode offers), Patents, Scholar (search + profiles + author + author citation + cite formats), Images, Videos, Finance, AI Mode, Lens, Local Pack, Shorts, Flights, Products, Ads Transparency Center
• Vinted scraping - Search items, item details, user profiles, brands, colors, markets
• Web scraping - Anti-bot bypass, JS rendering, screenshots, and selector/AI data extraction

Installation

npm install scrapebadger

yarn add scrapebadger

pnpm add scrapebadger

Quick Start

import { ScrapeBadger } from "scrapebadger";

const client = new ScrapeBadger({ apiKey: "your-api-key" });

// Get a tweet
const tweet = await client.twitter.tweets.getById("1234567890");
console.log(`@${tweet.username}: ${tweet.text}`);

// Scrape a website
const result = await client.web.scrape("https://scrapebadger.com", { format: "markdown" });
console.log(result.content);

// Get a user profile
const user = await client.twitter.users.getByUsername("elonmusk");
console.log(`${user.name} has ${user.followers_count.toLocaleString()} followers`);

Authentication

// Pass API key directly
const client = new ScrapeBadger({ apiKey: "sb_live_xxxxxxxxxxxxx" });

// Or use environment variable SCRAPEBADGER_API_KEY
const client = new ScrapeBadger();

Available APIs

| API | Description | Documentation | |-----|-------------|---------------| | Web Scraping | Scrape any website with JS rendering, anti-bot bypass, screenshots, and selector/AI extraction | Web Scraping Guide | | Twitter | 37+ endpoints for tweets, users, lists, communities, trends, and real-time streams | Twitter Guide | | Google | 20 products — Search, Maps, News, Hotels, Trends, Jobs, Shopping, Patents, Scholar, Images, Videos, Finance, AI Mode, Lens, Autocomplete, Local, Shorts, Flights, Products, Ads Transparency | Google Guide | | Vinted | Search items, get details, user profiles, and reference data across all Vinted markets | Vinted Guide | | Reddit | Search posts/subreddits/users, subreddit details, post comments, user profiles, wiki pages, trophies | Reddit Guide | | Instagram | User profile/about/related/posts/videos/reels/tagged/pinned/followers/following/stories/highlights, media detail/comments/replies/likers/oEmbed, search (users/hashtags/places/top/reels/music/autocomplete), hashtag/location/audio feeds | Instagram Guide | | Amazon | 14 endpoints — search, autocomplete, product detail, offers, reviews, bestsellers, new releases, deals, category browse, seller profile/products/feedback, markets, categories | Amazon Guide | | Shopee | 6 endpoints across 11 markets (sg, my, ph, id, vn, th, tw, br, co, cl, mx) — product search, category items, product detail, reviews, category tree, markets | Shopee Guide | | TikTok | 26 endpoints — user profile/videos/followers/following/liked/reposts, video detail/comments/replies/related/transcript, oEmbed, hashtag detail/videos, music detail/videos, search (general/videos/users/hashtags), trending (videos/hashtags/songs), ad library, regions | TikTok Guide | | eBay | 12 endpoints across 18 markets — search, search by image, completed/sold listings, item detail, item reviews, seller profile/items/feedback, category browse, categories, autocomplete, markets | eBay Guide | | Walmart | 11 endpoints (walmart.com, US-only) — search, category browse, deals feed, autocomplete, product detail, reviews, seller profile/products, store detail with nearby stores, markets | | Bing | 6 endpoints — web search (with ads + related searches), image search, video search, news vertical, autocomplete, markets | | DuckDuckGo | 7 endpoints — web search (with instant-answer abstract), image search, news vertical, video search, autocomplete, instant answers, regions | | Yahoo | 6 endpoints across 35 markets — web search (with ads + related searches), image search, video search, news vertical, autocomplete, markets | Yahoo Docs | | Yandex | 4 endpoints across 6 markets (tr/com/ru/by/kz/uz) — web search (organic + ads + sitelinks + inline media), image search, reverse-image (CBIR) search, markets | Yandex Docs | | YouTube | 39 endpoints — search, autocomplete, video detail/related/comments/replies/transcript/captions/streams/live chat/batch, channel detail/videos/shorts/streams/playlists/community/about/subscriber count/search/resolve, playlist detail/items, mixes, trending/shorts, hashtags, home, shorts detail/by sound, community posts/comments, music search, oEmbed, categories/languages/regions/markets | YouTube Guide | | Immobiliare | 8 endpoints across 4 markets (it, es, gr, lu) — autocomplete, search, listing detail, agency profile/listings, price stats, markets, reference | Immobiliare Guide | | LoopNet | 5 endpoints across loopnet.com/.ca/.co.uk/.fr/.es — commercial-real-estate search (for-lease/for-sale/auctions), listing detail, broker profile, markets, property types | LoopNet Guide | | Apartments.com | US rental listings with UNIT-LEVEL pricing — search by location with bed/price filters, plus per-unit rent, beds, baths, sqft and availability for every rentable unit | | Baidu | 4 endpoints (baidu.com, China's #1 search engine) — web search with language and date-window filters, news vertical, image search, autocomplete. Results carry the real target URL, not just Baidu's tracking redirect | Baidu Docs | | ChatGPT | Prompt the real chatgpt.com anonymously — structured answer with citations anchored to character offsets, the full retrieved search trail, and AEO/GEO brand-visibility analysis | ChatGPT Guide | | Gemini | Prompt the real gemini.google.com anonymously — structured answer with cited web sources, the full retrieved search trail, and AEO/GEO brand-visibility analysis | Gemini Docs |

Error Handling

import {
  ScrapeBadger,
  AuthenticationError,
  RateLimitError,
  NotFoundError,
  InsufficientCreditsError,
} from "scrapebadger";

const client = new ScrapeBadger({ apiKey: "your-api-key" });

try {
  const tweet = await client.twitter.tweets.getById("1234567890");
} catch (error) {
  if (error instanceof AuthenticationError) {
    console.error("Invalid API key");
  } else if (error instanceof RateLimitError) {
    console.error(`Rate limited. Retry after: ${error.retryAfter}`);
  } else if (error instanceof NotFoundError) {
    console.error("Tweet not found");
  } else if (error instanceof InsufficientCreditsError) {
    console.error("Out of credits");
  } else {
    throw error;
  }
}

Configuration

const client = new ScrapeBadger({
  // Required: Your API key (or use SCRAPEBADGER_API_KEY env var)
  apiKey: "your-api-key",

  // Optional: Custom base URL (default: https://scrapebadger.com)
  baseUrl: "https://scrapebadger.com",

  // Optional: Request timeout in milliseconds (default: 30000)
  timeout: 30000,

  // Optional: Maximum retry attempts (default: 10)
  maxRetries: 10,

  // Optional: Initial retry delay in milliseconds (default: 1000)
  retryDelay: 1000,
});

Retry Behavior

The SDK automatically retries requests that fail with server errors (5xx) or rate limits (429) using exponential backoff (1s, 2s, 4s, 8s, ...). Each retry prints a colored warning:

⚠ ScrapeBadger: 503 Service Unavailable — retrying in 4s (attempt 3/10)

Rate Limit Aware Pagination

When using *All pagination methods (e.g. searchAll, getFollowersAll), the SDK reads X-RateLimit-Remaining and X-RateLimit-Reset headers from each response. When remaining requests drop below 20% of your tier's limit, pagination automatically slows down to spread requests across the remaining window — preventing 429 errors:

⚠ ScrapeBadger: Rate limit: 25/300 remaining (resets in 42s), throttling pagination

This works transparently with all tier levels (Free: 60/min, Basic: 300/min, Pro: 1000/min, Enterprise: 5000/min).

Exceptions

• ScrapeBadgerError - Base exception class
• AuthenticationError - Invalid or missing API key
• RateLimitError - Rate limit exceeded
• NotFoundError - Resource not found
• ValidationError - Invalid request
• ServerError - Server error
• TimeoutError - Request timeout
• InsufficientCreditsError - Out of credits
• AccountRestrictedError - Account restricted
• WebSocketStreamError - WebSocket stream failure (auth, limit, or network)

Requirements

• Node.js 18+ (for native fetch support)
• TypeScript 5.0+ (for best type inference)

License

MIT License - see LICENSE for details.

Links

• Documentation
• Discord
• GitHub
• npm