@webpeel/sdk

Official TypeScript SDK for the WebPeel API — the fast, AI-ready web fetcher with stealth mode, JS rendering, screenshot capture, structured extraction, and more.

Installation

npm install @webpeel/sdk

Node.js 18+ is required (uses native fetch). Zero runtime dependencies.

Quick Start

import WebPeel from '@webpeel/sdk';

const client = new WebPeel({ apiKey: process.env.WEBPEEL_API_KEY! });
const result = await client.fetch('https://example.com');
console.log(result.content); // Clean Markdown

Authentication

Create an API key at webpeel.dev/dashboard. Keys start with wp_.

const client = new WebPeel({ apiKey: 'wp_your_key_here' });

We recommend storing your key in an environment variable:

export WEBPEEL_API_KEY=wp_your_key_here

Methods

client.fetch(url, options?)

Fetch a URL and return clean, structured content.

// Simple fetch → Markdown
const result = await client.fetch('https://example.com');
console.log(result.content);        // Markdown content
console.log(result.metadata.title); // Page title
console.log(result.metadata.wordCount);

// With JavaScript rendering (for SPAs, lazy-loaded content)
const result = await client.fetch('https://app.example.com', { render: true });

// With stealth mode (bypass bot detection)
const result = await client.fetch('https://protected.example.com', { stealth: true });

// Ask a question about the page
const result = await client.fetch('https://example.com', {
  question: 'What is this company's main product?',
});
console.log(result.answer); // Direct answer

// Token budget (stop early to save credits)
const result = await client.fetch('https://example.com', { budget: 4000 });

// Different output formats
const result = await client.fetch('https://example.com', { format: 'text' }); // plain text
const result2 = await client.fetch('https://example.com', { format: 'html' }); // raw HTML

Options:

| Option | Type | Default | Description | |--------|------|---------|-------------| | render | boolean | false | Use headless browser for JS-heavy pages | | stealth | boolean | false | Enable stealth mode to bypass bot detection | | question | string | — | Ask a question; get an answer in the result | | budget | number | — | Token budget limit | | format | 'markdown' \| 'text' \| 'html' \| 'json' | 'markdown' | Output format | | waitFor | string | — | CSS selector to wait for (requires render: true) | | waitMs | number | — | Extra wait time in ms (requires render: true) | | timeout | number | client default | Per-request timeout (ms) | | signal | AbortSignal | — | Cancellation signal |

Result:

{
  url: string;           // Final URL (after redirects)
  content: string;       // Page content in requested format
  metadata: {
    title?: string;
    description?: string;
    author?: string;
    publishedAt?: string;
    wordCount?: number;
    language?: string;
    siteName?: string;
    favicon?: string;
    ogImage?: string;
    canonical?: string;
  };
  answer?: string;       // Answer to your question (if asked)
  statusCode?: number;   // HTTP status
  contentType?: string;  // Content-Type header
  requestId?: string;    // For debugging
}

client.search(query, options?)

Search the web and return structured results.

const results = await client.search('best web scrapers 2026');
for (const r of results) {
  console.log(r.rank, r.title, r.url);
  console.log(r.description);
}

// With options
const results = await client.search('web scraping tools', {
  limit: 20,
  country: 'US',
  language: 'en',
  includeContent: true, // Fetch page content for each result
});

Options:

| Option | Type | Default | Description | |--------|------|---------|-------------| | limit | number | 10 | Max results to return | | country | string | — | Country code ('US', 'GB', ...) | | language | string | — | Language code ('en', 'fr', ...) | | includeContent | boolean | false | Include page content for each result |

client.screenshot(url, options?)

Take a screenshot of a URL.

const shot = await client.screenshot('https://example.com');
// shot.imageData is base64-encoded image
const buf = Buffer.from(shot.imageData, 'base64');
await fs.writeFile('screenshot.png', buf);

// Full-page screenshot in JPEG
const shot = await client.screenshot('https://example.com', {
  format: 'jpeg',
  quality: 90,
  fullPage: true,
  width: 1440,
});

Options:

| Option | Type | Default | Description | |--------|------|---------|-------------| | format | 'png' \| 'jpeg' \| 'webp' | 'png' | Image format | | quality | number | 80 | Quality 0–100 (JPEG/WebP only) | | fullPage | boolean | false | Capture full scrollable page | | width | number | 1280 | Viewport width | | height | number | 720 | Viewport height | | selector | string | — | CSS selector to screenshot | | waitForNetworkIdle | boolean | false | Wait for network idle |

client.crawl(url, options?)

Crawl a website, following internal links.

const result = await client.crawl('https://example.com', {
  depth: 2,
  limit: 100,
  onPage: (page) => {
    console.log(`[${page.depth}] ${page.url} — ${page.metadata.title}`);
  },
});

console.log(`Crawled ${result.totalPages} pages (${result.failedPages} failed)`);
for (const page of result.pages) {
  console.log(page.url, page.content.slice(0, 200));
}

Options:

| Option | Type | Default | Description | |--------|------|---------|-------------| | depth | number | 1 | Max crawl depth from start URL | | limit | number | 100 | Max pages to crawl | | include | string[] | — | URL patterns to include (glob/regex) | | exclude | string[] | — | URL patterns to exclude | | render | boolean | false | Use JS rendering for all pages | | onPage | (page) => void | — | Progress callback per page |

client.batch(urls, options?)

Fetch multiple URLs concurrently with controlled parallelism.

const result = await client.batch([
  'https://example.com',
  'https://example.org',
  'https://example.net',
], {
  concurrency: 5,
  onResult: (item) => {
    if (item.success) {
      console.log(`✓ ${item.url}`);
    } else {
      console.log(`✗ ${item.url}: ${item.error}`);
    }
  },
  fetchOptions: {
    render: true,
    format: 'markdown',
  },
});

console.log(`${result.succeeded} succeeded, ${result.failed} failed`);

Options:

| Option | Type | Default | Description | |--------|------|---------|-------------| | concurrency | number | 5 | Max concurrent requests | | onResult | (item) => void | — | Callback per completed URL | | fetchOptions | FetchOptions | — | Options applied to every fetch |

Error Handling

All errors extend WebPeelError. Use instanceof to handle specific cases:

import WebPeel, {
  AuthenticationError,
  RateLimitError,
  TimeoutError,
  BlockedError,
  ValidationError,
  NetworkError,
} from '@webpeel/sdk';

try {
  const result = await client.fetch('https://example.com');
} catch (err) {
  if (err instanceof AuthenticationError) {
    // Invalid/missing API key
    console.error('Check your API key:', err.message);
  } else if (err instanceof RateLimitError) {
    // Rate limit exceeded
    const waitSec = err.retryAfter ?? 60;
    console.error(`Rate limited. Retry after ${waitSec}s`);
  } else if (err instanceof TimeoutError) {
    // Request timed out
    console.error('Request timed out:', err.message);
  } else if (err instanceof BlockedError) {
    // Target site blocked the request
    console.error('Site blocked the request. Try { stealth: true }');
  } else if (err instanceof ValidationError) {
    // Bad request parameters
    console.error('Invalid parameters:', err.message);
  } else if (err instanceof NetworkError) {
    // No response received (network failure)
    console.error('Network error:', err.message);
  } else {
    throw err; // Re-throw unknown errors
  }
}

All errors have:

• err.message — Human-readable description
• err.type — Machine-readable type string
• err.status — HTTP status code (0 for network errors)
• err.hint — Optional fix suggestion
• err.requestId — Request ID for support (include this when filing issues)

Configuration

const client = new WebPeel({
  apiKey: 'wp_...',           // Required
  baseUrl: 'https://...',     // Override API base URL (default: https://api.webpeel.dev)
  timeout: 60_000,            // Default timeout in ms (default: 30000)
  maxRetries: 3,              // Max retries on 429/5xx (default: 2)
});

The SDK automatically retries requests on rate limit (429) and server errors (5xx) with exponential backoff, respecting the Retry-After header.

TypeScript

The SDK ships with full TypeScript support. All options and return types are exported:

import type {
  FetchOptions,
  FetchResult,
  SearchOptions,
  SearchResult,
  ScreenshotOptions,
  ScreenshotResult,
  CrawlOptions,
  CrawlResult,
  CrawledPage,
  BatchOptions,
  BatchResult,
  BatchItemResult,
  PageMetadata,
  WebPeelOptions,
} from '@webpeel/sdk';

License

MIT © WebPeel