Installation

npm install @webclaw/sdk

pnpm add @webclaw/sdk

yarn add @webclaw/sdk

bun add @webclaw/sdk

Quick Start

import { Webclaw } from "@webclaw/sdk";

const client = new Webclaw({ apiKey: "wc-YOUR_API_KEY" });

const result = await client.scrape({ url: "https://example.com", formats: ["markdown"] });
console.log(result.markdown);

Endpoints

Scrape

Extract content from a single URL. Supports multiple output formats, CSS selectors for targeting specific elements, and cache control.

const result = await client.scrape({
  url: "https://example.com",
  formats: ["markdown", "text", "llm", "json"],
  include_selectors: ["article", ".content"],
  exclude_selectors: ["nav", "footer"],
  only_main_content: true,
  no_cache: true,
});

result.url       // string
result.markdown  // string | undefined
result.text      // string | undefined
result.llm       // string | undefined
result.extraction // unknown | undefined (formats: ["json"])
result.metadata  // { title?, description?, language?, ... }
result.cache     // { status: "hit" | "miss" | "bypass" }
result.warning   // string | undefined

Vertical extractors

Site-specific extractors return structured JSON (GitHub, Reddit, Amazon, YouTube, PyPI, HuggingFace, Trustpilot, etc.) instead of generic markdown. See the catalog for the full list.

// Discover available extractors
const catalog = await client.listExtractors();
catalog.extractors.forEach((e) => console.log(e.name, "-", e.label));

// Run a specific extractor
const pr = await client.scrapeVertical(
  "github_pr",
  "https://github.com/rust-lang/rust/pull/123456",
);
console.log(pr.data); // { title, state, author, commits, reviews, ... }

// Amazon product as typed JSON
const product = await client.scrapeVertical(
  "amazon_product",
  "https://www.amazon.com/dp/B0C6KKQ7ND",
);
console.log(product.data.price, product.data.rating);

The data field is extractor-specific; call listExtractors() to discover what each returns.

Search

Web search with strict source filters, provider freshness/date/locale hints, pagination, and optional parallel scraping of each result page.

const result = await client.search({
  query: "website pain points",
  num_results: 10,
  include_domains: ["reddit.com"],
  include_url_prefixes: ["https://www.reddit.com/r/webdesign/comments/"],
  freshness: "month",
  page: 1,
  location: "Austin, Texas, United States",
  autocorrect: false,
  scrape: false,
  formats: ["markdown"],
  country: "us",
  lang: "en",
  no_cache: true,
});

for (const r of result.results) {
  console.log(r.title, r.url, r.snippet);
}

console.log(result.filtered_out_count); // provider hits rejected by strict filters
console.log(result.applied_filters);

Use published_after / published_before (YYYY-MM-DD) instead of freshness when you need explicit provider discovery hints; published_before is exclusive. These hints do not verify a result's actual publication date, so inspect the source timestamp when correctness matters. topic remains accepted for compatibility but is deprecated and ignored by the hosted API.

Map

Discover URLs from a site's sitemap.

const result = await client.map({ url: "https://example.com" });
console.log(`Found ${result.count} URLs`);
result.urls.forEach((url) => console.log(url));

Endpoints

Discover API endpoints embedded in a page's JavaScript — scans inline <script> bodies plus <script src> bundles for request paths, absolute URLs, GraphQL, and WebSocket endpoints. This surfaces the request layer that map (sitemap-based) can't see.

const result = await client.endpoints({
  url: "https://example.com",
  include_third_party: false, // default; set true to include other hosts
  max_bundles: 20,            // default & max; bundles fetched on top of inline JS
});

console.log(`${result.endpoint_count} endpoints across ${result.bundles_scanned} bundles`);
for (const ep of result.endpoints) {
  console.log(ep.kind, ep.value, ep.first_party ? "(1st-party)" : "(3rd-party)");
}
result.hosts      // distinct hosts seen, e.g. ["api.example.com"]
result.truncated  // true if results were capped by max_bundles

Security: endpoints, hosts, and their fields are extracted from page content (inline scripts and fetched bundles), which is attacker-influenced. The SDK does not sanitize them. Never feed a returned value or source into another request, shell command, eval, or SQL query without your own validation.

Batch

Scrape multiple URLs in parallel with configurable concurrency.

const result = await client.batch({
  urls: ["https://a.com", "https://b.com", "https://c.com"],
  formats: ["markdown"],
  concurrency: 5,
});

for (const item of result.results) {
  if ("error" in item) console.error(item.url, item.error);
  else console.log(item.url, item.markdown?.length);
}

Extract

LLM-powered structured data extraction. Provide a JSON schema for typed output, or a natural-language prompt for flexible extraction.

// Schema-based extraction
const result = await client.extract({
  url: "https://example.com/pricing",
  schema: {
    type: "object",
    properties: {
      plans: { type: "array", items: { type: "object" } },
    },
  },
});
console.log(result.data);

// Prompt-based extraction
const result2 = await client.extract({
  url: "https://example.com",
  prompt: "Extract all pricing tiers with names and prices",
});
console.log(result2.data);

Summarize

Generate a concise summary of a page's content.

const result = await client.summarize({
  url: "https://example.com/blog/long-article",
  max_sentences: 3,
});
console.log(result.summary);

Diff

Compare a page with your most recent cached extraction. Use the same API account for both calls; a missing or expired baseline returns an error.

// Establish the cached baseline once, then check for changes later.
await client.scrape({ url: "https://example.com", formats: ["json"] });
const result = await client.diff({ url: "https://example.com" });
console.log(result.status);    // "Same", "Changed", or "New"
console.log(result.text_diff); // unified diff, or null
console.log(result.metadata_changes);

To compare against a saved baseline instead, pass its complete extraction as previous, including metadata and content. An arbitrary title/body object is not accepted.

Brand

Extract brand identity information (name, colors, fonts, logos) from a URL.

const result = await client.brand({ url: "https://example.com" });
console.log(result); // { name, colors, fonts, logos, ... }

Research

Start an async deep research job. The SDK automatically polls until the job completes (up to 20 minutes by default; override with maxWait).

Note: every research job now runs in deep mode. The deep request flag is deprecated and ignored by the API — don't pass it.

const result = await client.research(
  {
    query: "How do modern web crawlers handle JavaScript rendering?",
    max_sources: 15,
  },
  { interval: 3_000 },
);

console.log(result.report);
console.log("Sources:", result.sources?.length);
console.log("Findings:", result.findings?.length);

To inspect an existing job independently, use its saved ID:

const status = await client.getResearchStatus("your-existing-job-id");
console.log(status.status);

Crawl

Start an async crawl job that discovers and scrapes pages from a root URL.

const job = await client.crawl({
  url: "https://example.com",
  max_depth: 3,
  max_pages: 100,
  use_sitemap: true,
});

console.log("Job ID:", job.id);

Poll with waitForCompletion, which resolves when the crawl finishes or fails:

const result = await job.waitForCompletion({
  interval: 2_000,   // polling interval in ms
  maxWait: 300_000,  // max wait time in ms (5 min)
});

console.log(`Status: ${result.status}`);
console.log(`${result.completed}/${result.total} pages`);
for (const page of result.pages) {
  console.log(page.url, page.markdown?.length);
}

Or check status manually at any time:

const status = await job.getStatus();
// or: const status = await client.getCrawlStatus(job.id);

Watch

Monitor URLs for content changes. Create watchers, check them on demand, and receive webhook notifications when content changes.

Create a watch

const watch = await client.watchCreate({
  url: "https://example.com/pricing",
  name: "Pricing page",
  interval_minutes: 60,
  webhook_url: "https://your-server.com/webhooks/webclaw",
});
console.log("Watch ID:", watch.id);

List all watches

const watches = await client.watchList(10, 0); // limit, offset
for (const w of watches.watches) {
  console.log(w.id, w.url, w.active);
}

Get a single watch

const watch = await client.watchGet("watch_abc123");
console.log(watch.last_checked_at, watch.last_changed_at);

Trigger an immediate check

const check = await client.watchCheck("watch_abc123");
console.log(check.status); // "checking"; the snapshot is produced asynchronously.
// Fetch watchGet later to inspect its snapshots.

Delete a watch

await client.watchDelete("watch_abc123");

X (Twitter) monitoring

Monitor X for new tweets — profiles, search queries, lists, or replies to a tweet — and receive webhook notifications when new matches appear. The X analog of Watch. You can also export an account's followers or following.

Paid feature. These endpoints return 403 (ScopeError) on free or lapsed accounts. Monitors and audience export are billed per X request at your plan rate (Starter 5, Growth 3, Pro 2, Scale 1 credits). Max 50 monitors per account.

Create a monitor

const monitor = await client.createXMonitor({
  kind: "profile",             // "profile" | "search" | "list" | "replies"
  target: "@webclaw",          // handle | search query | list id | tweet id (per kind)
  name: "Webclaw mentions",
  interval_minutes: 15,        // default 15, clamped 2..10080
  webhook_url: "https://discord.com/api/webhooks/...",
  include_retweets: true,      // default true
  include_replies: true,       // default true
  include_quotes: true,        // default true
  min_faves: 0,                // minimum likes to match
  keyword: "scraping",         // only match tweets containing this
  lang: "en",                  // only match this language
});
console.log("Monitor ID:", monitor.id);

List monitors

const { monitors } = await client.listXMonitors(10, 0); // limit, offset
for (const m of monitors) {
  console.log(m.id, m.kind, m.target, m.active);
}

Get a single monitor

const monitor = await client.getXMonitor("xmon_abc123");
console.log(monitor.last_checked_at, monitor.last_matched_at);

Update a monitor

const res = await client.updateXMonitor("xmon_abc123", {
  name: "Renamed",
  interval_minutes: 30,
  webhook_url: "https://hooks.slack.com/services/...",
  active: false,       // pause it
});
console.log(res.success); // true

Trigger an immediate check

const res = await client.checkXMonitor("xmon_abc123");
console.log(res.status); // "checking" — runs in the background

Delete a monitor

const res = await client.deleteXMonitor("xmon_abc123");
console.log(res.success); // true

Webhook payload

When a monitor matches new tweets, webhook_url receives:

{
  "event": "x.monitor.matched",
  "monitor_id": "xmon_abc123",
  "kind": "profile",
  "target": "webclaw",
  "new_count": 2,
  "tweets": [
    {
      "id": "1790000000000000000",
      "screen_name": "webclaw",
      "text": "…",
      "url": "https://x.com/webclaw/status/1790000000000000000",
      "created_at": "2026-06-12T10:00:00Z",
      "favorite_count": 42,
      "retweet_count": 7,
      "reply_count": 3,
      "lang": "en",
      "is_retweet": false,
      "is_reply": false,
      "is_quote": false
    }
  ],
  "checked_at": "2026-06-12T10:00:05Z"
}

Discord and Slack webhook URLs receive native embed/text formatting instead of this generic JSON.

Export an audience

Export an account's followers or following, cursor-paginated and metered per page at your plan rate (Starter 5, Growth 3, Pro 2, Scale 1 credits). To walk a full audience, call repeatedly — pass the returned user_id and next_cursor back in until next_cursor is null.

let cursor: string | null | undefined;
let userId: string | undefined;

do {
  const page = await client.exportXAudience({
    handle: userId ? undefined : "@webclaw", // resolved once (unbilled)
    user_id: userId,                          // reuse to skip re-resolving
    direction: "followers",                   // "followers" (default) | "following"
    cursor: cursor ?? undefined,
    max_pages: 2,                             // default 2, clamped 1..10
  });

  for (const u of page.users) {
    console.log(u.screen_name, u.followers, u.description);
  }

  userId = page.user_id;
  cursor = page.next_cursor;
} while (cursor !== null);

Firecrawl v2 compatibility

The API also exposes a Firecrawl-compatible surface at /v2/scrape, /v2/crawl, and /v2/search. These endpoints are not yet wrapped by this SDK (future work) — call them directly if you need Firecrawl drop-in compatibility today.

Error Handling

All errors extend WebclawError, so you can catch broadly or handle specific cases.

import {
  WebclawError,
  AuthenticationError,
  NotFoundError,
  RateLimitError,
  TimeoutError,
} from "@webclaw/sdk";

try {
  await client.scrape({ url: "https://example.com" });
} catch (err) {
  if (err instanceof RateLimitError) {
    console.error("Rate limited, retry after:", err.retryAfter, "s");
  } else if (err instanceof AuthenticationError) {
    console.error("Bad API key");
  } else if (err instanceof NotFoundError) {
    console.error("Resource not found");
  } else if (err instanceof TimeoutError) {
    console.error("Request timed out");
  } else if (err instanceof WebclawError) {
    console.error("API error:", err.message, err.status, err.body);
  }
}

Configuration

const client = new Webclaw({
  apiKey: process.env.WEBCLAW_API_KEY!,
  baseUrl: "https://api.webclaw.io", // default
  timeout: 60_000,                    // ms, default 30_000
});

| Option | Type | Default | Description | |--------|------|---------|-------------| | apiKey | string | required | Your Webclaw API key | | baseUrl | string | https://api.webclaw.io | API base URL | | timeout | number | 30000 | Request timeout in milliseconds |

TypeScript

Full type definitions are included for every request and response. All types are exported from the package root:

import type {
  ScrapeRequest,
  ScrapeResponse,
  CrawlRequest,
  CrawlStatusResponse,
  EndpointsRequest,
  EndpointsResponse,
  SearchRequest,
  SearchResponse,
  ExtractRequest,
  ExtractResponse,
  ResearchRequest,
  ResearchResponse,
  WatchCreateRequest,
  WatchResponse,
  CreateXMonitorRequest,
  UpdateXMonitorRequest,
  XMonitor,
  ListXMonitorsResponse,
  ExportXAudienceRequest,
  ExportXAudienceResponse,
  // ... and more
} from "@webclaw/sdk";

Highlights

• Zero runtime dependencies. Uses native fetch.
• ESM + CJS dual output via tsup.
• Full TypeScript types for every request and response.
• Automatic polling for async jobs (crawl, research).
• Node.js 18+.

Development

Use Node.js 20.19+ for the build and test tools. The SDK itself supports Node.js 18+; CI checks both ESM and CommonJS builds with real local HTTP requests on Node.js 18 and 20.

npm ci
npm run typecheck
npm test
npm run build
npm run check:package

License

MIT