Geonode Scraper SDK for TypeScript

TypeScript SDK for the Geonode Scraper API. Supports single-URL extraction, batch extraction, site crawling, URL mapping, job polling, and usage statistics.

Requirements

• A modern browser with fetch, or Node.js 18+

Installation

npm install @geonode_scraper/scraper-sdk

This is a scoped npm package. Use the full package name, including the @geonode_scraper/ prefix, in both npm install and import statements.

Configuration And Authentication

Create a client configuration with your API base URL and API key.

import { Configuration } from "@geonode_scraper/scraper-sdk"

const configuration = new Configuration({
  basePath: "https://api.example.com",
  apiKey: "your-api-key"
})

If you do not set basePath, the generated client defaults to http://localhost.

Quick Start

Synchronous extraction — blocks until the result is ready.

import {
  Configuration,
  ExtractionApi,
  OutputFormat,
  ProcessingMode
} from "@geonode_scraper/scraper-sdk"

const api = new ExtractionApi(
  new Configuration({
    basePath: "https://api.example.com",
    apiKey: "your-api-key"
  })
)

const response = await api.extractV1ExtractPost({
  extractRequest: {
    url: "https://example.com",
    formats: [OutputFormat.Markdown],
    processingMode: ProcessingMode.Sync
  }
})

if ("data" in response) {
  console.log(response.data?.markdown)
  console.log(response.tokensCharged)
}

Async Extraction Workflow

When processingMode is ProcessingMode.Async, the extract call returns an async job response with a job ID and status URL.

import { Configuration, ExtractionApi, ProcessingMode } from "@geonode_scraper/scraper-sdk"

const api = new ExtractionApi(
  new Configuration({
    basePath: "https://api.example.com",
    apiKey: "your-api-key"
  })
)

const submit = await api.extractV1ExtractPost({
  extractRequest: {
    url: "https://example.com",
    processingMode: ProcessingMode.Async
  }
})

if ("jobId" in submit) {
  const job = await api.getJobResultV1ExtractJobIdGet({ jobId: submit.jobId })
  console.log(job.status)
}

Use getJobResultV1ExtractJobIdGet({ jobId }) to poll a single job, or listJobsV1ExtractJobsGet(...) to inspect and filter job history.

Batch Extraction

Submit multiple URLs in one request and poll for results.

import { BatchApi, Configuration, OutputFormat } from "@geonode_scraper/scraper-sdk"

const api = new BatchApi(
  new Configuration({
    basePath: "https://api.example.com",
    apiKey: "your-api-key"
  })
)

const accepted = await api.createBatchV1BatchPost({
  batchRequest: {
    urls: ["https://example.com", "https://example.org"],
    formats: [OutputFormat.Markdown]
  }
})
console.log(accepted.jobId, accepted.acceptedUrls)

const status = await api.getBatchStatusV1BatchJobIdGet({
  jobId: accepted.jobId,
  page: 1,
  pageSize: 10
})
console.log(status.status, status.completedUrls, status.totalUrls)

Site Crawling

Crawl a website from a seed URL up to a configurable depth and page limit.

import { Configuration, CrawlApi, OutputFormat } from "@geonode_scraper/scraper-sdk"

const api = new CrawlApi(
  new Configuration({
    basePath: "https://api.example.com",
    apiKey: "your-api-key"
  })
)

const accepted = await api.createCrawlV1CrawlPost({
  crawlRequest: {
    url: "https://example.com",
    depth: 2,
    limit: 50,
    formats: [OutputFormat.Markdown]
  }
})
console.log(accepted.jobId, accepted.estimatedPages)

const status = await api.getCrawlStatusV1CrawlJobIdGet({
  jobId: accepted.jobId,
  page: 1,
  pageSize: 10
})
console.log(status.status, status.completedPages, status.totalPages)

URL Mapping

Discover all URLs under a base URL by combining sitemap parsing with HTML link extraction. Returns synchronously.

import { Configuration, MapApi } from "@geonode_scraper/scraper-sdk"

const api = new MapApi(
  new Configuration({
    basePath: "https://api.example.com",
    apiKey: "your-api-key"
  })
)

const result = await api.mapUrlsV1MapPost({
  mapRequest: { url: "https://example.com" }
})
for (const link of result.links ?? []) {
  console.log(link.url, link.source)
}

Error Handling

Non-2xx API responses throw ApiResponseError. The error includes:

• status: HTTP status code
• data: parsed error model for declared API responses such as ErrorResponse, ExtractionErrorResponse, HTTPValidationError, or HealthResponse
• body: raw parsed response body
• response: the original Response object

Transport failures still throw FetchError when no HTTP response is available.

import { ApiResponseError } from "@geonode_scraper/scraper-sdk"

try {
  await api.extractV1ExtractPost({
    extractRequest: { url: "https://example.com" }
  })
} catch (error) {
  if (error instanceof ApiResponseError) {
    console.error(error.status, error.data)
  }
}

Request Options

ExtractRequest supports the following fields:

• formats: output formats to return
• renderJs: use a headless browser for JavaScript-rendered pages
• processingMode: ProcessingMode.Sync or ProcessingMode.Async
• proxy: optional proxy settings for country and proxy type selection
• headers: optional request headers dictionary
• waitConfig: optional explicit browser wait policy (waitUntil, waitFor, waitTimeout)

Example with additional options:

import {
  OutputFormat,
  ProcessingMode,
  ProxyType,
  WaitUntil,
  type ExtractRequest
} from "@geonode_scraper/scraper-sdk"

const request: ExtractRequest = {
  url: "https://example.com",
  formats: [OutputFormat.Html, OutputFormat.Markdown],
  renderJs: true,
  processingMode: ProcessingMode.Sync,
  proxy: { country: "US", type: ProxyType.Residential },
  headers: { "User-Agent": "geonode-scraper-sdk-demo" },
  waitConfig: {
    waitUntil: WaitUntil.Networkidle,
    waitFor: "#content",
    waitTimeout: 2000
  }
}

API Reference

ExtractionApi (/v1/extract)

• extractV1ExtractPost({ extractRequest })
• getJobResultV1ExtractJobIdGet({ jobId })
• listJobsV1ExtractJobsGet({ jobId, url, status, output, startDate, endDate, page, pageSize })

BatchApi (/v1/batch)

• createBatchV1BatchPost({ batchRequest })
• getBatchStatusV1BatchJobIdGet({ jobId, page, pageSize })
• cancelBatchV1BatchJobIdDelete({ jobId })
• listBatchJobsV1BatchJobsGet({ status, startDate, endDate, page, pageSize })

CrawlApi (/v1/crawl)

• createCrawlV1CrawlPost({ crawlRequest })
• getCrawlStatusV1CrawlJobIdGet({ jobId, page, pageSize })
• cancelCrawlV1CrawlJobIdDelete({ jobId })
• listCrawlJobsV1CrawlJobsGet({ url, status, startDate, endDate, page, pageSize })

MapApi (/v1/map)

• mapUrlsV1MapPost({ mapRequest })
• listMapJobsV1MapJobsGet({ url, status, startDate, endDate, page, pageSize })
• getMapJobV1MapJobIdGet({ jobId })

StatisticsApi (/v1/statistics)

• getStatisticsV1StatisticsGet({ startDate, endDate })

SystemApi (/health)

• healthCheckHealthGet()

WebhooksApi (/v1/webhooks)

• listWebhooksV1WebhooksGet({ page, pageSize })
• createWebhookV1WebhooksPost({ webhookCreate })
• getWebhookV1WebhooksWebhookIdGet({ webhookId })
• updateWebhookV1WebhooksWebhookIdPatch({ webhookId, webhookUpdate })
• deleteWebhookV1WebhooksWebhookIdDelete({ webhookId })
• listDeliveriesV1WebhooksWebhookIdDeliveriesGet({ webhookId, page, pageSize, status })
• rotateSecretV1WebhooksWebhookIdRotateSecretPost({ webhookId })

Advanced Usage

You can inject a custom fetch implementation or middleware through Configuration.