n8n-nodes-scraperapi

Community n8n node package for the Geonode Scraper API. Wraps the public REST endpoints (/v1/extract, /v1/map, /v1/crawl, /v1/webhooks, /v1/statistics) and ships a webhook trigger that registers itself against /v1/webhooks automatically.

The default deployment target is the preprod environment:

https://scraper-preprod-34do9l3uke5.geonode.com

You can point it at any other deployment by editing the Base URL field on the credential.

Package structure

n8n-nodes-scraperapi/
├─ credentials/
│  └─ ScraperApiApi.credentials.ts   # X-Api-Key credential, tested against /v1/statistics
├─ nodes/
│  ├─ ScraperApi/
│  │  ├─ ScraperApi.node.ts          # Action node (7 resources, ~20 operations)
│  │  ├─ GenericFunctions.ts         # Auth + retry/backoff helpers + polling helper
│  │  ├─ scraperapi.svg              # Node icon
│  │  └─ descriptions/
│  │     ├─ ExtractDescription.ts    # Form schema for extract: run / getJob / listJobs
│  │     ├─ SearchDescription.ts     # Form schema for search: run / getJob / listJobs
│  │     ├─ BatchDescription.ts      # Form schema for batch: start / getStatus / listJobs / cancel
│  │     ├─ MapDescription.ts        # Form schema for map: discover URLs
│  │     ├─ CrawlDescription.ts      # Form schema for crawl: start / getStatus
│  │     ├─ WebhookDescription.ts    # Form schema for webhook CRUD + deliveries
│  │     └─ StatisticsDescription.ts # Form schema for statistics: get
│  └─ ScraperApiTrigger/
│     ├─ ScraperApiTrigger.node.ts   # Trigger node — auto-registers on activate
│     └─ scraperapi.svg
├─ docker-compose.n8n.yml            # Local n8n that mounts this package as a custom extension
├─ package.json                      # n8n entry points -> dist/**
├─ tsconfig.json                     # Compiles credentials/** + nodes/** -> dist/
├─ gulpfile.js                       # Copies SVG icons into dist/
├─ .eslintrc.js                      # eslint-plugin-n8n-nodes-base rules
└─ index.js                          # Empty — n8n loads via the "n8n" key in package.json

Component descriptions

Credential — ScraperApiApi

X-Api-Key: <key> — your personal API key, sent on every request.

Credential fields:

| Field | Default | When to fill | |----------|----------------------------------------------------|-------------------------------------------------------| | Base URL | https://scraper-preprod-34do9l3uke5.geonode.com | Root URL, no trailing slash. | | API Key | (empty) | Required. Sent as X-Api-Key. |

Clicking Test issues GET {baseUrl}/v1/statistics with the credentials applied. This endpoint is auth-gated.

Action node — ScraperApi

One node with seven resources, each exposing the relevant operations:

| Resource | Operation | What it does | |--------------|-------------------|----------------------------------------------------------------------------------------------------| | extract | Run Extraction | POST /v1/extract. Pick formats (html / markdown), JS rendering, sync vs async, proxy, headers. | | extract | Get Job | GET /v1/extract/{job_id} — fetch a single async job result. | | extract | List Jobs | GET /v1/extract/jobs with filters (status, URL, date range) and pagination. | | search | Run Search | POST /v1/search — query plus optional engine, locale, page, safe search and time range. | | search | Get Job | GET /v1/search/{job_id} — fetch a single search job. | | search | List Jobs | GET /v1/search/jobs with filters (query, status, date range) and pagination. | | batch | Start Batch | POST /v1/batch — many URLs in one job; formats, JS rendering, proxy, headers, invalid-URL policy.| | batch | Get Status | GET /v1/batch/{job_id} — paginated results; optional client-side polling until terminal status. | | batch | List Jobs | GET /v1/batch/jobs with filters (status, date range) and pagination. | | batch | Cancel | DELETE /v1/batch/{job_id} — stop a batch that has not finished. | | map | Discover URLs | POST /v1/map — sitemap + optional search keyword, returns links[]. | | crawl | Start Crawl | POST /v1/crawl — seed URL, depth, limit, formats, same-domain / subdomain rules, proxy. | | crawl | Get Status | GET /v1/crawl/{job_id} — paginated results; optional client-side polling until terminal status. | | webhook | Create | POST /v1/webhooks — register a target URL for an event. | | webhook | Get / List | GET /v1/webhooks/{id} and GET /v1/webhooks (paginated). | | webhook | Update | PATCH /v1/webhooks/{id} — change URL, description, event type, active flag. | | webhook | Delete | DELETE /v1/webhooks/{id}. | | webhook | Rotate Secret | POST /v1/webhooks/{id}/rotate-secret — issues a new HMAC signing secret. | | webhook | List Deliveries | GET /v1/webhooks/{id}/deliveries with status filter and pagination. | | statistics | Get | GET /v1/statistics for an optional date range. |

When a sync extract returns inline, the response is the raw {data, ...} body. For async (processing_mode=async) the API returns {job_id, status, status_url, estimated_tokens}; if Wait for Completion is on, the node polls status_url until the job hits completed | failed | cancelled and returns that final body.

The shared HTTP helper (GenericFunctions.scraperApiRequest) handles:

• X-Api-Key header injection via the credential
• Up to 5 retries on 429 and 5xx with exponential backoff
• Retry-After header support (seconds or HTTP-date)
• Conversion of non-retryable errors into NodeApiError so n8n surfaces them in the UI

Trigger node — ScraperApiTrigger

Use this when you want a workflow to run whenever the Scraper API emits an event (extract_completed, crawl_completed).

On workflow activation it:

1. Calls POST /v1/webhooks with n8n's public webhook URL and the chosen event type.
2. Stashes the returned id and secret in workflow static data.

On webhook delivery it (optionally) verifies X-Webhook-Signature: sha256=<hex> using HMAC-SHA256 + the stored secret, then emits {headers, body} as workflow input.

On deactivation it calls DELETE /v1/webhooks/{id} to clean up.

Running the node

A ready-made compose file lives at docker-compose.n8n.yml in this directory. It mounts this package into n8n's custom-extensions dir and exposes the editor on http://localhost:5678.

From this directory:

# 1. Install deps and build the package
npm install
npm run build        # produces ./dist/ which docker-compose mounts into n8n

# 2. Start n8n
docker compose -f docker-compose.n8n.yml up -d

The compose file mounts the package directory read-only into /home/node/.n8n/custom/n8n-nodes-scraperapi. n8n reads the n8n key from package.json and loads the compiled JS from dist/**, so npm run build must succeed before starting the container (or before restarting after code changes).

After the container is up:

1. Open http://localhost:5678 and finish the n8n owner setup.
2. Credentials → New → Scraper API.
   • Base URL: https://scraper-preprod-34do9l3uke5.geonode.com (the default).
   • API Key: your preprod key.
   • Click Test — expects 200 from /v1/health.
3. New workflow → search Scraper API → drop it in.

To pick up code changes:

npm run build
docker compose -f docker-compose.n8n.yml restart n8n

To stop / clean up:

docker compose -f docker-compose.n8n.yml down            # stop, keep data
docker compose -f docker-compose.n8n.yml down -v         # also drop the n8n_data volume

Smoke test

In a workflow, add Scraper API → resource extract → operation Run Extraction:

• URL: https://example.com
• Output Formats: html
• Render JavaScript: off
• Processing Mode: Sync (Block Until Done)

Click Execute Node — you should see a JSON output with data.html populated.

Then try the trigger: in a separate workflow add Scraper API Trigger, pick extract_completed, save and activate. The node will register a webhook against /v1/webhooks. Run an extract against the same deployment with processing_mode=async; when it finishes, the trigger workflow fires.

Scripts reference

npm run build      # tsc + copy SVG icons into dist/
npm run dev        # tsc --watch
npm run lint       # eslint with eslint-plugin-n8n-nodes-base
npm run lintfix    # eslint --fix
npm run format     # prettier on nodes/ and credentials/

Notes specific to this API

• Auth — X-Api-Key header (per-user). See app/api/security/auth_token_handler.py. Billing user resolution happens in app/api/security/user_resolver.py.
• Async extract — processing_mode=async returns 202 with {job_id, status, status_url, estimated_tokens} (see app/api/v1/extract/router.py). The node polls status_url when Wait for Completion is on.
• Throttling (HTTP 429) — scraperApiRequest honors Retry-After and retries up to 5 times with exponential backoff (also covers transient 5xx).
• Errors — surfaced as NodeApiError. The API returns {code, message, correlation_id, retryable} (see app/api/v1/extract/schemas.py).
• Webhook events — extract_completed, crawl_completed (see app/core/webhooks/event_types.py).