n8n-nodes-webscraping-ai

This is an n8n community node that integrates WebScraping.AI into your n8n workflows.

WebScraping.AI provides AI-powered web scraping with Chromium JavaScript rendering, rotating datacenter/residential/stealth proxies, and built-in HTML parsing — plus LLM-powered question answering and structured field extraction on any page. This node allows you to leverage these capabilities directly in your n8n automations.

Features

• AI Question: Ask questions about web page content using LLM
• AI Extract Fields: Extract structured data fields using AI
• Get HTML: Retrieve full HTML content of web pages
• Get Text: Extract visible text content
• Get Selected HTML: Extract HTML from specific CSS selectors
• Get Multiple Selections: Extract HTML from multiple CSS selectors
• Get Account Info: Retrieve account quota and usage information
• Search (SERP): Get parsed Google search results for a query, from a chosen location, one or several pages at a time
• Search Locations: Look up location names for Search (SERP) (free)
• Get Structured Data: Get structured JSON for a page on a supported site (e.g. YouTube, TikTok, X, LinkedIn, Instagram, Reddit) from its normal URL

Installation

Follow the installation guide in the n8n community nodes documentation.

Install from npm

npm install n8n-nodes-webscraping-ai

Manual Installation

1. Clone this repository
2. Run npm install to install dependencies
3. Run npm run build to build the node
4. Copy the dist folder to your n8n custom nodes directory

Prerequisites

You need to have an account with WebScraping.AI and obtain an API key from your dashboard. Sign up to get started — a free trial, no credit card required.

Credentials

Create a new credential of type "WebScraping.AI API" in n8n and enter your API key.

Operations

AI Question

Ask AI a question about a webpage. The LLM will analyze the page content and provide an answer.

Parameters:

• URL (required): The webpage to analyze
• Question (required): The question to ask about the page
• Response Format: JSON or Text

AI Extract Fields

Extract structured data from a webpage using AI.

Parameters:

• URL (required): The webpage to scrape
• Fields to Extract (required): JSON object defining fields and their descriptions
  • Example: {"title":"Main product title","price":"Current product price"}

Get HTML

Get the full HTML content of a webpage.

Parameters:

• URL (required): The webpage to retrieve
• Response Format: JSON or Text
• Return Script Result: Return result of custom JavaScript execution instead of HTML

Get Text

Get the visible text content of a webpage.

Parameters:

• URL (required): The webpage to retrieve
• Text Format: Plain, JSON, or XML
• Return Links: Include links in JSON response (works only with JSON format)

Get Selected HTML

Get HTML content of selected page areas by CSS selector.

Parameters:

• URL (required): The webpage to retrieve
• CSS Selector: The selector for target elements
• Response Format: JSON or Text

Get Multiple Selections

Get HTML content of multiple page areas by CSS selectors.

Parameters:

• URL (required): The webpage to retrieve
• CSS Selectors: Array of CSS selectors in JSON format
  • Example: ["h1", ".price", "#description"]

Get Account Info

Get your API account quota and usage information.

Returns:

• Email address
• Remaining API credits
• Next billing cycle reset time
• Remaining concurrent requests

Search (SERP)

Get parsed search engine results for a query. Query-shaped rather than URL-shaped: WebScraping.AI handles proxy routing and parsing, so the scraping Advanced Options below don't apply. Billed per results page returned (see pricing); failed searches are not charged.

Every search runs from one fixed location: Location, Latitude and Longitude, or UULE, otherwise the capital of the Country. So repeats and later pages are consistent, and Country defaults to the location's country. Pages fetches up to 5 consecutive pages in one call as one de-duplicated list.

Parameters:

• Query (required): The search query, e.g. coffee machines
• Search Options:
  • Country: Two-letter country code for the search (Google gl). Defaults to the country of Location, Latitude/Longitude or UULE, otherwise us
  • Engine: Search engine to query (currently google, the default)
  • Language: Two-letter language code for the results (Google hl, default en)
  • Latitude / Longitude: Search origin in decimal degrees (set both; sent as plain decimals with up to 7 fraction digits). A number or a decimal string from an expression; a blank value is an error, so leave the option out when there is none
  • Location: Where the search runs from, e.g. Austin,Texas,United States, Austin, TX or an id from Search Locations. Large cities also match local names, e.g. Москва. An unknown name returns a 400 that is not charged
  • Page: Results page number, 10 results per page (default 1)
  • Pages: Number of consecutive pages to fetch starting at Page, 1 to 5 (default 1). Duplicates across pages are removed, position counts through the combined list and each result gets a page field
  • Radius (Meters): Radius around Location or Latitude/Longitude, 1 to 1000
  • UULE: Google-encoded location (w+CAIQICI… or a+…)

Location, Latitude/Longitude and UULE are alternatives: use one of them. Invalid combinations and out-of-range values are rejected by the API with a 400 that is not charged.

Returns JSON with search_parameters (including location_used, the catalog place searched from: the Location you gave, a UULE's place, or the Country's capital when none is set, absent for Latitude/Longitude searches; location_requested, the Location or UULE as sent; lat/lon, the point searched from; and radius/pages when set), search_information, organic_results (each with position, title, link, domain, displayed_link, and optional snippet/date; page when Pages is more than 1), optional related_searches, and pagination (with pages_returned when Pages is more than 1).

Search Locations

Look up names for Search (SERP)'s Location option. Free, no credits used.

Parameters:

• Query: Part of a location name in English, e.g. austin. Without it, the most populous locations are returned
• Limit: Maximum number of locations to return, 1 to 10 (default 10)

Returns one item per location, most populous first, with id, google_id, google_parent_id, name, canonical_name, country_code, target_type (e.g. City, State, Country), reach (approximate population) and gps ([longitude, latitude]). Pass canonical_name (or id) as Location.

Get Structured Data

Get structured JSON for a public page on a supported site from its normal URL: for example a YouTube video, channel or playlist, a TikTok video or profile, an X post or profile, a LinkedIn company, job or profile, an Instagram post, reel or profile, or a Reddit post, subreddit or user. The site (provider) and page kind (type) are detected from the URL. These are examples: more sites are added on the server over time and work with this node without an update, so the node never checks the URL itself. An unsupported URL or page type returns a 400 that is not charged. Its message lists what is supported. For other sites, use AI Extract Fields. Priced per site (see pricing), including pages that parse empty (parse_failed) or no longer exist (not_found); unsupported URLs and failed fetches are not charged. The scraping Advanced Options below don't apply.

Parameters:

• URL (required): The page's normal URL, e.g. https://www.youtube.com/watch?v=dQw4w9WgXcQ. Surrounding whitespace is trimmed; otherwise it is sent unchanged.
• Country: Two-letter country code of the proxy used to fetch the page, us by default.
• Transcript: YouTube videos only. Also fetch the video's transcript into data.transcript. It's null when no matching captions are available. If the transcript fetch itself fails, the whole request fails with a 500 and is not charged.
• Transcript Language: Caption language to pick, e.g. en or de. Without it, English is preferred, then the first available track. If the video has no captions in that language, data.transcript is null.
• Extra Parameters: Name/value pairs sent to the API as-is, for site-specific parameters added after this node version. url, api_key, a name that repeats one of the fields above, or a name used in two rows is an error.

Returns JSON with request_parameters (url, provider, type), parse_status (ok, parse_failed or not_found) and data, whose snake_case fields depend on provider and type (data can be null).

Advanced Options

All URL-based scraping operations support these advanced options:

Browser & Rendering

• Enable JavaScript: Execute on-page JavaScript (default: true)
• JavaScript Timeout: Max JS rendering time in ms (default: 2000)
• Wait For Selector: CSS selector to wait for before returning content
• Custom JavaScript: Execute custom JavaScript code on the page
• Device: Desktop, Mobile, or Tablet emulation

Proxy Settings

• Proxy Type: Datacenter or Residential
• Country: Select proxy country (US, GB, DE, IT, FR, CA, ES, RU, JP, KR, IN)
• Custom Proxy: Use your own proxy in "http://user:password@host:port" format

Request Configuration

• Headers: Custom HTTP headers as JSON object
• Timeout: Max page retrieval time in ms (default: 10000, max: 30000)
• Error on 404: Return error on 404 status
• Error on Redirect: Return error on redirects

Usage Examples

Extract Product Data from E-commerce Site

1. Add WebScraping.AI node to your workflow
2. Select "AI Extract Fields" operation
3. Enter product URL
4. Set Fields to Extract:

{
  "title": "Product title",
  "price": "Current price",
  "description": "Product description",
  "rating": "Product rating",
  "availability": "In stock or out of stock"
}

Monitor Website Content Changes

1. Use "Get Text" operation with JSON format
2. Store the result in a database
3. Schedule the workflow to run periodically
4. Compare new results with previous ones to detect changes

Scrape Google Search Results

1. Use "Search (SERP)" operation
2. Enter the search query (e.g., coffee machines)
3. Optionally set Country, Language, and Page under Search Options
4. Split out organic_results to process each result (position, title, link, snippet)

Search From a City, Two Pages at Once

1. Optionally use "Search Locations" with Query austin and pick the canonical_name you want (e.g. Austin,Texas,United States)
2. Use "Search (SERP)" with Query coffee machines
3. Under Search Options set Location to Austin, TX (or the canonical_name from step 1) and Pages to 2
4. organic_results holds both pages as one list; each result's page says where it came from

Get YouTube Video Details and Transcript

1. Use "Get Structured Data" operation
2. Enter the video URL (e.g., https://www.youtube.com/watch?v=dQw4w9WgXcQ)
3. Turn on Transcript to also get data.transcript
4. Read data.title, data.views, data.channel and so on from the output

Error Handling

The node handles the following error scenarios:

• 400: Parameter validation errors - Check your input parameters
• 402: Billing/quota exceeded - Check your account balance
• 403: Invalid API key - Verify your credentials
• 429: Rate limiting - Reduce request frequency
• 500: Target page errors - The target website returned an error
• 504: Timeout errors - Increase timeout parameter

Enable "Continue on Fail" in the node settings to handle errors gracefully in your workflow.

Live smoke test (development)

scripts/smoke.ts sends one live API request for each of the 10 operations on https://example.com (the SERP operation searches for "coffee machines"; Search Locations looks up austin; Get Structured Data fetches https://www.youtube.com/watch?v=dQw4w9WgXcQ), a second SERP request with Location Austin, TX and Pages 2, plus one Get Structured Data request on https://example.com/ that must come back as the server's 400 with a message containing Unsupported URL, proving the node has no client-side site filter. Each request is built by the node's real buildRequest helper. The script then adds api_key to the query string, as the credential does, and encodes the query string the way n8n's httpRequest helper does with the arrayFormat: 'repeat' that buildRequest sets: arrays go out as repeated keys (selectors=h1&selectors=p; the API ignores bracketed selectors[0]= arrays), objects as bracketed keys (fields[title]=..., headers[X-Foo]=...). It checks results, not just status codes: SERP must return non-empty organic_results with search_parameters.q matching the query (and, for the Austin request, search_parameters.location_used containing Austin), Search Locations a non-empty list whose first canonical_name contains Austin, Get Structured Data must return parse_status ok, request_parameters.provider youtube and a non-empty data.title, Selected Multiple must return at least one non-empty inner array (the API answers mis-encoded selectors with [[]]), AI Fields a non-empty object, and the rest a non-empty body. It prints ok/FAIL for each operation (never the API key) and exits non-zero if any operation fails. An optional WEBSCRAPING_AI_API_URL overrides the base URL and must include the scheme (http:// or https://).

WEBSCRAPING_AI_API_KEY=your-key npm run smoke

It uses real credits; see the script header. The script is compiled with tsc -p tsconfig.smoke.json into .smoke/, which is gitignored. It isn't part of dist/ or the npm package, and it sits outside the nodes/ and credentials/ paths that lint and the n8n community-package scanner check.

Resources

• WebScraping.AI — features, pricing, signup
• WebScraping.AI Documentation
• API Reference
• n8n Community Nodes
• Other official clients: Python · JavaScript · Ruby · PHP · Go · Java · .NET · CLI · MCP server

License

MIT

Support

For issues related to this n8n node, please open an issue on GitHub.

For WebScraping.AI API issues, contact [email protected].