Web Scraper (Free)

A no-API-key, DuckDuckGo-based search & scraping node for n8n.
Perform searches and extract page content—all without paying for an API or maintaining credentials.

🚀 Features

1. Search URLs only
   Perform a DuckDuckGo search and return the top N result URLs.

2. Extract single page
   Fetch any web page and return its title, H1 headers, and paragraphs.

3. Filter page content
   Fetch a page and return only the parts you care about:

   • Headings (H1, H2, H3)
   • Paragraphs
   • Bold text
   • All of the above

4. Query + Full Scrape
   Combines “Search URLs only” and “Extract single page”:

   • Search DuckDuckGo for a query
   • Fetch and parse each resulting URL

5. Browser-like HTTP requests

   • Custom User-Agent header
   • Configurable timeout to prevent hanging workflows

6. Built-in blacklist
   Automatically skips links from social, video, and other noise domains (e.g. YouTube, Facebook, Instagram, TikTok, Reddit, etc.).

7. Zero dependencies
   No external APIs or credentials required—just install once and start scraping.

📦 Installation

1. In your n8n instance, open Settings → Community Nodes → Install
2. In npm Package Name enter:

n8n-nodes-free-web-scrapping

3. Click Install and restart n8n if prompted.

🔧 Configuration

When you add the Web Scraper (Free) node, you’ll see:

1. Operation

| Value | Description | |-----------------|-------------------------------------------------------| | searchUrls | DuckDuckGo search → return URLs only | | extractPage | Fetch a single page → extract headers & paragraphs | | filterContent | Fetch a single page → return only filtered content | | fullScrape | DuckDuckGo search → fetch & extract content of URLs |

2. Parameters by Operation

Search URLs only & Full Scrape

• Search Query (string)
  Text to search for on DuckDuckGo.
• Maximum URLs (number)
  How many top links to process (1–20).

Extract single page & Filter page content

• Page URL (string)
  The URL of the page you want to scrape.

Filter page content only

• Filter Type (options)
• headings → H1, H2, H3
• paragraphs → <p> elements
• bolds → <b> and <strong>
• all → combinations of the above

📖 Examples

1. Search URLs only

[
{ "url": "https://openai.com/gpt-4" },
{ "url": "https://en.wikipedia.org/wiki/GPT-4" }
]

2. Extract single page

[
  {
    "url": "https://en.wikipedia.org/wiki/Web_scraping",
    "title": "Web scraping - Wikipedia",
    "h1": ["Web scraping"],
    "paragraphs": [
      "Web scraping, web harvesting, or web data extraction is data scraping used for extracting data from websites..."
    ]
  }
]

3. Filter page content

[
  {
    "url": "https://example.com",
    "filterType": "headings",
    "data": {
      "h1": ["Main Heading"],
      "h2": ["Section Title", "Another Section"],
      "h3": ["Subsection"]
    }
  }
]

4. Query + Full Scrape

[
  {
    "url": "https://example.com/page1",
    "title": "Example Page",
    "h1": ["Example Heading"],
    "paragraphs": ["First paragraph...", "Second paragraph..."]
  }
]

🔍 Blacklist

youtube.com, instagram.com, facebook.com, twitter.com,
tiktok.com, snapchat.com, linkedin.com, pinterest.com,
reddit.com, twitch.tv, vimeo.com, dailymotion.com,
bilibili.com, youku.com

⚙️ Under the Hood

• Uses got for HTTP requests.
• Parses DuckDuckGo’s HTML endpoint: https://html.duckduckgo.com/html?q=<query>
• Extracts links via regex on the result__a class.
• Cleans HTML with regex-based strip functions (scripts, styles, comments, tags).
• All operations return JSON—ready for downstream nodes.