n8n-nodes-unified-scraper

v4.1.0

Published

2 months ago

Generic n8n node for any scraper service with custom URL and API key

0High
0Medium
0Low

n8n n8n-community-node scraper twitter x facebook instagram linkedin tiktok threads reddit google news social-media webhook database

@mymp/n8n-nodes-unified-scraper v4.0.0

Unified n8n community node for 12 scrapers with automatic database insertion.

Supported Scrapers (v4.0.0)

Scraper X (Twitter/X)
Scraper Facebook
Scraper Instagram
Scraper LinkedIn
Scraper TikTok
Scraper Threads
Scraper News
Scraper News Go
Scraper Google News
Scraper Reddit Go
Scraper Google Search
Instagram API

New in v4.0.0

12 Scrapers: Expanded from 3 to 12 scrapers
DB Insertion: Optional automatic insertion to scraper-db-service
Retry Logic: Configurable retries (0-5) with exponential backoff
PostgreSQL Credential: Direct DB access support
Standardized Operations: All scrapers support search, trending, getById

Usage

Basic Scraping

Add the Unified Scraper node to your workflow
Select your scraper and operation
Enter your query/ID
(Optional) Enable "Insert to Database"
Execute

With Database Insertion

Configure credentials:
- Unified Scraper API (DB service API key required)
- PostgreSQL (optional, for direct queries)
Enable "Insert to Database" in node parameters
Results will be automatically stored

Credentials

Unified Scraper API

Scraper DB Service Base URL (default: http://scraper-db-service:8088)
Scraper DB Service API Key (required for DB insert)
Individual scraper API keys (optional)

PostgreSQL

Host, Port, Database, User, Password

Output Format

{
  "success": true,
  "scraper": "scraper-x",
  "operation": "search",
  "results": [...],
  "dbInserted": true,
  "dbStats": {
    "added": 10,
    "updated": 2,
    "skipped": 0
  },
  "errors": [],
  "retryCount": 0
}

📦 Installation

Method 1: Direct Copy (Recommended)

# Clone the repository
git clone https://github.com/mymp/n8n-nodes-unified-scraper.git
cd n8n-nodes-unified-scraper

# Build the node
npm install
npm run build

# Copy to n8n custom nodes directory
cp -r dist/* /path/to/n8n/custom-nodes/

# Restart n8n
docker-compose restart
# or
npm restart

Method 2: From NPM (Coming Soon)

cd /path/to/n8n/custom-nodes
npm install @mymp/n8n-nodes-unified-scraper

For Docker Users

Add to your docker-compose.yml:

services:
  n8n:
    image: n8nio/n8n
    volumes:
      - ./custom-nodes:/custom-nodes
    environment:
      - N8N_CUSTOM_NODES_DIR=/custom-nodes
      - NODE_FUNCTION_ALLOW_BUILTIN=crypto,fs

🔑 Configuration

Step 1: Create Credentials

In n8n UI, go to Credentials → New
Select Unified Scraper API
Configure all platforms:

Scraper X (Twitter/X):

Scraper X Base URL: http://localhost:8082
Scraper X API Key: (leave empty if no auth)

Scraper Facebook:

Scraper Facebook Base URL: http://localhost:8083
Scraper Facebook API Key: (leave empty if no auth)

Instagram API:

Instagram API Base URL: http://localhost:8080
Instagram API Key: (leave empty if no auth)

Scraper DB Service (v4.0.0):

Scraper DB Service Base URL: http://scraper-db-service:8088
Scraper DB Service API Key: (required for DB insert)

Step 2: Create Workflow

Add a new node → Unified Scraper
Select platform (12 options in v4.0.0)
Select operation (options change based on platform)
Configure parameters
Enable "Insert to Database" (optional, v4.0.0)
Connect to other nodes

📖 Usage Examples

Example 1: Get Malaysia Trending Topics

{
  "scraper": "scraper-x",
  "operation": "trending",
  "region": "malaysia",
  "limit": 20,
  "headless": true,
  "insertToDatabase": true,
  "retryCount": 3
}

Example 2: Scrape Facebook Page with DB Insert

{
  "scraper": "scraper-facebook",
  "operation": "page",
  "pageUrl": "https://www.facebook.com/meta",
  "limit": 20,
  "headless": true,
  "insertToDatabase": true
}

Example 3: Instagram Login + Get User Info

// Node 1: Login
{
  "scraper": "instagram-api",
  "operation": "igLogin",
  "igUsername": "your_username",
  "igPassword": "your_password"
}

// Node 2: Get User Info
{
  "scraper": "instagram-api",
  "operation": "igUser",
  "igUsername": "natgeo",
  "insertToDatabase": true
}

Example 4: LinkedIn Search (v4.0.0)

{
  "scraper": "scraper-linkedin",
  "operation": "search",
  "query": "software engineer",
  "limit": 25,
  "insertToDatabase": true,
  "retryCount": 2
}

Example 5: Multi-Platform Monitoring with DB

Cron Trigger (every 30 minutes)
  ↓
Split in Batches
  ↓     ↓          ↓           ↓
X Trending   FB Page   IG Hashtag   LinkedIn Jobs
  ↓           ↓          ↓             ↓
          Merge ←──────┴─────────────┘
            ↓
      IF (high engagement)
            ↓
    Insert to Database
            ↓
      Slack (alert)

🎛️ Node Parameters

Common Parameters (All Operations)

| Parameter | Type | Default | Description | |-----------|------|---------|-------------| | scraper | Options | scraper-x | Platform selector (12 options) | | limit | Number | 20 | Max items to return (1-100) | | headless | Boolean | true | Run browser headless (X/FB/LinkedIn only) | | insertToDatabase | Boolean | false | Insert results to DB (v4.0.0) | | retryCount | Number | 0 | Retry attempts with backoff (0-5, v4.0.0) |

Scraper-X Parameters

| Operation | Parameters | |-----------|-----------| | trending | region (malaysia, singapore, indonesia, united-states, india, uk, canada, australia, japan, philippines, thailand) | | search | keyword | | comments | tweetUrl | | crawl | entityName, keyword, sources | | getById | id (v4.0.0 standardized) | | demoTrending | region | | demoSearch | keyword |

Scraper-Facebook Parameters

| Operation | Parameters | |-----------|-----------| | page | pageUrl | | group | groupUrl | | search | query | | post | postUrl, includeComments | | watch | — | | hashtag | hashtag | | getById | id (v4.0.0 standardized) |

Instagram API Parameters

| Operation | Parameters | |-----------|-----------| | igLogin | igUsername, igPassword | | ig2fa | igUsername, ig2faCode | | igAuthStatus | — | | igUser | igUsername | | igUserFeed | igUsername, limit | | igFollowers | igUsername, limit | | igFollowing | igUsername, limit | | igFollow / igUnfollow | igUsername | | igMedia | igMediaId | | igMediaByUrl | igUrl | | igLike / igUnlike | igMediaId | | igComments | igMediaId, limit | | igComment | igMediaId, igCommentText | | igSave | igMediaId | | igDmThreads | limit | | igDmMessages | igThreadId, limit | | igDmSend | igUserId or igDmUsername, igDmText | | igTimeline / igExplore | limit | | igSearchUsers / igSearchHashtags | igQuery, limit | | igHashtag | igHashtag, igTab (top/recent), limit | | igBatchLike | igMediaIds (comma-separated), igDelay | | igBatchFollow | igUsernames (comma-separated), igDelay | | igWebhookRegister | igWebhookUrl, igWebhookEvents, igWebhookSecret | | igWebhookList | — | | igWebhookTest | igWebhookUrl, igWebhookSecret | | igWebhookDelete | igWebhookId |

Additional Scrapers (v4.0.0)

| Scraper | Operations | |---------|-----------| | Scraper LinkedIn | search, trending, getById | | Scraper TikTok | search, trending, getById | | Scraper Threads | search, trending, getById | | Scraper News | search, trending, getById | | Scraper News Go | search, trending, getById | | Scraper Google News | search, trending, getById | | Scraper Reddit Go | search, trending, getById | | Scraper Google Search | search, trending, getById |

🌐 API Endpoints

Scraper-X API

Base URL: http://localhost:8082
Auth: X-API-Key header or Authorization: Bearer <token>
Endpoints: POST /trending, POST /search, POST /comments, POST /crawl, POST /demo/trending, POST /demo/search

Scraper-Facebook API

Base URL: http://localhost:8083
Auth: X-API-Key header or Authorization: Bearer <token>
Endpoints: POST /page, POST /group, POST /search, POST /post, POST /watch, POST /hashtag

Instagram API

Base URL: http://localhost:8080
Auth: X-API-Key header or Authorization: Bearer <token>
Swagger UI: http://localhost:8080/docs
Endpoints: 41 routes across Auth, User, Media, Upload, Direct, Feed, Search, Hashtag, Batch, Webhooks

Scraper DB Service (v4.0.0)

Base URL: http://scraper-db-service:8088
Auth: X-API-Key header (required for writes)
Swagger UI: http://scraper-db-service:8088/docs
Endpoints: POST /scrape, GET /health, GET /stats

🧪 Testing

Test Scraper-X

curl -X POST http://localhost:8082/trending \
  -H "X-API-Key: your-key" \
  -H "Content-Type: application/json" \
  -d '{"region": "malaysia", "limit": 10}'

Test Scraper-Facebook

curl -X POST http://localhost:8083/page \
  -H "X-API-Key: your-key" \
  -H "Content-Type: application/json" \
  -d '{"page_url": "https://www.facebook.com/meta", "limit": 10}'

Test Instagram API

# Login
curl -X POST http://localhost:8080/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username": "your_user", "password": "your_pass"}'

# Get user info
curl http://localhost:8080/user/natgeo \
  -H "X-API-Key: your-key"

# Register webhook
curl -X POST http://localhost:8080/webhooks \
  -H "X-API-Key: your-key" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://your-server.com/webhook", "events": ["message"]}'

Test DB Service (v4.0.0)

# Health check
curl http://scraper-db-service:8088/health

# Scrape with DB insert
curl -X POST http://scraper-db-service:8088/scrape \
  -H "X-API-Key: your-db-key" \
  -H "Content-Type: application/json" \
  -d '{
    "scraper": "scraper-x",
    "operation": "trending",
    "params": {"region": "malaysia", "limit": 10},
    "insertToDb": true
  }'

🍪 Cookie Requirements

For authenticated scraping (Facebook groups, X search, Instagram):

Save Cookies via API (X/FB)

# Scraper-X
curl -X POST http://localhost:8082/auth/save \
  -H "X-API-Key: your-key" \
  -H "Content-Type: application/json" \
  -d '{"cookies": [...], "session_name": "x-session"}'

# Scraper-Facebook
curl -X POST http://localhost:8083/auth/save \
  -H "X-API-Key: your-key" \
  -H "Content-Type: application/json" \
  -d '{"cookies": [...], "session_name": "fb-session"}'

Instagram Auth

Instagram API uses username/password login with session persistence. Just use the igLogin operation — sessions are saved automatically in ./sessions/.

🐛 Troubleshooting

Node Not Showing

Restart n8n completely
Check files are in /custom-nodes/ directory
Check n8n logs for compilation errors
Verify TypeScript build completed successfully (npm run build)

401 Unauthorized

Verify API keys are correct in credentials
Test API endpoints with curl first
Check base URLs are accessible
Verify authentication format (X-API-Key header)
For DB insert, verify Scraper DB Service API Key

Empty Results

Check if logged in (for authenticated endpoints)
Verify URLs/usernames are correct and public
Try with headless: false to debug (X/FB only)
Check API logs for errors
Verify retryCount is not causing delays

DB Insert Fails (v4.0.0)

Verify Scraper DB Service is running (curl http://scraper-db-service:8088/health)
Check API key is valid for writes
Verify data format matches expected schema
Check logs for constraint violations

📁 File Structure

n8n-nodes-unified-scraper/
├── nodes/
│   └── UnifiedScraperNode.ts       # Main unified node (12 scrapers)
├── credentials/
│   ├── UnifiedScraperApi.ts        # Credentials for all platforms
│   └── PostgreSQL.ts                # PostgreSQL credential (v4.0.0)
├── package.json                     # NPM package config
├── tsconfig.json                    # TypeScript config
├── README.md                        # This file
├── LICENSE                          # MIT License
└── .gitignore                      # Git ignore

🚀 Production Status

| Component | Status | URL | |-----------|--------|-----| | Scraper-X API | ✅ Ready | http://localhost:8082 | | Scraper-Facebook API | ✅ Ready | http://localhost:8083 | | Instagram API | ✅ Ready | http://localhost:8080 | | Scraper DB Service | ✅ Ready | http://scraper-db-service:8088 | | n8n Node | ✅ Ready | v4.0.0 |

📄 License

MIT License — see LICENSE file for details

🤝 Contributing

Contributions welcome! Please:

Fork the repository
Create a feature branch
Make your changes
Test thoroughly with all platforms
Submit a pull request

🎉 Summary

One node, twelve platforms, endless possibilities!

✅ Single unified interface for 12 scrapers
✅ 60+ operations across all platforms
✅ Automatic database insertion with retry logic
✅ Webhook support for Instagram events
✅ Full TypeScript support
✅ Easy installation