@mymp/n8n-nodes-unified-scraper

Unified n8n community node for Scraper-X (Twitter/X), Scraper-Facebook, and Instagram API.

🎯 Features

Single Node, Multiple Platforms

• ✅ Scraper X (Twitter/X) — Trending topics, search, comments, crawl
• ✅ Scraper Facebook — Pages, groups, search, posts, watch, hashtags
• ✅ Instagram API — User info, media, DMs, upload, webhooks, batch actions
• ✅ Unified Interface — One node for all platforms
• ✅ Dynamic Operations — Context-aware parameters based on selection

Supported Operations

| Platform | Operations | |----------|-----------| | Scraper-X | trending, search, comments, crawl, demo/trending, demo/search | | Scraper-Facebook | page, group, search, post, watch, hashtag | | Instagram API | login, 2fa, auth status, user, feed, followers, following, follow/unfollow, media, like/unlike, comments, save, DM threads/messages/send, timeline/explore, search users/hashtags, hashtag posts, batch like/follow, webhooks |

📦 Installation

Method 1: Direct Copy (Recommended)

# Clone the repository
git clone https://github.com/mymp/n8n-nodes-unified-scraper.git
cd n8n-nodes-unified-scraper

# Build the node
npm install
npm run build

# Copy to n8n custom nodes directory
cp -r dist/* /path/to/n8n/custom-nodes/

# Restart n8n
docker-compose restart
# or
npm restart

Method 2: From NPM (Coming Soon)

cd /path/to/n8n/custom-nodes
npm install @mymp/n8n-nodes-unified-scraper

For Docker Users

Add to your docker-compose.yml:

services:
  n8n:
    image: n8nio/n8n
    volumes:
      - ./custom-nodes:/custom-nodes
    environment:
      - N8N_CUSTOM_NODES_DIR=/custom-nodes
      - NODE_FUNCTION_ALLOW_BUILTIN=crypto,fs

🔑 Configuration

Step 1: Create Credentials

1. In n8n UI, go to Credentials → New
2. Select Unified Scraper API
3. Configure all platforms:

Scraper X (Twitter/X):

• Scraper X Base URL: http://localhost:8082
• Scraper X API Key: (leave empty if no auth)

Scraper Facebook:

• Scraper Facebook Base URL: http://localhost:8083
• Scraper Facebook API Key: (leave empty if no auth)

Instagram API:

• Instagram API Base URL: http://localhost:8080
• Instagram API Key: (leave empty if no auth)

Step 2: Create Workflow

1. Add a new node → Unified Scraper
2. Select platform:
   • Scraper X (Twitter/X) for Twitter/X operations
   • Scraper Facebook for Facebook operations
   • Instagram API for Instagram operations
3. Select operation (options change based on platform)
4. Configure parameters
5. Connect to other nodes

📖 Usage Examples

Example 1: Get Malaysia Trending Topics

{
  "scraper": "scraper-x",
  "operation": "trending",
  "region": "malaysia",
  "limit": 20,
  "headless": true
}

Example 2: Scrape Facebook Page

{
  "scraper": "scraper-facebook",
  "operation": "page",
  "pageUrl": "https://www.facebook.com/meta",
  "limit": 20,
  "headless": true
}

Example 3: Instagram Login + Get User Info

// Node 1: Login
{
  "scraper": "instagram-api",
  "operation": "igLogin",
  "igUsername": "your_username",
  "igPassword": "your_password"
}

// Node 2: Get User Info
{
  "scraper": "instagram-api",
  "operation": "igUser",
  "igUsername": "natgeo"
}

Example 4: Instagram DM Automation

// Node 1: Send DM
{
  "scraper": "instagram-api",
  "operation": "igDmSend",
  "igDmUsername": "friend_username",
  "igDmText": "Hello from n8n!"
}

// Node 2: Register webhook for replies
{
  "scraper": "instagram-api",
  "operation": "igWebhookRegister",
  "igWebhookUrl": "https://n8n.yourserver.com/webhook/instagram-dm",
  "igWebhookEvents": ["message"]
}

Example 5: Multi-Platform Monitoring

Cron Trigger (every 30 minutes)
  ↓
Split in Batches
  ↓     ↓          ↓
X Trending   FB Page   IG Hashtag
  ↓           ↓          ↓
Merge ←──────┴──────────┘
  ↓
IF (high engagement)
  ↓
Slack (alert)

🎛️ Node Parameters

Common Parameters (All Operations)

| Parameter | Type | Default | Description | |-----------|------|---------|-------------| | scraper | Options | scraper-x | Platform selector | | limit | Number | 20 | Max items to return (1-100) | | headless | Boolean | true | Run browser headless (X/FB only) |

Scraper-X Parameters

| Operation | Parameters | |-----------|-----------| | trending | region (malaysia, singapore, indonesia, united-states, india, uk, canada, australia, japan, philippines, thailand) | | search | keyword | | comments | tweetUrl | | crawl | entityName, keyword, sources | | demoTrending | region | | demoSearch | keyword |

Scraper-Facebook Parameters

| Operation | Parameters | |-----------|-----------| | page | pageUrl | | group | groupUrl | | search | query | | post | postUrl, includeComments | | watch | — | | hashtag | hashtag |

Instagram API Parameters

| Operation | Parameters | |-----------|-----------| | igLogin | igUsername, igPassword | | ig2fa | igUsername, ig2faCode | | igAuthStatus | — | | igUser | igUsername | | igUserFeed | igUsername, limit | | igFollowers | igUsername, limit | | igFollowing | igUsername, limit | | igFollow / igUnfollow | igUsername | | igMedia | igMediaId | | igMediaByUrl | igUrl | | igLike / igUnlike | igMediaId | | igComments | igMediaId, limit | | igComment | igMediaId, igCommentText | | igSave | igMediaId | | igDmThreads | limit | | igDmMessages | igThreadId, limit | | igDmSend | igUserId or igDmUsername, igDmText | | igTimeline / igExplore | limit | | igSearchUsers / igSearchHashtags | igQuery, limit | | igHashtag | igHashtag, igTab (top/recent), limit | | igBatchLike | igMediaIds (comma-separated), igDelay | | igBatchFollow | igUsernames (comma-separated), igDelay | | igWebhookRegister | igWebhookUrl, igWebhookEvents, igWebhookSecret | | igWebhookList | — | | igWebhookTest | igWebhookUrl, igWebhookSecret | | igWebhookDelete | igWebhookId |

🌐 API Endpoints

Scraper-X API

• Base URL: http://localhost:8082
• Auth: X-API-Key header or Authorization: Bearer <token>
• Endpoints: POST /trending, POST /search, POST /comments, POST /crawl, POST /demo/trending, POST /demo/search

Scraper-Facebook API

• Base URL: http://localhost:8083
• Auth: X-API-Key header or Authorization: Bearer <token>
• Endpoints: POST /page, POST /group, POST /search, POST /post, POST /watch, POST /hashtag

Instagram API

• Base URL: http://localhost:8080
• Auth: X-API-Key header or Authorization: Bearer <token>
• Swagger UI: http://localhost:8080/docs
• Endpoints: 41 routes across Auth, User, Media, Upload, Direct, Feed, Search, Hashtag, Batch, Webhooks

🧪 Testing

Test Scraper-X

curl -X POST http://localhost:8082/trending \
  -H "X-API-Key: your-key" \
  -H "Content-Type: application/json" \
  -d '{"region": "malaysia", "limit": 10}'

Test Scraper-Facebook

curl -X POST http://localhost:8083/page \
  -H "X-API-Key: your-key" \
  -H "Content-Type: application/json" \
  -d '{"page_url": "https://www.facebook.com/meta", "limit": 10}'

Test Instagram API

# Login
curl -X POST http://localhost:8080/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username": "your_user", "password": "your_pass"}'

# Get user info
curl http://localhost:8080/user/natgeo \
  -H "X-API-Key: your-key"

# Register webhook
curl -X POST http://localhost:8080/webhooks \
  -H "X-API-Key: your-key" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://your-server.com/webhook", "events": ["message"]}'

🍪 Cookie Requirements

For authenticated scraping (Facebook groups, X search, Instagram):

Save Cookies via API (X/FB)

# Scraper-X
curl -X POST http://localhost:8082/auth/save \
  -H "X-API-Key: your-key" \
  -H "Content-Type: application/json" \
  -d '{"cookies": [...], "session_name": "x-session"}'

# Scraper-Facebook
curl -X POST http://localhost:8083/auth/save \
  -H "X-API-Key: your-key" \
  -H "Content-Type: application/json" \
  -d '{"cookies": [...], "session_name": "fb-session"}'

Instagram Auth

Instagram API uses username/password login with session persistence. Just use the igLogin operation — sessions are saved automatically in ./sessions/.

🐛 Troubleshooting

Node Not Showing

1. Restart n8n completely
2. Check files are in /custom-nodes/ directory
3. Check n8n logs for compilation errors
4. Verify TypeScript build completed successfully (npm run build)

401 Unauthorized

• Verify API keys are correct in credentials
• Test API endpoints with curl first
• Check base URLs are accessible
• Verify authentication format (X-API-Key header)

Empty Results

• Check if logged in (for authenticated endpoints)
• Verify URLs/usernames are correct and public
• Try with headless: false to debug (X/FB only)
• Check API logs for errors

📁 File Structure

n8n-nodes-unified-scraper/
├── nodes/
│   └── UnifiedScraperNode.ts       # Main unified node (X + FB + IG)
├── credentials/
│   └── UnifiedScraperApi.ts        # Credentials for all 3 platforms
├── package.json                     # NPM package config
├── tsconfig.json                    # TypeScript config
├── README.md                        # This file
├── LICENSE                          # MIT License
└── .gitignore                      # Git ignore

🚀 Production Status

| Component | Status | URL | |-----------|--------|-----| | Scraper-X API | ✅ Ready | http://localhost:8082 | | Scraper-Facebook API | ✅ Ready | http://localhost:8083 | | Instagram API | ✅ Ready | http://localhost:8080 | | n8n Node | ✅ Ready | v1.1.0 |

📄 License

MIT License — see LICENSE file for details

🤝 Contributing

Contributions welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Test thoroughly with all platforms
5. Submit a pull request

🎉 Summary

One node, three platforms, endless possibilities!

• ✅ Single unified interface for X, Facebook, and Instagram
• ✅ 30+ operations across all platforms
• ✅ Webhook support for Instagram events
• ✅ Full TypeScript support
• ✅ Easy installation