app-manager-edge-worker

Distributed Scraper Edge Worker for Apps Manager. Run on your local laptop, desktop, VPS, or Android phone (via Termux) to offload web scraping, visual element selection, and bypass Cloudflare WAF restrictions with zero proxy cost.

✨ Why Distributed Edge Workers?

Cloud datacenter IPs (Render, AWS, DigitalOcean, Hetzner) are frequently blocked or challenged by Cloudflare, Akamai, and anti-bot systems. app-manager-edge-worker turns your existing hardware into an autonomous residential scraping cluster:

• 🌐 True Residential IP Routing: Requests originate directly from your home/office broadband or mobile data (Airtel, Jio, Comcast, etc.).
• 🔒 Zero Port Forwarding: Persistent outbound WebSockets (wss://.../workers) effortlessly traverse home Wi-Fi routers, firewalls, and carrier-grade NAT (CGNAT).
• ⚡ Autonomous Chromium Engine: Checks your system for Chrome/Edge or automatically installs stable Chromium without any manual intervention.
• 📱 Android Phone Support: Transform an old spare Android phone into a 24/7 scraping powerhouse using Termux and Wake Lock.
• ⏰ Automated Render Anti-Sleep Keep-Alive: Periodically pings the central server (every 7 minutes by default) to keep Render free-tier instances warm and eliminate cold-start spindowns.
• 🚀 Smart Single-Flight Deduplication: Concurrent scrapes for the exact same target URL coalesce into a single execution flight.

⚡ Quick Start

Option 1: Run Instantly via NPX (Recommended)

npx app-manager-edge-worker

On first run, the CLI interactively asks for your Worker Key. Paste it from your Apps Manager dashboard, hit Enter, and you are immediately connected! Credentials are automatically saved to ~/.app-manager-edge-worker/config.json for subsequent runs.

Option 2: Run with Flags Directly

npx app-manager-edge-worker --key wkey_live_your_key_here --server https://apps-manager.onrender.com

Option 3: Global NPM Installation

npm install -g app-manager-edge-worker

# Run directly
app-manager-edge-worker

📱 Turn Any Android Phone into a 24/7 Scraping Node (Termux)

Turn any spare or everyday Android phone into an ultra-low-power, 24/7 residential edge scraping node using Termux (download free from F-Droid):

Quick 1-Liner Setup (Copy & Paste into Termux)

pkg update -y && pkg install -y x11-repo && pkg update -y && pkg install -y nodejs-lts chromium termux-tools && termux-wake-lock && npx app-manager-edge-worker --server https://apps-manager.onrender.com --key <YOUR_WORKER_KEY>

Step-by-Step Installation

# 1. Update Termux core packages
pkg update -y

# 2. Enable x11-repo (Termux hosts the Chromium ARM64 binary inside x11-repo)
pkg install -y x11-repo
pkg update -y

# 3. Install Node.js LTS, Chromium browser, and Termux tools
pkg install -y nodejs-lts chromium termux-tools

# 4. Acquire wake-lock (keeps CPU active when screen turns off)
termux-wake-lock

# 5. Point Puppeteer to Termux's native Chromium binary (auto-detected, or export explicitly)
export PUPPETEER_EXECUTABLE_PATH=/data/data/com.termux/files/usr/bin/chromium-browser
export PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true

# 6. Launch the distributed edge worker
npx app-manager-edge-worker --server https://apps-manager.onrender.com --key <YOUR_WORKER_KEY>

[!IMPORTANT] Why x11-repo is mandatory: On Android aarch64, Termux maintains the chromium package within the x11-repo repository. Running pkg install -y x11-repo enables this repository so pkg install chromium succeeds without "package not found" errors.

[!TIP] Android Battery Optimization:

1. In Termux, run termux-wake-lock or swipe down your Android notification panel and tap "Acquire Wake Lock" (our worker CLI also automatically acquires the wake-lock on startup).
2. In Android Settings → Apps → Termux → Battery → set to "Unrestricted" (or "Don't optimize"). This prevents Android's Doze mode or manufacturer task killers (Samsung, Xiaomi, OnePlus) from putting Termux to sleep when the screen is locked.
3. For low-RAM phones (2GB–3GB RAM), add --concurrency 1 or --concurrency 2 to keep memory consumption ultra-lean.

🔄 Production Background Service (PM2)

To run the worker continuously as a daemon on Linux, macOS, or Windows:

# 1. Install PM2 and the worker CLI globally
npm install -g pm2 app-manager-edge-worker

# 2. Start the scraper worker as a managed process
pm2 start app-manager-edge-worker --name "apps-worker" -- --server https://apps-manager.onrender.com --key <YOUR_WORKER_KEY>

# 3. Configure auto-start on machine boot
pm2 save
pm2 startup

Useful PM2 commands:

• Check status: pm2 status
• View live stream logs: pm2 logs apps-worker
• Restart worker: pm2 restart apps-worker

🐳 Docker Container

Run in an isolated container on any VPS or local Docker host:

docker run -d \
  --name app-manager-edge-worker \
  --restart unless-stopped \
  -e SERVER_URL="https://apps-manager.onrender.com" \
  -e WORKER_KEY="<YOUR_WORKER_KEY>" \
  node:20-slim sh -c "apt-get update && apt-get install -y chromium && npx app-manager-edge-worker"

🌐 Autonomous Chromium Detection & Self-Installation

When app-manager-edge-worker initializes, it executes a multi-layer verification routine to detect a usable Chromium executable:

1. Explicit Environment Variables: Checks PUPPETEER_EXECUTABLE_PATH, CHROME_BIN, CHROME_PATH, CHROMIUM_PATH.
2. Pre-Installed System Browsers: Scans standard OS locations for Google Chrome, Microsoft Edge, Brave, or Chromium, and checks system PATH.
3. Bundled/Cached Browsers: Scans ~/.cache/puppeteer using @puppeteer/browsers.
4. Android Termux Detection: Automatically detects /data/data/com.termux/files/usr/bin/chromium-browser.

What happens if NO Chromium is found?

The worker automatically downloads and installs it according to your operating system:

| Operating System | Auto-Installation Action | |------------------|--------------------------| | Windows (x64/x86) | Downloads official stable Chrome for Testing from Google CDN with a progress bar into ~/.cache/puppeteer. | | macOS (Intel / Apple Silicon M1-M4) | Detects system architecture (mac or mac_arm) and downloads matching stable Chrome from Google CDN. | | Linux Desktop / VPS (Ubuntu, Debian, Fedora, Arch) | Downloads official Linux x64/arm64 Chrome for Testing via @puppeteer/browsers. | | Android (Termux) | Automatically enables x11-repo and invokes Termux package manager (pkg update -y && pkg install -y x11-repo && pkg update -y && pkg install -y nodejs-lts chromium termux-tools). | | Alpine Linux | Automatically invokes Alpine package manager (apk add --no-cache chromium). |

No manual setup required!

⚙️ CLI Options & Flags

| Flag | Description | Default | |------|-------------|---------| | -k, --key <key> | Worker authorization key (wkey_live_...) | Prompted / Stored | | -s, --server <url> | Apps Manager Server URL | https://apps-manager.onrender.com | | -c, --concurrency <n> | Max concurrent Chromium browser tabs | 3 | | -n, --name <name> | Custom display name for this worker | Device hostname | | -p, --keep-alive <minutes> | Keep-alive ping interval to prevent Render free-tier cold sleep | 7 | | --no-keep-alive | Disable keep-alive ping loop | - | | --reset | Clear stored credentials and prompt again | - | | -h, --help | Display help screen | - |

🕐 Log Output Format

Every log line is prefixed with a timestamp in YYYY-MM-DD HH:mm:ss format:

[2026-09-15 14:32:07] [INFO] Connected to Apps Manager server: https://apps-manager.onrender.com
[2026-09-15 14:32:07] [SUCCESS] Chromium detected: /usr/bin/chromium-browser
[2026-09-15 14:32:10] [TASK] SCRAPE_CHROMIUM -> https://example.com/product (1243ms)
[2026-09-15 14:32:11] [WARN] Scrape returned no data for selector: .price-value
[2026-09-15 14:32:11] [ERROR] Socket connection lost, reconnecting in 5s...

Log levels: [INFO] · [SUCCESS] · [WARN] · [ERROR] · [TASK] (job dispatch with optional duration).

🔒 Security & Sandboxing

• Outbound Only: All socket tunnels are initiated outbound to the central Apps Manager server over TLS (wss://). No public IP, dynamic DNS, or port forwarding required.
• Granular Scoping: Workers can be dedicated to a specific application or organization team, or shared with the public pool.
• Incognito Sandboxing: Every scraping job and visual selector preview is executed within an isolated browser context that is cleanly destroyed upon task completion.

📝 Changelog

v1.0.4

• Timestamped logs: every log line now starts with [YYYY-MM-DD HH:mm:ss] for debugging and production monitoring
• Version fix: banner, help text, and user-agent string now correctly report v1.0.4

v1.0.3

• Improved element inspector selector generation (CSS/XPath ranked candidates)

v1.0.2

• Comprehensive CSS/XPath selector generation for visual element picker

v1.0.1

• Chunked streaming for large payloads (256 KB chunks over Socket.IO)

v1.0.0

• Initial release — SCRAPE_CHROMIUM, SCRAPE_HTTP, PREVIEW_PAGE, INSPECT_ELEMENT handlers
• Multi-layer Chromium auto-detection + auto-install (Windows/macOS/Linux/Alpine/Termux)
• Keep-alive ping, single-flight deduplication, PM2/Docker support

👤 Author

Chetan Mohite

• GitHub: @codes-by-chetan
• Platform: Apps Manager

📄 License

MIT © Chetan Mohite