📰 Google News Extractor

Search Google News and Google Search to extract article content with automatic URL resolution. Queries both sources in parallel and merges results.

Quick Start (npx)

# Install and run in one command
npx google-news-extractor -q "AI" -m 5

# Or install globally
npm install -g google-news-extractor
google-news -q "AI" -m 5

The first run will automatically download Chromium for URL resolution.

Command Line Usage

# Search news + web results (default: both sources)
google-news -q "OpenAI" -m 5

# News RSS only
google-news --news-only -q "OpenAI" -m 5

# Google Search only (web results)
google-news --web-only -q "GPU benchmarks" -m 5

# Browse a topic
google-news -t technology -m 10

# JSON output piped to jq
google-news -q "AI" -f json 2>/dev/null | jq

# Save to file
google-news -q "climate" -o results.md

# Sources only (fast, no content extraction)
google-news -q "business" -s

Options

--query, -q <term>        Search for specific news
--topic, -t <topic>       Browse: world, nation, business, technology,
                          entertainment, sports, science, health
--max, -m <number>        Max articles (default: 10)
--format, -f <type>       Output: pretty, markdown, json, text
--output, -o <file>       Save to file
--sources-only, -s        Only extract source links
--news-only               Only query Google News RSS
--web-only                Only query Google Search
--help, -h                Show help

Web Server

# Start the web server
./start.sh
# or
node server.mjs

# Open http://localhost:3000

API

curl -X POST http://localhost:3000/api/search \
  -H "Content-Type: application/json" \
  -d '{"query": "AI", "maxArticles": 5}'

Features

• ✅ Dual Sources - Queries Google News RSS + Google Search in parallel, merges & deduplicates
• ✅ Stealth Browser - Full anti-detection (webdriver, plugins, WebGL, human mouse/scroll)
• ✅ DuckDuckGo Fallback - If Google blocks, falls back to DuckDuckGo HTML search
• ✅ URL Resolution - Resolves Google News redirect URLs to actual article links via Playwright
• ✅ Content Extraction - Extracts article paragraphs using readability-style scoring
• ✅ Multiple Formats - Markdown, JSON, text, pretty terminal output
• ✅ Topic Browsing - Browse by category (technology, business, sports, etc.)
• ✅ Paywall Handling - Graceful fallback for paywalled content
• ✅ Puppeteer Fallback - Falls back to headless Chrome when HTTP parsing isn't enough
• ✅ Auto Setup - Automatically installs Chromium on first run

Requirements

• Node.js 18+
• Playwright Chromium (auto-installed on first run)

Project Structure

├── src/
│   ├── cli.ts            # CLI entry point
│   ├── index.ts          # Main library API
│   ├── rss.ts            # RSS feed parser + URL resolver orchestration
│   ├── google-search.ts  # Google Search scraper (HTTP + Puppeteer fallback)
│   ├── extractor.ts      # Content extraction
│   ├── fetcher.ts        # HTTP fetch with fallback
│   ├── gnews-resolver.ts # Playwright/Puppeteer URL resolver
│   ├── formatters.ts     # Output formatters
│   ├── utils.ts          # Shared utilities
│   └── types.ts          # TypeScript types
├── public/
│   └── index.html        # Web UI
├── server.mjs            # Web server
├── start.sh              # Quick start script
└── package.json

License

MIT