Scraper CLI Tool

A beautiful command-line tool that scrapes chapters from URLs and saves them as PDFs. Works on any machine with Node.js!

Features

• 🎨 Beautiful CLI interface using @clack/prompts
• 📚 Scrapes multiple chapters from a URL pattern
• 📄 Generates well-formatted PDFs
• ⚡ Progress tracking for each chapter
• 🛡️ Error handling and validation
• 📁 Flexible output path configuration
• 🖥️ Cross-platform - Works on Windows, Mac, and Linux
• ⏱️ Smart rate limiting (2-second delay between requests)
• 🚀 One-command execution with npx

Requirements

• Node.js 14.0.0 or higher
• npm (comes with Node.js)

Quick Start

Run on any machine with just one command:

npx @venkat/scraper-cli

That's it! No installation needed. The tool will:

1. Download and run automatically
2. Guide you through the setup
3. Scrape chapters and generate PDFs

Alternative Usage

Local Installation (Optional)

If you want to install it locally:

# Install globally
npm install -g @venkat/scraper-cli

# Run from anywhere
scraper

Development Mode

# Install dependencies
npm install

# Run locally
npm start

How it works

1. Welcome Message: The tool greets you with a beautiful intro
2. Output Path: Specify where the PDF should be saved
3. URL & Chapters: Provide the base URL and number of chapters to scrape
4. Progress Tracking: Watch as each chapter is scraped with real-time progress
5. PDF Generation: The tool generates a formatted PDF with table of contents

Example

📚 Chapters Scraper CLI

? Where should the PDF be saved? ./output
? Enter the base URL for the chapters: https://example.com/chapter-1
? How many chapters to scrape? 5
? Ready to scrape 5 chapters from https://example.com/chapter-1? Yes

⠋ Starting chapter scraping...
⠋ Scraping chapter 1/5... (Collecting chapters)
⠋ Scraping chapter 2/5... (Collecting chapters)
⠋ Scraping chapter 3/5... (Collecting chapters)
⠋ Scraping chapter 4/5... (Collecting chapters)
⠋ Scraping chapter 5/5... (Collecting chapters)
✓ Successfully scraped 5 chapters!
⠋ Generating PDF...
✓ PDF generated successfully!

🎉 Your PDF has been saved to: ./output/scraped-book-2025-01-15.pdf

Cross-Platform Support

Windows

npx @venkat/scraper-cli

Mac/Linux

npx @venkat/scraper-cli

Path Examples

• Windows: C:\Users\YourName\Documents\output
• Mac/Linux: ./output or /home/user/books
• Relative: ./output/my-book

Project Structure

scraper/
├── index.js              # Main CLI entry point
├── utils/
│   └── scraper.js        # Scraping and PDF generation utilities
├── package.json          # Dependencies and scripts
└── README.md             # This file

Dependencies

• @clack/prompts - Beautiful CLI prompts
• axios - HTTP client for web scraping
• pdfkit - PDF generation
• chalk - Terminal styling
• cheerio - HTML parsing

Error Handling

The tool includes comprehensive error handling:

• Input validation for all user inputs
• Network timeout handling
• Graceful failure for individual chapters
• Path validation and directory creation
• Uncaught exception handling

File Naming

Book titles are automatically sanitized for cross-platform compatibility:

| Original Title | Filename | | ------------------- | -------------------------------- | | My Novel: Part 1 | my-novel-part-1-2025-01-15.pdf | | Book #1 (Special) | book-1-special-2025-01-15.pdf | | The "Best" Story! | the-best-story-2025-01-15.pdf |

Troubleshooting

Common Issues

Issue: "npx: command not found"

• Solution: Install Node.js from https://nodejs.org/
• Make sure to restart your terminal after installation

Issue: "Cannot find module"

• Solution: Run npx @venkat/scraper-cli again - it will download dependencies automatically

Issue: PDF not saving to specified location

• Solution: Make sure you have write permissions to the output directory
• Try using an absolute path

Issue: Slow scraping or getting blocked

• Solution: The tool has a built-in 2-second delay between requests
• Don't reduce this delay or you might get IP banned

Performance Tips

For Best Results:

1. Stable Internet: Use a stable internet connection
2. VPN: Consider using a VPN if you're scraping many chapters
3. Batch Size: Scrape in batches (e.g., 10-20 chapters at a time)
4. Timing: Avoid peak hours when websites are busy

License

ISC

Happy Scraping! 📚✨

Just run npx @venkat/scraper-cli and start scraping!