HotUKDeals Scraper

A TypeScript-based web scraper for HotUKDeals using Cheerio for HTML parsing. This application scrapes deal listings from HotUKDeals and saves them as beautifully formatted HTML files.

Features

• TypeScript: Full type safety and modern JavaScript features
• Cheerio: Server-side HTML parsing with jQuery-like syntax
• Interactive CLI: User-friendly command-line interface with input validation
• Beautiful HTML Output: Modern, responsive HTML files with CSS styling
• Rate Limiting: Configurable delays between requests to be respectful to servers
• Error Handling: Robust error handling and detailed logging
• Data Extraction: Extracts deal titles, prices, merchants, links, images, and more

Prerequisites

• Node.js (version 16 or higher)
• npm or yarn

Installation

1. Clone or download the project files
2. Install dependencies:

npm install

Usage

Development Mode

Run the scraper directly with TypeScript:

npm run dev

Production Mode

Build and run the compiled JavaScript:

npm run build
npm start

Interactive Prompts

The application will prompt you for:

1. Start Page Number: The first page to scrape
2. End Page Number: The last page to scrape
3. Delay: Time to wait between requests (in milliseconds)

Example Output

=== HotUKDeals Scraper ===
A TypeScript scraper using Cheerio for HTML parsing

? Enter start page number: 1
? Enter end page number: 3
? Enter delay between requests (in milliseconds): 1000

=== HotUKDeals Scraper (TypeScript + Cheerio) ===

Starting scrape from page 1 to page 3...

Scraping page 1...
✓ Saved page 1 to page_1.html
Scraping page 2...
✓ Saved page 2 to page_2.html
Scraping page 3...
✓ Saved page 3 to page_3.html

=== Scraping Complete ===
Successfully scraped 3 out of 3 pages
HTML files saved in current directory:
- page_1.html
- page_2.html
- page_3.html

Project Structure

├── src/
│   ├── index.ts          # Main entry point with CLI interface
│   ├── scraper.ts        # Core scraping logic
│   └── types.ts          # TypeScript type definitions
├── dist/                 # Compiled JavaScript (generated)
├── package.json          # Project dependencies and scripts
├── tsconfig.json         # TypeScript configuration
└── README.md            # This file

Data Extraction

The scraper extracts the following information from each deal:

• Title: Deal title with link
• Price: Deal price
• Merchant: Store/merchant name
• Link: Direct link to the deal
• Image: Deal image URL
• Description: Deal description
• Temperature: Deal popularity score
• Comment Count: Number of comments

HTML Output

Each scraped page is saved as a standalone HTML file with:

• Modern, responsive CSS styling
• Deal cards with hover effects
• Page statistics
• Timestamp information
• Clean, readable layout

Configuration

TypeScript Configuration

The tsconfig.json file is configured for:

• ES2020 target
• CommonJS modules
• Strict type checking
• Source maps for debugging

Package Scripts

• npm run dev: Run in development mode with ts-node
• npm run build: Compile TypeScript to JavaScript
• npm start: Run the compiled JavaScript
• npm run clean: Remove the dist directory

Dependencies

Production Dependencies

• axios: HTTP client for making requests
• cheerio: HTML parsing library
• inquirer: Interactive command-line interface

Development Dependencies

• typescript: TypeScript compiler
• ts-node: TypeScript execution engine
• @types/node: Node.js type definitions
• @types/inquirer: Inquirer type definitions

Error Handling

The scraper includes comprehensive error handling:

• Network request failures
• Invalid JSON responses
• Missing HTML content
• File system errors
• Input validation

Rate Limiting

To be respectful to the HotUKDeals servers, the scraper includes:

• Configurable delays between requests
• User-Agent headers
• Request timeouts
• Error recovery

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests if applicable
5. Submit a pull request

License

MIT License - feel free to use this project for personal or commercial purposes.

Disclaimer

This scraper is for educational purposes. Please respect the website's terms of service and robots.txt file. Use responsibly and consider implementing proper rate limiting for production use.