@registered/fetch-files

A TypeScript tool that reads CSV files, extracts URLs, and downloads the referenced data to a local directory.

Features

• 📖 CSV Processing: Reads CSV files and extracts URLs from any column
• 🔗 URL Validation: Validates URLs before attempting to download
• ⬇️ Smart Downloads: Downloads files with intelligent filename generation
• 🔄 Retry Logic: Automatic retry with exponential backoff for failed downloads
• 📁 File Management: Creates output directories and handles file conflicts
• 📊 Progress Tracking: Shows download progress and statistics

Installation

Global Installation (Recommended)

npm install -g @registered/fetch-files

Local Installation

npm install @registered/fetch-files

Usage

Global Installation Usage

After installing globally, you can use the fetch-files command from anywhere:

# Basic usage
fetch-files <csv-file-path> [output-directory]

# Examples
fetch-files data/urls.csv
fetch-files data/urls.csv downloads/
fetch-files data/urls.csv --dry-run

# Get help
fetch-files --help

Local Installation Usage

If installed locally, use npx to run the command:

# Basic usage
npx @registered/fetch-files <csv-file-path> [output-directory]

# Examples
npx @registered/fetch-files data/urls.csv
npx @registered/fetch-files data/urls.csv downloads/
npx @registered/fetch-files data/urls.csv --dry-run

# Get help
npx @registered/fetch-files --help

Command Options

• <csv-file-path>: Path to your CSV file (required)
• [output-directory]: Directory to save downloaded files (default: "downloads")
• --dry-run: Preview URLs without downloading
• --help or -h: Show help message

Examples

Download Files from CSV

# Download files to default 'downloads' directory
fetch-files data/urls.csv

# Download files to a custom directory
fetch-files data/urls.csv my-downloads/

# Preview URLs without downloading (dry-run mode)
fetch-files data/urls.csv --dry-run

Sample Output

🚀 Starting CSV URL fetcher...
📁 CSV file: data/urls.csv
📂 Output directory: /path/to/downloads
📖 Reading CSV file...
🔗 Found 5 URLs in CSV
⬇️  Starting downloads...
[1/5] Fetching: https://github.com
✅ Saved: github_com_1b.bin
[2/5] Fetching: https://google.com
✅ Saved: google_com_1b.bin
...
🎉 Download complete!
✅ Successfully downloaded: 4 files
❌ Failed downloads: 1 files
📂 Files saved to: /path/to/downloads

CSV Format

The tool can extract URLs from any column in your CSV file. URLs can be in any format:

name,url,description
GitHub,https://github.com,Code hosting platform
Google,https://google.com,Search engine
Stack Overflow,https://stackoverflow.com,Developer Q&A site

Features

URL Extraction

• Automatically finds URLs in any CSV column
• Supports both HTTP and HTTPS URLs
• Removes duplicate URLs
• Validates URL format

File Download

• Intelligent filename generation based on URL structure
• Handles files without extensions
• Skips existing files to avoid re-downloading
• Supports all file types (images, documents, etc.)
• Consistent target directory resolution (absolute paths)

Error Handling

• Retries failed downloads up to 3 times
• Graceful error handling with detailed messages
• Continues processing even if some downloads fail

Progress Tracking

• Shows download progress for each file
• Displays success/failure statistics
• Provides clear feedback throughout the process

Dry-Run Mode

• Preview URLs found in CSV without downloading
• Useful for validating CSV content before downloading
• Shows numbered list of all URLs found

Error Handling

The application handles various error scenarios:

1. CSV File Not Found: Clear error message with file path
2. Invalid URLs: URLs are validated and invalid ones are skipped
3. Network Errors: Automatic retry with exponential backoff
4. File System Errors: Graceful handling of permission and disk space issues
5. Download Failures: Individual file failures don't stop the entire process

License

ISC License