File Deduplication Tool

A comprehensive command-line tool for finding and removing duplicate files using content-based hashing. This NPX package provides extensive parameters and validation for advanced file management scenarios.

Quick Start

# Install and run
npx @tb.p/dd

# Find duplicates in specific directory
npx @tb.p/dd --targets /path/to/directory

# Scan multiple directories
npx @tb.p/dd --targets "dir1;dir2;dir3"

# Dry run to see what would be moved
npx @tb.p/dd --dry-run

Features

🔍 Advanced Detection

• Content-based duplicate detection using BLAKE3 hashing (default)
• Support for multiple hash algorithms (SHA-1, SHA-256, SHA-512, MD5, BLAKE2, BLAKE3)
• Parallel processing for improved performance

📁 Multi-Directory Support

• Scan multiple directories simultaneously
• Semicolon-separated directory lists
• Recursive directory traversal

🛡️ Safety Features

• Dry-run mode to preview operations
• Confirmation prompts before moving files
• Safe file operations (move to subdirectory, never delete)
• System file exclusion

🎯 Advanced Filtering

• File extension filtering (include/exclude)
• Regex pattern matching
• Size-based filtering (min/max size)
• Date-based filtering (newer/older than)

⚡ Duplicate Handling

• Automatic Move: Duplicates are automatically moved to duplicates/ subdirectory preserving path structure (default)
• Dry Run: Use --dry-run to preview what would be moved without making changes

📊 Output Formats

• Table (human-readable)
• JSON (programmatic processing)
• CSV (spreadsheet analysis)
• XML (structured data)
• YAML (configuration-like)

🔧 Advanced Options

• Configurable parallel processing
• Memory limit controls
• SQLite database storage for persistent data
• Caching and resume functionality
• Detailed reporting and statistics
• Comprehensive validation
• Minimal output by default with verbose mode for detailed progress

Installation

NPX (Recommended)

npx @tb.p/dd

Global Installation

npm install -g @tb.p/dd

Basic Usage

Find Duplicates

# Current directory
npx @tb.p/dd

# Specific directory
npx @tb.p/dd --targets /path/to/photos

# Multiple directories
npx @tb.p/dd --targets "C:\Photos;D:\Backup;E:\Archive"

Duplicate Handling

# Move duplicates to duplicates/ subdirectory (automatic)
npx @tb.p/dd

# Move without confirmation (force)
npx @tb.p/dd --force

# Move with custom duplicates directory
npx @tb.p/dd --target-dir ./duplicates

Advanced Filtering

# Only image files larger than 1MB
npx @tb.p/dd --extensions jpg,png,gif --min-size 1MB

# Exclude system and hidden files
npx @tb.p/dd --exclude-system --exclude-hidden

# Files modified in last 7 days
npx @tb.p/dd --newer-than 7d

Requirements

• Node.js 14.0 or higher
• Sufficient disk space for temporary files
• Read access to source directories
• Write access to target directories (for move/copy actions)

License

MIT

Contributing

Contributions are welcome! Please read the project specification and validation rules before submitting pull requests.

Support

For issues and questions, please refer to the documentation or create an issue in the project repository.