S3 CLI JS

A TypeScript-based npm package that replaces AWS CLI for S3 operations using presigned URLs for enhanced performance.

Features

• Fast file operations: Uses presigned URLs + direct HTTP requests instead of AWS SDK methods
• Concurrent operations: Supports parallel uploads/downloads for improved performance
• Advanced sync: Intelligent sync with timestamp comparison, size checking, and conflict resolution
• Complete S3 functionality: Supports all major S3 operations (ls, cp, mv, rm, sync, mb, rb)
• TypeScript: Fully typed for better development experience
• CLI interface: Can be run directly with npx
• Progress indicators: Shows progress for file uploads/downloads with overall sync progress
• Flexible configuration: Supports AWS credentials from environment variables or AWS credentials file

Installation

Global installation

npm install -g s3-cli-js

Run with npx (no installation required)

npx s3-cli-js --help

Configuration

The tool uses standard AWS credential configuration methods:

Environment Variables

export AWS_ACCESS_KEY_ID=your_access_key
export AWS_SECRET_ACCESS_KEY=your_secret_key
export AWS_DEFAULT_REGION=us-east-1

# custom endpoint for custom S3 implementations
export AWS_ENDPOINT_URL=https://xxx.r2.cloudflarestorage.com

AWS Credentials File

Create ~/.aws/credentials:

[default]
aws_access_key_id = your_access_key
aws_secret_access_key = your_secret_key

[profile]
region = us-east-1

Usage

List buckets

s3-cli ls

List objects in a bucket

s3-cli ls s3://my-bucket
s3-cli ls s3://my-bucket/prefix --recursive

Upload files

# Upload single file
s3-cli cp file.txt s3://my-bucket/

# Upload directory recursively
s3-cli cp ./local-dir s3://my-bucket/remote-dir --recursive

# Upload with custom concurrency (default: 5)
s3-cli cp ./local-dir s3://my-bucket/remote-dir --recursive --concurrency 10

Download files

# Download single file
s3-cli cp s3://my-bucket/file.txt ./

# Download directory recursively
s3-cli cp s3://my-bucket/remote-dir ./local-dir --recursive

# Download with custom concurrency
s3-cli cp s3://my-bucket/remote-dir ./local-dir --recursive --concurrency 8

Move files

s3-cli mv file.txt s3://my-bucket/
s3-cli mv s3://my-bucket/file.txt ./

Remove objects

# Remove single object
s3-cli rm s3://my-bucket/file.txt

# Remove directory recursively
s3-cli rm s3://my-bucket/directory/ --recursive

Create bucket

s3-cli mb s3://my-new-bucket

Remove bucket

# Remove empty bucket
s3-cli rb s3://my-bucket

# Force remove bucket and all contents
s3-cli rb s3://my-bucket --force

Sync directories (Advanced)

# Basic sync with intelligent timestamp comparison
s3-cli sync ./local-dir s3://my-bucket/remote-dir
s3-cli sync s3://my-bucket/remote-dir ./local-dir

# Sync with high concurrency for large datasets
s3-cli sync ./local-dir s3://my-bucket/remote-dir --concurrency 15

# Sync with delete (remove files not in source)
s3-cli sync ./local-dir s3://my-bucket/remote-dir --delete

# Size-only comparison (faster, ignores timestamps)
s3-cli sync ./local-dir s3://my-bucket/remote-dir --size-only

# Exact timestamp matching (no tolerance)
s3-cli sync ./local-dir s3://my-bucket/remote-dir --exact-timestamps

# Force overwrite all files
s3-cli sync ./local-dir s3://my-bucket/remote-dir --force

# Dry run to see what would be synced
s3-cli sync ./local-dir s3://my-bucket/remote-dir --dry-run

Command Options

Global Options

• --region <region>: Override AWS region
• --profile <profile>: Use specific AWS profile
• --endpoint-url <url>: Use custom S3 endpoint

Common Options

• --recursive, -r: Perform operation recursively
• --dry-run: Show what would be done without actually doing it
• --exclude <patterns>: Exclude files matching patterns
• --include <patterns>: Include only files matching patterns
• --concurrency <number>: Number of concurrent operations (default: 5, applies to cp, mv, sync)

Sync Options

• --delete: Delete files in destination that do not exist in source
• --size-only: Compare files by size only (skip timestamp comparison)
• --exact-timestamps: Require exact timestamp matches (default allows 1s tolerance)
• --force: Force overwrite all files regardless of timestamps
• --checksum: Compare files using checksums (not implemented yet)

List Options

• --human-readable: Display file sizes in human readable format
• --summarize: Display summary information

Bucket Options

• --force: Force operation (for rb command)

Examples

# List all buckets
npx s3-cli-js ls

# List objects with human readable sizes
npx s3-cli-js ls s3://my-bucket --human-readable --summarize

# Upload with exclusions
npx s3-cli-js cp ./src s3://my-bucket/src --recursive --exclude "*.log" "*.tmp"

# Dry run to see what would be copied
npx s3-cli-js cp ./data s3://my-bucket/data --recursive --dry-run

# Sync with custom endpoint
npx s3-cli-js sync ./website s3://my-bucket --endpoint-url https://s3.example.com

# High-performance sync with increased concurrency
npx s3-cli-js sync ./large-dataset s3://my-bucket/data --concurrency 20

# Advanced sync with delete and size-only comparison
npx s3-cli-js sync ./website s3://my-bucket/site --delete --size-only

# Exact timestamp sync for critical data
npx s3-cli-js sync ./backup s3://my-bucket/backup --exact-timestamps

Development

Setup

git clone <repository>
cd s3-cli-js
npm install

Build

npm run build

Test

npm test

Lint

npm run lint

Architecture

This tool is designed for performance and uses:

• Presigned URLs: Generated using AWS SDK for secure, time-limited access
• Direct HTTP requests: Uses axios for file uploads/downloads instead of AWS SDK streaming
• TypeScript: Provides type safety and better development experience
• Modular design: Each command is implemented as a separate module

Sync Algorithm

The advanced sync feature uses intelligent comparison logic:

File Comparison Strategy

1. Size Comparison: Files with different sizes are always synced
2. Timestamp Comparison: By default, allows 1-second tolerance for timestamp differences
3. Exact Timestamps: With --exact-timestamps, requires exact timestamp matches
4. Size-Only Mode: With --size-only, skips timestamp comparison entirely
5. Force Mode: With --force, overwrites all files regardless of timestamps

Sync Process

1. Discovery: Lists all files in source and destination
2. Comparison: Compares each file using the selected strategy
3. Planning: Creates a sync plan showing what will be uploaded/downloaded/deleted
4. Execution: Performs operations concurrently with progress tracking
5. Summary: Reports results including errors and statistics

Conflict Resolution

• Newer Source: Source file is newer → Upload/Download
• Same Size & Timestamp: Files are identical → Skip
• Force Mode: Always overwrite destination → Upload/Download
• Delete Mode: Files only in destination → Delete (with --delete flag)

Performance Benefits

By using presigned URLs + direct HTTP requests instead of AWS SDK methods:

• Faster upload/download speeds
• Better progress tracking
• Reduced memory usage for large files
• More control over HTTP request configuration
• Concurrent operations for improved throughput on large datasets
• Intelligent sync with advanced file comparison strategies

License

MIT