Aspect Data Sync

A CLI tool to sync files from external services (via rclone) to the Aspect platform.

Features

• Rclone Integration: Sync files from any rclone-supported remote (Dropbox, Google Drive, S3, etc.)
• Directory Structure Preservation: Maintains folder hierarchy from source to destination
• Idempotent Syncs: Safe to re-run - automatically skips already-uploaded files and reuses existing directories
• Multipart Upload: Efficient chunked uploads with retry logic (20MB chunks)
• Dynamic CLI Progress: Real-time upload progress display with in-place updates
• Concurrent Uploads: Configurable parallel chunk uploads for optimal performance
• Session Isolation: Auto-generated nanosecond-precision sessions for safe concurrent syncs
• Automatic Cleanup: Removes local files after successful upload
• Retry Logic: Automatic retry for failed uploads (part-level and file-level)

Prerequisites

1. Node.js 22+ installed
2. rclone installed and configured
   • Install: https://rclone.org/install/
   • Configure your remote: rclone config
3. Aspect API Key for authentication

Configuring Rclone

Before using this tool, you must configure rclone with your cloud storage provider.

Initial Setup

1. Run the rclone configuration wizard:

   rclone config

2. Choose n for "New remote"

3. Give it a name (e.g., dropbox, gdrive, my-s3)

   • This name is what you'll use for the --remote parameter
   • Use only letters, numbers, hyphens, and underscores
   • Don't include special characters or colons

4. Select your storage provider from the list (Dropbox, Google Drive, S3, etc.)

5. Follow the prompts to authenticate with your cloud provider

Verify Configuration

Check your configured remotes:

rclone listremotes

Output example:

dropbox:
gdrive:
my-s3:

Important: The remote names are shown with a colon (:) but you use them without the colon in the --remote parameter.

Examples:

• If you see dropbox: → use --remote dropbox
• If you see my-dropbox: → use --remote my-dropbox
• If you see gdrive: → use --remote gdrive

Test Your Remote

Before syncing, test that your remote works:

# List files in the root of your remote
rclone ls dropbox:

# List files in a specific folder
rclone ls dropbox:my-folder

This should list files in your cloud storage. If you see files listed, your remote is configured correctly!

Installation

Install globally via npm:

npm install -g aspect-sync

Usage

Basic Command

If installed globally:

aspect-sync \
  --remote dropbox \
  --path /my-videos \
  --directory-id <aspect-directory-id> \
  --project-id <aspect-project-id> \
  --api-key <your-api-key>

Or with npx (no installation):

npx aspect-sync \
  --remote dropbox \
  --path /my-videos \
  --directory-id <aspect-directory-id> \
  --project-id <aspect-project-id> \
  --api-key <your-api-key>

Command Options

| Option | Description | Required | Default | |--------|-------------|----------|---------| | --remote <name> | rclone remote name from your rclone config (without colon). Use rclone listremotes to see your configured remotes. | Yes | - | | --path <path> | Remote path to sync from | Yes | - | | --directory-id <id> | Aspect directory ID to upload to | Yes | - | | --project-id <id> | Aspect project ID | Yes | - | | --api-key <key> | Aspect API key (or set ASPECT_API_KEY env var) | Yes | - | | --api-url <url> | Aspect API URL (or set ASPECT_API_URL env var) | No | https://api.aspect.inc | | --concurrent <number> | Max concurrent chunk uploads | No | 16 | | --batch-size <size> | Run the migration in download/upload batches (e.g., 500GB, 1TB) so data is uploaded as soon as each batch finishes downloading. | No | Disabled | | --keep-local | Keep local files after upload (for debugging) | No | false | | --check | Run in check-only mode (list remote files and compare with Aspect without downloading/uploading) | No | false | | --temp-dir <path> | Base temporary directory for synced files | No | ~/.aspect/sync | | --session <name> | Session name for isolation. If not provided, auto-generates timestamp with nanosecond precision (YYYY-MM-DD_HHh-MMm-SSs-NNNNNNNNNns). Creates subdirectory in temp-dir. | No | Auto-generated | | --rclone-extra <arg> | Additional rclone argument appended to the sync command. Repeat the flag to pass multiple values. | No | Not set |

Repeat --rclone-extra for each passthrough flag you need. Example: --rclone-extra --drive-shared-with-me.

Environment Variables

You can set API credentials via environment variables:

export ASPECT_API_KEY=your-api-key
export ASPECT_API_URL=https://api.aspect.inc

aspect-sync \
  --remote dropbox \
  --path /my-videos \
  --directory-id <directory-id> \
  --project-id <project-id>

Examples

Sync from Dropbox

aspect-sync \
  --remote dropbox \
  --path /Camera Uploads \
  --directory-id abc-123-def \
  --project-id xyz-789-uvw \
  --api-key your-api-key

Sync from Google Drive with Custom Concurrency

aspect-sync \
  --remote gdrive \
  --path /Videos/2024 \
  --directory-id abc-123-def \
  --project-id xyz-789-uvw \
  --api-key your-api-key \
  --concurrent 8 \
  --rclone-extra --drive-shared-with-me

Sync from S3 Bucket

aspect-sync \
  --remote s3 \
  --path mybucket/videos \
  --directory-id abc-123-def \
  --project-id xyz-789-uvw \
  --api-key your-api-key

Concurrent Syncs

Multiple syncs can run simultaneously using sessions for isolation. Each session gets its own subdirectory.

Auto-generated sessions (default):

Each sync automatically gets a unique nanosecond-precision timestamp, allowing unlimited concurrent syncs without conflicts.

# Terminal 1
aspect-sync --remote dropbox --path /project-a --directory-id abc-123 --project-id p1 --api-key key
# Session: 2025-10-21_20h-15m-30s-487123456ns

# Terminal 2
aspect-sync --remote dropbox --path /project-b --directory-id def-456 --project-id p2 --api-key key
# Session: 2025-10-21_20h-15m-30s-487945821ns

# Terminal 3
aspect-sync --remote gdrive --path /project-c --directory-id ghi-789 --project-id p3 --api-key key
# Session: 2025-10-21_20h-15m-30s-488201937ns

Custom sessions:

# Terminal 1
aspect-sync --remote dropbox --path /videos --directory-id abc-123 --project-id p1 --session prod-sync --api-key key
# Session: prod-sync

# Terminal 2
aspect-sync --remote gdrive --path /images --directory-id def-456 --project-id p2 --session dev-sync --api-key key
# Session: dev-sync

Finding your temp files:

Sessions are stored in ~/.aspect/sync/{session-name} by default. The full path is shown in the logs at startup.

How It Works

1. Session Creation: Creates isolated session directory with nanosecond-precision timestamp
2. Rclone Sync: Downloads files from the remote to the session directory
3. Directory Discovery: Fetches existing directory tree from Aspect to avoid duplicates
4. Directory Creation: Creates only missing directories, reuses existing ones
5. Asset Deduplication: Checks if each asset already exists before uploading
6. File Upload: Uploads new assets using multipart upload (20MB chunks)
7. Progress Display: Shows real-time upload progress in the CLI
8. Cleanup: Deletes local files after successful upload (unless --keep-local is set)

Idempotent Behavior

The syncer is fully idempotent - you can safely re-run the same sync multiple times:

✅ Existing directories are reused - The tool fetches the directory tree first and only creates missing directories

✅ Existing assets are skipped - Before uploading, checks if an asset with the same name, size, and location already exists

• If found and fully uploaded: Skips upload
• If found but incomplete: Deletes incomplete asset and re-uploads
• If not found: Proceeds with upload

✅ Clean re-runs - If a sync fails halfway:

# First run - uploads 50 of 100 files then crashes
aspect-sync --remote dropbox --path /videos --directory-id abc --project-id xyz --api-key key

# Re-run - skips the 50 uploaded files, uploads remaining 50
aspect-sync --remote dropbox --path /videos --directory-id abc --project-id xyz --api-key key

Output on re-run:

✓ video1.mp4 (100%)
⊘ video2.mp4 (Skipped - already uploaded)  ← Already uploaded
⊘ video3.mp4 (Skipped - already uploaded)  ← Already uploaded
↑ video4.mp4 (45%)                        ← Uploading now

Total: 100 | Success: 1 | Failed: 0 | Skipped: 98 | In Progress: 1 | Queued: 0

CLI Progress Display

The tool shows dynamic, in-place progress updates:

✓ video1.mp4 (100%)
✓ video2.mp4 (100%)
⊘ video3.mp4 (Skipped - already uploaded)
↑ video4.mp4 (47%)
↑ video5.mp4 (23%)

Total: 10 | Success: 2 | Failed: 0 | Skipped: 1 | In Progress: 2 | Queued: 5 | Speed: 45.3 Mb/s | Time remaining: 2m 15s

Status Indicators:

• ✓ = Successfully uploaded
• ✗ = Failed upload
• ⊘ = Skipped (already exists on server)
• ↑ = Currently uploading

Only files that are actively uploading are shown (not the entire queue).

Error Handling

Retry Logic

• Part-level retries: Each 20MB chunk is retried up to 3 times on failure
• File-level retries: Each file upload is retried up to 3 times on failure
• Exponential backoff: Delays increase exponentially between retries

Common Errors

rclone not found:

Error: rclone is not installed or not in PATH

Solution: Install rclone from https://rclone.org/install/

Authentication failed:

Error: 401 Unauthorized

Solution: Check your --api-key is correct

Directory not found:

Error: 404 Not Found - Directory not found

Solution: Verify --directory-id exists and you have access

Troubleshooting

Keep local files for debugging

Use --keep-local to prevent deletion of synced files:

aspect-sync \
  --remote dropbox \
  --path /test \
  --directory-id abc-123 \
  --project-id xyz-789 \
  --api-key your-key \
  --keep-local

Increase concurrency

If you have high bandwidth, increase --concurrent:

aspect-sync \
  --remote dropbox \
  --path /test \
  --directory-id abc-123 \
  --project-id xyz-789 \
  --api-key your-key \
  --concurrent 16

Check rclone configuration

Verify your rclone remote is configured:

rclone listremotes
rclone ls dropbox:

License

MIT