Aspect Data Sync

A CLI tool to sync files from external services (via rclone) to the Aspect platform.

Features

• Rclone integration: Lists and downloads from any rclone-supported remote.
• Aspect-compatible path planning: Preserves source hierarchy while sanitizing names that Aspect rejects.
• Duplicate preservation when supported: Preserves duplicate file and folder names when the rclone provider exposes stable IDs.
• Idempotent reruns: Reuses existing directories and skips assets that are already uploaded.
• Batched migrations: Downloads and uploads large remotes in size-limited batches.
• Progress display: Shows download, upload, skip, and failure progress in the terminal.
• Session isolation: Uses unique session directories so concurrent syncs do not collide.
• Automatic cleanup: Removes local staged files after successful upload unless --keep-local is set.

Prerequisites

1. Node.js 22+ installed
2. rclone installed and configured
   • Install: https://rclone.org/install/
   • Configure your remote: rclone config
3. Aspect API Key for authentication

Configuring Rclone

Before using this tool, you must configure rclone with your cloud storage provider.

Initial Setup

1. Run the rclone configuration wizard:

   rclone config

2. Choose n for "New remote"

3. Give it a name (e.g., dropbox, gdrive, my-s3)

   • This name is what you'll use for the --remote parameter
   • Use only letters, numbers, hyphens, and underscores
   • Don't include special characters or colons

4. Select your storage provider from the list (Dropbox, Google Drive, S3, etc.)

5. Follow the prompts to authenticate with your cloud provider

Verify Configuration

Check your configured remotes:

rclone listremotes

Output example:

dropbox:
gdrive:
my-s3:

Important: The remote names are shown with a colon (:) but you use them without the colon in the --remote parameter.

Examples:

• If you see dropbox:, use --remote dropbox
• If you see my-dropbox:, use --remote my-dropbox
• If you see gdrive:, use --remote gdrive

Test Your Remote

Before syncing, test that your remote works:

# List files in the root of your remote
rclone lsjson dropbox: --recursive

# List files in a specific folder
rclone lsjson dropbox:my-folder --recursive

This should list files in your cloud storage. If you see files listed, your remote is configured correctly!

Installation

Install globally via npm:

npm install -g aspect-sync

Usage

Basic Command

If installed globally:

aspect-sync \
  --remote dropbox \
  --path /my-videos \
  --directory-id <aspect-directory-id> \
  --project-id <aspect-project-id> \
  --api-key <your-api-key>

Or with npx (no installation):

npx aspect-sync \
  --remote dropbox \
  --path /my-videos \
  --directory-id <aspect-directory-id> \
  --project-id <aspect-project-id> \
  --api-key <your-api-key>

Command Options

Required:

| Option | Description | |--------|-------------| | --remote <name> | rclone remote name from your rclone config, without the trailing colon. | | --path <path> | Remote path to sync from. | | --directory-id <id> | Aspect directory ID to upload into. | | --project-id <id> | Aspect project ID. | | --api-key <key> | Aspect API key. Can also be set with ASPECT_API_KEY. |

Aspect API options:

| Option | Description | Default | |--------|-------------|---------| | --api-url <url> | Aspect API URL. Can also be set with ASPECT_API_URL. | https://api.aspect.inc | | --edge-worker-url <url> | Aspect edge worker URL. Can also be set with ASPECT_EDGE_WORKER_URL. | https://mackinac.aspect.inc |

Sync behavior:

| Option | Description | Default | |--------|-------------|---------| | --upload-concurrent <number> | Maximum concurrent chunk uploads to Aspect. | 16 | | --batch-size <size> | Enables batched mode with a maximum batch size, for example 500GB or 1TB. | Disabled | | --keep-local | Keep local staged files after upload for debugging. | false | | --check | Compare remote files against Aspect without downloading or uploading, including missing, extra, incomplete, and size-mismatched files. | false | | --temp-dir <path> | Base temporary directory for sync sessions. | ~/.aspect/sync | | --session <name> | Session name. If omitted, a nanosecond-precision timestamp is generated. | Auto-generated |

rclone tuning:

| Option | Description | Default | |--------|-------------|---------| | --rclone-transfers <number> | Parallel file transfers for bulk rclone sync. Also controls rclone checkers. | 4 | | --rclone-multi-thread-streams <number> | Streams per large file for rclone transfers. | 4 | | --rclone-multi-thread-chunk-size <size> | rclone multi-thread chunk size. | 64MiB | | --rclone-multi-thread-cutoff <size> | Minimum file size for rclone multi-threading. | 64MiB | | --rclone-buffer-size <size> | rclone buffer size per transfer. | 64M | | --rclone-no-mmap | Disable rclone memory-mapped I/O. | false | | --rclone-extra <arg> | Extra rclone argument. Repeat for multiple args. | Not set |

Repeat --rclone-extra for each passthrough value. Example:

aspect-sync \
  --remote gdrive \
  --path /Videos \
  --directory-id <directory-id> \
  --project-id <project-id> \
  --api-key <api-key> \
  --rclone-extra --drive-root-folder-id \
  --rclone-extra <folder-id>

Environment Variables

You can set API credentials via environment variables:

export ASPECT_API_KEY=your-api-key
export ASPECT_API_URL=https://api.aspect.inc
export ASPECT_EDGE_WORKER_URL=https://mackinac.aspect.inc

aspect-sync \
  --remote dropbox \
  --path /my-videos \
  --directory-id <directory-id> \
  --project-id <project-id>

Analytics is enabled only when a PostHog key is available at build time or runtime. Runtime values can be supplied with ASPECT_POSTHOG_KEY/ASPECT_POSTHOG_HOST or POSTHOG_KEY/POSTHOG_HOST. Set ASPECT_DISABLE_POSTHOG=true or POSTHOG_DISABLED=true to disable analytics.

Examples

Sync from Dropbox

aspect-sync \
  --remote dropbox \
  --path /Camera Uploads \
  --directory-id abc-123-def \
  --project-id xyz-789-uvw \
  --api-key your-api-key

Sync from Google Drive with Custom Concurrency

aspect-sync \
  --remote gdrive \
  --path /Videos/2024 \
  --directory-id abc-123-def \
  --project-id xyz-789-uvw \
  --api-key your-api-key \
  --upload-concurrent 8 \
  --rclone-extra --drive-shared-with-me

Sync a Specific Google Drive Folder by ID

aspect-sync \
  --remote gdrive \
  --path / \
  --directory-id abc-123-def \
  --project-id xyz-789-uvw \
  --api-key your-api-key \
  --rclone-extra --drive-root-folder-id \
  --rclone-extra 1truv6wvzgXM3CkGMzuC8c-hrpMk5RAAs

Sync from S3 Bucket

aspect-sync \
  --remote s3 \
  --path mybucket/videos \
  --directory-id abc-123-def \
  --project-id xyz-789-uvw \
  --api-key your-api-key

Concurrent Syncs

Multiple syncs can run simultaneously using sessions for isolation. Each session gets its own subdirectory.

Auto-generated sessions (default):

Each sync automatically gets a unique nanosecond-precision timestamp, allowing unlimited concurrent syncs without conflicts.

# Terminal 1
aspect-sync --remote dropbox --path /project-a --directory-id abc-123 --project-id p1 --api-key key
# Session: 2025-10-21_20h-15m-30s-487123456ns

# Terminal 2
aspect-sync --remote dropbox --path /project-b --directory-id def-456 --project-id p2 --api-key key
# Session: 2025-10-21_20h-15m-30s-487945821ns

# Terminal 3
aspect-sync --remote gdrive --path /project-c --directory-id ghi-789 --project-id p3 --api-key key
# Session: 2025-10-21_20h-15m-30s-488201937ns

Custom sessions:

# Terminal 1
aspect-sync --remote dropbox --path /videos --directory-id abc-123 --project-id p1 --session prod-sync --api-key key
# Session: prod-sync

# Terminal 2
aspect-sync --remote gdrive --path /images --directory-id def-456 --project-id p2 --session dev-sync --api-key key
# Session: dev-sync

Finding your temp files:

Sessions are stored in ~/.aspect/sync/{session-name} by default. The full path is shown in the logs at startup.

How It Works

1. Session creation: Creates an isolated local session directory.
2. Remote listing: Uses rclone lsjson to list files and directories with IDs when the remote exposes them.
3. Path planning: Sanitizes names for Aspect and assigns unique targets for duplicate files and folders.
4. Directory planning: Fetches the existing Aspect directory tree and creates only missing directories.
5. Asset existence check: Skips assets that already exist and are fully uploaded.
6. Download: Downloads unchanged files with bulk rclone sync --files-from; downloads renamed or duplicate files by ID when needed.
7. Upload: Uploads staged files through @aspect/upload-core and the Aspect edge worker.
8. Cleanup: Deletes local staged files after successful upload unless --keep-local is set.

Aspect upload chunk size is controlled by the server. There is no CLI flag for upload chunk size.

Name Sanitization And Duplicates

Aspect enforces cross-platform file and folder names. Sanitization works for all rclone providers because it happens inside aspect-sync after remote listing. During planning, each path segment is sanitized before upload:

• forbidden characters such as :, *, ?, ", <, >, and | are replaced with _
• trailing dots/spaces are trimmed
• reserved Windows names such as CON, AUX, and COM1 are prefixed with _
• names are truncated to the backend byte limit

Some providers, especially Google Drive, allow duplicate names in the same folder. aspect-sync preserves those without changing the source when rclone exposes stable IDs:

• duplicate files use parenthesized suffixes, for example IMG_1891.MOV, IMG_1891 (2).MOV
• duplicate folders use numeric suffixes, for example Shoot, Shoot 2

Provider behavior:

• Providers without duplicate folder ambiguity work normally.
• Duplicate files are preserved when rclone exposes stable file IDs. If identity is required but no file ID is available, the sync fails instead of guessing.
• Duplicate folders are preserved when rclone exposes stable folder IDs and supports querying children by folder ID. Google Drive is the primary supported provider for this flow.
• Duplicate folders without stable folder IDs fail with a clear error instead of silently merging folders.

When a duplicate Google Drive folder is detected, aspect-sync queries each folder by its Drive ID and downloads descendants by file ID so the source folder identity is preserved.

Idempotent Behavior

The syncer is fully idempotent - you can safely re-run the same sync multiple times:

Existing directories are reused - The tool fetches the directory tree first and only creates missing directories.

Existing assets are skipped - Before uploading, checks if an asset with the same name, size, and location already exists.

• If found and fully uploaded: Skips upload
• If found but incomplete: Deletes incomplete asset and re-uploads
• If not found: Proceeds with upload

Clean re-runs - If a sync fails halfway:

# First run - uploads 50 of 100 files then crashes
aspect-sync --remote dropbox --path /videos --directory-id abc --project-id xyz --api-key key

# Re-run - skips the 50 uploaded files, uploads remaining 50
aspect-sync --remote dropbox --path /videos --directory-id abc --project-id xyz --api-key key

Output on re-run:

video1.mp4 (100%)
video2.mp4 (Skipped - already uploaded)
video3.mp4 (Skipped - already uploaded)
video4.mp4 (45%)

Total: 100 | Success: 1 | Failed: 0 | Skipped: 98 | In Progress: 1 | Queued: 0

CLI Progress Display

The tool shows dynamic, in-place progress updates:

video1.mp4 (100%)
video2.mp4 (100%)
video3.mp4 (Skipped - already uploaded)
video4.mp4 (47%)
video5.mp4 (23%)

Total: 10 | Success: 2 | Failed: 0 | Skipped: 1 | In Progress: 2 | Queued: 5 | Speed: 45.3 Mbps | Time remaining: 2m 15s

Only files that are actively uploading are shown (not the entire queue).

Error Handling

Retry Logic

• Chunk retries: Upload chunks are retried on transient failures.
• File retries: Failed files can be retried by rerunning the sync.
• Exponential backoff: Delays increase exponentially between retries

Common Errors

rclone not found:

Error: rclone is not installed or not in PATH

Solution: Install rclone from https://rclone.org/install/

Authentication failed:

Error: 401 Unauthorized

Solution: Check your --api-key is correct

Directory not found:

Error: 404 Not Found - Directory not found

Solution: Verify --directory-id exists and you have access

Troubleshooting

Keep local files for debugging

Use --keep-local to prevent deletion of synced files:

aspect-sync \
  --remote dropbox \
  --path /test \
  --directory-id abc-123 \
  --project-id xyz-789 \
  --api-key your-key \
  --keep-local

Increase concurrency

If you have high bandwidth, increase --upload-concurrent:

aspect-sync \
  --remote dropbox \
  --path /test \
  --directory-id abc-123 \
  --project-id xyz-789 \
  --api-key your-key \
  --upload-concurrent 16

Check rclone configuration

Verify your rclone remote is configured:

rclone listremotes
rclone lsjson dropbox: --recursive

License

MIT