@maydotinc/s3-syncer

Sync local folders to S3-compatible storage such as AWS S3, Cloudflare R2, and MinIO.

s3-syncer is designed for real app repos and monorepos:

• reusable s3-syncer.json config
• interactive setup and one-off sync flows
• dry runs and validation before mutating remote state
• streaming uploads and hashing for large asset sets
• bounded concurrency and retries for production reliability
• pull support with incremental diffing and dry-run previews

Quick Start

Most teams can start with:

npx @maydotinc/s3-syncer sync

If s3-syncer.json is missing, the CLI offers to:

• create a reusable config, or
• run a one-time sync without writing config

If credentials are missing, it can:

• load them from .env or --env-file, or
• prompt for them interactively

Requirements

• Node.js 20+
• S3-compatible credentials with:
  • ListObjectsV2
  • PutObject
  • GetObject for pull
  • optionally DeleteObject when delete: true

Common Commands

# Interactive setup
npx @maydotinc/s3-syncer init --config-only

# Validate config, env placeholders, credentials, and read access
npx @maydotinc/s3-syncer validate --env

# Preview uploads/deletes without mutating remote state
npx @maydotinc/s3-syncer sync --dry-run --env

# Run an actual sync
npx @maydotinc/s3-syncer sync --env

# Watch local files and rerun sync on changes
npx @maydotinc/s3-syncer sync --watch --env

# Preview a pull without downloading files
npx @maydotinc/s3-syncer pull --dry-run -o ./pulled --env

# Pull only changed/new files
npx @maydotinc/s3-syncer pull -o ./pulled --env

Core Features

Sync

• fingerprints local files
• lists remote objects
• compares local hashes to remote ETags
• uploads changed/new files
• optionally deletes stale remote files
• retries transient failures
• ends with a summary instead of aborting on the first bad file

Pull

• lists remote objects under a target/prefix
• compares remote files against local files in the output directory
• downloads only new or changed files
• skips unchanged files
• supports --dry-run
• honors target exclude rules

Production controls

Each target can define:

• exclude (glob patterns, extension shorthands like .mp4, contains:lqip, re:^drafts/, or object form { extension } / { contains } / { regex } / { glob })
• concurrency
• retry
• maxFileSize
• watch
• envFile
• ${ENV_VAR} placeholders for bucket, endpoint, and credentials

Guides

Start here for the level of depth you need:

• Commands and config reference
• Notifications
• Watch Mode
• Monorepo setup (basic)
• Advanced setup (single target)
• Agent Skills

Agent Skills

This repo includes portable Agent Skills under skills/:

• s3-syncer-integration
• shared-cdn-assets
• s3-syncer-troubleshooting

They package the integration, architecture, and troubleshooting workflows that agents tend to get wrong when they only see a generic README.

Minimal Config Example

{
  "targets": [
    {
      "directory": "public",
      "bucket": "my-assets",
      "region": "auto",
      "endpoint": "https://<account>.r2.cloudflarestorage.com",
      "prefix": "",
      "delete": false,
      "exclude": [
        "**/*.map",
        ".next/cache/**",
        ".mp4",
        "contains:lqip",
        { "extension": "webm" },
        { "regex": "\\.tmp$" }
      ],
      "concurrency": {
        "fingerprint": 8,
        "upload": 4,
        "delete": 8
      },
      "retry": {
        "maxAttempts": 3,
        "baseDelayMs": 250,
        "maxDelayMs": 4000
      },
      "maxFileSize": {
        "warnBytes": 25000000,
        "errorBytes": 100000000
      },
      "watch": {
        "debounceMs": 500
      }
    }
  ],
  "branch": "main",
  "notifications": {
    "slack": false,
    "discord": false
  }
}

Environment Variables

Required for normal sync/pull usage unless target-level credential placeholders are used:

AWS_S3_ACCESS_KEY_ID=your_access_key
AWS_S3_SECRET_ACCESS_KEY=your_secret_key
AWS_S3_ENDPOINT=

Optional:

• SLACK_WEBHOOK_URL
• DISCORD_WEBHOOK_URL

Notes

• sync and pull both use MD5 vs remote ETag for change detection today.
• If remote objects were uploaded outside s3-syncer with multipart uploads or provider-specific ETag behavior, some unchanged files may still be treated as changed.
• Hashing and uploads are streamed, so large files are not fully buffered in memory.

License

MIT