SuperSync

A Node.js tool that watches a local directory and automatically syncs changes to a remote SSH destination using rsync.

Features

• Real-time file watching and syncing
• Efficient rsync-based file transfer
• Support for SSH/rsync remote destinations
• Configurable SSH port
• Configuration file support with named targets
• Customizable ignore patterns with .gitignore support
• Parallel transfer control
• Verbose logging mode
• Ignores common unnecessary files (dot files, node_modules, logs)
• Customizable hooks for file events and sync operations

Installation

npm install -g supersync

Or install locally in your project:

npm install supersync

Usage

Basic usage:

supersync --target user@host:/path/to/remote/dir

With custom SSH port:

supersync --target user@host:/path/to/remote/dir --port 2222

With verbose logging:

supersync --target user@host:/path/to/remote/dir --verbose

Limit parallel transfers:

supersync --target user@host:/path/to/remote/dir --max-parallel 5

Delete extraneous files on remote during initial sync:

supersync --target user@host:/path/to/remote/dir --delete

Include .gitignore patterns:

supersync --target user@host:/path/to/remote/dir --use-gitignore

Configuration

You can create a .supersyncrc file in your project root or home directory to configure named targets, ignore patterns, and other settings:

{
  "targets": {
    "prod": "user@production:/var/www/app",
    "staging": "user@staging:/var/www/app"
  },
  "ignore": [
    "/(^|[\/\\])\../",
    "/node_modules/",
    "**/*.log",
    "dist/*",
    "*.tmp"
  ],
  "maxParallel": 10,
  "useGitignore": true,
  "muteSyncs": false,
  "hooks": {
    "beforeSync": "echo 'Starting sync'",
    "afterSync": {
      "local": "echo 'Local sync complete'",
      "remote": "echo 'Remote sync complete'"
    }
  }
}

Then use named targets:

supersync --target prod

Hooks

The tool supports hooks that are executed at specific points during the synchronization process. Hooks can be used to run commands both locally and on the remote server.

Available Hooks

• beforeSync: Executed before the initial sync starts
• afterSync: Executed after the initial sync completes
• beforeFileAdd: Executed before a new file is synced
• afterFileAdd: Executed after a new file is synced
• beforeFileChange: Executed before a modified file is synced
• afterFileChange: Executed after a modified file is synced
• beforeFileDelete: Executed before a file is deleted (if implemented)
• afterFileDelete: Executed after a file is deleted (if implemented)
• beforeExit: Executed before the program exits

Hook Configuration

Hooks can be defined in three ways:

1. Simple string command (runs locally):

{
  "hooks": {
    "beforeSync": "echo 'Starting sync'"
  }
}

2. Object with local/remote commands:

{
  "hooks": {
    "afterSync": {
      "local": "echo 'Local sync complete'",
      "remote": "pm2 restart app"
    }
  }
}

3. Array of commands (executed in sequence):

{
  "hooks": {
    "beforeFileAdd": [
      "echo 'Adding file'",
      { "local": "npm run lint ${filepath}" },
      { "remote": "mkdir -p $(dirname ${filepath})" }
    ]
  }
}

Command Execution in Hooks

When using commands in hooks, you have several options for command execution:

1. Shell Commands:

{
  "hooks": {
    "beforeSync": "npm run build && echo 'Build complete'",
    "afterSync": {
      "local": "tar czf backup.tar.gz dist/",
      "remote": "systemctl restart nginx"
    }
  }
}

2. Multiple Commands with Pipes:

{
  "hooks": {
    "beforeFileAdd": "cat ${filepath} | grep 'TODO' || true",
    "afterFileChange": {
      "remote": "find /var/cache -type f | xargs rm -f"
    }
  }
}

3. Environment Variables:

{
  "hooks": {
    "beforeSync": "NODE_ENV=production npm run build",
    "afterSync": {
      "remote": "export APP_ENV=prod && pm2 reload app"
    }
  }
}

4. Command Substitution:

{
  "hooks": {
    "afterFileAdd": {
      "local": "echo \"File added at $(date)\" >> sync.log",
      "remote": "echo \"${filepath} deployed to $(hostname)\" >> /var/log/deploy.log"
    }
  }
}

5. Conditional Execution:

{
  "hooks": {
    "beforeFileChange": {
      "local": "test -f ${filepath} && npm run lint ${filepath} || echo 'File not found'",
      "remote": "[ -d /var/cache ] && rm -rf /var/cache/*"
    }
  }
}

6. Background Processes:

{
  "hooks": {
    "afterSync": {
      "local": "nohup npm run watch &",
      "remote": "(pm2 reload app && sleep 5 && pm2 reset app) &"
    }
  }
}

Remember:

• Commands in string format run locally by default
• Use local and remote properties to specify execution location
• Commands are executed in a shell environment
• Failed commands will stop the hook execution
• Use || true to ignore command failures
• Background processes should be managed carefully

Available Variables

Hooks can use variables that are replaced with actual values:

• ${filepath}: Path of the file being processed (relative to workspace)
• ${hookName}: Name of the current hook (e.g., "beforeSync")
• ${hookType}: Type of the hook without before/after prefix (e.g., "sync", "fileAdd")
• ${timestamp}: Current timestamp in ISO format

Variables can be used with both ${variable} and $variable syntax.

Example Use Cases

1. Build before sync:

{
  "hooks": {
    "beforeSync": "npm run build"
  }
}

2. Restart application after sync:

{
  "hooks": {
    "afterSync": {
      "remote": "pm2 restart app"
    }
  }
}

3. Validate files before adding:

{
  "hooks": {
    "beforeFileAdd": [
      { "local": "eslint ${filepath}" },
      { "local": "prettier --check ${filepath}" }
    ]
  }
}

4. Create directories and set permissions:

{
  "hooks": {
    "beforeFileAdd": {
      "remote": [
        "mkdir -p $(dirname ${filepath})",
        "chmod 755 $(dirname ${filepath})"
      ]
    }
  }
}

5. Notify on changes:

{
  "hooks": {
    "afterFileChange": {
      "local": "notify-send 'File ${filepath} updated'",
      "remote": "echo '${timestamp}: ${filepath} updated' >> /var/log/sync.log"
    }
  }
}

Options

• --target, -t: Target name from config or SSH URI (required)
• --port, -p: SSH port number (default: 22)
• --max-parallel, -m: Maximum number of parallel sync commands (default: 10)
• --delete, -d: Delete extraneous files from destination during initial sync (default: false)
• --use-gitignore, -g: Include .gitignore patterns in the ignore list (default: true)
• --verbose, -v: Enable verbose logging (default: false)
• --mute-syncs: Mute file sync notifications (default: false)
• --help: Show help

Requirements

• Node.js 12.0 or higher
• SSH access to the remote destination
• Rsync installed on both local and remote machines

Development

To develop and test the package locally:

1. Clone the repository:

git clone <repository-url>
cd supersync

2. Install dependencies:

npm install

3. Link the package globally:

npm link

This creates a symbolic link from the global node_modules to your local development directory.

4. Now you can run the command from anywhere:

supersync --target your-target

To unlink when done:

npm unlink -g supersync

Alternatively, use the dev script:

npm run dev -- --target your-target

How it Works

SuperSync uses Chokidar to watch for file system changes in the current directory and its subdirectories. When a file is added or modified, it automatically syncs the changes to the specified remote destination using rsync.

File Synchronization

The tool uses rsync for all file transfer operations:

1. Initial Sync:

   • Performs a full directory sync when starting
   • Uses rsync's efficient delta-transfer algorithm
   • Only transfers new or modified files
   • Preserves file attributes (permissions, timestamps, etc.)
   • Optionally removes extraneous files (with --delete flag)

2. Individual File Changes:

   • Uses rsync for syncing individual file changes
   • Automatically creates necessary directories
   • Preserves file attributes
   • Compresses data during transfer
   • Shows progress in verbose mode

Parallel Transfers

The tool implements a queue system to control the number of parallel rsync transfers, preventing system overload when many files change simultaneously. By default, it allows up to 10 parallel transfers, but this can be configured through the command line or configuration file.

Ignored Files

The tool uses a comprehensive ignore system combining:

• Built-in default ignores
• Configuration-based ignores from .supersyncrc
• Optional .gitignore patterns (when --use-gitignore is enabled)

This ensures that unnecessary files are not synced while providing flexibility to customize ignore patterns for your specific needs.

JavaScript Hooks

In addition to configuration-based hooks, you can create a supersync.hooks.js file in your project root to define hooks using JavaScript functions. This provides more flexibility and control over the hook behavior.

Hook Function Interface

Each hook function receives two parameters:

1. context - Object containing information about the current operation
2. utils - Utility object for executing commands and accessing configuration

module.exports = {
  async beforeSync(context, utils) {
    // Hook implementation
  }
};

Available Utilities

The utils object provides the following methods:

• utils.local(command) - Execute a local command
• utils.remote(command) - Execute a remote command via SSH
• utils.getContext() - Get the current hook context
• utils.getConfig() - Get the current configuration
• utils.getArgs() - Get command line arguments

Hook Context

The context object contains:

• filepath - Path of the file being processed (for file operations)
• hookName - Name of the current hook (e.g., "beforeSync")
• hookType - Type of the hook without before/after prefix
• timestamp - Current timestamp in ISO format

Example JavaScript Hooks

Create a supersync.hooks.js file:

module.exports = {
  // Build and prepare before sync
  async beforeSync(context, utils) {
    await utils.local('npm run build');
    await utils.remote('pm2 stop app');
  },

  // Restart application after sync
  async afterSync(context, utils) {
    await utils.remote('npm install --production');
    await utils.remote('pm2 restart app');
  },

  // Validate files before adding
  async beforeFileAdd(context, utils) {
    const { filepath } = context;
    if (filepath.endsWith('.js')) {
      await utils.local(`eslint ${filepath}`);
    }
  },

  // Update permissions after adding
  async afterFileAdd(context, utils) {
    const { filepath } = context;
    await utils.remote(`chmod 644 ${filepath}`);
  }
};

Hook Precedence

1. JavaScript hooks in supersync.hooks.js are checked first
2. If no JavaScript hook is found, falls back to configuration hooks
3. If neither exists, the hook is skipped

Example Use Cases

1. Complex Build Process:

async beforeSync(context, utils) {
  // Run tests
  await utils.local('npm test');
  
  // Build for production
  await utils.local('npm run build');
  
  // Clean remote dist
  await utils.remote('rm -rf dist/*');
}

2. Dynamic Cache Management:

async afterFileChange(context, utils) {
  const { filepath } = context;
  
  // Clear specific cache based on file type
  if (filepath.endsWith('.css')) {
    await utils.remote('rm -rf /tmp/css-cache');
  } else if (filepath.endsWith('.js')) {
    await utils.remote('rm -rf /tmp/js-cache');
  }
}

3. File Validation:

async beforeFileAdd(context, utils) {
  const { filepath } = context;
  
  // Run different validations based on file type
  if (filepath.endsWith('.js')) {
    await utils.local(`eslint ${filepath}`);
  } else if (filepath.endsWith('.css')) {
    await utils.local(`stylelint ${filepath}`);
  }
}

4. Application Management:

async afterSync(context, utils) {
  // Get current configuration
  const config = utils.getConfig();
  const env = config.environment || 'production';
  
  // Restart application with specific environment
  await utils.remote(`NODE_ENV=${env} pm2 restart app`);
}

License

MIT