Transfer documents between Firebase projects with support for subcollections, filtering, parallel transfers, and merge mode. Built with Bun for maximum performance.

Features

• Subcollection support - Recursively copy nested collections
• Document filtering - Filter documents with --where clauses
• Exclude patterns - Skip subcollections by name or glob pattern
• Merge mode - Update existing documents instead of overwriting
• Parallel transfers - Copy multiple collections simultaneously
• Clear destination - Optionally delete destination data before transfer
• Sync mode - Delete destination docs not present in source
• Document transform - Transform data during transfer with custom JS/TS functions
• Collection renaming - Rename collections in destination for backups or migrations
• ID modification - Add prefix or suffix to document IDs to avoid conflicts
• Webhook notifications - Send Slack, Discord, or custom webhooks on completion
• Resume transfers - Continue interrupted transfers from saved state
• Interactive mode - Guided setup with prompts for project and collection selection
• Progress bar - Real-time progress with speed (docs/s) and ETA
• Automatic retry - Exponential backoff on network errors
• Dry run mode - Preview changes before applying (enabled by default)
• Flexible config - INI, JSON, or CLI arguments
• Rate limiting - Control transfer speed to avoid quota issues
• Size validation - Skip oversized documents (>1MB)
• JSON output - Machine-readable output for CI/CD pipelines
• Post-transfer verification - Verify document counts after transfer

Installation

With Bun (recommended)

# Global install
bun add -g @fazetitans/fscopy

# Or run directly
bunx @fazetitans/fscopy --help

With npm

npm install -g @fazetitans/fscopy

Prerequisites

1. Bun or Node.js 18+
2. Google Cloud authentication: gcloud auth application-default login
3. Firestore read access on source project, write access on destination

Quick Start

# 1. Generate config file
fscopy --init config.ini

# 2. Edit config.ini with your project IDs and collections

# 3. Preview transfer (dry run)
fscopy -f config.ini

# 4. Execute transfer
fscopy -f config.ini -d false -y

Usage

Basic Transfer

# Using config file
fscopy -f config.ini

# Using CLI arguments
fscopy \
  --source-project my-source \
  --dest-project my-dest \
  -c users orders products

With Subcollections

# Include all subcollections
fscopy -f config.ini -s

# Exclude specific subcollections
fscopy -f config.ini -s --exclude logs --exclude "temp/*"

Filtering Documents

# Single filter
fscopy -f config.ini --where "status == active"

# Multiple filters (AND)
fscopy -f config.ini -w "active == true" -w "createdAt > 2024-01-01"

# Supported operators: ==, !=, <, >, <=, >=

Advanced Options

# Merge mode (update instead of overwrite)
fscopy -f config.ini --merge

# Parallel transfers (3 collections at once)
fscopy -f config.ini --parallel 3

# With logging
fscopy -f config.ini --log transfer.log

# Limit documents per collection
fscopy -f config.ini --limit 100

# Quiet mode (no progress bar)
fscopy -f config.ini -q

# Clear destination before transfer (DESTRUCTIVE)
fscopy -f config.ini --clear

# Sync mode: delete orphan docs in destination
fscopy -f config.ini --delete-missing

# Interactive mode with prompts
fscopy -i

# Transform documents during transfer
fscopy -f config.ini --transform ./transforms/anonymize.ts

# Rename collections in destination
fscopy -f config.ini -r users:users_backup -r orders:orders_2024

# Add prefix to document IDs
fscopy -f config.ini --id-prefix backup_

# Add suffix to document IDs
fscopy -f config.ini --id-suffix _archived

# Send notification to Slack/Discord
fscopy -f config.ini --webhook https://hooks.slack.com/services/...

# Resume an interrupted transfer
fscopy -f config.ini --resume

# Verify document counts after transfer
fscopy -f config.ini --verify

# Rate limit to 100 docs/second (avoid quota issues)
fscopy -f config.ini --rate-limit 100

# Skip documents larger than 1MB
fscopy -f config.ini --skip-oversized

# JSON output for CI/CD pipelines
fscopy -f config.ini --json

Collection Renaming

Rename collections during transfer for backups or migrations:

# Backup with dated collection names
fscopy -f config.ini -r users:users_2024_12_29 -r orders:orders_2024_12_29

# Multiple renames in one command
fscopy -f config.ini --rename-collection users:users_v2 --rename-collection products:catalog

Subcollections are automatically renamed along with their parent collection.

ID Modification

Add prefix or suffix to document IDs to avoid conflicts when merging:

# Add prefix: user123 → backup_user123
fscopy -f config.ini --id-prefix backup_

# Add suffix: user123 → user123_v2
fscopy -f config.ini --id-suffix _v2

# Combine both: user123 → old_user123_archived
fscopy -f config.ini --id-prefix old_ --id-suffix _archived

Document Transform

Transform documents during transfer using a custom function:

# Create a transform file
cat > anonymize.ts << 'EOF'
export function transform(doc, meta) {
    // Anonymize email addresses
    if (doc.email) {
        doc.email = `user_${meta.id}@example.com`;
    }
    // Remove sensitive fields
    delete doc.password;
    delete doc.ssn;
    // Return null to skip the document
    if (doc.deleted) return null;
    return doc;
}
EOF

# Use the transform
fscopy -f config.ini -t ./anonymize.ts

The transform function receives:

• doc - The document data as an object
• meta - Metadata with id (document ID) and path (full document path)

Return the transformed document, or null to skip it.

Security Warning: The --transform option executes arbitrary code from the specified file. Only use transform files from trusted sources. Never run transforms from untrusted or unverified files as they have full access to your system.

Webhook Notifications

Get notified when transfers complete (success or failure):

# Slack webhook
fscopy -f config.ini --webhook https://hooks.slack.com/services/XXX/YYY/ZZZ

# Discord webhook
fscopy -f config.ini --webhook https://discord.com/api/webhooks/123/abc

# Custom webhook (receives raw JSON payload)
fscopy -f config.ini --webhook https://api.example.com/webhook

The webhook receives a POST request with:

• source / destination - Project IDs
• collections - List of transferred collections
• stats - Documents transferred, deleted, errors
• duration - Transfer time in seconds
• dryRun - Whether it was a dry run
• success - Boolean status
• error - Error message (if failed)

Slack and Discord webhooks are automatically formatted with rich messages.

Resume Interrupted Transfers

Large migrations can be resumed if interrupted:

# Start a transfer (state is saved automatically to .fscopy-state.json)
fscopy -f config.ini -d false

# If interrupted (Ctrl+C, network error, etc.), resume from where it left off
fscopy -f config.ini --resume

# Use a custom state file
fscopy -f config.ini --state-file ./my-transfer.state.json
fscopy -f config.ini --resume --state-file ./my-transfer.state.json

The state file tracks:

• Completed document IDs per collection
• Transfer statistics
• Source/destination project validation

State files are automatically deleted on successful completion.

Configuration

INI Format (recommended)

fscopy --init config.ini

[projects]
source = my-source-project
dest = my-dest-project

[transfer]
collections = users, orders, products
includeSubcollections = true
dryRun = true
batchSize = 500
limit = 0

[options]
; where = status == active
; exclude = logs, cache, temp/*
merge = false
parallel = 1
clear = false
deleteMissing = false
; transform = ./transforms/anonymize.ts
; renameCollection = users:users_backup, orders:orders_2024
; idPrefix = backup_
; idSuffix = _v2

JSON Format

fscopy --init config.json

{
    "sourceProject": "my-source-project",
    "destProject": "my-dest-project",
    "collections": ["users", "orders"],
    "includeSubcollections": true,
    "dryRun": true,
    "batchSize": 500,
    "limit": 0,
    "where": ["status == active"],
    "exclude": ["logs", "cache"],
    "merge": false,
    "parallel": 1,
    "clear": false,
    "deleteMissing": false,
    "transform": null,
    "renameCollection": {},
    "idPrefix": null,
    "idSuffix": null
}

CLI Reference

| Option | Alias | Type | Default | Description | | -------------------------- | ----- | ------- | -------------------- | --------------------------------------- | | --init | | string | | Generate config template | | --config | -f | string | | Path to config file | | --source-project | | string | | Source Firebase project | | --dest-project | | string | | Destination project | | --collections | -c | array | | Collections to transfer | | --include-subcollections | -s | boolean | false | Include subcollections | | --where | -w | array | | Filter documents | | --exclude | -x | array | | Exclude subcollections | | --merge | -m | boolean | false | Merge instead of overwrite | | --parallel | -p | number | 1 | Parallel transfers | | --dry-run | -d | boolean | true | Preview without writing | | --batch-size | -b | number | 500 | Documents per batch | | --limit | -l | number | 0 | Limit docs (0 = no limit) | | --retries | | number | 3 | Retries on error | | --log | | string | | Log file path | | --quiet | -q | boolean | false | No progress bar | | --yes | -y | boolean | false | Skip confirmation | | --clear | | boolean | false | Clear destination before transfer | | --delete-missing | | boolean | false | Delete dest docs not in source | | --interactive | -i | boolean | false | Interactive mode with prompts | | --transform | -t | string | | Path to JS/TS transform file | | --rename-collection | -r | array | | Rename collection (source:dest) | | --id-prefix | | string | | Add prefix to document IDs | | --id-suffix | | string | | Add suffix to document IDs | | --webhook | | string | | Webhook URL for notifications | | --resume | | boolean | false | Resume from saved state | | --state-file | | string | .fscopy-state.json | State file path | | --verify | | boolean | false | Verify counts after transfer | | --rate-limit | | number | 0 | Limit docs/second (0 = unlimited) | | --skip-oversized | | boolean | false | Skip documents > 1MB | | --json | | boolean | false | JSON output for CI/CD | | --max-depth | | number | 0 | Max subcollection depth (0 = unlimited) | | --detect-conflicts | | boolean | false | Detect concurrent modifications | | --verify-integrity | | boolean | false | Verify document integrity with hash |

How It Works

1. Authentication - Uses Google Application Default Credentials (ADC)
2. Document counting - Counts total documents for progress bar
3. Batch processing - Transfers documents in configurable batches
4. Retry logic - Automatic retry with exponential backoff on failures
5. Subcollection discovery - Uses listCollections() to find nested data

Security

• Transform files execute arbitrary code - The --transform option uses dynamic imports to load and execute JavaScript/TypeScript files. Only use transform files you have written or thoroughly reviewed. Malicious transform files could access your filesystem, network, or credentials.

• Webhook URLs should use HTTPS - fscopy warns if you use HTTP webhooks (except localhost). Webhook payloads contain project names and transfer statistics that could be sensitive.

• Credentials via ADC - fscopy uses Google Application Default Credentials. Ensure you're authenticated with the correct account before running transfers.

Notes

• Dry run is ON by default - Use -d false for actual transfer
• Documents are overwritten - Use --merge to update instead
• Where filters apply to root only - Subcollections are copied in full
• Exclude patterns support globs - e.g., temp/*, */logs
• Progress bar shows ETA - Based on documents processed
• Clear is destructive - --clear deletes all destination docs before transfer
• Delete-missing syncs - --delete-missing removes orphan docs after transfer
• Transform applies to all - Transform function is applied to both root and subcollection docs
• Same project allowed - Source and destination can be the same project when using --rename-collection or --id-prefix/--id-suffix

Limitations

Firestore Special Types

When reading documents, Firestore sentinel values are resolved to their actual values:

| Sentinel | Behavior | | -------- | -------- | | serverTimestamp() | Resolved to actual Timestamp value | | increment() | Resolved to current numeric value | | arrayUnion() / arrayRemove() | Resolved to current array value |

These sentinels are write-time operations, not persistent values. fscopy transfers the resolved data, which is the expected behavior for data migration.

Document References

DocumentReference fields are transferred as-is. If the reference points to a document in the source project, it will still point to the source after transfer. Consider using --transform to update references if needed.

Collection and Document IDs

fscopy validates IDs according to Firestore rules:

• Cannot be empty
• Cannot be . or ..
• Cannot match __*__ pattern (reserved by Firestore)

Unicode characters, special characters (#, $, [, ]), and forward slashes in nested paths are all supported.

Subcollection Depth

Use --max-depth to limit recursion when copying deeply nested subcollections:

# Copy only first level of subcollections
fscopy -f config.ini -s --max-depth 1

# Copy up to 3 levels deep
fscopy -f config.ini -s --max-depth 3

Development

# Clone and install
git clone https://github.com/fazetitans/fscopy.git
cd fscopy
bun install

# Run locally
bun start -- -f config.ini

# Run tests
bun test

# Type check & lint
bun run type-check
bun run lint

License

MIT