Mongoose State Migrations

Production-ready MongoDB migration tool with state tracking, conflict detection, and proper rollback support.

Features

• ✅ State Tracking: Automatic schema snapshot management
• ✅ Conflict Detection: Git-based conflict resolution
• ✅ Type Safety: Full TypeScript support
• ✅ Lock Mechanism: Prevent concurrent migrations
• ✅ Transaction Support: Auto-detect replica sets
• ✅ Rollback Support: Complete up/down migration system
• ✅ Consistency Checks: Verify database vs code state
• ✅ CLI & API: Both command-line and programmatic interfaces

Installation

npm install mongoose-state-migrations

Quick Start

1. Create Configuration

Create mongoose-migrate.config.js in your project root:

module.exports = {
  mongoUrl: process.env.MONGODB_URL,
  migrationsDir: './migrations',
  modelsPath: './src/lib/database/models/index.js',  // Must export all models
  
  // Optional
  lockTimeout: 600000,        // 10 minutes
  useTransactions: 'auto',    // 'auto' | 'always' | 'never'
  
  // Warnings/reminders
  warnings: {
    autoIndex: true  // Remind to set mongoose autoIndex: false
  }
};

2. Export Your Models

Create src/lib/database/models/index.js:

const User = require('./user.model');
const Post = require('./post.model');

module.exports = {
  User,
  Post
};

3. Create Your First Migration

npx migrate create add_user_email

4. Run Migrations

npx migrate up

CLI Commands

Create Migrations

# Create new migration
migrate create add_user_email

# Create empty migration (for seeders)
migrate create seed_initial_data --empty

# Rename migration
migrate rename old_name new_name

Run Migrations

# Run next pending migration
migrate up

# Run all pending migrations
migrate latest

# Run up to specific migration
migrate up --to=20251015120000_add_email

# Run specific number of migrations
migrate up --steps=3

Rollback Migrations

# Rollback last migration
migrate down

# Rollback to specific migration (keeps it applied)
migrate down --to=20251015120000_add_email

# Rollback specific number of migrations
migrate down --steps=2

# Rollback all migrations
migrate reset --confirm

Status & Verification

# Show migration status
migrate status

# Check consistency
migrate verify

# List all migrations
migrate list

Emergency Commands

# Mark migration as rolled back (doesn't execute down function)
migrate mark-rollback migration_name --force

Migration Structure

JavaScript Migration

/**
 * Migration: Add User Email
 * Created: 2025-10-15T12:00:00.000Z
 * 
 * IMPORTANT USAGE NOTES:
 * 
 * oldModels: Represents schema state BEFORE this migration
 * newModels: Represents schema state AFTER this migration
 * 
 * up() function:
 *   1. Start by querying/modifying using oldModels
 *   2. Apply transformations
 *   3. Use newModels for new structure (indexes, validations)
 * 
 * down() function:
 *   1. Start with newModels (current state)
 *   2. Reverse transformations
 *   3. Restore to oldModels state
 */

module.exports = {
  async up(db, { oldModels, newModels }) {
    // Add email field to existing users
    const users = await oldModels.User.find({});
    for (const user of users) {
      user.email = null;
      await user.save();
    }
    
    // Create index using new model
    await newModels.User.collection.createIndex({ email: 1 });
  },

  async down(db, { oldModels, newModels }) {
    // Remove index
    await newModels.User.collection.dropIndex({ email: 1 });
    
    // Remove email field
    await db.collection('users').updateMany({}, { $unset: { email: "" } });
  }
};

TypeScript Migration

import { Connection } from 'mongoose';

interface MigrationContext {
  oldModels: Record<string, any>;
  newModels: Record<string, any>;
}

const migration = {
  async up(db: Connection, { oldModels, newModels }: MigrationContext): Promise<void> {
    // Implementation here
  },

  async down(db: Connection, { oldModels, newModels }: MigrationContext): Promise<void> {
    // Implementation here
  }
};

export = migration;

Programmatic API

const { createDefaultAPI } = require('mongoose-state-migrations');

async function runMigrations() {
  const api = createDefaultAPI();
  
  try {
    // Check status
    const status = await api.status();
    console.log('Applied:', status.applied.length);
    console.log('Pending:', status.pending.length);
    
    // Run migrations
    await api.latest();
    
    // Verify consistency
    const isValid = await api.verify();
    console.log('Consistent:', isValid);
    
  } finally {
    await api.close();
  }
}

Best Practices

1. Disable AutoIndex

// In your app initialization
mongoose.set('autoIndex', false);

2. Keep Migrations Small

// Good: Single focused change
migrate create add_user_email

// Bad: Multiple unrelated changes
migrate create add_user_email_and_posts_and_comments

3. Test Both Directions

Always test both up() and down() functions before committing.

4. Use Batching for Large Updates

async up(db, { oldModels, newModels }) {
  const batchSize = 1000;
  let skip = 0;
  
  while (true) {
    const users = await oldModels.User.find({}).skip(skip).limit(batchSize);
    if (users.length === 0) break;
    
    for (const user of users) {
      user.newField = 'default';
      await user.save();
    }
    
    skip += batchSize;
  }
}

5. Handle Online Migrations

async up(db, { oldModels, newModels }) {
  // Add new field (safe for online)
  await db.collection('users').updateMany(
    { newField: { $exists: false } },
    { $set: { newField: null } }
  );
  
  // Create index in background (non-blocking)
  await newModels.User.collection.createIndex(
    { newField: 1 },
    { background: true }
  );
}

Conflict Resolution

Git Conflicts

When two developers create migrations simultaneously, git will show conflicts in migrations/changelog.json:

{
  "migrations": [
    {"timestamp": "20251015120000", "name": "add_email", "createdAt": "2025-10-15T12:00:00Z"},
<<<<<<< HEAD
    {"timestamp": "20251015130000", "name": "add_role", "createdAt": "2025-10-15T13:00:00Z"}
=======
    {"timestamp": "20251015125000", "name": "add_status", "createdAt": "2025-10-15T12:50:00Z"}
>>>>>>> feature-branch
  ]
}

Resolution Steps:

1. Rollback conflicting migration:

   git checkout <commit-hash>  # Go to commit with your migration
   migrate down --to=<previous-migration>

2. Accept incoming changes:

   git checkout <your-branch>
   # Resolve changelog.json conflict

3. Rename your migration:

   migrate rename add_status add_status_v2

4. Run migrations:

   migrate latest

Missing Migration Files

If a migration exists in the database but not in code:

ERROR: Migration inconsistency detected!

The following migrations are in the database but missing from code:
  - 20251015120000_add_email.js (ran at 2025-10-15 12:00:00)

RECOMMENDED SOLUTION:
  1. Find the commit where this migration was added:
     git log --all --full-history -- "migrations/20251015120000_add_email.js"
  
  2. Checkout that commit:
     git checkout <commit-hash>
  
  3. Rollback the migration:
     migrate down --to=<previous-migration>
  
  4. Return to current branch:
     git checkout <your-branch>
  
  5. Resolve conflicts and run migrations:
     migrate latest

Configuration Options

| Option | Type | Default | Description | |--------|------|---------|-------------| | mongoUrl | string | - | MongoDB connection string (required) | | migrationsDir | string | - | Directory for migration files (required) | | modelsPath | string | - | Path to models index file (required) | | lockTimeout | number | 600000 | Lock timeout in milliseconds | | useTransactions | string | 'auto' | 'auto' | 'always' | 'never' | | warnings.autoIndex | boolean | true | Show autoIndex warning |

Database Collections

migrations

Tracks applied migrations:

{
  _id: ObjectId,
  name: "20251015120000_add_email",
  timestamp: "20251015120000",
  batch: 1,
  codeHash: "abc123",
  runAt: ISODate,
  rolledBackAt: null
}

migration_lock

Prevents concurrent migrations:

{
  _id: "migration_lock",
  locked: true,
  lockedAt: ISODate,
  lockedBy: "hostname:pid",
  process: "up"
}

Schema Snapshots

Schema state is automatically saved to .migration-state/:

.migration-state/
├── schema-20251015120000.json
├── schema-20251015130000.json
└── schema-latest.json

Each snapshot contains:

{
  "migrationName": "add_email",
  "timestamp": "20251015120000",
  "schema": {
    "collections": {
      "users": {
        "fields": {
          "email": { "type": "String", "required": false }
        },
        "indexes": [
          { "keys": { "email": 1 }, "options": {} }
        ]
      }
    }
  },
  "createdAt": "2025-10-15T12:00:00.000Z"
}

TypeScript Support

The tool automatically detects TypeScript projects and generates .ts migration files when tsconfig.json is present.

Error Handling

Common Errors

1. Configuration Missing

   Configuration file not found. Please create mongoose-migrate.config.js

2. Models Not Found

   Models file must export an object with all models

3. Migration Locked

   Migration is already running (hostname:pid) since 2025-10-15 12:00:00

4. Consistency Failed

   Consistency check failed. Fix issues before running migrations.

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests
5. Submit a pull request

License

MIT License - see LICENSE file for details.

Support

• 📖 Documentation
• 🐛 Issues
• 💬 Discussions