@push.rocks/smartmigration

A unified migration runner for MongoDB (via @lossless.org/client/nosqldb) and S3 (via @lossless.org/client/objectstorage) — designed to be invoked at SaaS app startup.

Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit community.foss.global/. This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a code.foss.global/ account to submit Pull Requests directly.

Installation

pnpm add @push.rocks/smartmigration
pnpm add @lossless.org/client @aws-sdk/client-s3

Both are optional peer dependencies, and both are optional in the package-manager sense only — nothing installs them for you.

@lossless.org/client covers both resources: @lossless.org/client/nosqldb for the database and @lossless.org/client/objectstorage for the bucket. Install it when your migrations touch either. The database handed to db is the SmartdataDb from @lossless.org/client/nosqldb (a NoSqlConnection obtained from LosslessOrgClient.nosqldb() is one, too). The bucket handed to bucket is the Bucket from @lossless.org/client/objectstorage, opened through a SmartBucket (an ObjectStorageConnection obtained from LosslessOrgClient.objectstorage() is one, too). The supported peer range is >=1.4.1 <2.0.0 (the client's breaking line is its major, so the range widens deliberately at each client major); development and the repository lockfile use 1.4.1.

@aws-sdk/client-s3 is the S3 driver the objectstorage family runs on, and the client declares it an optional peer of its own. Install it even if you only migrate MongoDB: the client's objectstorage type declarations import the SDK at module scope, and this package's declarations reach them through IMigrationContext.bucket, so a document-only consumer that leaves it out fails to type-check with an unresolved @aws-sdk/client-s3 rather than at runtime. The client's next major cuts that reach, and this peer goes with it.

What is smartmigration?

smartmigration is the missing piece for SaaS apps in the push.rocks ecosystem: a deterministic, idempotent way to evolve persistent data in lockstep with code releases. You define a chain of migration steps with from/to semver versions, hand the runner your nosqldb SmartdataDb and/or Bucket, and call run() from your app's startup path. The runner figures out which steps need to execute, runs them sequentially, and stamps progress into a ledger so subsequent boots are fast no-ops.

Key features

| Feature | Description | |---|---| | Builder-style API | Fluent step definition: step('id').from('1.0.0').to('1.1.0').up(handler) | | Unified mongo + S3 | A single semver represents your combined data version. One step can touch both. | | Drivers exposed via context | ctx.db, ctx.mongo, ctx.bucket, ctx.s3 — write any operation you can write directly | | Idempotent | run() is a no-op when data is at targetVersion. Costs one ledger read on the happy path. | | Resumable | Mark a step .resumable() and it gets ctx.checkpoint.read/write/clear for restartable bulk operations; successful runs clear the step checkpoint automatically | | Lockable | Mongo-backed lock uses atomic updates plus TTL heartbeats to serialize concurrent SaaS instances — safe for rolling deploys | | Fresh-install fast path | Configure freshInstallVersion to skip migrations on a brand-new database | | Target bridging | Optionally bridge schema/data version to the app version without hand-written no-op steps | | Reversible | Give a step .down(handler) and a target behind the ledger walks the chain back. A path crossing a step without one fails closed before anything runs | | Deletable history | Declare oldestSupportedVersion and released steps can be deleted: a ledger older than the floor is refused by name instead of being skipped forward | | Dry-run | dryRun: true or .plan() returns the execution plan without writing anything | | Safe operational errors | Unknown handler, dependency, and driver failures cross one fail-closed sanitizer and become SmartMigrationError instances with stable boundary codes |

Downgrades

A run is entirely one direction: the target is either ahead of the ledger or behind it. When targetVersion is behind the current data version, the runner walks the chain backwards and calls each step's .down() handler, newest step first, stamping the ledger to each step's fromVersion as it goes. Steps come back with status reverted.

migration
  .step('lowercase-emails')
  .from('1.0.0')
  .to('1.1.0')
  .down(async (ctx) => {
    // undo just enough that 1.0.0 code is correct again
  })
  .up(async (ctx) => {
    // ...
  });

.down() is optional and must precede .up(), which is terminal. Three rules make a downgrade safe to reach for:

• Irreversible by default. A step without .down() cannot be walked back. If a downgrade path crosses one, the whole run fails with DOWNGRADE_STEP_IRREVERSIBLE before any handler runs, so you never end up half way down a chain. Auto-generated bridge steps do nothing and so need no handler.
• The target must be a step boundary. Rewinding to a version no step starts at fails with TARGET_NOT_REACHABLE rather than guessing which half of a step to undo.
• No skip-forward in reverse. A forward run tolerates a ledger sitting between two steps by re-running an idempotent handler. Reverse has no equivalent — nothing describes how to undo a partially applied version — so that state fails closed.

Two more behaviours worth knowing:

• Bridge mode walks back through the bridge. With targetVersionStrategy: 'bridge' the ledger normally sits above the last step, at the app version. The generated bridge step is reversible by definition (it does nothing), so a rewind crosses it and continues into the real steps.
• Fresh installs never run down handlers. A database that never had a step applied has nothing to undo, so a target below freshInstallVersion is simply stamped.

plan() and dryRun report a downgrade exactly as they report an upgrade, without executing or writing anything.

Quick start

import { SmartdataDb } from '@lossless.org/client/nosqldb';
import { SmartBucket } from '@lossless.org/client/objectstorage';
import { SmartMigration } from '@push.rocks/smartmigration';
import { commitinfo } from './00_commitinfo_data.js';

// 1. set up your resources as you normally would
const db = new SmartdataDb({
  mongoDbUrl: process.env.MONGODB_URL!,
  mongoDbName: 'myapp',
});
await db.init();

const sb = new SmartBucket({
  accessKey: process.env.S3_ACCESSKEY!,
  accessSecret: process.env.S3_SECRETKEY!,
  endpoint: process.env.S3_ENDPOINT!,
  region: 'us-east-1',
});
const bucket = await sb.getBucketByName('myapp-uploads');

// 2. construct the runner with your app's target version
const migration = new SmartMigration({
  targetVersion: commitinfo.version,
  db,
  bucket,
});

// 3. register migrations in execution order, with from/to validating the chain
migration
  .step('lowercase-emails')
    .from('1.0.0').to('1.1.0')
    .description('Lowercase all user emails')
    .up(async (ctx) => {
      await ctx.mongo!.collection('users').updateMany(
        {},
        [{ $set: { email: { $toLower: '$email' } } }],
      );
    })
  .step('reorganize-uploads')
    .from('1.1.0').to('2.0.0')
    .description('Move uploads/ to media/')
    .resumable()
    .up(async (ctx) => {
      const cursor = ctx.bucket!.createCursor('uploads/');
      const startToken = await ctx.checkpoint!.read<string>('cursorToken');
      if (startToken) cursor.setToken(startToken);
      while (cursor.hasMore()) {
        const { keys } = await cursor.next();
        for (const key of keys) {
          await ctx.bucket!.fastMove({
            sourcePath: key,
            destinationPath: 'media/' + key.slice('uploads/'.length),
            overwrite: true,
          });
        }
        await ctx.checkpoint!.write('cursorToken', cursor.getToken());
      }
    });

// 4. run on startup — fast no-op once the data is at targetVersion
const result = await migration.run();
console.log(`smartmigration: ${result.currentVersionBefore ?? 'fresh'} → ${result.currentVersionAfter} (${result.stepsApplied.length} steps in ${result.totalDurationMs}ms)`);

Core concepts

The data version

A single semver string represents the combined state of your mongo and S3 data. It is not the same as your app version (though you usually want them to match): the app version is what's running, the data version is what's on disk. The runner's job is to bring the data version up to the app version.

Steps and the chain

Each migration is a step with:

• A unique id (string) — used as the ledger key
• A from semver — the data version this step expects to start from
• A to semver — the data version this step produces
• An up handler — the actual migration logic
• An optional down handler — reverses the step, making it downgradable
• An optional description, resumable flag

Steps execute in registration order. The runner validates that the chain is contiguous: step[N].to === step[N+1].from. This catches gaps and overlaps at definition time, before any handler runs.

Resume modes

When run() reads the ledger and finds a current version, it computes a plan: the subset of steps needed to advance from currentVersion to targetVersion. Two resume modes are supported:

1. Exact resume — currentVersion === step.fromVersion for some step. The normal case, where the ledger sits exactly at a step's starting point (because the previous step's to was written to the ledger when it completed).

2. Skip-forward resume — currentVersion > step.fromVersion but currentVersion < step.toVersion. The orphan case: the ledger was stamped to an intermediate version that no registered step starts at. This typically happens when an app configures freshInstallVersion: targetVersion across several releases that didn't add any migrations — fresh installs get stamped to whatever commitinfo.version was at install time, not to the last step's to. When a migration is finally added, those installs have a ledger value that doesn't match any step's from.

   In skip-forward mode, the planner picks the first step whose toVersion > currentVersion and runs it (and all subsequent steps) normally. The step's handler is being invoked against data that may already be partially in the target shape, so step handlers must be idempotent (use $set over $inc, check existence before insert, filter-based updateMany over cursor iteration where possible). A log line at INFO level announces when a step runs in skip-forward mode.

If no step's toVersion is greater than currentVersion (the ledger is past the end of the chain), the runner throws TARGET_NOT_REACHABLE.

Target version strategies

By default, smartmigration is strict: the registered chain must contain a real step whose toVersion exactly matches targetVersion. This is best when you maintain an explicit schema/data version.

If your app uses its package version as targetVersion, releases without schema changes can leave the migration chain behind the app version. In that case, enable bridge mode:

const migration = new SmartMigration({
  targetVersion: commitinfo.version,
  db,
  freshInstallVersion: commitinfo.version,
  targetVersionStrategy: 'bridge',
});

Bridge mode appends an internal no-op step from the last registered migration version to targetVersion when the chain ends before the target. It stamps the ledger to the app version without requiring hand-written no-op migrations for every release.

Important behavior:

• Existing real migrations still run first.
• If the ledger is already past the last registered migration but below targetVersion, only the internal bridge runs.
• Future real migrations still work because skip-forward resume runs the first real step whose toVersion is above the ledger version.
• Downgrades and steps that overshoot the target still fail.

Deleting released steps

Steps accumulate. Once every environment has passed a migration, its handler is dead weight — often a large cutover engine nobody wants to keep compiling. Deleting it is safe only if no database can still be behind it, and by default nothing checks that:

skip-forward resume enters the first step whose toVersion is above the ledger. Delete the oldest steps and a database still sitting below the remaining chain is not refused — it silently runs the truncated remainder and ends up stamped at targetVersion as if the deleted work had happened.

oldestSupportedVersion closes that hole. It is the oldest data version this release can still migrate:

const migration = new SmartMigration({
  targetVersion: '32.0.0',
  db,
  freshInstallVersion: '32.0.0',
  oldestSupportedVersion: '30.0.0', // steps below 30.0.0 were deleted in this release
});
migration
  .step('split-tenant-index').from('30.0.0').to('31.0.0').up(async (ctx) => { /* ... */ })
  .step('backfill-usage').from('31.0.0').to('32.0.0').up(async (ctx) => { /* ... */ });

The floor is the start of the chain this release ships — the fromVersion of the oldest step still registered. A ledger below it is refused by name on the first read: before the lock is taken, before anything is written, before any step runs. A ledger at or above it plans exactly as it did without a floor.

The floor judges the ledger, and only the ledger. A resource with no ledger at all has no recorded version to judge: an empty one is bootstrapped as it always was (through freshInstallVersion when set, otherwise to the chain start), and a populated one is assumed to sit at the chain start — which is the floor — and migrates forward from there. Pinning the floor to the chain start is what makes that safe: every version a bootstrap can stamp is at or above the floor, so an interrupted first run is resumed on the next start rather than refused by the release that wrote it.

Deleting steps safely

1. Pick the boundary to keep, and confirm with getCurrentVersion() that every environment's ledger is at or above it.
2. Delete the steps below that boundary. The remaining steps must still form a contiguous chain.
3. Set oldestSupportedVersion to the fromVersion of the oldest remaining step — the boundary you just picked. A floor above it would declare a step this release still ships as unsupported; a floor below it, or inside a step's span, would admit data the remaining chain cannot start from. With targetVersionStrategy: 'bridge' and the chain emptied entirely, the floor is the start of the generated bridge: the version the previous release stamped.
4. Keep the last release that still contains the deleted steps deployable. That is the intermediate release operators are told to run.

The configuration is refused at startup, not at the first unlucky database:

| Code | Cause | |---|---| | INVALID_VERSION | oldestSupportedVersion is not valid semver | | OLDEST_SUPPORTED_VERSION_ABOVE_TARGET | the floor is ahead of targetVersion | | OLDEST_SUPPORTED_VERSION_ABOVE_FRESH_INSTALL | the floor is ahead of freshInstallVersion, so this release would refuse the database it just created | | OLDEST_SUPPORTED_VERSION_NOT_THE_CHAIN_START | the floor is not the fromVersion of the oldest registered step: above it, steps this release still ships are declared unsupported; below it or inside a step's span, data on the floor would be skip-forward resumed — the truncation the floor exists to refuse |

What operators see

A database that is too old fails startup with LEDGER_BELOW_OLDEST_SUPPORTED_VERSION and a message that names the ledger version, the floor, and the way out:

The migration ledger is at "29.2.0", which is below the oldest supported version "30.0.0".
The steps for versions below "30.0.0" are no longer part of this release. Run an intermediate
release that migrates this data to at least "30.0.0", then run this release again.

error.details carries currentVersion (the ledger version that was refused), oldestSupportedVersion and targetVersion.

plan() and dryRun raise the same refusal, and every successful result echoes the configured floor as oldestSupportedVersion (null when unset), so a CI probe can see both.

The ledger

The ledger is the source of truth for "what data version are we at, what steps have been applied, who holds the lock right now." It is persisted in one of two backends:

• mongo (default when db is provided) — stored as a single raw document in the nosqldb SmartdataEasyStore collection. Lock acquisition / renewal uses atomic mongo updates and works safely across multiple SaaS instances. Recommended.
• s3 (default when only bucket is provided) — a single JSON object at <bucket>/.smartmigration/<ledgerName>.json. Lock is best-effort because S3 has no atomic CAS without additional infrastructure; do not use for multi-instance deployments without external coordination.

If you pass both db and bucket, mongo is used.

The Mongo backend talks directly to the raw collection so lock operations can use atomic filters. Its stored shape is:

{
  _id: ObjectId, // assigned by MongoDB
  nameId: `smartmigration:${ledgerName}`,
  data: {
    currentVersion: string | null,
    steps: Record<string, IMigrationLedgerEntry>,
    lock: {
      holder: string | null,
      acquiredAt: string | null,
      expiresAt: string | null,
    },
    checkpoints: Record<string, Record<string, unknown>>,
  },
}

Unlike ordinary documents created through SmartdataDb.createEasyStore(), the migration document is not expected to contain EasyStore model metadata such as ephemeral or lastEdit. Both forms intentionally coexist in SmartdataEasyStore; nameId is unique and the migration key is smartmigration:<ledgerName>.

The migration context

Each up handler receives a IMigrationContext with:

interface IMigrationContext {
  db?: SmartdataDb;            // high-level nosqldb database
  bucket?: Bucket;             // high-level objectstorage Bucket
  mongo?: mongodb.Db;          // raw mongo Db (db.mongoDb)
  s3?: S3Client;               // raw AWS SDK v3 S3Client
  step: { id, fromVersion, toVersion, description?, isResumable };
  log: Smartlog;
  isDryRun: boolean;
  checkpoint?: IMigrationCheckpoint; // only when step is .resumable()
  startSession(): mongodb.ClientSession; // throws if no db
}

Use whichever level fits your migration:

• High-level (ctx.db, ctx.bucket) for working with nosqldb models / objectstorage files
• Raw drivers (ctx.mongo, ctx.s3) for any operation the client's families don't wrap (renameCollection, dropIndex, S3 lifecycle policies, etc.). ctx.s3 is the S3Client behind the bucket, reached through Bucket.getStorageClient(), the escape hatch @lossless.org/client/objectstorage retains
• startSession() for transactional mongo migrations

Error sanitization

Every unknown failure crossing a package-owned operational boundary passes through one synchronous sanitization function. Covered boundaries are ledger initialization/read/write, fresh-install and version planning, lock acquisition/renewal/release, migration run and context setup, and migration step handlers. Without an errorSanitizer, any smartmigration-generated text derived from such a failure uses only this generic record:

{ message: 'An operation failed; error details were redacted.' }

Configure errorSanitizer only when an application has approved, non-sensitive error text to expose:

const migration = new SmartMigration({
  targetVersion: '2.0.0',
  db,
  errorSanitizer: (_error, context) => ({
    message: context.stepId
      ? `Migration step ${context.stepId} failed.`
      : `Migration ${context.phase} operation failed.`,
  }),
});

The callback receives the original unknown throw and an IErrorSanitizerContext. Its phase is one of 'ledger-initialization', 'ledger-read', 'ledger-write', 'planning', 'lock-acquire', 'lock-renew', 'lock-release', 'run-setup', or 'migration-step'; step-related phases also include stepId. It is invoked once for each unknown failure at the first matching boundary. An existing SmartMigrationError is already safe and passes through without another sanitizer call.

The sanitizer output must be a plain synchronous { message: string; stack?: string } record with a non-empty message no longer than ERROR_SANITIZER_MAX_MESSAGE_LENGTH (1,024 code units) and an optional stack no longer than ERROR_SANITIZER_MAX_STACK_LENGTH (8,192 code units). Thrown callbacks, Promises, thenables, accessors, malformed records, and oversized values fail closed to GENERIC_SANITIZED_ERROR.

Smartmigration never reads getters or calls toString() while normalizing an unknown throw. For thrown wrappers, the original value is retained only as the non-enumerable cause of SmartMigrationError; it is excluded from the wrapper message, details, generated stack, logs, persisted ledger records, and JSON serialization. Ordinary package validation and planning errors are already-safe SmartMigrationError instances and retain their specific code and message. This guarantee covers smartmigration-produced output; a migration handler or custom logger that independently emits the raw value remains application-owned.

A lock-release failure is sanitized and logged at warn, but it never changes a successful migration into a failure and never masks an earlier failure. The unreleased lock remains protected by its TTL and can be recovered manually after confirming that no migration runner is active.

Common use cases

Mongo schema rename

migration
  .step('rename-user-email-field')
  .from('1.0.0').to('1.1.0')
  .up(async (ctx) => {
    await ctx.mongo!.collection('users').updateMany(
      { emailAddress: { $exists: true } },
      { $rename: { emailAddress: 'email' } },
    );
  });

Backfilling a derived field with a transaction

migration
  .step('backfill-account-tier')
  .from('1.5.0').to('2.0.0')
  .up(async (ctx) => {
    const session = ctx.startSession();
    try {
      await session.withTransaction(async () => {
        const users = ctx.mongo!.collection('users');
        await users.updateMany(
          { tier: { $exists: false }, plan: 'free' },
          { $set: { tier: 'free' } },
          { session },
        );
        await users.updateMany(
          { tier: { $exists: false }, plan: { $in: ['pro', 'enterprise'] } },
          { $set: { tier: 'paid' } },
          { session },
        );
      });
    } finally {
      await session.endSession();
    }
  });

Resumable S3 reprefix

migration
  .step('move-attachments')
  .from('2.0.0').to('2.1.0')
  .resumable()
  .up(async (ctx) => {
    const cursor = ctx.bucket!.createCursor('attachments/legacy/', { pageSize: 200 });
    const start = await ctx.checkpoint!.read<string>('cursor');
    if (start) cursor.setToken(start);
    while (cursor.hasMore()) {
      const { keys } = await cursor.next();
      for (const key of keys) {
        await ctx.bucket!.fastMove({
          sourcePath: key,
          destinationPath: 'attachments/' + key.slice('attachments/legacy/'.length),
          overwrite: true,
        });
      }
      await ctx.checkpoint!.write('cursor', cursor.getToken());
    }
  });

If the process crashes mid-migration, the next call to run() will resume from the last persisted cursor token. If the step completes successfully, smartmigration clears that step's checkpoint bag automatically.

Mongo + S3 in lockstep

migration
  .step('split-avatars-out-of-users')
  .from('2.1.0').to('2.5.0')
  .up(async (ctx) => {
    // 1. for every user with an inline avatarBytes field, write the bytes to S3
    const users = ctx.mongo!.collection('users');
    const cursor = users.find({ avatarBytes: { $exists: true } });
    for await (const user of cursor) {
      const buf = Buffer.from(user.avatarBytes.buffer);
      await ctx.bucket!.fastPut({
        path: `avatars/${user._id}.bin`,
        contents: buf,
        overwrite: true,
      });
      await users.updateOne(
        { _id: user._id },
        { $set: { avatarKey: `avatars/${user._id}.bin` }, $unset: { avatarBytes: '' } },
      );
    }
  });

Fresh-install fast path

const migration = new SmartMigration({
  targetVersion: '5.0.0',
  db,
  freshInstallVersion: '5.0.0', // brand-new DB jumps straight to 5.0.0
});
// ... register your full chain of steps ...
await migration.run(); // for a fresh DB, runs zero steps and stamps version 5.0.0

A resource is a fresh install when its ledger has no version and it holds no data. Data is judged per backend:

• MongoDB — a document in any collection except views, system.* namespaces and SmartdataEasyStore. In SmartdataEasyStore, a document is data when it is not a ledger document (its nameId does not start with smartmigration:) and its data holds at least one key; the empty document an EasyStore read creates is not.
• S3 — any object outside the .smartmigration/ prefix.

With both db and bucket, the install is fresh only when neither holds data. Collections and indexes are structure, not data: a database that model preparation or a first read has given collections and indexes, or whose documents were all deleted, is still fresh. Detection stops at the first document that is data.

Call run() before preparing models, and before anything writes a document. Steps may rename collections or reshape indexes that preparation would otherwise install first, and a document written before run() — seeded settings, an EasyStore value, a bootstrap record — is data, so it turns an empty database into one that walks the chain from its first step.

Dry run / planning

const planned = await migration.plan();
console.log(`would apply ${planned.stepsSkipped.length} steps:`,
  planned.stepsSkipped.map((s) => `${s.id}(${s.fromVersion}→${s.toVersion})`).join(' → '));
console.log(`predicted version after run: ${planned.currentVersionAfter}`);

// or — same thing, via dryRun option
const m = new SmartMigration({ targetVersion: '2.0.0', db, dryRun: true });
// ...
const result = await m.run(); // returns plan, doesn't write

plan() / dryRun resolve the same fresh-install shortcut that run() would use, so the returned currentVersionAfter is the predicted post-run version. For S3-backed ledgers, planning does not create the .smartmigration/<ledger>.json sidecar.

API reference

new SmartMigration(options: ISmartMigrationOptions)

| Option | Type | Default | Description | |---|---|---|---| | targetVersion | string | — (required) | Semver representing the data version this code expects | | db | SmartdataDb | undefined | Required if any step uses ctx.db/ctx.mongo or for the mongo ledger | | bucket | Bucket | undefined | Required if any step uses ctx.bucket/ctx.s3 or for the S3 ledger | | ledgerName | string | "smartmigration" | Logical name; lets multiple migrations coexist on the same db/bucket | | ledgerBackend | 'mongo' | 's3' | mongo if db, else s3 | Where to persist the ledger | | freshInstallVersion | string | undefined | When no resource holds data, jump straight to this version. See Fresh-install fast path | | oldestSupportedVersion | string | undefined | Oldest data version this release can still migrate; must be the fromVersion of the oldest registered step. See Deleting released steps | | targetVersionStrategy | 'strict' | 'bridge' | 'strict' | Require the chain to end exactly at targetVersion, or auto-bridge to it | | lockWaitMs | number | 60_000 | How long to wait for a stale lock from another instance | | lockTtlMs | number | 600_000 | How long this instance's own lock auto-expires after | | dryRun | boolean | false | If true, run() returns the plan without executing or locking | | logger | Smartlog | module logger | Custom logger; defaults to a Smartlog with a local destination | | errorSanitizer | IErrorSanitizer | generic fail-closed record | Synchronous sanitizer for unknown failures at package-owned operational boundaries |

The constructor throws SmartMigrationError with one of these codes on bad input:

• INVALID_OPTIONS — options object missing
• MISSING_TARGET_VERSION — targetVersion not set
• INVALID_VERSION — targetVersion, freshInstallVersion or oldestSupportedVersion is not a valid semver
• NO_RESOURCES — neither db nor bucket provided
• LEDGER_BACKEND_MISMATCH — explicit ledgerBackend doesn't match the resources you provided
• INVALID_LEDGER_NAME — ledgerName is blank or not a string
• INVALID_LOCK_WAIT_MS — lockWaitMs is not an integer >= 0
• INVALID_LOCK_TTL_MS — lockTtlMs is not an integer >= 1
• INVALID_TARGET_VERSION_STRATEGY — targetVersionStrategy is not strict or bridge
• INVALID_ERROR_SANITIZER — errorSanitizer is present but is not a function
• OLDEST_SUPPORTED_VERSION_ABOVE_TARGET — the floor is ahead of targetVersion
• OLDEST_SUPPORTED_VERSION_ABOVE_FRESH_INSTALL — the floor is ahead of freshInstallVersion

migration.step(id: string).from(v).to(v).[description(t)].[resumable()].up(handler)

The step builder. .from(), .to(), and .up() are required; .description() and .resumable() are optional. .up() is the terminal call: it commits the step to the parent runner and returns the runner so you can chain .step('next').

migration.run(): Promise<IMigrationRunResult>

The startup entry point. Acquires a lock, computes the plan, executes pending steps in order, and releases the lock. Returns:

interface IMigrationRunResult {
  currentVersionBefore: string | null;
  currentVersionAfter: string;
  targetVersion: string;
  oldestSupportedVersion: string | null; // the configured floor, null when unset
  wasUpToDate: boolean;        // true if no steps ran
  wasFreshInstall: boolean;    // true if startup took the fresh-install shortcut
  stepsApplied: IMigrationStepResult[];
  stepsSkipped: IMigrationStepResult[];
  totalDurationMs: number;
}

Throws SmartMigrationError with these codes:

• LEDGER_INITIALIZATION_FAILED — ledger backend initialization failed
• LEDGER_READ_FAILED / LEDGER_WRITE_FAILED — a raw ledger operation failed
• PLANNING_FAILED — an unknown dependency/driver failure occurred while planning; ordinary planning errors retain their specific code
• RUN_SETUP_FAILED — an unknown failure occurred while setting up the run or a step context
• LOCK_ACQUIRE_FAILED — the lock backend failed while attempting acquisition
• LOCK_TIMEOUT — could not acquire lock within lockWaitMs
• LOCK_LOST — the runner lost its lock while a migration was still in flight
• STEP_FAILED — a step's handler threw; the failure is persisted to the ledger
• CHAIN_*, DUPLICATE_STEP_ID, NON_INCREASING_STEP, TARGET_NOT_REACHABLE — chain validation / planning errors
• OLDEST_SUPPORTED_VERSION_NOT_THE_CHAIN_START — the configured floor is not the start of the registered chain; raised before the ledger is read
• LEDGER_BELOW_OLDEST_SUPPORTED_VERSION — the ledger is older than oldestSupportedVersion; raised on the first ledger read, before the lock and before anything is written
• DOWNGRADE_STEP_IRREVERSIBLE — a reverse plan crosses a step with no .down() handler; raised during planning, so nothing has run
• DOWNGRADE_NOT_SUPPORTED, DOWNGRADE_PLAN_INVALID (legacy) — a plan was computed in the wrong direction for the versions given. Neither is reachable through the public API since the runner picks the direction itself; they guard direct VersionResolver use

Lock-release failures do not throw a public error code. They produce a sanitized warning and preserve the run's existing success or failure outcome.

migration.plan(): Promise<IMigrationRunResult>

Same as run() but does not acquire the lock or execute anything. Useful for --dry-run style probes in CI. stepsSkipped are the steps that would run, and currentVersionAfter is the predicted post-run version after fresh-install resolution.

migration.getCurrentVersion(): Promise<string | null>

Returns the current data version from the ledger, or null if the ledger has never been initialized.

Troubleshooting

LOCK_TIMEOUT on every startup

Another instance crashed while holding the lock. Wait for lockTtlMs (default 10 minutes) for the lock to expire, or manually clear the lock field on the ledger document.

For Mongo, inspect SmartdataEasyStore for { nameId: 'smartmigration:<ledgerName>' }. Only after confirming that the recorded holder is no longer running, clear the exact observed holder atomically:

db.SmartdataEasyStore.updateOne(
  {
    nameId: 'smartmigration:smartmigration',
    'data.lock.holder': '<observed-holder-id>',
  },
  {
    $set: {
      'data.lock': { holder: null, acquiredAt: null, expiresAt: null },
    },
  },
)

Replace the second smartmigration with a custom ledgerName when configured. For S3, make the equivalent edit to .smartmigration/<ledgerName>.json. Prefer waiting for expiresAt; manual clearing can allow two runners to overlap if the holder is still active.

LOCK_LOST during a migration

The runner stopped renewing its lock or another instance replaced the lock document while a step was still running. Check for long-running steps, make sure lockTtlMs is comfortably larger than the expected renewal cadence, and investigate other processes touching the same ledger.

CHAIN_GAP at startup

Two adjacent steps have mismatched versions: step[N].to !== step[N+1].from. Steps must form a contiguous chain in registration order. Fix the version on the offending step.

CHAIN_NOT_AT_CURRENT (legacy)

Retained in the error vocabulary for backward compatibility with downstream consumers that previously branched on it, but no longer thrown by computePlan in normal operation. Prior versions of smartmigration required an exact fromVersion === currentVersion match when resolving the plan; the current planner supports skip-forward resume and handles intermediate-version ledger stamps transparently.

TARGET_NOT_REACHABLE

Forward, either (a) a step in the plan upgrades to a version past targetVersion without any step ending exactly at targetVersion, or (b) the ledger's currentVersion is past the end of the registered chain but has not reached targetVersion. Case (a) means the chain has a mid-step that overshoots — add the missing final step or adjust targetVersion. Case (b) means the chain needs a new step extending it toward targetVersion.

In reverse the causes are different, and extending the chain is the wrong response to all of them: the target is not a step boundary (pick a version some step starts at), or the ledger sits between two steps so no step describes how to undo it. With targetVersionStrategy: 'bridge' a ledger above the chain end is walked back through the generated bridge, so that case plans normally.

LEDGER_BELOW_OLDEST_SUPPORTED_VERSION at startup

This database is older than the oldest version this release still carries steps for, because those steps were deleted. Deploy the intermediate release named in the message (the last one that still contains them), let it finish migrating, then deploy this release again. The refusal lands on the first read of the ledger, before the lock is taken and before anything is written, so the database is exactly as it was before startup — the one exception is another instance stamping a sub-floor version while this run waits for the lock, where the refusal comes from the re-read under the lock and the lock document exists. Lowering oldestSupportedVersion is not a fix unless the deleted steps come back with it.

OLDEST_SUPPORTED_VERSION_NOT_THE_CHAIN_START at startup

The configured floor is not the start of the registered chain. Move it to the fromVersion of the oldest remaining step: a floor above that declares a step this release still ships as unsupported, and a floor below it, or inside a step's span, admits data the chain cannot start from. The error message prints the registered chain.

S3-only deployments and concurrent instances

The S3 ledger's lock is best-effort. If you run multiple SaaS instances against the same S3 bucket, use external coordination (e.g. Redis lock, leader election) before calling run(). The mongo backend has no such limitation.

Client peer verification

Start the repository's configured Mongo service, then verify the declared peer floor:

gitzone services start
pnpm run verify:client:peer # one exact version inside the declared peer range

The command takes one exact @lossless.org/client version and refuses anything the declared peer range in package.json does not contain, so the range and the verifier can never drift apart. It creates an isolated temporary copy, installs the selected version from the configured registry without file:, link:, or workspace links, builds the package, type-checks the compatibility probe, and runs it against Mongo. The probe verifies SmartdataDb.mongoDb, raw sessions, the nosqldb EasyStore, the atomic smartmigration ledger document, and coexistence of both document types in SmartdataEasyStore. The temporary copy carries qenv.yml, whose required: list qenv resolves while the test helpers construct their Qenv, and it reads those names from .nogit/env.json when available or from the equivalent service environment variables in CI; it is removed after success, failure, SIGINT, or SIGTERM. Child commands have a 10-minute default timeout, receive forwarded termination signals as one process group, and escalate to SIGKILL after five seconds. Override those bounds with positive integer SMARTMIGRATION_VERIFY_TIMEOUT_MS and SMARTMIGRATION_VERIFY_TERMINATION_GRACE_MS values.

Best practices

1. Prefer idempotent migrations. Even with the lock, machines crash, networks partition, and migrations should be safe to re-run. Use $set over $inc, check existence before insert, etc.
2. Use .resumable() for any step that processes more than a few hundred records. Crashes happen; resumability avoids redoing work.
3. Use transactions for multi-collection mongo migrations via ctx.startSession() + session.withTransaction().
4. Don't mix S3 and Mongo writes inside a single conceptual operation — S3 has no transactions, so design migrations so each S3 write is observably idempotent on its own.
5. Keep the targetVersion linked to your app's package.json version via the autocreated 00_commitinfo_data.ts. That way every release automatically signals the data version it expects.
6. Freeze applied migrations in source control — don't edit a step that has been applied to production data, even if the new version is "more correct." Add a new step instead.
7. Declare oldestSupportedVersion whenever you delete released steps. Without it, a database that never ran the deleted steps is skipped forward in silence. See Deleting released steps.

License and Legal Information

This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the repository license file.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.

Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.

Company Information

Task Venture Capital GmbH Registered at District Court Bremen HRB 35230 HB, Germany

For any legal inquiries or further information, please contact us via email at [email protected].

By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.