@fullstackhouse/open-mercato-data-sync-durable

A drop-in replacement for Open Mercato's core data_sync module that runs sync runs as durable work. Same tables, same REST API, same adapter contract, same events, same UI.

The problem

A data_sync run is one long stream over a cursor. On core's engine:

• a deploy in the middle of a multi-day backfill kills the run
• a worker that dies leaves the run running forever
• one transient error at hour nine fails the whole run, throwing away hour one through eight

Install

yarn add @fullstackhouse/open-mercato-data-sync-durable @fullstackhouse/open-mercato-durable-work pg-boss

Name durable-work explicitly: it is a peer of this package, not a dependency, and yarn does not install peers for you. That is not tidiness — it exports a process-wide registry, so a second copy in the tree would mean job kinds register into one while the worker reads the other, silently. pg-boss is the client for the default transport; swap it for bullmq and ioredis to run on Redis.

The two packages release in lockstep at one version, and this package's peer range is rewritten to ^<that version> on every release — upgrade them together (yarn up '@fullstackhouse/open-mercato-*'). Still 0.x: pin the exact version in production.

src/modules.ts — the whole change is one from:

{ id: 'durable_work', from: '@fullstackhouse/open-mercato-durable-work' },
{ id: 'data_sync',    from: '@fullstackhouse/open-mercato-data-sync-durable' },  // was '@open-mercato/core'

Then yarn generate && yarn db:migrate — a no-op for existing data_sync history, because the migration classes keep core's names — and run yarn mercato durable_work worker.

Reverting is the same line.

What changes

A run gets a lease. It commits its cursor under that lease, hands back at a batch boundary when its slice budget is spent or the process is stopping, and resumes from the committed cursor. The batch that spends the budget is always committed first, however long it took — the hand-back lands at the start of the next one — so every slice makes progress even when a single batch outlasts the whole budget. A transient error retries the slice instead of failing the run. A worker that dies is noticed by the reconciler within about a minute. One live run per integration, entity and direction.

That makes lease.sliceBudgetMs a soft limit: a slice runs until the first batch boundary past it, so it can outlast the budget by a whole batch — size the budget against the slowest batch, not the average one. Two to three times the slowest batch leaves room; a budget that a single batch can already exceed does not.

A slice that never reaches a boundary at all — an adapter stuck on a socket or a lock that never returns — is abandoned after DATA_SYNC_MAX_SLICE_MS. The run is re-driven from its last committed cursor, and parked as poison if it hangs every time, which releases the stream for a new start. The broker's in-flight timeout follows the deadline, so a slice that is merely slow is never redelivered while it runs.

Runs no job carries

A run started through core's start path becomes durable when its queue message is delivered. If that never happens — the process died before the enqueue, the message was lost, or every adoption was refused — the run would sit pending or running with no job, and core's overlap check would refuse every later start of that stream. A sweep on the reconciler's tick settles those runs once they have been quiet for the grace period:

• its job already finished: the run takes the job's outcome
• no job, and quiet for less than the adoption limit: a job is created and the run resumes
• no job, and quiet for longer: the run is failed rather than resumed from a stale cursor
• another live run holds the stream: the run is failed with that reason

A run it fails gets core's finalize tail — progress job, integration health, lifecycle event — and every action is logged as data_sync.jobless_run_<action>.

| Variable | Default | What it does | |---|---|---| | DATA_SYNC_JOBLESS_RUN_GRACE_MS | 900000 | how long a run may sit with no live job before the sweep acts; never below the 15-minute pending TTL | | DATA_SYNC_JOBLESS_RUN_MAX_ADOPT_AGE_MS | 86400000 | a jobless run quiet for longer than this is failed, not adopted | | DATA_SYNC_MAX_SLICE_MS | 1800000 | how long one slice may run before it is abandoned and re-driven; never below twice the 5-minute slice budget |

"Quiet" is measured from the run's updated_at, which every committed batch moves.

What does not change

Your adapters. They import core's deep paths, which this package re-exports, so there is one process-wide adapter registry and one entity class. sync_excel, sync_akeneo and anything you wrote keep working untouched — including runs they start themselves, which are adopted when the queue delivers them.

The UI, the REST API, the events and the ACL features are core's, unchanged.

How it works

It does not fork core's engine. Core's engine already survives being displaced by another worker: it asks a cancellation question at every batch boundary, and it stays silent when a terminal write is refused. This package answers those two questions differently and routes the cursor commit through the lease fence. See docs/adr/0004.

Every seam it relies on is asserted against the installed core in CI, on both the released and the development channel, so an upstream change fails there rather than in your app.

Versioning

One release line per core minor: data-sync-durable 0.7.x mirrors @open-mercato/core 0.7.x. Bump both together. Hosts that have ejected data_sync into @app are out of scope.

MIT. Part of open-mercato-durable.