@recordtimelabel/core

Shared RecordTimeLabel data model, normalization, merge logic, operation reducer, and sync engine.

Scope

This package is intentionally platform-neutral. It must not import React, Firebase, Chrome extension APIs, Electron APIs, or app-specific UI code.

Current consumers:

• D:\Programming\RecordTimeLabel
• D:\Programming\YouTwitchDownloader

During local development an app can consume a sibling checkout with:

"@recordtimelabel/core": "file:../recordtimelabel-core"

For release builds, consume a fixed npm package, git tag, or private registry version so builds do not depend on a sibling folder path. The release target is 0.9.0; verify that its registry tarball and lockfile integrity are available before updating consumers:

"@recordtimelabel/core": "0.9.0"

If this checkout's package.json is ahead of the published version, publish the new package before updating consumers to that version.

The 0.9.0 release preserves authoritative content during moves, adds explicit field replacement with write protocol 5 and field-replacement-v1, and exports bounded lifecycle tombstone index builders for Gateway bootstrap reads. Existing patch, range, lifecycle, and operation-proof contracts remain supported. The index retains every authoritative tombstone and requires a separate controlled rebuild and enablement.

The 0.8.1 patch recovers newer mixed rejection ACKs through bounded catch-up instead of a fresh bootstrap, and keeps a ready subscription alive across transient catch-up failures by retrying the gap in place with backoff. Protocol, schema, and capabilities stay unchanged.

The 0.8.0 minor adds additive record ranges, write protocol 4 / bulk protocol 3, range-preserving legacy snapshot and transfer handling, and queued local admission checks. Existing points are not backfilled. It retains the published 0.7.1 rejection recovery, persistent retry deadlines, and committed-workspace cache. See the range contract below before enabling range writers.

The 0.7.1 patch settles rejection ACKs already covered by the committed baseline without another bootstrap. Transient rejection/gap recovery persists the original batch retry deadline across restarts. It also includes the mainline committed-workspace snapshot cache; protocol, schema, and capabilities stay unchanged. Plan 081 record ranges are not included.

The 0.7.0 artifact adds the Plan 072 operation-deduplication/v3 delivery contract and trusted-intent migration below. It retains the atomic snapshot-replacement admission and /transfer entrypoint introduced by 0.6.22. The already published 0.6.22 artifact remains immutable.

Ordinary cold sync uses bounded operations and per-operation acknowledgements; it never promotes pending work to a snapshot replacement.

Plan 072 delivery contract

New durable intents carry deliveryContract: operation-dedup-v1 and local deliveryContractVersion: 1, committed with the intent. Loading or recovering unversioned prototype/legacy work assigns version 0 and persists syncMeta.operationDedupMigrationVersion: 1; it never upgrades that work to fresh. Only both trusted fields qualify for operationAdmission.freshOperationIds. The fields are excluded from the immutable wire hash. Existing server proofs or verified batch receipts can acknowledge historical work; missing proof remains operation_commit_unknown, with the original intent retained in quarantine.

The new ordinary apply contract is v3; reads/bootstrap/bulk remain v2. Deploy legacy-safe hold and drain all proof-free writers before releasing clients that create trusted intents. Downgrades must never reopen a proof-free writer. The published 0.6.22 does not provide these guarantees. The full cutover checklist is in RecordTimeLabel plans/072-sync-recovery-and-operation-dedup.md.

Public API

• normalizeState(input)
• normalizeRecords(records, options)
• mergeLocalRemote({ localState, remoteState, pendingOps, clientId, now })
• applyOperation(state, operation)
• applyRecordTimeLabelOperation(state, operation)
• createRecordTimeLabelSyncEngine({ storage, cloud, session, client, clock, logger })
• RTL_MAX_SYNC_DRAIN_ROUNDS
• RTL_SYNC_DRAIN_RETRY_DELAY_MS
• RECORD_TIMELABEL_DURABLE_ENGINE_CAPABILITIES
• RECORD_TIMELABEL_CAPABILITY_CLOUD_FAILURE_STATE
• RECORD_TIMELABEL_CLOUD_FAILURE_CLASSES
• RECORD_TIMELABEL_CLOUD_RECOVERY_ACTIONS
• normalizeRecordTimeLabelCloudFailure(input)
• classifyRecordTimeLabelCloudRecovery(input)
• toRecordTimeLabelCloudFailureError(input)
• RECORD_TIMELABEL_PROTOCOL_CAPABILITIES
• toRecordTimeLabelWireOperation(operation)
• buildRecordTimeLabelRequestId(namespace, operations)
• createRecordTimeLabelTransportFailureResults(operations, failure, options)
• normalizeRecordTimeLabelOperationResults(operations, results, options)
• normalizeRecordTimeLabelEnvelopeResponse(operations, response, options)
• createSyncEngine({ storageAdapter, cloudAdapter, clientId, clock, logger })
• createRecordTimeLabelController({ storageAdapter, cloudAdapter, clientId, settingKeys, clock, logger })
• buildRecordTimeLabelSyncPayload({ data, previousCloudData, pendingOps, syncMode, clientId, now, groupOrderNormalizer })
• buildSyncPayloadWithOperations({ data, previousCloudData, pendingOps, syncMode, clientId, now, groupOrderNormalizer })
• flushPendingOperations({ operations, storageAdapter, cloudAdapter, clientId, now })
• mergeRemoteStateIntoLocal({ localData, remoteData, pendingOps, settingKeys, clientId, now })
• buildLocalStateFromStorage(data, options)
• buildStoragePatchFromState(state, options)
• buildFirestoreV1UserPatch(state, options)
• buildFirestoreV2LogicalPaths(userId)
• buildFirestoreV2DocumentsFromState(state, options)
• buildFirestoreV2DocumentChangeSet(previousDocuments, nextDocuments, options)
• buildStateFromFirestoreV2Documents(documents, options)
• validateRecordTimeLabelOperationBatch(body)
• buildFirestoreV2OperationReadPlan(operations)
• extendFirestoreV2OperationReadPlanWithRecords(readPlan, records)
• extendFirestoreV2OperationReadPlanWithTrash(readPlan, trashEntries)
• planFirestoreV2OperationChanges({ documents, operations, localState, now, requireLifecycleGeneration })
• estimateFirestoreV2WriteUnits(changes, overhead)
• buildOperationsFromSnapshotDiff({ previousState, nextState, now, operationIdPrefix, batchSize })（一般操作在 batches，folder.delete 在 bulkOperations）
• RTL_SYNC_PROTOCOL_VERSION
• hasMeaningfulRecordTimeLabelCloudState(data, options)
• buildRecordTimeLabelContentFingerprint(data)
• normalizeRecordTimeLabelDomainState(input)
• migrateRecordTimeLabelExpandedGroups({ pendingOperations, currentView })
• buildMigratedRecordTimeLabelV2State(legacyState, options)
• RECORD_TIMELABEL_CLOUD_SCHEMAS
• RECORD_TIMELABEL_SYNC_MODES
• createOperation(type, payload, options)
• OPERATION_TYPES
• getActiveTrashEntries(entries, now)
• RTL_TRASH_RETENTION_MS
• buildFirestoreV2SnapshotRoot({ currentRoot, incomingRoot, mode })
• applyRecordTimeLabelSnapshot(currentState, incomingState, mode)
• composeRecordTimeLabelHydratedState({ remote, pendingOps, importJobs, localNavigation })
• applyFirestoreV2ResolvedChangeBatch({ cache, changes, resolvedDocuments, targetRevision })
• FIRESTORE_V2_BOOTSTRAP_REASONS

The package exports five intentional entrypoints. The root (@recordtimelabel/core) keeps the complete backwards-compatible surface, /protocol contains only shared acknowledgement protocol helpers, /firestore-v2 contains platform-neutral document and operation planners, /domain contains group identity, Twitch VOD matching, legacy import, and channel-folder planning, and /compat contains the legacy createSyncEngine and createRecordTimeLabelController APIs.

RECORD_TIMELABEL_DOMAIN_CAPABILITIES is a local release contract. It is not a gateway protocol capability.

Durable sync engine

createRecordTimeLabelSyncEngine is the platform-neutral durable workspace API. Its storage port has load() and save(workspace) methods (adapters may consume the optional second save(workspace, fenceContext) argument to enforce the session/epoch atomically), and its cloud port has bootstrap(context, attemptOptions), applyOperations(envelope, context), and subscribe(listener, context) methods. The session port provides current(), subscribe(listener), and isCurrent(sessionToken, uid, workspaceEpoch). sessionToken is an opaque fence for equality only. Core never aliases current.id or the whole session object as an authorization token, and bootstrap/catchUp/apply context no longer carries token. Platform adapters obtain Firebase credentials from their own credential provider.

normalizeRecordTimeLabelCloudFailure preserves status, reason, code, message, retryable, and retryAfterMs, and assigns exactly one class: transient, bootstrap-required, auth-transition-required, terminal, or stale-session. Classification uses only an explicit class, HTTP status, retryable, bootstrapRequired, and other platform-neutral fields. Adapters may keep a Firebase reason string, but Core does not interpret Firebase or JWT taxonomies. {error: "reason"} keeps that string as reason, code, and message. Capability cloud-failure-state-v1 is part of RECORD_TIMELABEL_PROTOCOL_CAPABILITIES; clients must not invent that string. An auth-transition-required bootstrap or catch-up failure latches the captured session identity (sessionToken + UID + workspace epoch) so repeated init() calls do not invoke cloud.bootstrap again. A changed SessionPort identity, UID, or workspace epoch clears the latch. An ID token refresh that keeps the same sessionToken must not change session identity or current work. recoverBaseline falls back from catch-up only for explicit bootstrap-required / cursor-gap outcomes; auth and transient failures keep their class and must not start a second full walk.

The persisted workspace is versioned and contains only durable data:

{
  schemaVersion: 1,
  ownerUid: 'user-id',
  workspaceEpoch: 3,
  remoteBaseline: { state, revision: 12, changeCursor: 'cursor-12' },
  pendingOperations: [],
  rejectedOperations: {},
  syncMeta: {
    // Core-owned resume fence; the cursor itself remains opaque adapter data.
    changeCursorBinding: {
      ownerUid: 'user-id', uid: 'user-id', workspaceEpoch: 3,
      authSessionBinding: 'auth-session-3'
    }
  }
}

remoteBaseline.changeCursor is an opaque string owned by the cloud adapter and must remain byte-for-byte stable across persistence. Core stores the owner, workspace-epoch, and non-credential auth binding separately in syncMeta.changeCursorBinding so a committed workspace can be resumed without embedding session credentials in the cursor.

Every normalized pending operation also carries ownerUid and a non-negative workspaceEpoch. Dispatch stamps these fields from the captured session; legacy operations missing them inherit the migrated workspace identity. An operation whose explicit identity does not match the current workspace is moved to rejectedOperations with a stable identity-mismatch reason before any cloud apply.

Pending operations also carry an internal syncBatchId. Operations dispatched in one call share a batch boundary, while legacy pending operations are assigned one stable boundary during migration. The engine sends only the first FIFO batch boundary, capped at 20 operations, and derives a stable request ID from that boundary and the ordered canonical wire operations. Durable retry metadata does not affect request identity, while reusing an operation ID with different wire content is quarantined. A bounded history of completed operation fingerprints is kept in syncMeta; syncBatchId, identity fences, and retry scheduling fields stay outside canonical wire operations.

Bootstrap attempts use initial-hydration, rejection-rebase, or gap-recovery modes with a unique attemptId; all require a fresh logical attempt. Structured failures, missing state, and missing/invalid revisions fail closed. UID or workspace epoch changes clear the visible workspace and require init() before another dispatch or sync; token refresh for the same UID/epoch only refreshes the subscription fence. An operation acknowledgement records its revision diagnostically and advances remoteBaseline.revision with a complete state, or with an all-applied no-state ACK at exactly the next revision; larger revision gaps trigger a fresh bootstrap. Re-dispatching the same operation ID and wire content is reported as deduplicated, while different wire content is quarantined as an ID conflict. Rejected entries are terminal diagnostics in this API; an intentional retry currently requires a new operation ID.

The engine exposes init(), dispatch(operations), sync(reason), syncUntilIdle(reason, options), waitForIdle(), waitForRemoteReady(), getSnapshot(), subscribe(listener), and destroy(). A cloud subscribe(listener, context) disposer may carry a .ready promise. After init() settles, hosts can await waitForRemoteReady() to prove that the current subscription's initial root observation and any bounded catch-up have settled; this wait rejects when the listener fails before readiness and fails closed across a session generation change or destroy. init() deliberately does not await this promise because the initial listener callback may itself enter the engine queue. Anonymous/no-subscription use is immediately ready. After readiness, a transient catch-up failure keeps the subscription and retries the gap in place with backoff; it is not reported as a listener failure. The server retryAfterMs is a floor, and newer root events wait for the active retry deadline. A non-transient failure on that retry stops the subscription and emits remote_error with origin: 'remote_gap_retry'; hosts route that event into their listener-failure recovery. Adapters must report a missing root only from a server-confirmed snapshot, never from an offline cache event.

sync() preserves the one-FIFO-batch boundary. syncUntilIdle() drains subsequent batches up to RTL_MAX_SYNC_DRAIN_ROUNDS; it stops on retry/deferred/protocol conditions and reports syncDrainLimitReached as a retryable failure instead of silently reporting success. Hosts may provide beforeRound({round, pendingCount, snapshot}) to block gateway writes while a host-specific hydration/import gate is active, and runRound({round, pendingCount, run}) to serialize that gate check and the gateway call with host mutations. getSnapshot().state is derived by replaying pending operations over remoteBaseline.state; rejected operations are kept in diagnostics and are not replayed. Session tokens are used for fencing but are never persisted. The legacy createSyncEngine and app adapters remain available and are not implicitly migrated by this API.

expandedGroups is a local view projection, not durable cloud domain state. Durable workspaces, content fingerprints, and Firestore v2 root documents exclude it. The migration helper consumes legacy expandedGroups.update operations in order, returns the resulting view projection, and is idempotent; legacy root fields are preserved verbatim when a compatibility document is rewritten so older clients can continue reading the view until they migrate.

Operation-result protocol

The root package and @recordtimelabel/core/protocol export the shared acknowledgement normalizers. normalizeRecordTimeLabelOperationResults(operations, results) verifies count, request membership, unique IDs, and completeness before returning results in request order. The only normalized statuses are applied, noop, rejected, and retryable; retryable is derived from that status, while the legacy id and applied fields remain available alongside operationId, reason, and retryAfterMs.

normalizeRecordTimeLabelEnvelopeResponse(operations, response) requires an explicit operationResults array by default. Malformed, duplicate, missing, unknown, or ambiguous IDs throw stable protocol errors so callers can reject the entire acknowledgement before changing durable state.

Gateway-only compatibility code may opt into successful legacy envelopes with {allowLegacySuccessWithoutResults: true}. New durable clients never enable that fallback. Capabilities operation-conflict-quarantine, strict-operation-results, and lifecycle-generation-fence gate the corresponding 0.4.4 behavior. Single record.restore, folder.restore, and trash.purge operations require a positive expectedGeneration; batch lifecycle operations retain their existing contract. The gateway validator derives enforcement from client.capabilities. The planner is strict by default; the time-limited legacy gateway path must explicitly pass requireLifecycleGeneration: false for clients that did not declare the capability. A supplied but stale generation is always rejected.

Firestore v1 Compatibility

Phase 1 keeps the existing users/{uid} document shape:

• folders
• records
• groupOrder
• folderOrder
• expandedGroups
• settings

The core also normalizes and preserves these compatibility metadata fields:

• rtlSyncMeta
• deletedRecordTombstones
• deletedFolderTombstones
• trashEntries

Deletion must be represented by tombstones or operations. Do not reintroduce the old heuristic that treats "local exists but cloud missing for more than five minutes" as deletion.

mergeLocalRemote resolves deletions by lifecycleGeneration, in both directions: a tombstone or trash entry is dropped when either side carries that entity as active and newer, and restore operations are what bump the generation. When generations tie — an active copy written outside the delete/restore path — the active copy's own timestamp breaks the tie against deletedAt. A stale snapshot on either side therefore cannot re-delete an entity that was already restored.

Consumers must not reimplement this reconciliation in an app-level wrapper. In particular, never clear a local tombstone just because the remote snapshot still carries the entity as active: that is presence-based inference, the same anti-pattern as the five-minute heuristic above, and it is not order-independent, so two clients using different rules will not converge on shared cloud data.

Firestore v2 Gateway Contract

The package exposes platform-neutral v2 document helpers so app adapters can share the same record/folder/settings/ops shape before wiring Firebase SDK or REST calls:

• root document users/{uid}/recordTimeLabel/main: settings, order arrays, tombstones, and sync metadata
• users/{uid}/recordTimeLabel/main/records/{recordId}: flattened per-record documents with folderId
• users/{uid}/recordTimeLabel/main/folders/{folderId}: per-folder documents
• users/{uid}/recordTimeLabel/main/trash/{trashEntryId}: 30-day record/folder snapshots
• users/{uid}/recordTimeLabel/main/lifecycleTombstones/{id}: content-free anti-revival markers
• users/{uid}/recordTimeLabel/main/ops/{opId}: pending operation documents

The v2 validator, bounded read planner, pure mutation planner, cost estimator, and snapshot-diff batch builder are also platform-neutral. They do not import Firebase and do not perform network writes. Functions and clients must use this package instead of maintaining app-specific planner copies.

Record ranges and additive migration

Records retain their existing playback start and may own rangeDurationSeconds: absent means a legacy point with no range opinion, a positive safe integer means an interval, and null is an explicit clear. Never backfill old points or change an existing operation ID/payload/hash during upgrade. Existing media-length fields durationSeconds / videoDurationSeconds keep their original meaning.

Only fresh record.create / record.update payloads with recordRangeIntent: {version: 1, durationSeconds: number | null} authorize range changes. Legacy full-record echoes preserve the authoritative value, including null, through updates, moves, duplicate creates, and trash operations. Range intent is hash-bound operation data; it is never a Firestore record field.

Platform adapters may pass a synchronous void assertAdmission(snapshot) to durable engine.dispatch. Check the local engine capability queued-local-intent-admission-v1 before relying on it; older engines must fail closed for guarded calls. Core runs the assertion inside its serialized queue, after awaited identity checks and before cloning, normalizing or saving the new intent. The snapshot is a detached copy. Adapters synchronously assert captured owner/session/source bindings against this latest snapshot and throw their own stale-operation error on mismatch. Async/thenable or other non-void assertions are rejected with recordtimelabel_invalid_local_admission_assertion; no new operation, pending entry or durable write is created. This local admission hook adds no wire field or schema and never rewrites existing pending work.

The record-range-v1 capability uses ordinary write protocol 4, with the same permanent operation proof/admission contract as dedup protocol 3. Pure points may continue using 3. getRecordTimeLabelWriteProtocolVersion and the durable engine retain a minimum protocol floor through partial ACKs and restarts. Collection, read, and global schema versions stay at 2. A range snapshot uses bulk protocol 3 and an immutable recordRangeContractVersion: 1; propagate that marker separately from normalized state into the stored job and canonical body/hash.

applyRecordTimeLabelSnapshot(current, incoming, mode, options) and buildOperationsFromSnapshotDiff({..., recordRangeContractVersion}) share this contract. Legacy same-ID snapshot records retain authoritative range data; omitted IDs in replace still follow existing deletion semantics. Explicitly different lifecycle generations never inherit a range. New legacy IDs carrying unknown range data throw record_range_confirmation_required before mutations. validateRecordTimeLabelRangeSnapshot preflights a complete job, while preserveRecordTimeLabelRange applies the same rule to a worker's documents. If the document builder filled a missing generation, pass incomingLifecycleGenerationKnown: false based on the raw incoming record.

Direct snapshots and gateway workers use resolveRecordTimeLabelSnapshotRecord for same-ID selection. A missing legacy fence keeps the admitted current fence; an explicitly older fence keeps the entire current record. Merge chooses by the existing lifecycle/time order, while replace uses the admitted incoming record. A genuinely later lifecycle never inherits range data from an older lifecycle. Callers retain their existing tombstone admission; the selector cannot authorize a previously blocked recreation.

The snapshot diff API still consumes a fully materialized target state. Its optional mode selects same-ID merge/replace behavior; callers materialize a merge before building the diff. It uses the shared selector for operations and returned documents. If a target requires advancing an existing record lifecycle, ordinary updates cannot express it: the builder throws snapshot_lifecycle_generation_requires_bulk without returning partial work. Preserve the original input and use the snapshot bulk path; never fabricate a restore operation or silently ACK an unchanged generation.

snapshot.replace diffs replace record, folder, and settings content with immutable payload.replaceFields: true on their existing update operations. These require write protocol 5 and field-replacement-v1; ordinary patches still preserve omitted keys. snapshot.merge retains ordinary patch semantics. Replacement also removes fields added concurrently after the diff was built. Identity, lifecycle, schema, and update timestamps retain their existing Core rules, and record ranges keep their explicit-intent contract. Partial ACKs retain the protocol floor through restart. An older server's unsupported-protocol response holds the original outbox with a retry deadline rather than downgrading the request. The existing deletion/trash/undo and empty-outbox replacement admission are unchanged.

record.move changes placement and its update timestamp, never content from a caller snapshot. Existing pending payloads and operation hashes stay unchanged. A successful planner batch containing a legacy payload.record move returns compatibilityChangeWrites: 1: the gateway must commit an empty change at revision R+1 and the actual mutation/root/change/ACK at R+2 in the same transaction, and pass that extra write to estimateFirestoreV2WriteUnits. This forces older compact-ACK clients to recover authoritative state. The current engine also recovers a legacy move's compact ACK from an older server instead of promoting a divergent local result. Already completed proofs replay without another barrier or mutation.

Run the actual old/new engine matrix against an immutable installed 0.8.1 entrypoint: RTL_MOVE_LEGACY_CORE_ENTRYPOINT=/path/to/core/src/index.js npm test. Also set RTL_LEGACY_CORE_PATH=/path/to/unpacked-0.6.22/package to exercise the official 0.6.22 engine's catch-up, bootstrap fallback, restart, and receipt replay.

resolveRecordTimeLabelPlaybackRange returns {kind, startSeconds, endSeconds, durationSeconds}; Twitch's present, valid numeric VOD offset takes precedence. Missing/invalid anchors return kind invalid, never a guessed zero. A range endpoint is always derived from that playback start plus its duration.

Editors use resolveRecordTimeLabelRangeEdit(record, edits) with only dirty liveTime / rangeDurationSeconds keys. liveTime stays on the raw capture clock; a valid Twitch VOD anchor shifts by the raw start delta. The helper returns {record, patch, recordRangeIntent?}: duration writes use the immutable intent, while an end-only edit leaves raw time and the playback anchor alone. It keeps the existing vodOffsetNeedsRefresh flag and invalidates stale matching metadata. A start edit with a valid VOD anchor freezes a missing original raw reference in originalLiveTime; it does not rewrite capture dates or offsets. Validation applies to the final combined candidate, so moving a start and shortening the duration together is checked once against the final playback end. buildRecordTimeLabelKnownVodEditContext qualifies a manually shifted, already admitted Twitch anchor; unchanged historical anchors do not receive this context. Pass its optional knownVodId / native-safe knownVodOffsetSeconds into the shared scorer, or use applyRecordTimeLabelKnownVodAnchor on an admitted provider result. The override applies only to that same selected VOD, after existing matching admission, and rejects an anchor outside a known candidate length. This prevents a provider refresh from reinterpreting an edited raw time as a playback time.

Share v2 and backup 3.0 retain nullable ranges. Point-only outputs stay share v1 and backup 2.0. Readers retain transfer provenance outside normalized domain state; a legacy backup with unknown range fields requires explicit preview confirmation. Share imports keep ifAbsent and put range intent in the same create, preserving duplicate and tombstone admission. Anonymous range evidence is saved in the same durable commit as its original intent and survives restart; missing evidence holds login migration without rewriting the source workspace.

Exporting/repacking an unconfirmed legacy transfer never grants range authority. buildRecordTimeLabelBackupData checks its transfer wrapper before normalization and rejects record_range_confirmation_required; it neither strips the field nor silently emits backup 3.0. When callers select records or normalize a parsed transfer first, pass recordRangeConfirmationRequired / transferSourceVersion in the backup/share exporter options so provenance survives that transform. Only an explicitly confirmed recordRangeContractVersion: 1 clears this hold. Ordinary validated visible state continues to export nullable ranges normally. Authoritative share v2 / backup 3.0 readers and new exporters validate safe playback endpoints and known media bounds before preview or job creation. Legacy backups retain their unknown range data and confirmation requirement instead of treating stale same-ID fields as new intent.

Test

npm test