@bsv/overlay

The core engine and storage contracts for BSV Overlay Services. The engine admits transactions through topic managers, maintains UTXO state, serves lookup services, and supports SHIP, SLAP, GASP, and BASM synchronization.

Lookup history selectors retain their selected topical predecessors even when a transaction is confirmed. The response keeps its current subject last in BRC-62 BEEF and includes the selected available history within existing traversal and byte budgets. A lookup with no history selector returns its original proof bytes. UMP hosts must request this lineage to let clients follow token updates past an older WAB support pin.

Use @bsv/overlay-express when you want the standard HTTP server, operational endpoints, edge policy, and health checks. Use this package directly when you are embedding the engine in another runtime or implementing a custom transport.

Knex storage maps SQL NULL block heights to absent confirmation metadata. It never invents a confirmed height; malformed non-null values still fail engine validation.

Requirements

• Node.js 22 or newer
• @bsv/sdk installed as a peer dependency
• A Storage implementation
• A ChainTracker, or the explicit 'scripts only' validation mode

Install

npm install @bsv/overlay @bsv/sdk

Create an engine

import { Engine, KnexStorage } from '@bsv/overlay'
import type { LookupService, TopicManager } from '@bsv/overlay'
import knex from 'knex'

const database = knex({
  client: 'pg',
  connection: process.env.DATABASE_URL
})

const topicManagers: Record<string, TopicManager> = {
  tm_example: exampleTopicManager
}

const lookupServices: Record<string, LookupService> = {
  ls_example: exampleLookupService
}

const engine = new Engine(
  topicManagers,
  lookupServices,
  new KnexStorage(database),
  chainTracker,
  'https://overlay.example'
)

await engine.submit({
  beef: transaction.toBEEF(),
  topics: ['tm_example']
})

const answer = await engine.lookup({
  service: 'ls_example',
  query: { txid }
})

The constructor also accepts SHIP/SLAP trackers, broadcasters, an advertiser, sync configuration, a logger, a topic-anchor header resolver, and BASM/unproven state controls. Type declarations document the complete configuration surface.

Public API

The root entry point exports:

• Engine
• KnexStorage and KnexStorageMigrations
• the topic-manager, lookup-service, storage, advertisement, and sync contracts
• BASM utilities and types
• safe structured-log serializers

@bsv/overlay/storage exports the Storage contract. Existing supported deep imports remain available through the documented package export map, but new applications should prefer the root entry point wherever possible.

Optional persistence capability

AdmissionStorage defines an additive v1 atomic admission contract for future adapters. storageHasAdmission(storage) reports whether the optional admission field is present. getAdmissionStorage(storage) detects an explicit provider with both commit and reconciliation methods. Existing Knex and injected legacy adapters remain supported; their individual methods do not imply atomic submission. When getAdmissionStorage(storage) observes a complete overlay-admission-v1 provider, Engine.submit builds an admission plan and returns the saved STEAK only after majority commit. The SQL/Knex path and its early STEAK callback are unchanged; that callback is not a durable commit receipt.

The contract separates local commit, index visibility and propagation. It binds operation identity to verified transaction, topic/policy and off-chain context; uses ready payload references and outbox intents; and fences recovery by both chain epoch and topic history generation. Helper functions and shared fixtures pin exact integers, deterministic identity, leases and cursor eligibility. See the persistence specification. No consumer migration, storage-default change or database migration is included in this release candidate.

Optional Mongo foundation

The package also contains an opt-in MongoDB foundation for schema bootstrap, content-addressed payload publication, reference guards, payload collection, and an explicit AdmissionStorage adapter. Mongo is not the default Engine storage selection; importing @bsv/overlay alone does not load MongoDB.

Applications using a Mongo deep entry point install the optional peer first:

npm install @bsv/overlay mongodb@^7.5.0

The initial entry points are @bsv/overlay/storage/mongo/MongoSchema, @bsv/overlay/storage/mongo/MongoPayloadStore, @bsv/overlay/storage/mongo/MongoAdmissionStorage, and @bsv/overlay/storage/mongo/MongoOverlayStorage. They require an explicitly operated unsharded replica set; the supported deployment profile is three members. Payload publication makes GridFS bytes physically published before the guarded payload row becomes ready; caller-session reference and GC operations share that row guard. See the Mongo v1 foundation for operational bounds, recovery rules, and the opt-in admission path.

Runtime and package formats

The package supports both module systems:

import { Engine } from '@bsv/overlay'

const { Engine } = require('@bsv/overlay')

ES modules load from dist/esm; CommonJS loads from dist/cjs. Each condition has matching declarations. Published artifacts contain compiled output, the README, and the license only—tests, compiler caches, workspace source, and lock files are excluded.

Security and operations

The engine is transport-neutral. Authentication, CORS, CSP, body limits, timeouts, rate or concurrency controls, and administrative authorization belong at the HTTP or application boundary.

Overlay endpoints are commonly public protocol services used by browsers, mobile wallets, WUI, and applications on previously unknown origins. A wrapper should therefore remain public-by-default unless an operator deliberately configures an exact-origin allowlist. CORS is not an authentication mechanism, and CSP for a hosted UI should be configured independently.

For production deployments:

• validate all untrusted request data before invoking the engine;
• use a durable storage implementation and tested database migrations;
• configure transaction broadcast and proof providers;
• protect administrative and callback routes with explicit credentials;
• avoid logging raw secrets, authorization headers, or unbounded payloads;
• monitor readiness, proof acquisition, synchronization, and unproven state.

Run every KnexStorageMigrations migration before serving traffic. The topical uniqueness migration deliberately stops when duplicate (txid, outputIndex, topic) output rows or (txid, topic) applied-transaction rows already exist; operators must inspect and reconcile those records rather than letting a migration discard security-relevant state. The engine serializes submissions inside one process, while database uniqueness and conditional unspent updates enforce the same spend/admission boundary across processes.

Prefer roll-forward after these migrations. Older package versions do not know their migration names, so a bare image rollback can fail migration-list validation. To restore an older version, stop writes and either use the new migration source to reverse spentBy and topical uniqueness in reverse order after exporting and reconciling every spent/spentBy association, or restore coordinated pre-migration SQL and lookup-store backups. Never drop spentBy or restore only one store without preserving that spend evidence.

Submission is intentionally not a single all-or-nothing transaction across primary storage and external lookup indexes. The Engine validates before mutation, propagates failures, and never invokes the success callback until all attempted writes complete, but a late failure does not roll back an earlier committed write or third-party index update. A rejected submission therefore does not assert rollback. Operators and custom adapters must detect and reconcile partially applied work before replaying or exposing affected state.

GASP v1 bidirectional submitNode omits the spentBy parent outpoint. A receiver must already have the relevant parent, request it in a subsequent sync round, or reject the graph; it must never infer the edge from the child alone. Pull-only operation avoids that assumption. The Overlay pull adapter binds topic, graph, raw transaction, output, parent edge, proof, resource limits, and historical spend state before finalization.

Remote GASP/BASM peers and propagation/header providers are separate network authorities. Production adapters accept credential-free public HTTPS, pin DNS addresses to the requested origin, reject redirects, bound streamed bodies and deadlines, and correlate every transaction, topic, height, index, and proof to its request. Private or HTTP targets belong only in explicit isolated local development configurations.

Reorg event streams are hints rather than independent chain-state authority. Before demoting a proven admission for a reported orphaned block, the Engine requires the configured canonical header resolver to return a different hash for that exact height. An unavailable resolver or a hash that is still canonical rejects the event before any durable mutation.

SHIP tracker answers are discovery hints, not trusted routing authority. Before using a discovered endpoint for GASP, the Engine now requires a canonical identity-linked advertisement signature, a one-satoshi token, exact BEEF/TXID correlation, and the requested topic. The endpoint still passes through the same public-HTTPS and DNS-pinning controls as an explicitly configured peer.

Custom Storage, LookupService, TopicManager, advertiser, chain-tracker, and header-resolver implementations are trusted local components, but their runtime results are still checked before the Engine mutates or returns state. Storage queries must return only records bound to the requested outpoint, topic, height, block hash, and score window. Lookup formulas use a default 1,000-result limit and an unconditional 100,000-result safety ceiling; history depth, context bytes, stored BEEF, graph fan-out, and aggregate traversal work are also bounded. maxLookupResults: -1 disables the operator-selected lower limit, not the hard safety ceiling. BASM anchor stores must return strictly ordered, request-bound, canonical hash records.

Public component metadata is copied through a bounded own-data-property schema; names, descriptions, versions, and HTTP(S) links that are malformed, accessor backed, or oversized fall back to a minimal local registry description. Component Markdown documentation is limited to 1 MiB before it crosses the Engine/HTTP boundary.

@bsv/overlay-express supplies these standard HTTP controls while preserving public protocol access by default.

BASM peer validation and current recovery limits

BASM uses the current BRC-136 ordered admitted subset and block-anchored TAC. The five existing JSON POST routes remain compatible; empty tips remain { topic, blockHeight: -1, tac: <zero hash> }. Unsupported storage capabilities are errors, not empty histories. The remote client validates response shape, topic/height/hash binding, ordered unique admitted positions, contiguous returned ranges, and complete proof/raw response ID sets before use.

BASMRemote retains its injectable third fetch argument and accepts optional limits as a fourth argument. Defaults are 64 MiB per decoded response, 8 MiB per proof, 32 MiB per raw transaction, 100,000 admitted entries, 1,000 requested txids, 1,024 requested anchor heights, and 30 seconds per request including its body. Aggregate response limits also apply to hex-encoded transactions. These are configurable local acceptance limits, not consensus rules. Standard fetch bodies are bounded while streaming; legacy injected text() implementations are checked after buffering. Classified errors expose code, including BASM_UNSUPPORTED, BASM_RESOURCE_LIMIT, and BASM_TIMEOUT.

Reconciliation requires a canonical header resolver as well as a ChainTracker. An optional TopicAnchorHeader.blockTransactionCount must come independently from the trusted canonical provider and refer to that exact blockHash. It enables full-block count/index bounds and odd-duplication checks. The sync report's positionValidation is canonical-count only when that evidence was available for every checked proof; legacy providers yield encoded-offset-only. A Merkle root plus an encoded offset alone cannot disambiguate Bitcoin's duplicate-last-leaf position ambiguity. No provider is required to add the field, and the engine does not download full blocks to infer it.

Forward sync pages now contain at most 1,000 anchors to fit the standard HTTP server. Proof height, requested original index, canonical hash/root, raw byte identity, TAC continuity, and repeated peer anchors are checked before historical submission. Claimed admitted-list indices are bound to the compound path whenever a remote list is used as evidence, including when every remote txid is already local. Inclusion uses the chain tracker root/height check rather than MerklePath.verify, which also enforces coinbase 100-block spendability. Because inclusion is proven independently, admission submits in the historical-tx-no-spv mode so Transaction.verify does not re-apply that coinbase rule; the public historical-tx mode keeps full SPV verification. Admission still applies the local TopicManager, and because every admitted transaction carries its extracted Merkle path, neither network broadcast nor overlay propagation occurs. Automatic BASM sync remains disabled by default.

This is bounded protocol hardening, not durable recovery. An empty local node whose topic genesis precedes the recent bootstrap window now refuses the untrusted TAC prefix; this intentionally replaces the old unchecked tail behavior. A block above 1,000 admitted entries reaches a request-limit error until proof/raw chunking is implemented. Equal-height/local-ahead divergence, whole-target bootstrap, durable cursors/leases, atomic revision fencing, and truthful per-topic agreement status remain required follow-up work. A successful legacy report does not establish global completeness, current unspentness, or durable recovery completion. See BASM details.

Development

From the repository root:

pnpm --filter @bsv/overlay format:check
pnpm --filter @bsv/overlay lint
pnpm --filter @bsv/overlay typecheck
pnpm --filter @bsv/overlay test
pnpm --filter @bsv/overlay test:coverage
pnpm --filter @bsv/overlay pack:check

pack:check verifies the actual npm tarball with publint, strict type resolution, and clean ESM/CommonJS consumer projects.

License

Current TS Stack changes are licensed under the Open BSV License Version 6; see LICENSE.txt. This package also retains pre-uniformization code under the Open BSV License Version 4. Redistributors must preserve THIRD_PARTY_NOTICES.md and the applicable text in LICENSES/.