@happyvertical/smrt-facts

Knowledge base with semantic deduplication, provenance tracking, and confidence scoring for the s-m-r-t framework. Facts evolve through parent-child chains, are linked to sources and subjects, and undergo 3-zone reconciliation to prevent duplicates.

Installation

pnpm add @happyvertical/smrt-facts

Usage

import {
  Fact, FactCollection,
  FactSource, FactSourceCollection,
  FactSubject, FactSubjectCollection,
  FactContent, FactContentCollection,
  FactTag, FactTagCollection,
  calculateConfidence, normalizeText,
} from '@happyvertical/smrt-facts';

// Create a fact with provenance
const facts = await FactCollection.create({ db });
const fact = await facts.create({
  textRefined: 'The Eiffel Tower is 330 meters tall',
  type: 'measurement',
  domain: 'landmarks',
  status: 'active',
});

// Attach a source with credibility score
const sources = await FactSourceCollection.create({ db });
await sources.create({
  factId: fact.id,
  sourceUrl: 'https://example.com/eiffel-tower',
  sourceTitle: 'Tourism Board',
  credibility: 0.9,
});

// Recalculate confidence from all sources
await facts.recalculateConfidence(fact.id);

// 3-zone semantic reconciliation
// Exact normalized statements first reuse an active fact only within the same
// tenant, domain, and type. PostgreSQL callers are serialized on that key, so
// this identity guarantee also holds when embeddings fail or calls overlap.
// >= 0.85 similarity: auto-merge (same fact, update metadata)
// 0.60-0.85: optional typed decision, otherwise AI disambiguation
// Typed decisions merge only when their selected merge probability meets the
// conservative 0.75 operating threshold; uncertainty and provider errors branch.
// < 0.60: create new fact
const result = await facts.reconcile({
  rawInput: 'The Eiffel Tower stands 330m tall',
  type: 'measurement',
  domain: 'landmarks',
  source: { sourceUrl: 'https://another-source.com', credibility: 0.8 },
});
// result.action: 'created' | 'merged' | 'branched'

// Evolution chains: branch creates a successor fact linked via previousFactId
const successor = await facts.branch(fact.id, {
  textRefined: 'The Eiffel Tower is 330 meters tall including the antenna',
}, 'correction');
// 'correction' and 'contradiction' mark the predecessor as superseded

// Walk evolution: root -> current (via previousFactId)
const chain = await facts.getEvolutionChain(successor.id);
// Find highest-confidence leaf (via getSuccessors)
const latest = await facts.getLatestInChain(fact.id);
// Full tree (BFS with cycle detection)
const tree = await facts.getEvolutionTree(fact.id);

// Per-fact navigation: getPredecessor / getSuccessors / hasPredecessor
const previous = await successor.getPredecessor();
const successors = await fact.getSuccessors();
if (successor.hasPredecessor()) {
  // ...
}

// Entity briefing: all facts for a given entity
const briefing = await facts.getEntityBriefing('Place', placeId);
// { facts, totalCount, byType, byStatus }

API

Models

| Export | Description | |--------|------------| | Fact | Knowledge unit with textRefined, type, status, confidence, previousFactId for evolution chains (predecessor pointer; not a structural hierarchy edge), and auto-generated embeddings | | FactSource | Provenance record with sourceUrl, sourceType, credibility (0-1), extractedAt | | FactSubject | Polymorphic entity link (entityType + entityId), conflictColumns: ['fact_id', 'entity_type', 'entity_id'] | | FactContent | Join table linking facts to Content, conflictColumns: ['fact_id', 'content_id', 'relationship'] | | FactTag | Tag association for a fact |

Collections

| Export | Description | |--------|------------| | FactCollection | Query by status/type/domain, reconcile(), branch(), evolution traversal, recalculateConfidence(), getEntityBriefing(), findWithGlobals(tenantId) | | FactSourceCollection | Source management with getForFact() | | FactSubjectCollection | Subject links with getForFact(), getForEntity() | | FactContentCollection | Fact-to-content relationships | | FactTagCollection | Fact tag management |

Functions

| Export | Description | |--------|------------| | calculateConfidence | Weighted formula: base 0.5 + source volume (max 0.3) + credibility (0.2) + recency (0.1, decays over 10 days) + corroboration (0.1), clamped to [0, 1] | | normalizeText | Trim, collapse whitespace, lowercase -- used for dedup comparison |

Key Types

FactType (assertion/observation/measurement/definition/relationship/event/opinion/prediction), FactStatus (pending/active/disputed/superseded/archived/retracted), EvolutionType (original/correction/refinement/contradiction/extension/merge), SubjectRole, ReconcileAction, ReconcileOptions, ReconcileResult, FactContentRelationship, EntityBriefing

Dependencies

• @happyvertical/smrt-core -- ORM, code generation, and semantic search
• @happyvertical/smrt-tenancy -- optional multi-tenant scoping with findWithGlobals()
• @happyvertical/ai -- AI disambiguation in reconciliation

Contributor guide

See AGENTS.md for package architecture, invariants, validation, and contributor guidance.

Catalog pagination

browseCatalog(query, { limit, offset, latestOnly, tenantId }) defaults to 25 results and resolves evolution chains unless latestOnly: false is supplied. Empty-query browsing and successful semantic search hydrate only the requested SQL page. Empty-query candidate eligibility remains bounded to offset + 2 * limit; resolving and deduplicating chains can therefore return a short page. Semantic candidate retrieval remains bounded to offset + limit. These SQL windows use the requested pagination values rather than the collection defaultListLimit. The full tenant-visible successor set is available to chain resolution, even when a successor lies outside the candidate window or status filter. Implicit-scope confidence ties retain the newest successor, matching the previous ordered read.

Without an explicit tenant, the default candidate status is active; with an explicit tenant, tenant and global candidates exclude only superseded. includeSuperseded removes that status filter. Active tenant context continues to constrain implicit reads, and requesting another tenant is rejected. STI child collections constrain both candidates and successors to their discriminator.

Catalog reads apply normal beforeList authorization predicates to candidates, readiness checks, and the complete successor graph, including empty queries. Explicit tenant/global reads use withTenantGlobalRead, which grants the built-in tenancy list hook that narrow scope while business hooks keep the original user, permissions, tenant context, and system/non-system identity. Actual system calls remain system calls. Interceptor rejection and application SQL failures propagate to the caller. Only unavailable embedding configuration or query-embedding provider failure permits text fallback; a semantic authorization failure never does.

When query embeddings are unavailable, text fallback matches the exact JavaScript expression `${textRefined} ${textRaw}`.toLowerCase().includes(query.toLowerCase()). It uses persisted catalogSearch storage and the same bounded SQL page traversal. The storage encodes lowercased UTF-16 units as aligned ASCII tokens, preserving Unicode, whitespace, literal wildcard characters, and code-unit boundaries without SQL collation or case-folding differences. A scalar readiness check precedes text search; it throws with backfill instructions if any permitted row is unbackfilled. The database may scan matching rows and evolution edges internally, but only the requested Fact page crosses the database boundary. Arbitrary substring matching cannot use a normal B-tree index; no misleading search-column index is added.

Existing deployment migration

This is an explicit schema and data migration; ordinary reads never create schema. Stop old application writers, deploy the new manifest/code, and run smrt db:migrate (and smrt db:status --parity). This adds nullable catalog_search; historical rows remain NULL. Before enabling catalog text reads, run the following with your application's database configuration and repeat until remaining is zero:

import { FactCollection } from '@happyvertical/smrt-facts';
import { withSystemContext } from '@happyvertical/smrt-tenancy';

const facts = await FactCollection.create({ db });
await withSystemContext(async () => {
  while ((await facts.backfillCatalogSearch(100)).remaining > 0) {
    // Each call is independently resumable; record progress in your job runner.
  }
});

Backfill requires explicit system context, processes at most 100 rows per call by default (maximum 1000), and updates only NULL values whose source texts still match the read snapshot. It changes no source text, timestamps, embeddings, or revisions. A crash or concurrent write is safe to retry; sustained old writers must be stopped so the operation can finish. A subtype collection backfills only its discriminator; use the base collection for the full deployment. Back up before schema changes; roll forward by rerunning migration/backfill rather than dropping historical data.

Fact.save(), collection create/get-or-insert/get-or-upsert, and generated model updates maintain search storage from the source fields. catalogSearch is derived; callers must not author it, and generated transport surfaces exclude it using readonly/sensitive field metadata. Derivation uses the final persistence row after mutable beforeSave hooks and the complete subclass transformJSON chain, keeping persisted text and search storage consistent. If a custom transform omits a source column, returns a value other than a string or explicit NULL, or returns a string with an unpaired UTF-16 surrogate, the write retains its ordinary adapter semantics (retention, defaults, or coercion); search storage is invalidated to NULL rather than guessed from instance values. Catalog text reads then fail with the readiness error until privileged bounded backfill reads the actual persisted columns. Run that backfill after such custom writes before resuming catalog text reads. Only well-formed strings and explicit NULL source values are known before persistence. Query encoding remains exact UTF-16 code-unit matching, including queries containing an unpaired surrogate. Plain/public serialization retains the saved marker instead of recomputing from pre-transform instance text. Direct SQL writers must set catalog_search = NULL whenever either source text changes, then run backfill before text reads resume. If a custom writer or a pre-release implementation produced a known stale non-NULL search value, explicitly set that affected row's catalog_search to NULL and run the same bounded backfill. Backfill intentionally selects NULL markers; it does not scan or repair non-NULL values. This is a targeted repair, not an additional step for a fresh column migration.

Do not keep old application writers active after backfill: they cannot maintain this new invariant. Future changes to JavaScript lowercasing/encoding require an explicit new backfill; the format is not locale dependent.

PostgreSQL and SQLite run canonical pagination integration tests. DuckDB query coverage uses an explicitly identified SQL-only fixture: canonical Fact schema creation currently rejects its evolution self-reference, tracked in #2830. DuckDB hydration issues one DESCRIBE plus one data query per page; PostgreSQL and SQLite use one data query after semantic candidate retrieval, if any. Text fallback also performs one scalar readiness query.

Catalog reads preserve hydrated afterList policies on every final bounded page, including empty pages, after raw-query hooks and similarity annotations. The same list context and original caller identity reach before/after hooks. Filtering or redaction may shorten a page; browsing does not refill it. Policy rejection propagates and never triggers the embedding-unavailable text fallback.

Text fallback is selected by core's provider-origin availability result: only missing embedding configuration or a failed query embedding enables it. Errors from authorization, ranking, SQL, hydration, or result hooks propagate unchanged, even when a caller throws EmbeddingUnavailableError from those operations.

Catalog text-read readiness preserves database permission, transient and unknown errors unchanged. Only recognized missing-table/column driver diagnostics receive the migration/backfill instruction, retaining the original error as cause. Fact declares catalog_search as its permitted final derived column; custom normalization must preserve that declaration and cannot replace framework identity, tenant, revision or conflict columns.

Reconciliation access scopes

reconcile({ rawInput, tenantId, accessScope: 'public' }) partitions candidates by both tenant and an exact, application-defined scope token. Exact matching and semantic ranking exclude other scopes before candidate text is hydrated or sent to disambiguation. Created facts and evolution branches retain that scope; branch() rejects an explicit scope change. Use public collection predicates such as facts.list({ where: { tenantId, accessScope: 'public' } }) and scoped get() calls when reusing results. Generic catalog and evolution APIs do not implicitly authorize a scope: the application must authorize access and apply its scope predicate before returning evidence or associating article facts.

Tokens are case-sensitive, nonblank strings of at most 256 characters without control characters or unpaired surrogates. They are partition identifiers, not authorization credentials. Applications must derive them from trusted authorization state; never accept an arbitrary client-selected token as permission to read a scope. A fact must only aggregate sources with the visibility represented by its scope. Source metadata does not establish candidate visibility.

accessScope is server-managed: generated REST/MCP create and update inputs strip it while allowing ordinary writable fields. Trusted server-side FactCollection.create(), reconcile() and branch() retain persistence access. Owner-authorized migration code can assign the field and call save() in a controlled system context with reconciliation writers stopped; no generated reclassification endpoint is exposed. Do not expose that operator procedure or forward untrusted request bodies to direct collection writes.

Run the standard smrt db:migrate before deploying writers using this field. The nullable Fact.accessScope column leaves existing rows unclassified. Omitting accessScope (or passing null) searches only this legacy NULL partition; explicit scopes never include legacy rows. No migration guesses visibility from provenance metadata. Classify legacy facts only through an owner-authorized data migration with reconciliation writers stopped, or leave them separate and re-extract into the appropriate scope. A scope change is not an ordinary reconciliation operation.

PostgreSQL reconciliation retains its transactional fact/source write and per-identity lock. SQLite retains its existing partial-write contract: a failed source write may leave the fact, and retry finds the same scoped exact fact. Do not record an application extraction receipt until all writes succeed.