@cson-sec/idempotent-rag

Generic idempotent document ingestion. Apache-2.0, Copyright 2026 Duong Cong Son (cson).

The package uses only the two requested workspace libraries and Node.js built-ins. No framework, company schema, endpoints, database connection, or provider credentials are embedded.

Usage

import { DocumentIndexer, GuardRedactor, InMemoryVectorStore } from '@cson-sec/idempotent-rag';
import type { EmbeddingProvider } from '@cson-sec/idempotent-rag';

const embeddings: EmbeddingProvider = {
  async embed(maskedText) {
    // Replace with your own provider. This example is a local fixture only.
    return [maskedText.length, 1];
  },
};
const store = new InMemoryVectorStore();
const indexer = new DocumentIndexer(embeddings, store, new GuardRedactor());
const metadata = { documentId: 'sample-document', aclHash: 'sample-acl-revision-1' };

await indexer.indexDocument('Generic sample handbook', metadata); // Indexed, version 1
await indexer.indexDocument('Generic sample handbook', metadata); // Skipped, version 1
await indexer.indexDocument('Generic sample handbook', {
  ...metadata, aclHash: 'sample-acl-revision-2',
}); // Indexed, version 2; old chunks inactive

The actual result is { status: 'Indexed' | 'Skipped', documentId, combinedHash, version }.

Pipeline and provider contract

1. Snapshot metadata, then call the injected Redactor.mask(content).
2. Calculate exactly sha256(aclHash + "::" + sha256(maskedContent)) using UTF-8.
3. Look up the active version of this document. An identical active hash returns Skipped without embedding or writing. A matching hash on another document or an inactive historical version cannot cause a skip.
4. Split the masked text into Unicode-safe chunks (chunkSize, default 1000 code points), embed those masked chunks, and validate consistent vector dimensions and finite values.
5. Call VectorStoreProvider.replaceDocument with expectedVersion. The provider must atomically compare the version, insert metadata and chunks, deactivate all old document chunks, and activate the replacement. Provider failure must roll back the whole operation. Same-hash concurrent commits return Skipped; different-hash stale commits reject without writing. Retry a conflict only after deciding which source revision should win.

InMemoryVectorStore is a single-process reference implementation and test store. It holds inactive historical versions in memory and provides detached inspection snapshots. It has no JSON/disk persistence or search API. For SQL/vector databases, implement VectorStoreProvider with a transaction and per-document concurrency control. The indexer cannot enforce transactions in a third-party provider implementation.

GuardRedactor adapts the existing piiSanitizer.sanitize export from @cson-sec/vn-llm-guard; that package does not currently export a Redactor class. Custom synchronous or asynchronous redactors can implement the exported interface. Masking precedes both hashing and every embedding call. No input text is logged by this package. Metadata is not redacted: do not put sensitive raw content or credentials in metadata.

ACL and retrieval boundary

aclHash must be updated by the host whenever any effective document permission changes. Prefer a deterministic digest of a canonicalized source ACL, including deny rules and relevant policy revisions; it is not the requesting subject's computePermissionHash. The optional resource field uses ResourceMetadata from @cson-sec/abac-rag-policy; its id must equal documentId.

This is an ingestion library, not a retrieval authorization layer. The host must validate source ACLs, apply active-version filters and policy filters before top-k, then use policy point decisions on returned chunks. getVersions is an administrative inspection API, not a user-facing authorized query. A published resource does not imply that every subject may access it.

The requested hash covers masked content and aclHash only. Changing a redacted value to another value with the same mask intentionally skips. Changing the embedding model, chunking configuration, or redaction implementation is not automatically detected if masked content and ACL hash stay unchanged; use a new index namespace/store or an explicit operator-driven rebuild. Old inactive versions are retained, not purged. Redaction coverage is whatever your injected guard/redactor supplies; it is not a promise to detect all secrets.

Build, test, package

Dependencies are exactly workspace:* as requested. Use a package manager that understands this protocol; pnpm-workspace.yaml is provided at the monorepo root. The existing npm lockfile is not a pnpm lockfile, and npm / Yarn Classic installation does not support this protocol.

cd /Users/it03/Desktop/cson-ai-security
pnpm install
pnpm --filter @cson-sec/vn-llm-guard build
pnpm --filter @cson-sec/abac-rag-policy build
pnpm --filter @cson-sec/idempotent-rag test
pnpm --filter @cson-sec/idempotent-rag typecheck
# Creates a local tarball; does not publish. pnpm rewrites workspace:* to package versions.
pnpm --filter @cson-sec/idempotent-rag pack --pack-destination /tmp/cson-packages

Build emits ESM/CJS JavaScript plus matching .d.ts / .d.cts. node:test tests execute built artifacts using fake embeddings and the local reference store. They require neither a database server nor real OpenAI/Gemini calls. Build the guard and policy dependencies first in a fresh checkout.

Install in another application

After version 0.1.1 is publicly available on npm:

npm install @cson-sec/[email protected]

ESM import and CommonJS require use the package name; do not import internal src or dist paths. See the workspace consumer guide for an example and offline tarball installation. This npm command is for consumer applications; use pnpm in the source monorepo to resolve workspace:*.