@semiont/content

Working-tree storage for project resources, with optional git staging, plus PDF text-layer extraction.

Installation

npm install @semiont/content

Architecture Context

Infrastructure Ownership: In production applications, the working tree store is created and managed by @semiont/make-meaning's startMakeMeaning() function, which serves as the single orchestration point for all infrastructure components. Gateway code accesses it as knowledgeBase.content.

The quick start example below shows direct instantiation for testing, CLI tools, or content management scripts.

Quick Start

import { WorkingTreeStore, deriveStorageUri } from '@semiont/content';
import { SemiontProject } from '@semiont/core/node';

const project = new SemiontProject('/path/to/project', {
  anchoredTextDir: process.env.SEMIONT_ANCHORED_TEXT_DIR!,
});
const store = new WorkingTreeStore(project);

// Derive a stable file:// URI from a resource name
const uri = deriveStorageUri('My Document', 'text/markdown');
// => "file://my-document.md"

// Write content to the working tree (API/GUI/AI path)
const stored = await store.store(Buffer.from('# My Document\n'), uri);
console.log(stored.checksum);  // SHA-256 hex of the content
console.log(stored.byteSize);  // 14

// Register a file that is already on disk (CLI path)
const registered = await store.register('file://docs/overview.md');

// Read content back by URI
const content = await store.retrieve(uri);
console.log(content.toString()); // "# My Document\n"

// Move and remove files
await store.move(uri, 'file://docs/my-document.md');
await store.remove('file://docs/my-document.md');

Working Tree Storage

The working tree (project root) is the source of truth for file content. Resources are identified by their file:// URI, which is stable across content changes; moves are tracked by events.

my-project/                  ← project root
├── .semiont/                ← project config and event log
└── docs/
    └── overview.md          ← storageUri "file://docs/overview.md"

There are two write paths:

• store(content, storageUri) — write bytes to disk. Used when the file does not yet exist and the caller provides content (API/GUI/AI path).
• register(storageUri, expectedChecksum?) — read an existing file and record its metadata (CLI path). If expectedChecksum is provided and does not match, throws ChecksumMismatchError.

Both return the same metadata:

interface StoredResource {
  storageUri: string;    // file:// URI (e.g. "file://docs/overview.md")
  checksum: string;      // SHA-256 hex of content
  byteSize: number;      // Size in bytes
  created: string;       // ISO 8601 timestamp
}

Git Integration

When the project has [git] sync = true in .semiont/config, the store keeps the git index up to date automatically:

• store() / register() run git add
• move() runs git mv
• remove() runs git rm (or git rm --cached with keepFile: true)

Every method accepts { noGit: true } to skip staging for a single call. Without git sync, the store falls back to plain filesystem operations.

PDF Extraction

EXTRACTORS['pdf-text-layer'] turns a PDF into text plus the geometry that indexes it, routing by what the document actually holds:

| Class | Document | Read by | |---|---|---| | A | native text layer | pdf.js, directly | | B | scanned — pixels only | OCR (Tesseract) | | C | hybrid — some pages scanned | both; pages still unread are reported | | D | tables | grid pages rewritten as markdown rows | | E | forms | AcroForm values folded in, anchored to their widgets | | F / G | encrypted, corrupt | declined by name, from the parser error |

import { EXTRACTORS } from '@semiont/content';
import { textExtractionOf, locate } from '@semiont/core';

const extracted = await EXTRACTORS[textExtractionOf('application/pdf')]!
  .extract(pdfBytes, 'application/pdf');

if (!('declined' in extracted)) {
  extracted.text;          // reading-order text
  extracted.items;         // positioned runs indexing it
  extracted.method;        // 'pdf-text-layer' | 'ocr' | 'table' | 'form'
  extracted.unreadPages;   // class C: pages no reader could recover
}

A decline is named ('no-text-layer' | 'encrypted' | 'corrupt' | 'too-large') rather than a bare null, so a caller can settle with the reason. 'no-text-layer' means recognition ran and came up empty — not that it was never attempted.

extractPdfTextLayer() is the lower-level reader underneath class A, returning null for a document with no text operators anywhere.

Coordinates are PDF points, origin bottom-left; the Y-flip to canvas pixels happens in the browser. The vocabulary these produce — AnchoredText, PdfTextItem — and the locate / textUnder pair that reads it are exported from @semiont/core, so the browser can reason over geometry without importing this package's extraction stack.

Anchored-text store

OCR costs ~2.9 s per scanned page and six consumers read the same document, so what the engine produced is kept rather than re-derived:

import { createAnchoredTextStore } from '@semiont/content';

const store = createAnchoredTextStore(dir, logger);
await store.write(checksum, { text, items });
const map = await store.read(checksum);   // null on any miss

Derived values only, keyed by content checksum and stamped with the versions of this package, the engine and its traineddata. A stamp mismatch, a corrupt file and an absent one are all the same answer: a miss. The store may make things faster, never make them fail.

See ANCHORING.md for the pipeline this sits in.

Utilities

import {
  calculateChecksum,       // SHA-256 hex of a string or Buffer
  verifyChecksum,          // Compare content against an expected checksum
  deriveStorageUri,        // ("My Doc", "text/markdown") → "file://my-doc.md"
} from '@semiont/content';

deriveStorageUri takes a SupportedMediaType; the media-type registry — which types are admitted, their extensions, and their capabilities — lives in @semiont/core's media-types.ts. See docs/mime-types.md.

Documentation

• API Reference - Complete API documentation
• Architecture - Design principles

Development

# Install dependencies
npm install

# Build package
npm run build

# Run tests
npm test

# Type checking
npm run typecheck

License

Apache-2.0