@context-action/sem-doc

sem-doc is a publishable sem-centered analysis tool for answering what context an engineer needs before changing TypeScript code. It combines sem's semantic graph with a TSDoc-style Markdown document convention and a revision-pinned Git diff.

The package identity is @context-action/sem-doc. The former @tsdoc-edge/sem-doc name is retained only in the changelog as migration history.

The specification is in spec/sem-doc.md.

sem-doc is not an architecture registry or verification gate. Architecture Governance remains a consumer-owned control-plane package; the two packages may share SEM/Foundation primitives but keep independent report contracts and runtime dependencies. Use the boundary guide to choose the right tool when both are present in a consumer repository.

In that split, sem-doc is the operational Context Plane: it resolves symbols, work context, document bindings, and Git change evidence. It does not author or evaluate the Context-Action architecture convention; that experimental control-plane responsibility belongs to architecture-governance.

Symbol Context SSOT

sem-doc-work-context.v5 is the SSOT for a symbol-centered work context: target identity, bounded relationships, dependent files, affected tests, SEM provenance, Git revision, and document definitions/backlinks. Consumers should reuse this serialized report instead of issuing duplicate impact/context queries or rebuilding document bindings.

This does not make sem-doc the architecture registry or API documentation source. Capability intent, owner/role evidence, package policy, complete snapshots, and history belong to @context-action/architecture-governance; signatures and public API pages remain in the TypeDoc pipeline. sem-doc emits the canonical operational sem-doc-context-scope.v3 projection for implementer-facing grouping. It is derived from one work-context report and is not a replacement for the revision-bound, snapshot-backed Architecture Governance ContextScope contract.

ContextScope projection

context-scope maps the bounded work-context inventory into a deterministic context grouping. The result keeps the canonical projectId/filePath/entityId identity, includes context and project groups, records the exact work-context request provenance, and records SEM dependency evidence. A project ID is required so serialized scopes remain interoperable with snapshot and commit-diff consumers. If the bounded inventory contains symbols without corresponding edge evidence, the scope is emitted as incomplete with disconnected-nodes; it is intentionally structural and does not infer render, read, write, runtime order, or call-graph semantics.

The scope also carries a SHA-256 digest of the source work-context, the exact SEM impact/context argument vectors, and execution provenance (phase, ownerId, state, timeoutMs, maxOutputBytes, usedOutputBytes, and measured elapsedMs). parseContextScope strictly rejects unknown fields, duplicate entries, oversized collections, non-canonical symbol paths, missing revision provenance, and contradictory completion status before a consumer uses the artifact. Use --execution-owner-id <id> (or executionOwnerId in the API) when a caller needs a stable logical owner in serialized evidence; it is not a process identifier and is never used for retry or durable-operation identity.

For a screen, API, transaction, or workflow composed from more than one entry point, pass a sem-doc-context-manifest.v1 file with --manifest. The resulting scope keeps one context bubble, multiple typed anchors, and a source.workContexts provenance item for every anchor. context-scope-diff compares two serialized scopes by symbol, edge, and group identity. context-scope-history materializes the same scope at each bounded Git commit and stores adjacent diffs; historical worktrees are removed after each analysis. History collection can use --output <path> to write snapshots as NDJSON instead of retaining every commit in memory; the history writer retains only scalar entry accounting. The array-returning readContextScopeHistoryStream helper materializes records for validation and branch comparison, so callers processing very large streams should consume NDJSON incrementally. --aggregate-* controls the whole range, while --commit-* controls each individual commit; both budgets are enforced together. The stream starts with exactly one base record followed by unique commit entries. Branch intersection reports changed symbols/edges/groups common to both histories, not final snapshot membership.

The pinned sem 0.21.0 scanner excludes node_modules in its default mode. sem-doc keeps that safe default unless --include-node-modules-surface is explicitly supplied to work-context, context-scope, or context-scope-history. The option passes sem's --no-default-excludes to the underlying impact/context queries so graph-referenced package files can actually be observed; sem-doc then projects only the graph-referenced direct one-hop node_modules surface and discards deeper package internals before work-context, scope, history, or branch-intersection artifacts are serialized. Because sem's flag is intentionally broad (it can also expose generated/vendor/fixture files), the option is an explicit review-time policy, not a new default. Markdown indexing still skips node_modules, build output, and repository metadata directories.

pnpm --filter @context-action/sem-doc exec node dist/cli.js context-scope \
  Dashboard --file src/Dashboard.tsx --context-id dashboard --kind screen --project-id example --max-anchors 64 --json

pnpm --filter @context-action/sem-doc exec node dist/cli.js context-scope-diff before.json after.json --json
pnpm --filter @context-action/sem-doc exec node dist/cli.js context-scope-history HEAD~10 HEAD Dashboard \
  --project-id example --file src/Dashboard.tsx --aggregate-max-output-bytes 268435456 \
  --commit-max-output-bytes 33554432 --output managed/history.ndjson --json
pnpm --filter @context-action/sem-doc exec node dist/cli.js context-scope-compare \
  managed/left-history.ndjson managed/right-history.ndjson --json

Use this projection to prepare a visual scope or review context. Use Architecture Governance when the scope must be bound to a complete revision snapshot, manifest digest, and CI/reviewer gate.

The history API currently analyzes one repository-relative project/entity per request. Shared Git analysisProjects traversal belongs to @context-action/sem-foundation-repository; sem-doc does not claim a multi-project complete snapshot until a dedicated adapter contract is introduced.

Scope

• Uses the external sem executable as a read-only analysis engine.
• Normalizes sem JSON into versioned sem-advisory.v1 evidence.
• Provides sem-doc-work-context.v5 with bounded-hop symbols, a separate complete affected-test list, dependent usage files, context, document definitions, and backlinks.
• Provides sem-doc-context-scope.v3 as the canonical operational grouping view over one or more work-context reports.
• Includes separate documentEvidence[] in ContextScope so document binding readiness is visible without conflating it with graph completeness.
• Provides sem-doc-context-manifest.v1, ContextScope snapshot diffs, and bounded Git-history scope materialization.
• Provides sem-doc-context-scope-history-stream.v1 NDJSON output and branch changed-symbol intersections.
• Exposes parseContextScope so JSON scopes loaded from Git history or another process are validated before graph/diff consumers use them.
• Publishes the machine-readable contract at spec/context-scope.schema.json.
• Publishes companion context-manifest.schema.json, context-scope-diff.schema.json, context-scope-history.schema.json, context-scope-history-stream.schema.json, and context-scope-branch-compare.schema.json contracts.
• Uses sem entity IDs to distinguish same-named symbols and preserves their definition file and range.
• Provides sem-doc-git-diff.v1 with Git status, hunks, additions/deletions, and revision provenance.
• Limits semantic diff decisions to typed changes emitted by sem; native Git hunks remain factual evidence rather than semantic classification.
• Keeps sem entity IDs and revisions separate from any compiler-resolved canonical graph.
• Exposes foundationSymbolSnapshotEntry so parsed entities can be handed to the shared context-action/[email protected] serializer without adding an LSP or second AST index. The conversion is delegated to the required canonical @context-action/sem-foundation-contracts createSymbolSnapshotEntry primitive.
• Rejects repository-outside paths, stale revisions, malformed JSON, and invalid semantic ranges/counts.
• Does not attempt a complete inventory of function-local functions, constants, variables, or parameters.
• Does not embed the @microsoft/tsdoc parser; document bindings use the package's Markdown/frontmatter convention.
• Does not implement an LSP server, unsaved-overlay analysis, or mutating CodeAction.

Sem-derived output is advisory evidence. A caller must make a separate policy decision before promoting it to a canonical graph or lint violation.

Install and run

sem-doc requires Node.js 24. The published package includes @ataraxy-labs/[email protected] as a runtime dependency, so a clean npm install also obtains the matching platform binary. It also installs @context-action/sem-foundation-contracts and @context-action/sem-foundation-repository for canonical symbol and Git revision contracts. The CLI resolves the package-local sem binary by default, even when analysis changes the subprocess cwd to the Git repository root. Set SEM_BIN only when using a different executable.

For a consumer repository, install the published package and invoke its CLI directly:

npm install --save-dev @context-action/sem-doc@^0.1.2
npx sem-doc version
npx sem-doc work-context SemClient --file src/sem-client.ts --docs-root spec --json

For local workspace development:

pnpm install
pnpm --filter @context-action/sem-doc build
pnpm --filter @context-action/sem-doc exec node dist/cli.js work-context SemClient --file src/sem-client.ts --docs-root spec --json
pnpm --filter @context-action/sem-doc verify:poc

The package is a canonical package in this tooling workspace. This repository does not use Lerna; the dedicated GitHub Actions workflow publishes the Foundation contracts first and sem-doc second. The current 0.1.2 npm artifact is legacy and still points to the consumer repository, so it is not evidence that the tooling cutover is complete.

Release through GitHub Actions

Package publication is managed by .github/workflows/publish-packages.yml. It can be started from the GitHub Actions UI with Publish Packages, with gh workflow run publish-packages.yml --ref main, or by pushing a v* tag. The workflow runs the package export, tarball, type, test, documentation, and sem-doc contract gates before publishing the Foundation contracts and sem-doc in dependency order. Manual runs select oidc (npm Trusted Publishing, the default) or token (the NPM_TOKEN Actions secret) through the publish_auth input. Tag runs use OIDC. Each package must have a matching npm Trusted Publisher configuration before its first OIDC publish. The workflow stores package publication logs and runs a clean-consumer CLI smoke test after npm metadata becomes visible. This workflow publishes npm packages; it does not create a GitHub Release entry.

To use a different sem executable for the direct CLI commands below, run from the context-action root and set SEM_BIN explicitly:

export SEM_BIN="$PWD/packages/sem-doc/node_modules/.bin/sem"

Git diff

# HEAD vs working tree, including untracked files
pnpm --filter @context-action/sem-doc exec node dist/cli.js diff --json

# Native Git diff for selected paths
pnpm --filter @context-action/sem-doc exec node dist/cli.js diff src spec --context 1

# HEAD vs index; untracked files are excluded by default
pnpm --filter @context-action/sem-doc exec node dist/cli.js diff --staged --json

# Raw sem semantic diff remains explicit
pnpm --filter @context-action/sem-doc exec node dist/cli.js sem-diff --format json

sem-doc diff records the Git HEAD, an on-disk working-tree digest, file status, hunks, and addition/deletion counts. Binary files are reported without text parsing. Very large text files use a coarse prefix/suffix diff and declare that precision. The service checks the repository revision before and after analysis and rejects a stale result.

The subprocess buffer defaults to 64 MiB for Git and 32 MiB for direct sem calls; both are bounded by a 1 GiB safety maximum. Git request paths are limited to 4,096 entries (4,096 characters each) and hunk context to 4,096 lines; files with more than 20,000 combined old/new lines use the declared coarse diff path. Work-context timeout/output options use the same 1 hour/1 GiB maximum and retain the aggregate budget across version, impact, and context calls.

Work context

# Complete 2-hop symbol inventory, token-budgeted context, and document backlinks
pnpm --filter @context-action/sem-doc exec node dist/cli.js work-context SemClient --file src/sem-client.ts --docs-root spec

# Direct relationships only
pnpm --filter @context-action/sem-doc exec node dist/cli.js work-context SemClient --file src/sem-client.ts --depth 1

# Explicitly include the direct node_modules surface (package internals remain bounded to one hop)
pnpm --filter @context-action/sem-doc exec node dist/cli.js work-context SemClient \
  --file src/sem-client.ts --include-node-modules-surface --json

# Index TSDoc H1 [[Symbol]] definitions and backlinks
pnpm --filter @context-action/sem-doc exec node dist/cli.js docs index spec --json

# Validate classified SSOT bindings against the current sem entity catalog
pnpm --filter @context-action/sem-doc exec node dist/cli.js docs validate-bindings spec --strict --json

--depth accepts only 1 or 2 and is passed to both sem impact traversal and context --hops. The underlying sem CLI defaults to impact depth 2 but context hops 0 (unbounded, token-budgeted); sem-doc deliberately overrides both through its own default depth 2 so work-context collection is bounded and reproducible. --budget limits source excerpts, not the complete symbol inventory. Interactive consumers SHOULD request 2 hops once and derive a 1-hop view with selectWorkContextHops(report.symbols, 1) instead of issuing a second sem query. The work-context report preserves sem version, exact arguments, Git revision, sem payloads, and matching document definitions/backlinks.

--include-node-modules-surface is deliberately opt-in. It is recorded in the exact SEM argument provenance (--no-default-excludes) and therefore remains visible in serialized scopes and history snapshots. The one-hop projection is a collection boundary, not a claim that every file in a package was analyzed; use the repository-level entity snapshot lane when a complete revision inventory is required.

For conventional static TypeScript imports, sem entity identity and scope resolution are sufficient for the required source mapping. Named/default/namespace aliases resolve to the original entity ID, while parameters and local bindings that shadow an import do not resolve to that external entity. Same-named entities in different files remain distinct because work-context preserves entity ID, type, definition file, and startLine/endLine. Exact reference-site line/column tracking is not a requirement. Dynamic or computed patterns that sem cannot statically resolve are not treated as guaranteed mappings.

Documents use a unique double-bracket H1 checkpoint for routing and bind it to code through semDocumentKind: code, plus semEntityId, semEntityName, semEntityType, and semEntityFile frontmatter. Work-context never falls back to checkpoint or entity name alone. Missing exact provenance produces unresolved, while incomplete metadata and duplicate checkpoint/entity bindings fail indexing. See ADR-0003.

The repository convention distinguishes code-backed SSOT documents from conceptual or process guides. Any document intended to explain one implementation symbol MUST set semDocumentKind: code, use the four binding fields, and resolve in the work-context for that change. Conceptual, architecture, process, and tooling documents MAY remain document-only; external-reference documents may describe dependency surface evidence but do not own node_modules symbols. Use strict binding validation and a representative work-context run as review evidence.

The normative rules are in the Document Entity Binding Convention. Binding validation emits sem-doc-binding-validation.v2; --strict requires every document to declare a kind, requires exact bindings for code documents, and rejects bindings on non-code documents. Document-only checkpoints remain allowed when explicitly classified.

The accepted scope boundary is recorded in ADR-0001: work-context uses only entities exposed by sem and does not add a second AST index to enumerate every declaration inside a function scope.

--depth bounds sem's transitive impact traversal; impact.entities[].depth is the authoritative hop. Direct dependencies and dependents are 1-hop. Affected tests are separate evidence and are not assigned a hop unless the same entity also appears in the bounded impact result. The same depth bounds sem context collection through --hops.

usageFiles is the sorted, deduplicated file list of sem-reported dependent entities for the target symbol. It is a structural file-level signal, not exact reference locations or a runtime call graph.

Scoped performance benchmark

# Adapter/orchestration baseline with fake sem
pnpm --filter @context-action/sem-doc benchmark:scope

# Real sem engine measurement
SEM_BIN=/path/to/sem pnpm --filter @context-action/sem-doc benchmark:scope

The lane measures individual 1-hop and 2-hop work-context, separate 1+2-hop queries, one shared 2-hop query with a derived 1-hop view, and typed sem entity diff. It records timing distributions and environment metadata without a wall-clock pass/fail threshold. Fake-sem results must not be interpreted as real sem engine performance. The diff and benchmark decision is recorded in ADR-0002.

Development and verification

pnpm --filter @context-action/sem-doc typecheck
pnpm --filter @context-action/sem-doc verify:boundary
pnpm --filter @context-action/sem-doc lint
pnpm --filter @context-action/sem-doc build
pnpm --filter @context-action/sem-doc verify:pack
pnpm --filter @context-action/sem-doc test

# Use the pinned repository binary
pnpm --filter @context-action/sem-doc verify:poc

# Or override it with another sem build
SEM_BIN=/path/to/sem pnpm --filter @context-action/sem-doc verify:poc

packages/sem-doc is the implementation home for the canonical tooling package. The former standalone checkout and the old tsdoc-edge copy are no longer source locations; new sem-doc features should land in this workspace package first. The package keeps @context-action/sem-foundation-* as explicit runtime dependencies and is published by the tooling repository workflow, not by a consumer Lerna flow.