weavatrix-scan

A deterministic, path-safe repository scanner — written in Rust, exposed to Node.js and Bun through Node-API.

The default API produces a manifest: normalized paths, byte sizes, optional content hashes, an aggregate revision, ignore-rule provenance, typed evidence for everything it skipped, and hard bounds it will not exceed. scanPaths is the ignore-aware sorted path list when you do not need that report. It executes no repository code.

npm install weavatrix-scan
# or
bun add weavatrix-scan

const { scanPaths, scanRepository } = require('weavatrix-scan')

const paths = await scanPaths(process.cwd())
const report = await scanRepository(process.cwd(), {
  extensions: ['js', 'ts', 'rs'],
  selectedFilesOnly: true,
  maxFileBytes: 2_000_000,
})

report.revision        // one hash for the whole selection
report.files           // [{ relative, bytes, content_hash?, binary_checked }]
report.skipped         // why each excluded entry was excluded
report.complete        // false when a bound stopped the scan

The package is CommonJS. ESM and TypeScript import the same bindings:

import { scanPaths, scanPathsSync, scanRepository } from 'weavatrix-scan'

| Need | Call | | --- | --- | | Sorted relative paths only | scanPaths / scanPathsSync | | Sizes, hashes, revision, skip evidence | scanRepository / scanRepositorySync | | Fastest useful manifest | scanRepository(..., { metadataOnly: true, selectedFilesOnly: true }) | | Watcher updates on one live snapshot | ScanSession |

Why a manifest instead of a list of paths

Two runs over the same tree must produce the same bytes, or nothing built on top can be cached, diffed, or trusted. So the report carries:

• revision — an aggregate hash of the selection. Equal revisions mean equal selections, without comparing file lists.
• skipped — typed evidence for every exclusion, so "the file is missing from your output" always has an answer.
• ignore_sources — which ignore files were consulted and their content hashes, so a selection can be explained and reproduced.
• complete / termination — an explicit statement that a bound was hit, instead of a silently short list.

API

scanPaths(root, options?) → Promise<string[]>

Sorted repository-relative paths from native code. Same ignore and extension selection as a scan, without building or JSON-encoding a portable report. Runs on the Node-API worker pool.

scanPathsSync(root, options?) → string[]

The blocking form of scanPaths.

How to use it

const path = require('node:path')
const { scanPaths, scanPathsSync } = require('weavatrix-scan')

const root = process.cwd()

// Ignore-aware, vendor dirs skipped, every regular file.
const all = scanPathsSync(root)

const sources = await scanPaths(root, {
  extensions: ['ts', 'tsx', 'js', 'jsx'],
  skipHidden: true,
})

// Re-include a vendor tree that standardSkips would drop.
const dts = scanPathsSync(root, {
  extensions: ['d.ts'],
  overrideRules: ['!node_modules/@types/**'],
})

// No gitignore, no node_modules skip — closer to a bare walker.
const raw = scanPathsSync(root, {
  ignorePolicy: 'none',
  standardSkips: false,
})

const absolute = sources.map((relative) => path.join(root, ...relative.split('/')))

const controller = new AbortController()
const pending = scanPaths(root, { signal: controller.signal })
controller.abort()
await pending.catch((error) => {
  // error.code === 'GenericFailure'
})

• Paths use / on every platform and are already sorted.
• Default selection matches scanRepository: .gitignore (and the configured ignore set), standardSkips, plus any extensions or overrideRules.
• It does not read contents, hash, compute revision, or explain skips.
• metadataOnly, selectedFilesOnly, compact, hashFileContents, maxFileBytes, and maxTotalBytes are ignored. Oversized files still appear.
• signal cancels the walk and the promise rejects. scanRepository instead returns complete: false.
• maxEntries / maxDepth / parallelism still apply. Hitting maxEntries can return a shorter list without throwing.
• Join with path.join(root, ...relative.split('/')) only when a host API needs a native path. Keep the / form for anything you persist or compare.

On a tree that does not hit size limits, scanPathsSync(root) equals scanRepositorySync(root, { metadataOnly: true, selectedFilesOnly: true }) .files.map((file) => file.relative).

scanRepository(root, options?) → Promise<ScanReport>

Runs on the native worker pool; the JavaScript event loop stays free.

scanRepositorySync(root, options?) → ScanReport

The blocking form, for CLIs and controlled startup paths.

ScanSession

Keeps the last native report and applies a watch plan without asking JavaScript to hold two giant JSON trees. Prefer ScanSession.open() / applyWatchPlan() so the constructor and update can run off the JavaScript thread. files({ batchSize }) reads one snapshot generation; an update between pages throws instead of mixing manifests.

exportScanCache(root, options?) → Promise<ScanCache>

Returns local reusable hash evidence (format_version, root, entries). This is not a compact manifest and is empty in metadata-only mode.

scanDiagnostics()

Reports the Rust core version, npm package version, target triple, and that musl is unsupported.

| Parameter | Type | Notes | | --- | --- | --- | | root | string | Repository root. Must be an existing directory. | | options | ScanOptions | See below. |

ScanOptions

| Option | Type | Default | Effect | | --- | --- | --- | --- | | extensions | string[] | all | Restricts selection to these extensions, without a leading dot. | | overrideRules | string[] | — | Gitignore-syntax rules applied above discovered ignore files. A leading ! re-includes. | | metadataOnly | boolean | false | Skips content reads: no hashing, no binary detection. The fastest useful mode. | | selectedFilesOnly | boolean | false | Returns only selected files and drops per-entry skip records, which keeps memory flat on very large trees. | | skipHidden | boolean | scanner default | Whether dotfiles and dot-directories are skipped. | | standardSkips | boolean | true | Skip generated/vendor directories such as node_modules. | | ignorePolicy | string | repository | repository, none, or gitCompatible. | | hashFileContents | boolean | true | Set false for metadata-only consumers. | | compact | boolean | false | Return a compact manifest (files, revision, complete, termination) instead of the portable report. Use exportScanCache() for the local hash cache. | | signal | AbortSignal | — | Cancels the native scan. scanRepository then has an explicit termination; scanPaths throws. | | maxFileBytes | number | scanner default | Files above this are skipped with typed evidence rather than read. | | maxEntries | number | unbounded | Hard entry bound. Hitting it sets complete: false and termination. | | maxTotalBytes | number | unbounded | Hard byte bound, same reporting. | | maxDepth | number | unbounded | Traversal depth bound. | | parallelism | number | available | Worker count. |

ScanReport

The report is the crate's portable report, so its field names are the serialized Rust names.

| Field | Type | Meaning | | --- | --- | --- | | files | ScannedFile[] | The selection, in deterministic order. | | skipped | { relative, kind, detail_hash? }[] | One record per exclusion. kind names the reason. | | warnings | { relative?, message_hash }[] | Non-fatal problems, hashed so a report never leaks message text. | | ignore_sources | { kind, repository_relative?, content_hash }[] | Which ignore inputs shaped this selection. | | revision | string | Aggregate hash of roots, paths, and content. | | complete | boolean | false when a bound stopped the scan. | | termination | string \| undefined | Which bound: entries, total bytes, timeout, or cancellation. | | selection_portable | boolean | Whether the selection is reproducible on another platform. |

ScannedFile

| Field | Type | Meaning | | --- | --- | --- | | relative | string | Forward-slash path relative to root, on every platform. | | bytes | number | File size. | | content_hash | string \| undefined | Present unless metadataOnly was set. | | binary_checked | boolean | Whether binary detection actually ran on this file. |

Errors

| code | Cause | | --- | --- | | InvalidArg | Unknown option key, or malformed option JSON. | | GenericFailure | Root missing, unreadable, or not a directory. scanPaths also uses this when signal cancels or a timeout fires. |

What ships

| | | | --- | --- | | Runtimes | Node.js 18+ (Node-API 8), Bun 1.4+ | | Platforms | Windows x64/arm64, macOS x64/arm64, glibc Linux x64/arm64 | | Install script | none | | Network at install | none | | Runtime dependencies | none | | Platform packages | none — all six bindings are in this one tarball | | Writes to disk | none |

Measured

benchmark/RESULTS.md is generated from the weavatrix-benchmarks harness, which forces both sides to return the identical array before either is timed. The competitor is fdir, the fastest widely used Node crawler.

Medians of three independent runs over 20,000 files:

| Contract | Node 24 | Bun 1.3 | | --- | ---: | ---: | | Sorted relative paths (scanPaths) | 1.33x (1.31–1.50) | 1.53x (1.52–1.74) | | Sorted paths plus byte sizes | 8.25x (7.92–8.32) | 12.20x (11.80–13.74) |

The first row times scanPaths against fdir: same sorted path array, no portable report. The second row is the equal consumer-facing manifest contract, where fdir needs one statSync per path.

Scan owns its repository, package, release evidence, and MIT license, and can be used entirely on its own.

Repository: Weavatrix/weavatrix-scan · Rust crate: crates.io/crates/weavatrix-scan · License: MIT