omnifs

v0.1.0

Published

11 days ago

Rust-backed filesystem discovery engine for Node.js focused on streaming traversal, incremental rescans, and scalable directory indexing.

Downloads

107

# OmniFS

🏗️ Architecture & Intent

omnifs is a low-level filesystem engine designed to bridge the performance gap between Node.js and native system calls. By offloading recursive traversal, glob filtering, and hashing to a multi-threaded Rust core via napi-rs, it reduces pressure on the Node.js event loop during large traversal workloads and minimizes garbage collection overhead.

Key Capabilities

Parallel Traversal: Adaptive worker pooling to saturate available I/O bandwidth.
Ignore Semantics: Native support for recursive .gitignore hierarchy.
Incremental Mode: JSONL-backed indexing to skip unchanged subtrees in subsequent scans.
Streaming API: Async Iterator interface with backpressure-aware batching.
Integrity Pipeline: Optional blake3 hashing integrated directly into traversal.

⚖️ Capability Comparison

| Capability | Path Walkers | Glob Utilities | omnifs | | :--- | :---: | :---: | :---: | | Rust-Powered Engine | ❌ | ❌ | ✅ | | Async Streaming API | ⚠️ Partial | ✅ | ✅ | | Ignore-aware Traversal | ⚠️ Limited | ⚠️ Pattern-only | ✅ | | Incremental Rescans | ❌ | ❌ | ✅ | | Native Hashing Pipeline | ❌ | ❌ | ✅ | | Deterministic Ordering | ❌ | ❌ | ✅ |

📊 Performance Characteristics

omnifs is optimized for balanced throughput, low latency streaming, and stable memory usage.
Observed ranges below were measured on a ~100,000 file dataset.

Performance varies depending on filesystem type, dataset structure, CPU, and storage medium.

Ignore-aware Traversal: ~600–700ms total runtime.
Latency to First Result: ~2–5ms.
Memory Footprint: ~120–180MB peak RSS under sustained load.
Incremental Rescan: ~80ms via metadata-based subtree skipping.
Hashing (blake3): ~4–6s for full-content hashing across 100k files.

🚀 Usage

Installation

omnifs ships a JS wrapper plus platform-specific native binaries selected automatically at install/runtime.

npm install omnifs

Native binaries load lazily at runtime. No build step is required on supported platforms.

⚠️ Troubleshooting (Linux CI / Docker)

Some Linux CI environments skip optional native dependencies by default. If the native engine fails to load:

npm install --include=optional omnifs

If npm fails to recover optional dependencies due to lockfile state:

rm -rf node_modules package-lock.json
npm install

This resolves known npm optional-dependency edge cases in strict CI setups.

If you're using pnpm, ensure pnpm is not configured to ignore optional dependencies.

Basic Discovery

Returns an AsyncGenerator<FileMetadata>.

import { discover } from "omnifs";

for await (const file of discover("./src")) {
  console.log(`${file.path} — ${file.size} bytes`);
}

Batched Processing

import { discoverBatched } from "omnifs";

for await (const batch of discoverBatched(".", { batchSize: 512 })) {
  await Promise.all(batch.map(file => process(file)));
}

📦 Module Compatibility

omnifs works across modern Node.js module systems.

ESM

import { discover } from "omnifs";

CommonJS

const { discover } = require("omnifs");

TypeScript

import type { FileMetadata } from "omnifs";

📄 File Metadata

Each emitted entry contains:

path: string
size: number
mtimeMs: number
identity: string
hash?: string | null

⚙️ Configuration Options

🧩 Logical Semantics

Modes

auto — Default optimized path.
crawl — Full traversal.
glob — Pattern-focused mode.
ignore — Exclusion-focused traversal.

Hash & Incremental Rules

Incremental + No Hash → Metadata comparison only.
Incremental + blake3 → Hash changed files.
Standard + blake3 → Hash all emitted files.

⚠️ When NOT to use omnifs

If you only need simple glob expansion without metadata or streaming guarantees, lightweight JS glob libraries may be faster.

🛠️ Requirements & Platform Support

Node.js >=20
ESM / CommonJS / TypeScript supported
Linux: x64/arm64 (gnu + musl)
macOS: x64/arm64
Windows: x64/arm64