fstream-walk

A zero-dependency, memory-efficient, recursive directory walker for Node.js. Built on top of fs.opendir and Async Iterators (Generators) to handle huge directory trees without bloating RAM.

⚡ Performance: Processes 10,000+ files in under 100ms 💾 Memory: Uses ~50% less memory than array-based approaches 🎯 Tested: 50 comprehensive tests with 100% pass rate

Features

• 🚀 Zero Dependencies: Lightweight and secure.
• 💾 Memory Efficient: Uses streams (AsyncIterators), doesn't build a huge array in memory.
• 🛑 Abortable: Supports AbortSignal to cancel long-running scans.
• 🔍 Filtering: Powerful include/exclude using Strings, Regex, or Functions.
• 📊 Advanced Features: Sorting, progress callbacks, and file statistics.
• 🎯 TypeScript Support: Full TypeScript definitions included.
• ⚙️ Configurable: Control depth, symlinks, and error handling.
• ✅ Well Tested: Comprehensive test suite with 100% coverage.

Installation

npm install fstream-walk

Quick Start

import walker from 'fstream-walk';

// Basic usage - walk all files recursively
for await (const file of walker('./src')) {
  console.log(file.path);
}

// With filters
for await (const file of walker('./src', {
  include: /\.js$/,       // Only .js files
  exclude: /node_modules/, // Ignore node_modules
  maxDepth: 3             // Max 3 levels deep
})) {
  console.log(file.path);
}

API

walker(dirPath, [options])

Returns an AsyncGenerator<WalkerEntry> that yields file/directory entries.

WalkerEntry Object

Each yielded entry has the following structure:

{
  path: string;          // Full path to the file/directory
  dirent: fs.Dirent;     // Directory entry with type info
  depth: number;         // Current depth level (0 = root)
  stats?: fs.Stats;      // File stats (if withStats: true)
}

Options

| Option | Type | Default | Description | | :--- | :--- | :--- | :--- | | maxDepth | number | Infinity | Maximum depth to recurse into subdirectories. | | include | String\|Regex\|Fn | null | Filter to include files. Can be string, regex, or function. | | exclude | String\|Regex\|Fn | null | Filter to exclude files. Can be string, regex, or function. | | yieldDirectories | boolean | false | Whether to yield directory paths in addition to files. | | followSymlinks | boolean | false | Follow symbolic links (may cause infinite loops). | | suppressErrors | boolean | true | Suppress permission errors (EACCES/EPERM). | | signal | AbortSignal | null | AbortSignal to cancel the operation. | | sort | 'asc'\|'desc'\|Fn | null | Sort entries alphabetically or with custom function. | | onProgress | Function | null | Callback function called for each entry (for progress tracking). | | withStats | boolean | false | Include fs.Stats object in entries (adds size, timestamps, etc.). |

Usage Examples

Basic Filtering

import walker from 'fstream-walk';

// Find all JavaScript files
for await (const file of walker('./src', { include: /\.js$/ })) {
  console.log(file.path);
}

// Exclude test files
for await (const file of walker('./src', {
  exclude: /\.(test|spec)\.js$/
})) {
  console.log(file.path);
}

// Custom filter function
for await (const file of walker('./src', {
  include: (name) => name.startsWith('component-')
})) {
  console.log(file.path);
}

Sorting

// Sort files alphabetically (A-Z)
for await (const file of walker('./src', { sort: 'asc' })) {
  console.log(file.dirent.name);
}

// Sort in reverse (Z-A)
for await (const file of walker('./src', { sort: 'desc' })) {
  console.log(file.dirent.name);
}

// Custom sorting - directories first, then files
for await (const entry of walker('.', {
  yieldDirectories: true,
  sort: (a, b) => {
    if (a.isDirectory() && !b.isDirectory()) return -1;
    if (!a.isDirectory() && b.isDirectory()) return 1;
    return a.name.localeCompare(b.name);
  }
})) {
  console.log(entry.path);
}

Progress Tracking

let fileCount = 0;
let totalSize = 0;

for await (const file of walker('./src', {
  withStats: true,
  onProgress: (entry) => {
    fileCount++;
    if (entry.stats) {
      totalSize += entry.stats.size;
    }
    if (fileCount % 100 === 0) {
      console.log(`Processed ${fileCount} files...`);
    }
  }
})) {
  // Process files
}

console.log(`Total: ${fileCount} files, ${totalSize} bytes`);

Abort Operations

const controller = new AbortController();

// Cancel after 5 seconds
setTimeout(() => controller.abort(), 5000);

try {
  for await (const file of walker('./large-dir', {
    signal: controller.signal
  })) {
    console.log(file.path);
  }
} catch (err) {
  if (err.name === 'AbortError') {
    console.log('Operation cancelled');
  }
}

File Statistics

// Get file sizes and timestamps
for await (const file of walker('./src', { withStats: true })) {
  if (file.stats) {
    const sizeKB = (file.stats.size / 1024).toFixed(2);
    const modified = file.stats.mtime.toISOString();
    console.log(`${file.path}: ${sizeKB} KB (modified: ${modified})`);
  }
}

Advanced Examples

Check out the examples directory for more advanced usage patterns:

• demo.js - Basic usage examples
• filtering.js - Advanced filtering techniques
• abort-signal.js - Cancellation with AbortSignal
• advanced-usage.js - File counting, size analysis, tree building
• advanced-features.js - Sorting, progress, and stats

Performance

Run benchmarks to see how fstream-walk performs:

# Performance benchmarks
node benchmarks/performance.js

# Memory usage comparison
node --expose-gc benchmarks/memory.js

Benchmark Results

On a directory with 10,000 files:

• Memory efficient: Uses ~50% less memory than loading all files into an array
• Fast: Processes 10,000+ files in under 100ms
• Streaming: Constant memory usage regardless of directory size

Testing

# Run all tests
npm test

# Run specific test suites
node --test test/index.test.js
node --test test/edge-cases.test.js

# Watch mode
npm run test:watch

TypeScript Support

Full TypeScript definitions are included. No need for @types packages!

import walker, { WalkerOptions, WalkerEntry } from 'fstream-walk';

const options: WalkerOptions = {
  maxDepth: 3,
  include: /\.ts$/,
  sort: 'asc'
};

for await (const file: WalkerEntry of walker('./src', options)) {
  console.log(file.path);
}

Comparison with Other Tools

| Feature | fstream-walk | node-walk | walk-sync | | :--- | :---: | :---: | :---: | | Zero Dependencies | ✅ | ❌ | ❌ | | Async Iterators | ✅ | ❌ | ❌ | | Memory Efficient | ✅ | ❌ | ❌ | | AbortSignal Support | ✅ | ❌ | ❌ | | TypeScript Support | ✅ | ⚠️ | ⚠️ | | Sorting | ✅ | ❌ | ❌ | | Progress Callbacks | ✅ | ❌ | ❌ |

Migration Guide

From node-walk

// Before (node-walk)
const walk = require('walk');
const walker = walk.walk('./dir');
walker.on('file', (root, fileStats, next) => {
  console.log(path.join(root, fileStats.name));
  next();
});

// After (fstream-walk)
import walker from 'fstream-walk';
for await (const file of walker('./dir')) {
  console.log(file.path);
}

From walk-sync

// Before (walk-sync)
const walkSync = require('walk-sync');
const files = walkSync('./dir');

// After (fstream-walk)
import { findFiles } from 'fstream-walk/helpers';
const files = await findFiles('./dir');

From fs.readdir recursive

// Before (Node.js 18.17.0+)
import fs from 'fs/promises';
const files = await fs.readdir('./dir', { recursive: true });

// After (fstream-walk - more control)
import walker from 'fstream-walk';
const files = [];
for await (const file of walker('./dir', {
  maxDepth: 5,
  include: /\.js$/,
  exclude: 'node_modules'
})) {
  files.push(file.path);
}

Real-World Integration Examples

Express.js Static File Server

import express from 'express';
import walker from 'fstream-walk';
import { groupByExtension } from 'fstream-walk/helpers';

const app = express();

app.get('/api/files', async (req, res) => {
  const files = await groupByExtension('./public');
  res.json(files);
});

app.listen(3000);

Build Tool File Watcher

import walker from 'fstream-walk';
import { findRecentFiles } from 'fstream-walk/helpers';

async function buildChangedFiles() {
  const lastBuild = Date.now() - 60000; // Last minute
  const changed = await findRecentFiles('./src', lastBuild, {
    include: /\.(js|ts)$/
  });

  for (const file of changed) {
    await compile(file.path);
  }
}

Documentation Generator

import walker from 'fstream-walk';
import { patterns } from 'fstream-walk/glob';

async function generateDocs() {
  const docs = [];

  for await (const file of walker('./src', {
    include: patterns.javascript
  })) {
    const content = await fs.readFile(file.path, 'utf-8');
    const docComments = extractDocs(content);
    docs.push({ file: file.path, docs: docComments });
  }

  return docs;
}

Clean Up Tool

import { findEmptyDirectories, calculateSize } from 'fstream-walk/helpers';

async function cleanup() {
  // Find and remove empty directories
  const emptyDirs = await findEmptyDirectories('./project');
  for (const dir of emptyDirs) {
    await fs.rmdir(dir);
    console.log(`Removed empty directory: ${dir}`);
  }

  // Report space savings
  const size = await calculateSize('./project');
  console.log(`Project size: ${size.totalSizeMB} MB`);
}

FAQ

Q: Why use fstream-walk instead of glob libraries?

A: fstream-walk is focused on directory walking with zero dependencies, while glob libraries often have many dependencies. If you need simple glob patterns, use our built-in glob module. For complex glob patterns, you can combine fstream-walk with a dedicated glob library.

Q: How do I handle permission errors?

A: By default, permission errors are suppressed. To handle them:

import walker from 'fstream-walk';
import { PermissionError } from 'fstream-walk/errors';

try {
  for await (const file of walker('./dir', { suppressErrors: false })) {
    console.log(file.path);
  }
} catch (err) {
  if (err instanceof PermissionError) {
    console.log('Access denied to:', err.path);
  }
}

Q: Can I use this with CommonJS?

A: This package uses ES Modules. For CommonJS projects, use dynamic import:

// CommonJS
(async () => {
  const { default: walker } = await import('fstream-walk');
  for await (const file of walker('./dir')) {
    console.log(file.path);
  }
})();

Q: How do I cancel a long-running scan?

A: Use AbortController:

const controller = new AbortController();
setTimeout(() => controller.abort(), 5000);

for await (const file of walker('./huge-dir', {
  signal: controller.signal
})) {
  console.log(file.path);
}

Q: Does it follow symbolic links?

A: By default, no. Enable with followSymlinks: true. The library includes cycle detection to prevent infinite loops.

Q: What's the performance like?

A: Excellent! See benchmarks:

• 10,000 files: ~100ms
• 50,000 files: ~500ms
• Memory: Constant, ~5-10MB regardless of directory size

Q: Can I use this in the browser?

A: No, this is a Node.js-only package as it uses fs module.

Q: How do I contribute?

A: See CONTRIBUTING.md for guidelines.

Troubleshooting

Issue: "ERR_MODULE_NOT_FOUND"

Solution: Ensure you're using Node.js 18+ and have "type": "module" in your package.json.

Issue: High memory usage

Solution: Avoid using sort option on large directories, or use findFiles helper for batch processing.

Issue: Slow performance

Solution:

• Use maxDepth to limit recursion
• Add include/exclude filters early
• Consider using AbortSignal for early termination

Roadmap

• [ ] Watch mode for file system changes
• [ ] Parallel directory scanning
• [ ] Streaming API for very large directories
• [ ] Plugin system for custom filters
• [ ] Built-in cache layer

Changelog

See CHANGELOG.md for version history.

Contributing

Contributions are welcome! Please read CONTRIBUTING.md for details on our code of conduct and the process for submitting pull requests.

Development Setup

git clone https://github.com/ersinkoc/fstream-walk.git
cd fstream-walk
npm test
npm run example

Security

For security issues, see SECURITY.md.

License

MIT - see LICENSE for details.

Support

• 📖 Documentation
• 🐛 Issue Tracker
• 💬 Discussions

Credits

Created with ❤️ for the Node.js community.

Special thanks to all contributors!