📄 readline-pager

⚡ High-performance paginated file reader for Node.js. Process large text files efficiently without loading them into memory.

• 📦 Zero dependencies
• ⚡ Up to ~3× faster than Node.js readline
• 🚀 Up to ~6× faster with optional native C++ acceleration
• 🔁 Async (for await...of) and sync (for...of) iteration
• 📄 Page-based reading with manual control (next, nextSync)
• 🔀 Forward and backward reading support
• 🧪 Fully typed with over 90% test coverage

Important:
Performance depends heavily on the chunkSize option. Tune it for your storage device. A value of 64 KiB is usually a good starting point. Increasing it may improve throughput until you reach the best value for your hardware.

📦 Installation

npm install readline-pager

🚀 Quick start

import { createPager } from "readline-pager";

for await (const page of createPager("./bigfile.txt")) {
  console.log(page[0]);
}

Other usage patterns

import { createPager, createNativePager } from "readline-pager";

// Sync iteration
for (const page of createPager("./bigfile.txt")) {
}

// Manual async
const pager = createPager("./bigfile.txt");
while (true) {
  const page = await pager.next();
  if (!page) break;
}

// Manual sync
let page;
const pager = createPager("./bigfile.txt");
while ((page = pager.nextSync()) !== null) {}

// Native C++
for (const page of createNativePager("./bigfile.txt")) {
}

⚙️ Options

createPager(filepath, {
  chunkSize?: number,           // default: 64 * 1024 (64 KiB)
  pageSize?: number,            // default: 1_000
  delimiter?: string,           // default: "\n"
  prefetch?: number,            // default: 8
  backward?: boolean,           // default: false
  output?: "string" | "buffer", // default: "string"
});

createNativePager(filepath, {
  pageSize?: number,            // default: 1_000
  delimiter?: string,           // default: "\n"
  backward?: boolean,           // default: false
  output?: "string" | "buffer", // default: "string"
});

• chunkSize — Number of bytes read per I/O operation.
• pageSize — Number of lines per page.
• delimiter — Line separator.
• prefetch — Maximum number of pages buffered internally.
• backward — Read the file from end to start.
• output — Controls the page data type.

Note: createNativePager requires x86 AVX2 or ARM NEON CPU instruction set extensions and will throw if they are not available. It also does not support multi-character delimiters because it uses fast SIMD-based scanning.

📚 API

pager.next(): Promise<string[] | Buffer | null>

Returns the next page asynchronously.

Returns null when the end of the file is reached.

Empty lines are preserved.

Note: Unlike Node.js readline, which may skip empty files or leading empty lines, readline-pager always returns all lines.

• A completely empty file (0 bytes) produces [""] on the first read.
• A file containing multiple empty lines returns each line as an empty string.

pager.nextSync(): string[] | Buffer | null

Synchronous version of pager.next().

Returns the next page immediately or null when the end of the file is reached.

pager.close(): Promise<void>

Stops reading and releases resources asynchronously. Safe to call at any time.

📊 Benchmark

Run the benchmark locally:

npm run benchmark:node

# or customize with args
node test/benchmark.ts --lines=20000 --page-size=500 --backward

Test setup: generated text files (UUID lines), NVMe SSD, Node.js runtime. Results are averaged across multiple runs. Actual performance depends on hardware.

⚡ Throughput (MB/s)

| Method | 1M lines (35 MB) | 10M lines (353 MB) | 100M lines (3.5 GB) | 1B lines (35.3 GB) | | ---------------------- | ---------------: | -----------------: | ------------------: | -----------------: | | readline | ~370 MB/s | ~460 MB/s | ~460 MB/s | ~460 MB/s | | readline-pager (JS) | ~1100 MB/s | ~1300 MB/s | ~1300 MB/s | ~1150 MB/s | | readline-pager (C++) | ~2200 MB/s | ~2500 MB/s | ~2500 MB/s | ~2450 MB/s |

🛠 Development & Contributing

• Minimum supported Node.js: v18.12
• Development/test environment: Node v25.8 and TypeScript v6.0

Run tests:

npm i
npm test

Contributions are welcome. Open an issue or submit a PR.

📜 License

MIT — © Morteza Jamshidi