Persistence

Persistence - a performant, durable filesystem storage layer.

Introduction

Persistence is a filesystem persistent storage layer. It provides hierarchical read/write locking, durability, and atomic‑style writes. You can use Persistence as a drop‑in library to safely coordinate reads and writes to the filesystem. Reads on the same file are concurrent; however, reads are partitioned by writes and writes are processed in order of arrival (FIFO).

Features

• A zero-dependency filesystem storage layer.
• Coordinates reads and writes with hierarchical path locks.
• Provides atomic-style file replacement (temp file + rename).
• Flushes directory metadata for stronger durability on write/delete when durability is enabled.
• FIFO: for any two conflicting operations where at least one is a write, acquisition respects arrival order.

It is intentionally minimal: one lock manager, one client, one clear set of semantics.

Usage

Setup

import { once } from "node:events";
import { Client, LockManager } from "@far-analytics/persistence";

const manager = new LockManager();
const client = new Client({ manager, durable: true });

Write to a file

await client.write("/tmp/example.json", JSON.stringify({ message: "Hello, World!" }));

Read from a file

const data = await client.read("/tmp/example.json", "utf8");
console.log(JSON.parse(data)); // { message: "Hello, World!" }

Collect directory contents

const entries = await client.collect("/tmp", { encoding: "utf8", withFileTypes: false });
console.log(entries); // ['example.json']

Delete a file or directory

await client.delete("/tmp/example.json");

Create a write stream and write to a file

const ws = await client.createWriteStream("/tmp/example.json");
ws.write(JSON.stringify({ message: "Streaming Hello, World!" }) + `\n`);
ws.end();
await once(ws, "finish");

Create a read stream and read from a file

const rs = await client.createReadStream("/tmp/example.json");
rs.pipe(process.stdout); // {"message":"Streaming Hello, World!"}
await once(rs, "close");

Examples

"Hello, World!"

Please see the Usage section above or the Hello, World! example for a working implementation.

Locking model

• Per-operation hierarchical locking within a single LockManager instance.
• Write partitioned FIFO: for any two conflicting operations where at least one is a write, acquisition respects arrival order.
• Read/collect operations can overlap other reads on the same path or within the same ancestor/descendant subtree.
• Write/delete operations are exclusive: a write on a path excludes all reads and writes on that path and any ancestor/descendant paths until the write is complete.

Scalability

Persistence can scale across multiple clients as long as all operations route through a single authoritative LockManager (for example, a shared in-process instance or a single lock service accessed over RPC). If multiple independent LockManager instances are used, they do not coordinate and safety guarantees no longer hold.

Durability

When a client instance is instantiated with { durable: true }, writes are flushed and parent directories are fsync’d to reduce the chance of data loss after a crash. Durability guarantees are best‑effort and depend on filesystem and OS behavior.

Atomicity

Persistence supports atomic-style file replacement via temp file + rename for write and createWriteStream.

Limitations

• The directory structure that Persistence operates on is assumed to be hierarchical.
• Hence, symlinks/aliases are not supported.
• When durability is enabled, fsync on directories is considered best‑effort and behaves differently on different filesystems.
• Distributed locking or coordination across multiple independent LockManager instances is not supported.
• Protection against external processes that bypass the client and write directly to disk.
• Testing to date has been limited to Linux on ext4.

Planned

• Optional lock timeout / abort signal.
• Temp file cleanup utility (i.e. in the event of a crash or power loss).
• A couple more tests for symlink/path edge cases.

API

The Persistence API provides a path-aware lock manager and a filesystem client that uses it for safe reads and writes.

The Client class

new persistence.Client(options)

• options <ClientOptions> Options passed to the Client.
  • manager <LockManager> The lock manager instance used to coordinate access.
  • tempSuffix <string> Optional temp filename suffix used during atomic-style writes. Default: "tmp"
  • durable <boolean> If true, use directory fsync and flush writes for stronger durability. Default: false
  • errorHandler <typeof console.error> Optional error handler used for internal async stream errors. Default: console.error

Use a Client instance to read, write, list, and delete files with hierarchical locking.

public client.durable

• <boolean>

Whether durability mode is enabled for the client.

public client.collect(path, options)

• path <string> An absolute path to a directory.
• options <{ encoding: "buffer"; withFileTypes: true; recursive?: boolean }> Optional. Enables Dirent output with NonSharedBuffer names.

Returns: <Promise<Array<fs.Dirent<NonSharedBuffer>>>>

public client.collect(path, options?)

• path <string> An absolute path to a directory.
• options <{ encoding: Exclude<BufferEncoding, "buffer">; withFileTypes?: false; recursive?: boolean } | Exclude<BufferEncoding, "buffer"> | null> Optional.

Returns: <Promise<Array<string>>>

public client.collect(path, options)

• path <string> An absolute path to a directory.
• options <{ encoding: "buffer"; withFileTypes?: false; recursive?: boolean }> Optional.

Returns: <Promise<Array<NonSharedBuffer>>>

Lists the entries in a directory. All paths must be absolute.

public client.read(path, options)

• path <string> An absolute path to a file.
• options <{ encoding: BufferEncoding; flag?: fs.OpenMode } & Abortable | BufferEncoding> Read as text with the specified encoding.

Returns: <Promise<string>>

public client.read(path, options?)

• path <string> An absolute path to a file.
• options <{ encoding?: null; flag?: fs.OpenMode } & Abortable | null> Optional.

Returns: <Promise<NonSharedBuffer>>

Reads a file. All paths must be absolute.

public client.createReadStream(path, options?)

• path <string> An absolute path to a file.
• options <Object> Optional createReadStream options.
  • flags <string> File system flags. Default: "r"
  • encoding <string | null> Default: null
  • mode <integer> Default: 0o666
  • start <number> Start offset.
  • end <number> End offset (inclusive).
  • highWaterMark <number> Read buffer size.

Returns: <Promise<fs.ReadStream>>

Creates a read stream and holds a read lock for the stream lifetime. For the supported option list, see the Node.js fs.createReadStream documentation.

Notes:

• fd is not supported.
• autoClose must not be false.

public client.createWriteStream(path, options?)

• path <string> An absolute path to a file.
• options <Object> Optional createWriteStream options.
  • flags <string> File system flags. Default: "w"
  • encoding <string> Default: "utf8"
  • mode <integer> Default: 0o666
  • start <number> Start offset.
  • highWaterMark <number> Write buffer size.

Returns: <Promise<fs.WriteStream>>

Creates an atomic write stream (temp file + rename) and holds a write lock for the stream lifetime. For the supported option list, see the Node.js fs.createWriteStream documentation. When the Client is instantiated with durable: true, flush is forced to true regardless of the per‑call option.

Notes:

• The stream writes to a temp file in the target directory; after finish, the client attempts to rename it into place.
• The lock is held for the entire stream lifetime, so long-running writes will block conflicting operations.
• fd is not supported.
• autoClose must not be false.

public client.write(path, data, options?)

• path <string> An absolute path to a file.
• data <string | Buffer | TypedArray | DataView | Iterable | AsyncIterable | Stream> Data to write.
• options <Object | string> Optional writeFile options.
  • encoding <string | null> Default: "utf8"
  • mode <integer> Default: 0o666
  • flag <string> Default: "w"
  • flush <boolean> If true, flush data to disk after writing. Default: false
  • signal <AbortSignal> Abort an in‑progress write.

Returns: <Promise<void>>

Writes a file using a temp file + rename. In durable mode, writes are flushed and directories are fsync'd. For the full option list, see the Node.js fs.promises.writeFile documentation. When the Client is instantiated with durable: true, flush is forced to true regardless of the per‑call option.

public client.delete(path, options?)

• path <string> An absolute path to a file or directory.
• options <Object> Optional rm options.
  • recursive <boolean> Default: false
  • force <boolean> Default: false
  • maxRetries <number> Default: 0
  • retryDelay <number> Default: 100
  • signal <AbortSignal> Abort an in‑progress remove.

Returns: <Promise<void>>

Deletes a file or directory. In durable mode, the parent directory is fsync'd. For the full option list, see the Node.js fs.promises.rm documentation.

The LockManager class

new persistence.LockManager()

• options <LockManagerOptions> Optional options passed to the LockManager.
  • errorHandler <typeof console.error> Default: console.error.

Creates a hierarchical lock manager. The lock manager enforces per-operation locking for reads, writes, collects, and deletes.

public lockManager.acquire(path, type)

• path <string> An absolute path.
• type <"read" | "write" | "collect" | "delete"> The type of lock to acquire.

Returns: <Promise<number>>

Acquires a lock for a path and returns a lock id. Reads may overlap other reads; writes are exclusive across ancestors and descendants.

public lockManager.release(id)

• id <number> A lock id previously returned by acquire.

Returns: <void>

Releases a lock by id.

public lockManager.root

• <GraphNode>

The root node of the internal lock graph.

The GraphNode interface

persistence.GraphNode

• segment <string> The path segment for this node.
• parent <GraphNode | null> The parent node.
• children <Map<string, GraphNode>> Child nodes keyed by segment.
• writeTail <Promise<unknown> | null> Tail promise for write locks.
• readTail <Promise<unknown> | null> Tail promise for read locks.

The Artifacts interface

persistence.Artifacts

• locks <Array<Promise<unknown>>> Promises the lock acquisition must await.
• node <GraphNode> The graph node for the path.

Tests

How to run the test

Clone the repository.

git clone https://github.com/far-analytics/persistence

Change directory into the root of the repository.

cd persistence

Install dependencies.

npm install && npm update

Run the tests.

npm test

Support

For feature requests or issues, please open an issue or contact one of the authors.

• Adam Patterson