@zukhruf/fs

Helpers that write files, a lock on a file that the kernel frees when its holder dies, and a stamp that tells that a file changed. A reader never sees a part of a write. @zukhruf/mutex uses them for its lock files, @zukhruf/fencing for its fencing counters, and @zukhruf/election for the claim and the epoch of each election. @zukhruf/single-flight keeps no lock files of its own: its election claims a FileLock through @zukhruf/election.

The words in these documents have one meaning each. See the glossary in CONTEXT.md.

Replace a file in one step

A write that changes a file in place can stop halfway. Then a reader sees a part of the old content and a part of the new content. atomicWrite fills a draft beside the path, and then moves the draft over the path with one rename. A reader sees the old content or the new content.

import { atomicWrite } from '@zukhruf/fs';

await atomicWrite('/var/lib/app/queue', 'first\nsecond\n', {
  clock: globalThis,
});

Make the replacement durable

After a rename, the operating system can keep the new content in memory for some time. A power loss in that time can bring back the old content. durableWrite replaces the file in one step, like atomicWrite. It returns only after the new content and the rename are on the disk:

1. It writes the draft and syncs the draft to the disk.
2. It moves the draft over the path.
3. It syncs the directory, because the rename is a change of the directory.

import { durableWrite } from '@zukhruf/fs';

await durableWrite('/var/lib/app/counter', '42', { clock: globalThis });

Use durableWrite when a value must never go back after a crash, for example a counter that gives out fencing tokens. A counter that goes back gives the same token two times.

On Windows, a process cannot open a directory to sync it. Some file systems refuse a sync of a directory with EINVAL or ENOTSUP. In these cases, the rename is as durable as that system makes it. Each other error of the sync reaches the caller.

Write bytes, and set the mode of the file

atomicWrite and durableWrite take the same arguments:

atomicWrite(path: string, content: string | Uint8Array, options: { clock: Clock; mode?: number })
durableWrite(path: string, content: string | Uint8Array, options: { clock: Clock; mode?: number })

clock is the clock that the waits of the write run on. Give globalThis in production, and a fake clock in a test. A write waits only on Windows, between two tries of a rename that Windows refuses for now (see "Try again while Windows refuses a file for a moment"). Clock is the type of @zukhruf/async.

The content is a string or bytes. A Buffer is a Uint8Array, so a Buffer is bytes too. The write puts the bytes at the path with no change.

mode sets the POSIX file mode of the new file, for example 0o600 for a file that only its owner can read. The process umask applies to mode, as it does for writeFile. The write sets mode when it creates the draft, because the rename keeps the mode of the draft. Thus the file at the path never has a different mode, not even for a moment. When you do not give mode, the file gets the mode that writeFile gives a new file. A write never keeps the mode of the file that it replaces.

import { randomBytes } from 'node:crypto';

import { atomicWrite } from '@zukhruf/fs';

await atomicWrite('/var/lib/app/key', randomBytes(32), {
  clock: globalThis,
  mode: 0o600,
});

On Windows, a file has no POSIX mode. Node.js uses only the write bit of mode there, to make the file read-only.

Create a file only when it is absent

createExclusive creates the file when no file is at the path, and returns true. When a file is there, it changes nothing and returns false. It fills a draft first and then links the draft to the path. Thus the file is never empty: a reader of the new file sees all of its content.

import { createExclusive } from '@zukhruf/fs';

const created = await createExclusive('/var/lib/app/job.lock', 'pid 4242', {
  clock: globalThis,
});
if (!created) console.log('Another process holds the job.');

A failed write leaves no draft

When a write fails, it removes its draft. Thus failed writes do not fill the directory with drafts. The file at the path keeps its old content.

Keep room for the name of the draft

A draft has the name of the path plus draftSuffixLength characters. Most file systems limit a file name to 255 bytes. A caller that makes long file names, for example from user keys, keeps draftSuffixLength characters of room. Otherwise the name of the draft is too long, and the write fails with ENAMETOOLONG.

Name a file after any key

A caller can keep one file for each key, for example one lock file for each lock key. The caller must make a file name from the key. A key can hold /, be . or .., or be too long for a file name. safeFileName(key, longestSuffix) gives a name that is one path segment. The name plus longestSuffix more characters fits in 255 characters, the longest file name of most file systems.

import { join } from 'node:path';

import { createExclusive, draftSuffixLength, safeFileName } from '@zukhruf/fs';

const name = safeFileName('orders/2026', '.lock'.length + draftSuffixLength);
await createExclusive(join('/var/lib/app/locks', `${name}.lock`), 'pid 4242', {
  clock: globalThis,
});
// The file is orders%2F2026.lock

• A key that fits gets its URI encoding, with each . as %2E. Thus a name is never . or ... The name of job.lock is job%2Elock, so it is never the name of job plus the suffix .lock.
• The encoding also encodes each uppercase letter. The names of Sync, sync and SYNC are %53ync, sync and %53%59%4E%43. APFS and NTFS compare file names without case. On these file systems too, each key gets its own file, as on Linux.
• Windows refuses * in a file name, so the encoding encodes each * as %2A. Windows also gives some names to devices: con, prn, aux, nul, com0 to com9 and lpt0 to lpt9. Such a name stays a device name after a suffix, for example con.lock. When the encoding of a key is a device name, its first letter is encoded too. The name of con is %63on. Other names do not change: the name of console is console.
• A key that does not fit, or that is not well-formed Unicode, gets a hashed name: the first 32 characters of its encoding, %%, and the SHA-256 of its UTF-16 code units in hex. A hashed name has at most 98 characters. An encoding never holds %%, so a hashed name is never the name of a key that fits.

Try again while Windows refuses a file for a moment

Windows refuses a file for a moment while a delete of it is in progress, or while another program holds it open with no sharing, for example a virus scanner. The error codes are EPERM, EACCES, and EBUSY. A real permission error has the same codes. patiently runs an operation again while Windows refuses the file, for up to 1 second. After that, the error reaches the caller. On other systems, patiently runs the operation one time.

import { readFile } from 'node:fs/promises';

import { patiently } from '@zukhruf/fs';

const content = await patiently(
  () => readFile('/var/lib/app/job.lock', 'utf8'),
  { clock: globalThis },
);

Each wait between two tries runs on the clock that you give. The 1 second is a time on that clock too. A test gives a fake clock and moves the time, so that the test does not wait. See ADR 0005.

Each write of this package uses patiently for its rename or link, with the clock of the write.

Lock a file

A process can die while it holds a lock. When the lock is a file that is present, for example a file from createExclusive, the file stays after the process dies. Then the lock stays until somebody deletes the file. A FileLock is a lock that the kernel holds. When the process of the holder stops, however it stops, the kernel frees the lock at once.

Node.js has no flock. Thus a FileLock is an exclusive SQLite transaction on the file. SQLite is part of Node.js (node:sqlite), so this package has no dependencies.

import { FileLock } from '@zukhruf/fs';

using lock = FileLock.open('/var/lib/app/job.lock');
if (lock.tryLock()) {
  try {
    await runJob();
  } finally {
    lock.unlock();
  }
} else {
  console.log('Another process holds the job.');
}

• FileLock.open(path) opens the file. When no file is at the path, it creates an empty file. It takes no lock.
• tryLock() takes the lock and returns true. When another handle holds the lock, it returns false at once. It never waits: to wait, try again after a pause. Two handles in one process exclude each other too.
• When the file has other content, tryLock() throws. Only an empty file or a SQLite database can be a file lock.
• unlock() lets the lock go. The handle stays open, and it can lock again.
• A disposal closes the handle. A lock that the handle holds goes with it. A second disposal does nothing.
• A held lock stays held when nothing refers to its handle any more. Only unlock(), a disposal, or the end of the process lets it go.

FileLock.check(path) tells if a handle holds the lock now. It returns 'locked', 'unlocked', or 'missing' when no file is at the path. It reads the file one time and never creates it. For that read, it holds a shared lock for a moment, so a tryLock() at that moment can return false. The answer can be old when you use it. Other faults, for example a directory at the path, throw.

Obey these rules:

• Open the file only through FileLock. On Unix, the lock is a POSIX record lock of the process. When the process closes any other descriptor of the file, the kernel drops the lock and gives no error.
• Do not delete the file while processes use it. A new file at the path has no lock, so a second process can lock it while the first one holds the old file.
• Use assertLocalDirectory first. Other machines do not see the lock reliably on a network file system.

The journal of the transaction is in memory, so a holder that dies leaves no <path>-journal file. Other programs can hold an exclusive SQLite transaction on the same file in the default journal mode. Such a transaction and a FileLock exclude each other. A journal file that such a program leaves is deleted by the next tryLock().

Refuse a network directory

A lock that a file or SQLite holds works between the processes of one machine. On a network file system, for example NFS or SMB, other machines do not see that lock reliably. assertLocalDirectory(directory) rejects with NetworkDirectoryError when the directory is on a network file system. A directory that does not exist yet gets the result of its nearest parent that exists.

import { NetworkDirectoryError, assertLocalDirectory } from '@zukhruf/fs';

try {
  await assertLocalDirectory('/mnt/shared/locks');
} catch (error) {
  if (error instanceof NetworkDirectoryError) console.log(error.fileSystem); // 'NFS'
  throw error;
}

• On Linux, the check reads the file system type with statfs. FUSE counts as local.
• On Windows, a UNC path, for example \\server\share, is a network directory.
• On macOS, each directory counts as local, because macOS gives no stable file system type.
• When statfs fails, the check does not reject. The directory is used as if the check did not exist.

@zukhruf/mutex and @zukhruf/single-flight give this same NetworkDirectoryError class. A catch with instanceof works for each of them.

Tell that a file changed

A program that watches a file reads it again after each write. A read of the whole file on each check costs too much, and a file-change event can come late or not at all. Git has the same problem for each file in a working tree. Git compares the stat data of the file with the stat data that it kept: when they are equal, the file did not change. A FileStamp is that stat data.

import { FileStamp } from '@zukhruf/fs';

const before = await FileStamp.read('/var/lib/app/store.json');
// ... later
const now = await FileStamp.read('/var/lib/app/store.json');
if (!now.equals(before)) console.log('The store changed. Read it again.');

• A file stamp holds the device, the inode, the size, the time of the last write and the time of the last change of the file. Each time is to the nanosecond.
• A write changes the stamp. This is also true when the writer gives back the old write time, as rsync -t and tar do, because the kernel sets the time of the change. The writer cannot set that time.
• A new file at the path changes the stamp, also with the same size and write time. A program that replaces its file with a rename makes a new file.
• A missing file has a stamp of its own. Two missing stamps of one path are equal. When the file comes back, the stamp changes.
• A path through a file, for example file/store, is missing too (ENOTDIR).
• Each other error of stat reaches the caller, for example EACCES for a folder that the process may not search. A file that the process may not see is not a missing file.
• modifiedAt is the time of the last write. It is undefined for a missing file.
• FileStamp.read(path) reads the stamp. FileStamp.readSync(path) reads it and blocks.
• toString() gives the stamp as text, so that you can keep it.

FileStamps holds the stamps of several files that change together, for example a database and its journal, or the files of one store. A write of any file changes the whole. A missing file is a part of it.

import { FileStamps } from '@zukhruf/fs';

const stamps = await FileStamps.read([
  '/var/lib/app/History',
  '/var/lib/app/History-journal',
]);
console.log(stamps.modifiedAt); // the newest write of the files that are present

modifiedAt of FileStamps is the newest write of the files that are present. It is undefined when no file is present.

No package on npm does this job. ADR 0004 tells why each candidate failed.

Check the error code

isErrno(error, code) tells if error is a Node.js system error with that error code.

import { isErrno } from '@zukhruf/fs';

try {
  await readFile(path, 'utf8');
} catch (error) {
  if (!isErrno(error, 'ENOENT')) throw error;
}