fcrfs

A read-only remote file system with caching. A server serves an fsapi file system, and a client offers it as a read-only fsapi file system of its own. The two exchange JSON-RPC 2.0 messages and leave the transport to you. They are meant to run inside the inner JSON-RPC of a ksock connection, which carries them as BSON.

npm install @k13engineering/fcrfs @k13engineering/fsapi @k13engineering/hash-cache

fsapi is a peer dependency: fcrfs uses the fsapi of your project, so that its file systems and types are the ones your code uses.

hash-cache is the reference for the block cache of a client: its caches in memory, in a folder, and in tiers of both fit blockCache as they are.

Example

import nodeCrypto from "node:crypto";
import { createConvenienceApi, createRamFileSystem } from "@k13engineering/fsapi";
import { createRamHashCache } from "@k13engineering/hash-cache";
import { createClient, createServer, createSystemTimers } from "@k13engineering/fcrfs";

const served = createRamFileSystem();
await createConvenienceApi({ fileSystem: served })
  .writeFileOrThrow({ filepath: "/hello.txt", data: new TextEncoder().encode("hello") });

// on the server side
const server = createServer({
  fileSystem: served,
  timers: createSystemTimers(),
  sendMessage: ({ message }) => {
    // hand the message to the connection, which delivers it to client.receivedMessage
  }
});

// on the client side
// blocks are named by the MD5 of their data, in lowercase hex
const md5 = { name: "md5", digest: async ({ data }) => nodeCrypto.createHash("md5").update(data).digest("hex") };
const client = createClient({
  timers: createSystemTimers(),
  // keeps the data of blocks, here up to 256 MiB in memory; createFileSystemHashCache keeps them across restarts
  blockCache: createRamHashCache({ hash: md5, maxSize: 256 * 1024 * 1024, maxCount: 100_000, maxPendingSize: 16 * 1024 * 1024 }),
  sendMessage: ({ message }) => {
    // hand the message to the connection, which delivers it to server.receivedMessage
  }
});

const remote = createConvenienceApi({ fileSystem: client.fileSystem });
const data = await remote.readFileOrThrow({ filepath: "/hello.txt" });

examples/read-remote-files.ts runs a server and a client in one process. Their messages are encoded and decoded as BSON on the way. Run it with npm run example.

Connecting client and server

A server and a client are the two ends of one connection. Each one hands its messages to sendMessage and takes the messages of the other end through receivedMessage. A server serves exactly one client, so create one server for each connection.

The messages are plain JSON-RPC 2.0 objects. Binary data in them, the data of blocks, is a Uint8Array. A transport has to deliver them in order and keep binary data binary:

• ksock: the inner JSON-RPC of ksock encodes the messages as BSON. Decode them with binary data as bytes, e.g. with deserialize(bytes, { promoteBuffers: true }) of the bson package. Without that option, bson returns its own Binary objects, which the client does not accept as data.
• Other transports: JSON cannot carry a Uint8Array. Use a transport that can, e.g. BSON or the structured clone of postMessage.

receivedMessage returns { error } when a message is not a valid JSON-RPC message. It throws when a valid message breaks the protocol of fcrfs, e.g. an unknown notification or a patch that does not apply. Requests that break it are answered with a JSON-RPC error.

How it works

1. The client requests paths. When something opens a path on the client, the client asks the server for that path, its metadata, the first openReadaheadBytes of it, and the byte ranges being read, with a window ahead of sequential reads. It sends this as a JSON Patch of its requests (requestPatch). Requests that are opened and closed within the same task never reach the server.
2. The server sends the tree. The server reads what is requested and sends the client its view of the tree as JSON Patches (treePatch): a node for each path, and an inode with metadata, directory entries, the symlink target, or the blocks of a regular file. Blocks are identified by the MD5 hash of their data. Along with a path, the server sends the listing of its directory.
3. The server pushes the data that was asked for. The client asks for each range whether the server should push its data too (pushRegions, besides the fileRegions it reads). Before the patch that names the blocks of such a range, the server sends their data (hashNotify), each block once per connection, and only when it reads them for the request: a range whose blocks the server already sent in the tree is not pushed later. The client keeps the data in the block cache it is given, by hash, so a read finds it there when the patch arrives.
4. The client fetches the rest by hash. For the data of a block it does not have, e.g. one its cache dropped, the client sends requestDataSliceByHash, and the server answers with the bytes. The blocks of one read are fetched at once.
5. The server keeps the tree up to date. The server watches the inode of every path it has read, and the directories along the path, with the watch of fsapi. A reported change makes it read the path again right away, and the changes reach the client as patches. A changed mtime invalidates the cached blocks of a file. When the last link of an inode is gone, the inode leaves the tree.
6. A slow re-read backs up the watches. Every path is also read again once its last read is refreshAfterMs ago, a minute by default. If such a read finds a change that no watch reported, the server calls onWarning.

Requests ahead of need, like readahead, are marked background. The server reads the requests needed now first, and at most 16 paths in the background at once, so these never take the places of reads that are needed now.

The server reads files in blocks of 256 KiB. Files up to 64 KiB are read whole on the first request.

| Message | Direction | Kind | Parameters | Result | | --- | --- | --- | --- | --- | | requestPatch | client → server | notification | { patch }, a JSON Patch of the requests | | | requestDataSliceByHash | client → server | request | { hash } | { data } with a Uint8Array, or { error: { code, message } } | | treePatch | server → client | notification | { patch }, a JSON Patch of the tree | | | hashNotify | server → client | notification | { hash, data }, the data of a block ahead of time | |

API

createServer({ fileSystem, sendMessage, timers, refreshAfterMs?, staticTrees?, onWarning? })

• fileSystem: the fsapi file system to serve.
• sendMessage({ message }): sends a message to the client.
• timers: the clock and timers to use: createSystemTimers() for those of the platform, or fake ones in tests.
• refreshAfterMs: how long after its last read a path is read again even if its watch reported no change. Defaults to 60000.
• staticTrees: the static subtrees of fileSystem, made with createStaticTrees. Once they are hashed, the server sends their inodes with tree hashes, and the inodes of their roots without being asked. Several servers of the same file system can share them.
• onWarning({ message }): called when something goes wrong without failing, e.g. "the watch of /path missed a change". Defaults to dropping the warnings.

Returns:

• receivedMessage({ message }): takes a message of the client.
• close(): stops serving and closes fileSystem.

createClient({ sendMessage, blockCache, timers, trustDirectoryListings?, openReadaheadBytes?, readaheadInitialBytes?, readaheadMaxBytes?, pushData? })

• sendMessage({ message }): sends a message to the server.
• blockCache: where the client keeps the data of the blocks it got, by the MD5 of their data in lowercase hex. Each cache of hash-cache fits, with MD5 as its hash, e.g. one in tiers of memory and a folder, which keeps the blocks across restarts. Any { lookup({ hash }), put({ hash, data }) } will do:
  • lookup resolves to { error: undefined, hit: { data } } for a block the cache has, { error: undefined, hit: undefined } for one it does not have (anymore), and { error, hit: undefined } when it fails, e.g. with an I/O error. The client fetches what the cache does not give it.
  • put is a hint to keep a block; it returns right away, and the cache may drop it, or keep it only a while later. The client keeps the blocks that arrived in a session in memory as well, up to 64 MiB, so a cache that drops them or is slow to keep them costs no fetches.
• timers: like for the server.
• trustDirectoryListings: when true, opening a path whose name is missing from the known listing of its directory fails with ENOENT right away, without asking the server. The server sends the listing of the directory of every path the client asks for, so after one path of a directory, the others that do not exist cost no round trip. Only for served file systems whose listings name everything that can be opened. Defaults to false.
• openReadaheadBytes: how many bytes from the start of a file the client asks for when opening it, so that the server pushes them along with the file. The request covers directories and symlinks as well, which ignore it. 0 asks for none. Defaults to 64 KiB, which covers the files the server reads whole.
• readaheadInitialBytes and readaheadMaxBytes: the window of bytes the client asks for ahead of sequential reads of a file, so that the server pushes them before they are read. Reads that continue where the reads of a file ended are sequential, also when they arrive slightly out of order, and also when each comes with an open of its own, as from an NFS server. Once less than half of the window is left ahead, the client asks for more, and doubles the window if the reads used what it asked for ahead, up to readaheadMaxBytes. Any other read is random: it gets nothing ahead, and the window starts over. The client keeps this state for the 256 files read last. Default to 256 KiB and 4 MiB; a readaheadInitialBytes of 0 turns readahead off.
• pushData: whether the client asks the server to push the data of what it requests, for opens, reads and readahead alike. "adaptive", the default, decides it from how often the block cache already has the blocks that reads need: each block counts once, as a hit if the cache had it, and as a miss if it had to be fetched or only a push brought it. Data is pushed while the moving average of the hits is below one half, and it starts at 0. So a client that starts with an empty cache gets everything pushed, and one that restarts with the cache of an earlier run stops pushing after a few blocks and reads from its cache, fetching only what changed since. If many blocks are missing, pushing starts again. "always" and "never" decide it once. The client never tells the server which blocks it has.

Returns:

• receivedMessage({ message }): takes a message of the server.
• fileSystem: the read-only fsapi file system of the served files.
• recordRoute(): starts recording the route of the file system: the paths it is asked for, whether their metadata was needed, and the ranges read of them, in the order they were first used, including paths that turned out missing. stop() of what it returns ends the recording and returns the route, { version: 1, entries: [{ path, meta, ranges }] }, plain data to keep wherever the caller likes. Several recordings may run at once.
• prime({ route }): a hint, at any time and as often as wanted: asks the server for everything of a route at once, in the background, so that the paths are known before they are opened, e.g. when an application starts again the way it did before. Each request ends once the tree has what it asks for. Priming asks for no pushes, as nothing tells yet whether the cache has the data: once the tree has an entry, the client fetches the blocks of its ranges that the block cache lacks, so a filled cache costs no data, and an empty one one more round trip. A route that is out of date costs only reads of what is not needed; what to keep of routes, e.g. those of the last few starts, is up to the caller.

createStaticTrees({ fileSystem, roots, maxParallelReads? })

Hashes the subtrees below roots that the caller knows do not change, e.g. a read-only root file system. Only the one who serves a file system can know that, so it is up to the server; a client handles tree hashes whenever they come.

Each node below a root gets a tree hash: the MD5 of its tree node, an encoding of everything the inode holds, and for a directory the names, types, inode numbers and tree hashes of its entries. So a tree hash covers everything below its node. The encoding, fcrfs-tree-v1, is deterministic, and described in lib/tree-node.ts:

• It covers the inode number, nlink, mode, uid, gid, atime, mtime, ctime and birthtime; only the generation is left out.
• A file is described by its size and the hashes of its blocks of 256 KiB from offset 0.
• So the same files have the same tree hashes, as long as nothing of them changes, including their atime: serve only file systems whose atime does not change, e.g. mounted read-only or with noatime. A copy of the files, with other inode numbers and ctimes, has other tree hashes.

Hashing reads every file below the roots once, maxParallelReads nodes at once (8 by default); ready settles when it is done. The tree nodes are kept, and where each block is, so blocks can be read again by hash; the data is not kept. A change below a root is not noticed: a block whose file changed is not served anymore.

createSystemTimers()

The clock and timers of the platform, as createServer and createClient take them.

The types TBlockCache, TClient, TServer, TMessage, TTimers and TTimer are exported as well.

The client's file system

It behaves like a read-only local file system:

• Writes of any kind fail with EROFS.
• Reading a directory fails with EISDIR, and listing a file with ENOTDIR.
• Opening the content of a symlink fails with ELOOP, of a device or socket with ENXIO, and of a fifo with EOPNOTSUPP.
• Symlinks are not followed.
• statfs fails with ENOSYS.
• watch reports the changes of an inode as the server sends them: a new size, mtime or symlink target as content, other metadata as attributes, new directory entries as entries, and a new atime as access. Content ranges are not known. When the server drops the inode, the watch is removed.
• stat reports the generation of the served inode.
• A handle whose inode the server no longer has, or has with another type, is stale: its operations fail with EIO. So do reads whose data the server cannot provide.
• Operations wait for the server. While nothing arrives, the client checks again every 2 seconds.
• close waits until the blocks pushed so far went to the block cache, so the block cache can be closed right after, e.g. one in a folder.
• Static trees: a node the server sends with a tree hash is known in full to a client that has its tree node in the block cache, together with all tree nodes below it: the client opens paths below it, lists directories and reads symlinks without asking the server, and fetches the blocks its cache lacks by hash. A missing name is missing, without asking. Where a tree node on the way is not in the cache, e.g. after a change, the client asks the server for the path as for any other, and fetches the tree node by hash for later opens. The server pushes the tree node of a static path along with it while the client asks for pushes, so a cold start fills the cache.

Limitations

• Read-only.
• Freshness: changes reach the client as soon as the watch of the served file system reports them. File systems that cannot watch emulate it by polling, so changes then arrive with the next poll. A change of a file's data that keeps its mtime is read again, but blocks the server already cached for that mtime are reused.
• Memory: the server keeps every path it has read, a watch of each and of the directories along it, the data of every block, and the hashes of the blocks it pushed, for as long as it runs. Each path is also read again every refreshAfterMs. The client keeps up to 64 MiB of the blocks that arrived in memory, besides the block cache it is given; the tree it keeps is not limited.
• Flow control: neither end slows down when the other one cannot keep up. On ksock, its grants pace the messages on the connection.
• Symlinks have a size of 0, as the server does not send their size.
• Node.js: the tree store uses node:util, and the server uses node:crypto for hashing.

Development

npm ci
npm run build
npm run type-check
npm run lint
npm test

The tests are deterministic. They run on fake timers and in-memory file systems, and use no real timers, sockets or disk. npm test reports the coverage, which is 100%.

License

LGPL-2.1-only