@kilnjs/storage

One storage contract for KilnJS, a registry that resolves a driver by name, and the two drivers that need no package of their own.

| | | | --- | --- | | local | files on the disk the server runs on, under one root directory | | memory | nowhere, and what a test run gets whatever it asks for |

Anything that talks to a service is a package of its own, because a provider is a plugin: @kilnjs/storage-s3, and whatever else you install.

Install

yarn add @kilnjs/storage
yarn add @kilnjs/storage-s3     # only if you use it

// app.config.ts
import { storage } from "@kilnjs/storage";
import { s3 } from "@kilnjs/storage-s3";

export default {
  modules: [/* … */ storage({ drivers: [s3()] })],
} satisfies AppConfig;

import { Storage } from "@kilnjs/storage";

await Storage.put(body, "avatars/1.png");

Two ways in

Storage is both the default driver itself and a map of the installed ones:

await Storage.put(body, "avatars/1.png");          // whichever is default
await Storage.exists("avatars/1.png");

await Storage.local.put(body, "avatars/1.png");    // this driver, by name
await Storage.s3.testConnection();                 // including its own methods

Nothing has to be awaited to reach a driver, only to use one. Building a provider needs the credentials, and those come from app.config.ts, which is read asynchronously - so Storage.local is a handle that forwards each method to the provider, and the wait happens inside the call, where there is a promise already.

The handle is built when the driver registers, from the methods on its class, so a provider's own methods are reached the same way as the contract's. Two consequences worth knowing:

• Methods only. A public field or getter on a provider is not on the handle, because reading one would mean having built the provider already. A helper a provider wants off the handle should be #private, not private, which is a word the compiler keeps to itself.
• Storage.local instanceof LocalStorage is false - it is a handle, not the provider. storageDriver("local") resolves to the provider itself.

When a path comes back off a row, the driver it was stored with comes with it, and storageDriver(name) is also the door for a name that is only a string:

const file = await database.attachment.findById(id);
const driver = await storageDriver(file.driver);

const body = await driver.buffer(file.path);

Configuration

Four fields in app.config.ts, and two of them are all most servers set:

import type { StorageConfig } from "@kilnjs/storage";

export default {
  storage: {
    driver: "local",

    credentials: {
      local: { root: "storage" },
      s3: { bucket: "uploads", region: "eu-central-1" },
    },
  } satisfies StorageConfig,
} satisfies AppConfig;

That section is this package's, not core's. Core declares no settings for anything it is given; a module merges its own section into ModuleConfig, so storage exists in app.config.ts exactly as long as this package is installed - which is what naming StorageConfig above brings in.

Storage is configured here rather than by a modules entry because nothing of it happens at boot: there is no module to run, only drivers to resolve when something stores a file.

driver is what the bare actions use. credentials is what each driver is configured with, by the name it answers to - and anything left out falls back to that provider's own environment variables, field by field, so secrets can stay in the environment while the rest is declared here.

| | | | --- | --- | | driver | what the bare actions use | | credentials | what each driver is configured with | | substitute | whether a test run reaches the in-memory driver whatever was named. True in a test run unless this says otherwise | | keepExtension | whether a generated upload key ends in the extension the file arrived with |

substitute: false is for an integration suite that means to exercise a real driver: every other suite gets the in-memory one, which is what keeps a unit test from writing to a bucket.

Every part of it is typed:

error TS2561: 'rooot' does not exist in type 'LocalCredentials'. Did you mean 'root'?
error TS2353: ... and 's3' does not exist in type 'Partial<StorageCredentials>'
error TS2322: Type '"s3"' is not assignable to type '"local" | "memory"'

The second is the one worth noticing: configuring a driver you have not installed fails to compile, because the field only exists once its package does.

Providers behave like plugins

A provider package registers its class and merges itself into two interfaces, both of them this package's - core is not involved:

// @kilnjs/storage-s3
declare module "@kilnjs/storage" {
  interface StorageDrivers {
    s3: S3Storage;        // what it is
  }

  interface StorageCredentials {
    s3: S3Credentials;    // what it takes
  }
}

export function s3(): StorageDriver<"s3", S3Storage> {
  return {
    provider: S3Storage,
    build: (credentials) => new S3Storage(credentials),
  };
}

A driver is handed over, not installed. It used to register itself when its package was imported, so app.config.ts carried a bare import "@kilnjs/storage-s3" with a comment explaining that the import was the installation. That is a side effect of loading a file, which means the drivers a server has depend on what happened to get imported and in what order, and nothing declares it. Now importing a package is inert and storage({ drivers: [s3()] }) says in one line what the comment used to say. storage() registers them in setup, so every driver is a decision with a name in the configuration.

The class is passed rather than its name: the registry reads the name off SIGNATURE and the driver's methods off the prototype, which is what the handle is made of. The builder is handed whatever storage.credentials.s3 held - typed, because registerStorage reads that argument off the same interface the provider just merged into, keyed by the name on the class.

Storing an upload

const stored = await putUpload(upload, { prefix: `posts/${post.id}` });

await database.attachment.attach({
  storageKey: stored.key,
  filename: stored.filename,
  contentType: stored.contentType,
  size: stored.size,
});

| | | | --- | --- | | prefix | where under the root it lands. Required | | driver | a named driver, where it is not the default one | | keepExtension | end the key in the extension the file arrived with, over whatever app.config.ts declared | | key | build the whole key, where a generated one will not do |

keepExtension is for a CDN, which reads the key and nothing else: the extension is taken from the filename, lowercased, and kept only where it looks like one, so ../../escape.png still produces a generated key that happens to end in .png.

key is the escape hatch, and it is the one option that hands the naming back: whatever it answers is the key, so it is the caller's job to make sure a path somebody typed cannot reach it.

The key is generated and the filename is kept as data, which is the whole point of the function existing. Three applications wrote these four lines by hand, correctly, because whoever wrote them knew that a filename is a string somebody typed: ../../etc/passwd is a perfectly ordinary one. assertUsablePath already refuses a key that climbs out, so the remaining risks were quieter: a caller choosing where their file lands, two callers colliding on one name, and one overwriting the other. None of that is left to remember now.

The contract

abstract class StorageProvider {
  static readonly SIGNATURE: string;

  abstract put(body: Buffer, path: string): Promise<StoredObject>;
  abstract stream(path: string): Promise<Readable>;
  abstract exists(path: string): Promise<boolean>;
  abstract remove(path: string): Promise<boolean>;
  abstract downloadUrl(path: string, expiresIn?: number): Promise<string | null>;

  buffer(path: string): Promise<Buffer>;   // from stream, for free
}

put answers with a StoredObject - the driver, the path and the size - which is what an application stores on whatever row owns the file, so a later read knows who to ask and for what.

The path it answers with is the path that was given. Where that lands is the driver's business: the local driver roots it under a directory, the S3 one puts it under a key prefix, and both take the same string back for a read, a check or a delete. So a stored path means the same thing whichever driver wrote it, and an application can be moved from one to the other without rewriting a column.

Two guarantees

A test run never reaches a real store. Every driver, named or default, answers with the in-memory one, so no suite can touch a bucket or a disk. Anything a suite relies on should therefore be on the contract above rather than on one provider.

That is also why the shape of a usable path is on the contract: relative, no .. segment, no nul byte, refused by every driver including the one with nowhere to escape to. A rule that lived only in the local driver would be one no suite applies, so an application's own test for a traversal attempt would pass while the attempt "succeeded" in memory. The local driver still resolves and re-checks against its root afterwards, because a lexical rule cannot see a symbolic link.

A driver nothing registered fails everywhere. The registration is checked before that substitution, so a name that resolves nowhere cannot pass in a test and throw in production. The message names what is installed.

The local driver

Suited to development, to a single-host deployment and to a mounted volume. Not suited to more than one host: another one cannot read what this one wrote.

| | | | --- | --- | | root / LOCAL_STORAGE_ROOT | storage - the directory everything is kept under. A relative path resolves against the working directory | | url / LOCAL_STORAGE_URL | where the root is served from, if anything serves it. Unset means downloadUrl answers null |

Keep the root out of version control, and off the path your server publishes unless that is what url says it is.

A path cannot climb out of the root. A stored path is data - it came from an upload, a column or a URL - so ../, and an absolute path, resolve somewhere real on a disk. Every operation routes through one check, and a path that lands outside raises rather than reading back as "nothing is there":

GenericError: That path lands outside the storage root

A file is whole or absent. The bytes are written beside the target and renamed over it, and a rename within a filesystem is atomic, so a crash or a concurrent write leaves no half-file for a reader to find.

A missing file raises where the caller is. stream() checks first: a read stream on nothing emits its error later, to whoever happens to be listening, which is rarely the code that asked.

put identifies the object by the sha256 of its contents, which is what an ETag is for the stores that have one - equal ids mean equal bytes. downloadUrl ignores expiresIn, because the file is served by something else: for how long and to whom is that server's business, and signing would be a promise this driver cannot keep.

Development

yarn verify   # ts:check, lint, build, exports:check, test

The local driver's tests write to a real temporary directory, because what it has to get right is where a path lands on a disk. Nothing else touches one.