@vmlts/fs-node

Upload a file from Node or Bun, get back a URL on your own domain.

Day 0

Three things come from whoever runs filestore for you. They are made already — you do not create them, and you cannot make them from here.

| You hold | Looks like | What it is | |---|---|---| | A service URL | https://cdn.you.com | where you write to, and where the files are read from | | An API key | fs_a3f9... | it names your app, and it is the only way in | | A bucket name | photos | the namespace your files land in |

1. Install.

bun add @vmlts/fs-node      # or: npm i @vmlts/fs-node

2. Put the three in the environment.

export FILESTORE_URL=https://cdn.you.com
export FILESTORE_TOKEN=fs_a3f9...
export FILESTORE_BUCKET=photos

3. Upload a file.

import { Filestore } from "@vmlts/fs-node";

const filestore = Filestore.fromEnv();
const { key, url } = await filestore.put("./holiday.mp4");

console.log(url);   // https://cdn.you.com/blog/photos/holiday.mp4

4. Open that URL. The file is there. put returns only once the object is readable, so there is nothing to poll and nothing to wait for.

That is the whole of day 0. Everything below is detail you can read when you need it.

Prefer settings in code? new Filestore({ baseUrl, token, bucket }) takes the same three, and fromEnv({ bucket: "logos" }) overrides one of them.

Everything the SDK does

| You want to | Call | Notes | |---|---|---| | Upload a file on disk | put("./cat.jpg") | the path is read as a stream | | Upload bytes you hold | put(bytes, { key: "q3.pdf" }) | a Uint8Array or a Buffer | | Upload a large file | put("./film.mov") | over 64 MiB it splits and parallelises itself | | Name the object | put(x, { key: "reports/q3.pdf" }) | leave it out and the SDK builds one | | Follow the progress | put(x, { onProgress }) | bytes as they leave | | Stop an upload | put(x, { signal }) | the half-uploaded parts are discarded | | List objects | list({ prefix: "reports/" }) | an async iterator; it follows every page | | List into an array | listAll({ prefix }) | for a bucket that fits in memory | | Delete an object | delete("cat.jpg") | succeeds whether or not the key was there | | Build a read URL | url("cat.jpg") | no request; needs slug | | See your buckets | listBuckets() | every bucket of your app | | Make a bucket | createBucket("logos") | 1 to 32 characters of a-z, 0-9, - | | Drop a bucket | deleteBucket("logos") | refused while it still holds files | | Serve a file | GET the returned url | public, cached, no key needed |

const filestore = new Filestore({
  baseUrl: "https://cdn.you.com",
  token: "fs_...",
  bucket: "photos",      // every key below is relative to this
  slug: "blog",          // optional; only url() needs it
});

await filestore.put("./notes.md");
await filestore.put(bytes, { key: "reports/q3.pdf" });

for await (const object of filestore.list({ prefix: "reports/" })) {
  console.log(object.key, object.size, object.uploaded, object.url);
}

await filestore.delete("cat.jpg");
filestore.url("cat.jpg");

await filestore.listBuckets();
await filestore.createBucket("logos");
await filestore.deleteBucket("logos");

The key names an app, the bucket names a tenant

Your API key belongs to one app, and a bucket is a namespace inside that app. The app owns the first segment of every read URL and the bucket owns the second. You never spell the app — the service reads it off your key — but you do name the bucket, once, as a setting.

const { key, bucket, url } = await filestore.put("./cat.jpg", { key: "cat.jpg" });
key     // "cat.jpg"                                 bucket-relative, as you sent it
bucket  // "photos"                                  your setting, on the wire
url     // "https://cdn.you.com/blog/photos/cat.jpg" the app comes off your key

Give each tenant its own bucket and one client per tenant, and their keys cannot collide. Leave bucket out and your keys spell their own.

A bucket has to exist before a write names it. createBucket makes one, and so does the console at /admin. A write into a bucket your app does not have raises a 404 rather than making it, because a bucket carries settings of its own.

Turn on ensureBucket and the upload makes the bucket instead of raising, so a tenant who has never uploaded before needs no setup step of its own. It stays one request either way, and concurrent uploads into the same new bucket all land.

const filestore = Filestore.fromEnv({ bucket: `tenant-${id}`, ensureBucket: true });
await filestore.put("./cat.jpg");         // makes tenant-42 on the first upload

await filestore.put("./logo.png", { bucket: "logos", ensureBucket: true });  // or once

url() builds a read URL from a key you already hold, and it needs slug because the app slug is the first segment of every read URL. Every put and every list already carries the URL back, so most callers never need it.

Without a key of your own, the SDK builds one: twelve unguessable characters, then the filename, so the URL still says what it points at.

./holiday photos/sunset (1).jpg  ->  f7Kq2xBn9pWv/sunset--1-.jpg

Watching an upload

await filestore.put("./holiday.mp4", {
  onPlan: ({ size, parts, partSize }) => {
    console.error(`${size} bytes in ${parts} parts of ${partSize}`);
  },
  onProgress: ({ loaded, total, pool }) => {
    console.error(`${Math.round((loaded / total) * 100)}% at ${pool} parts at once`);
  },
});

onProgress fires as bytes leave, as often as the runtime sees them. A retry re-sends bytes already counted, so loaded can go backwards: that is the truth rather than a number that only ever rises.

Stopping

const stopping = new AbortController();
setTimeout(() => stopping.abort(), 5000);
await filestore.put("./huge.bin", { signal: stopping.signal });

A large file goes up in parts. Between the first request and the last, those parts sit in the store and nothing else knows about them. Any way out other than a successful finish discards them — a part that fails for good, a failed assemble, or a cancel.

A CLI wants that on SIGINT too, because the default handling ends the process where it stands and the parts outlive it. It is off by default, because taking the signal means taking the exit with it.

await filestore.put("./huge.bin", { abortOnSignals: true });

Settings

| Option | Environment | Default | |---|---|---| | baseUrl | FILESTORE_URL | required | | token | FILESTORE_TOKEN | required | | bucket | FILESTORE_BUCKET | none; your keys then spell their own bucket | | ensureBucket | FILESTORE_ENSURE_BUCKET | off; a write into a missing bucket raises | | slug | — | none; only url() needs it | | partSize | FILESTORE_PART_SIZE | 1/32 of the file, clamped to 5 MiB–100 MB | | concurrency | FILESTORE_CONCURRENCY | measured while uploading |

Leave the last two alone unless you have a reason. The part size aims for 32 parts, which keeps the pool full without splitting the file needlessly. The pool measures the link by doing it: nothing local can report bandwidth, so the upload times each generation of parts and doubles the pool while a wider pool still moves bytes faster.

Limits

| Limit | Value | |---|---| | One-request upload | up to 64 MiB; above that the SDK splits the file | | Part size | 1/32 of the file, held between 5 MiB and 100 MB | | Parts in flight | measured while uploading, up to 64 | | Largest object | 1000 GB. A larger file is refused before the first byte goes out | | Longest key | 1024 characters, counting the app and the bucket in front of it | | Delete visible everywhere | up to 1 hour, because reads are cached |

Failing

Everything raises FilestoreError.

try {
  await filestore.put("./cat.jpg");
} catch (err) {
  if (err instanceof FilestoreError) {
    err.status;    // 401, 403, 413... absent when no response came back
    err.detail;    // what the service said
    err.attempts;  // how many tries it took to give up
  }
}

A 4xx other than 408, 425 and 429 is not retried: the service read the request and refused it, so an identical second one earns an identical refusal.