express-asset-file-cache-middleware

A modest express.js middleware to locally cache assets (images, videos, audio, etc.) for faster access and proxying, for use in e.g. Electron apps.

TL;DR

For offline use of dynamic assets, e.g. in your Electron app or local express server.

Usage

const express = require("express");
const fileCacheMiddleware = require("express-asset-file-cache-middleware");

const app = express();

app.get(
  "/assets/:asset_id",
  async (req, res, next) => {    
    res.locals.fetchUrl = `https://cdn.example.org/path/to/actual/asset/${req.params.asset_id}`;

    res.locals.cacheKey = `${someExpirableUniqueKey}`;
    next();
  },
  fileCacheMiddleware({ cacheDir: "/tmp", maxSize: 10 * 1024 * 1024 * 1024 }),
  (req, res) => {
    res.set({
      "Content-Type": res.locals.contentType,
      "Content-Length": res.locals.contentLength
    });
    res.end(res.locals.buffer, "binary");
  }
);

app.listen(3000);

It works by fetching your asset in between two callbacks on e.g. a route, by attaching a fetchUrl onto res.locals. When the asset isn't cached on disk already, it will write it into a directory specified by the option cacheDir. If it finds a file that's alread there, it will use that.

On a cache miss the response body is streamed straight to disk (instead of being buffered entirely in memory), written to a temporary file and then atomically renamed into place, so an interrupted or partial download can never leave a corrupt cache entry. Non-2xx upstream responses (401/403/404/429/5xx, …) are never cached - the upstream status is proxied back to the client so the entry self-heals on the next request.

HTTP Range requests (video seeking, partial downloads)

Since 1.4.0 the middleware supports HTTP Range requests, which is what browsers use for <video>/<audio> timeline seeking. Use the bundled sendBuffer helper as your final handler to get Range-aware responses for free:

app.get(
  "/assets/:asset_id",
  async (req, res, next) => {
    res.locals.fetchUrl = `https://cdn.example.org/path/to/actual/asset/${req.params.asset_id}`;
    next();
  },
  fileCacheMiddleware({ cacheDir: "/tmp" }),
  fileCacheMiddleware.sendBuffer
);

sendBuffer emits Accept-Ranges: bytes, returns 206 Partial Content with a Content-Range header when the client sends a valid Range header, and falls back to 200 OK with the full body otherwise. Unsatisfiable ranges return 416 Range Not Satisfiable.

The middleware also now sets Accept-Ranges: bytes before calling next(), so even consumers that keep their own res.end(res.locals.buffer) response handler will get browser seek UI enabled (they still won't serve 206s, though — you need sendBuffer for that).

The asset's contentType and contentLength are stored base64 encoded in the filename, thus no offline database is necessary

Note that setting cacheKey and cacheDir isn't strictly necessary, it will fall back to res.local.fetchUrl and path.join(process.cwd(), "/tmp"), respectively.

Using it without express: cacheAsset

Since 1.6.0 the fetch+cache core is exported on its own, so callers that aren't serving an HTTP request (an Electron main process, a CLI precache, a queue worker) get the same streaming download, atomic rename, LRU accounting and progress reporting without standing up a local server:

const { cacheAsset } = require("express-asset-file-cache-middleware");

const result = await cacheAsset("https://cdn.example.org/big-video.mp4", {
  cacheDir: "/tmp",
  cacheKey: someExpirableUniqueKey, // optional, falls back to the URL
  onProgress: ({ received, total }) => {
    mainWindow.webContents.send("download:progress", { received, total });
  }
});

if (result.status === "cached") {
  console.log(result.path, result.contentType, result.contentLength);
}

It resolves to one of three shapes:

| status | Meaning | Fields | | ---------- | ------- | ------ | | "cached" | The asset is on disk and ready to read. | fromCache (false on a fresh download, true on a cache hit), path, contentType, contentLength | | "error" | Non-2xx upstream. Nothing was cached; the entry self-heals on the next call. | httpStatus, statusText, contentType, body | | "empty" | 2xx carrying no body (204/205). Nothing was cached. | httpStatus |

Filesystem and stream errors reject instead. cacheAsset unlinks its own temp file first, so a failed download never leaves a partial entry behind.

The options are the middleware's (cacheDir, maxSize, logger, onProgress) plus cacheKey, which the middleware takes from res.locals.cacheKey. The middleware is a thin adapter over this function, so both paths share one implementation and behave identically.

LRU Eviction

To avoid cluttering your device, an LRU (least recently used) cache eviction strategy is in place. Per default, when your cache dir grows over 1 GB of size, the least recently used (accessed) files will be evicted (deleted), until enough disk space is available again. You can change the cache dir size by specifying options.maxSize (in bytes) when creating the middleware.

Install

$ npm install express-asset-file-cache-middleware

or

$ yarn add express-asset-file-cache-middleware

API

Input

res.locals.fetchUrl (required)

The URL of the asset to cache.

res.locals.cacheKey (optional)

A unique, expireable cache key. If your asset contains a checksum/digest, you're already done, because it falls back to res.locals.fetchUrl.

Output

To further process the response, the following entries of res.locals are set:

res.locals.buffer

The cached asset as a binary buffer. Most likely, you will end the request chain with

res.end(res.locals.buffer, "binary");

res.locals.contentType and res.locals.contentLength

If you're serving your assets in the response, you'll need to set

res.set({
  "Content-Type": res.locals.contentType,
  "Content-Length": res.locals.contentLength
});

Options

You can pass the following options to the middleware:

cacheDir (optional)

The root directory where the file cache will be located. Falls back to path.join(process.cwd(), "/tmp").

logger (optional)

A logger to use for debugging, e.g. Winston, console, etc.

maxSize (optional)

The maximum size of the cache directory, from which LRU eviction is applied. Defaults to 1 GB (1024 * 1024 * 1024).

onProgress (optional)

A callback invoked repeatedly while an asset is being streamed to disk on a cache miss, useful for driving download-progress UIs. It receives an object { received, total }:

• received - the number of bytes written so far (monotonically increasing, ending at the full size).
• total - the expected size from the upstream Content-Length header, or null when the server doesn't send one (e.g. chunked responses).

fileCacheMiddleware({
  cacheDir: "/tmp",
  onProgress: ({ received, total }) => {
    const pct = total ? Math.round((received / total) * 100) : null;
    console.log(`downloaded ${received} bytes${pct !== null ? ` (${pct}%)` : ""}`);
  }
});

It is only called on a cache miss (an actual download); cache hits are served from disk and emit no progress events.

Tests

Run the test suite:

# install dependencies
$ npm install

# unit tests
$ npm test

License

The MIT License (MIT)

Copyright (c) 2019 Julian Rubisch

Contributors ✨

Thanks goes to these wonderful people (emoji key):

This project follows the all-contributors specification. Contributions of any kind welcome!