@hartl-services/payload-s3-cache-storage

Payload CMS 3 plugin that writes uploads to S3 and serves reads through a local TTL cache.

This package intentionally targets one long-lived Node.js instance with a persistent writable volume. It does not support serverless or multi-instance deployments because locking is process-local.

Neue Paketversion auf npm veröffentlichen

Das Paket wird öffentlich über npmjs.com bereitgestellt. Nach dem einmaligen Bootstrap prüft GitLab CI bei jedem Lauf auf dem Default-Branch, ob die Version aus package.json bereits auf npm existiert. Eine neue Version wird gebaut und per npm Trusted Publishing mit kurzlebigen GitLab-OIDC-Tokens veröffentlicht; unveränderte Versionen werden erfolgreich übersprungen.

Erhöhe für ein Release die Version ohne lokalen Release-Tag und bringe die Änderung auf den Default-Branch:

pnpm version patch --no-git-tag-version
git add package.json pnpm-lock.yaml
git commit -m "release: publish next version"
git push

Für ein Minor- oder Major-Release verwende entsprechend minor oder major.

Einmaliger npm-Bootstrap

Trusted Publishing kann erst in den Einstellungen eines bereits existierenden npm-Pakets aktiviert werden. Veröffentliche daher die erste Version einmalig mit einem npm-Account, der Pakete im Scope @hartl-services anlegen darf:

pnpm install --frozen-lockfile
pnpm build
pnpm pack --out /tmp/payload-s3-cache-storage.tgz
npm login --registry=https://registry.npmjs.org/
npm publish /tmp/payload-s3-cache-storage.tgz --registry=https://registry.npmjs.org/

Konfiguriere danach unter den npm-Paketeinstellungen den Trusted Publisher mit diesen Werten:

| Einstellung | Wert | | ---------------------- | ---------------------------------------- | | Provider | GitLab CI/CD | | Namespace | hartl-services-gmbh/pakete/payload-cms | | Project name | payload-s3-cache-storage | | Top-level CI file path | .gitlab-ci.yml | | Allowed action | npm publish |

Die Pipeline benötigt dafür einen GitLab.com Shared Runner; selbst betriebene Runner werden von npm Trusted Publishing derzeit nicht unterstützt. Nach erfolgreicher Einrichtung sollten in npm unter Publishing access klassische Publish-Tokens deaktiviert werden.

Schnellstart: öffentliche Paketinstallation

Für die öffentliche npm Registry sind weder eine .npmrc noch ein Registry-Token erforderlich:

pnpm add @hartl-services/payload-s3-cache-storage

Entferne in bestehenden Projekten die frühere Zuordnung von @hartl-services zur GitLab Package Registry sowie zugehörige GITLAB_NPM_TOKEN-Konfigurationen, damit der Scope wieder aus npmjs.org aufgelöst wird.

Compatibility

| Requirement | Supported | | ----------- | ----------------------------------------------------------------------- | | Payload | >=3.84.1 <4.0.0 | | Node.js | >=20.9.0 | | pnpm | 9, 10, or 11 | | Deployment | One long-lived Node.js instance with a writable persistent cache volume |

Payload 4 and Payload prerelease builds are not supported by version 0.1.x of this plugin.

Payload konfigurieren

Das Zielprojekt benötigt eine Payload-Version >=3.84.1 <4.0.0, eine Upload-Collection wie media, einen S3-kompatiblen Bucket und ein beschreibbares Cache-Verzeichnis. Nutze lokal ./.s3-cache und in Produktion ein persistentes Volume wie /data/s3-cache.

S3- und Cache-Variablen anlegen

Append these values to the consuming project's ignored .env and token-free .env.example:

S3_ACCESS_KEY_ID=
S3_SECRET_ACCESS_KEY=
S3_BUCKET=
S3_REGION=eu-central-1
S3_ENDPOINT=
S3_FORCE_PATH_STYLE=false
S3_PREFIX=
S3_CACHE_DIR=./.s3-cache
S3_CACHE_STORAGE_ENABLED=false

Fill S3_BUCKET and S3_REGION. For MinIO or another compatible service, also set S3_ENDPOINT and usually S3_FORCE_PATH_STYLE=true. S3_PREFIX can separate environments, for example production/. Set S3_CACHE_STORAGE_ENABLED=true when you want to exercise the plugin locally; false leaves Payload's existing storage configuration unchanged.

Static access keys are optional. If both credential values stay empty, the plugin uses the standard AWS SDK credential chain, including environment variables such as AWS_ACCESS_KEY_ID, instance roles, and workload identities.

Plugin in payload.config.ts aktivieren

The following example enables the plugin for the media upload collection. Add more collection slugs to collections when required.

import { s3CacheStorage } from '@hartl-services/payload-s3-cache-storage'
import { buildConfig } from 'payload'

const credentials =
  process.env.S3_ACCESS_KEY_ID && process.env.S3_SECRET_ACCESS_KEY
    ? {
        accessKeyId: process.env.S3_ACCESS_KEY_ID,
        secretAccessKey: process.env.S3_SECRET_ACCESS_KEY,
      }
    : undefined

export default buildConfig({
  collections: [Media],
  plugins: [
    s3CacheStorage({
      collections: ['media'],
      enabled: process.env.S3_CACHE_STORAGE_ENABLED === 'true',
      s3: {
        bucket: process.env.S3_BUCKET!,
        region: process.env.S3_REGION!,
        endpoint: process.env.S3_ENDPOINT || undefined,
        forcePathStyle: process.env.S3_FORCE_PATH_STYLE === 'true',
        prefix: process.env.S3_PREFIX || undefined,
        credentials,
      },
      cache: {
        dir: process.env.S3_CACHE_DIR || './.s3-cache',
        ttlMs: 1000 * 60 * 60,
        cleanupIntervalMs: 1000 * 60 * 15,
        expirationStrategy: 'absolute',
      },
    }),
  ],
})

Payload keeps handling imageSizes; each generated file is uploaded and cached under its own filename. The plugin sets disableLocalStorage: true, so new uploads are stored only in S3 and enter the local cache on their first read.

Lokale Integration prüfen

Regenerate Payload artifacts and build the consuming project:

pnpm generate:types
pnpm generate:importmap
pnpm build
pnpm dev

Then perform this smoke test:

1. Upload a new file through the Payload admin panel.
2. Confirm the object exists in the configured bucket.
3. Confirm .s3-cache does not contain the file before its first read.
4. Open the media URL once and confirm the file appears in .s3-cache.
5. Open it again and confirm the response still succeeds.
6. Delete the media document and confirm the S3 object and cached file disappear.

Produktion vorbereiten

For Docker and Dokploy, set S3_CACHE_DIR=/data/s3-cache, mount that path as a persistent volume, and keep the service at one replica. Continue with the production instructions below. The public npm package requires no registry credentials during build or at runtime.

Deploying a consuming Payload project with Dokploy

The recommended production flow is:

1. GitLab CI builds the consuming Payload project's Docker image.
2. GitLab CI pushes the finished image to the consuming project's Container Registry.
3. Dokploy pulls and runs that image through docker-compose.yml.

The npm package itself is public. Dokploy only needs separate credentials with read_registry for the private GitLab Container Registry.

1. Install dependencies in the Dockerfile

Add .env to the consuming project's .dockerignore:

.env
.env.*
!.env.example
node_modules
.next
.git

Use the following dependency stage in the existing Payload Dockerfile. Keep the syntax directive as its first line:

# syntax=docker/dockerfile:1.7
FROM node:20-alpine AS dependencies

ENV PNPM_HOME="/pnpm"
ENV PATH="$PNPM_HOME:$PATH"

WORKDIR /app

RUN corepack enable

COPY package.json pnpm-lock.yaml ./

RUN pnpm install --frozen-lockfile

Continue the existing multi-stage Dockerfile by copying node_modules from this stage into the builder stage. Do not copy .env into any image stage.

2. Build and push the Payload image in GitLab CI

Add this job to the consuming Payload project's .gitlab-ci.yml. If it already defines stages, add container to that list instead of declaring it twice.

stages:
  - container

build-payload-image:
  stage: container
  image: docker:27-cli
  services:
    - name: docker:27-dind
  variables:
    DOCKER_BUILDKIT: '1'
    DOCKER_TLS_CERTDIR: '/certs'
  before_script:
    - echo "$CI_REGISTRY_PASSWORD" | docker login --username "$CI_REGISTRY_USER" --password-stdin "$CI_REGISTRY"
  script:
    - docker build --tag "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA" .
    - docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"
    - docker tag "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA" "$CI_REGISTRY_IMAGE:latest"
    - docker push "$CI_REGISTRY_IMAGE:latest"
  rules:
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'

3. Run the finished image through Dokploy

Use this docker-compose.yml in the consuming Payload project:

services:
  payload:
    image: '${PAYLOAD_IMAGE}:${PAYLOAD_IMAGE_TAG:-latest}'
    restart: unless-stopped
    environment:
      DATABASE_URL: '${DATABASE_URL}'
      PAYLOAD_SECRET: '${PAYLOAD_SECRET}'
      S3_ACCESS_KEY_ID: '${S3_ACCESS_KEY_ID}'
      S3_BUCKET: '${S3_BUCKET}'
      S3_ENDPOINT: '${S3_ENDPOINT:-}'
      S3_FORCE_PATH_STYLE: '${S3_FORCE_PATH_STYLE:-false}'
      S3_REGION: '${S3_REGION}'
      S3_SECRET_ACCESS_KEY: '${S3_SECRET_ACCESS_KEY}'
      S3_CACHE_DIR: /data/s3-cache
    expose:
      - '3000'
    volumes:
      - payload-s3-cache:/data/s3-cache

volumes:
  payload-s3-cache:

In Dokploy, configure these service variables:

PAYLOAD_IMAGE=registry.gitlab.com/replace-with-your-payload-project-path
PAYLOAD_IMAGE_TAG=latest
DATABASE_URL=replace-with-your-production-database-url
PAYLOAD_SECRET=replace-with-a-long-random-secret
S3_ACCESS_KEY_ID=replace-with-your-access-key
S3_BUCKET=replace-with-your-bucket
S3_ENDPOINT=
S3_FORCE_PATH_STYLE=false
S3_REGION=eu-central-1
S3_SECRET_ACCESS_KEY=replace-with-your-secret-key

Create a GitLab Deploy Token with read_registry, add the GitLab Container Registry to Dokploy using that token's username and value, and select it for the Compose deployment. This credential is only for pulling the application image and is unrelated to npm.

Finally, configure the Dokploy domain to target port 3000, keep the Payload service at one replica, and deploy. The named payload-s3-cache volume preserves cached objects between deployments. See the official Dokploy documentation for Docker Compose variables, private build secrets, and production deployments.

Options

| Option | Required | Default | Description | | -------------------------- | -------: | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | | collections | yes | — | Upload collection slugs handled by the plugin. | | s3.bucket | yes | — | S3 bucket name. | | s3.region | yes | — | AWS region. | | s3.endpoint | no | SDK default | Endpoint for MinIO or another S3-compatible provider. | | s3.forcePathStyle | no | SDK default | Usually true for MinIO. | | s3.credentials | no | AWS SDK chain | Static access key and secret. Prefer environment or role credentials where possible. | | s3.prefix | no | empty | Global object-key prefix such as prod/; objects use <S3_PREFIX>/<collection-slug>/<filename> when set. | | cache.dir | no | ./.s3-cache | Local cache directory. | | cache.ttlMs | no | 3600000 | Freshness lifetime in milliseconds. | | cache.expirationStrategy | no | absolute | absolute counts from download; sliding refreshes mtime on a hit. | | cache.cleanupIntervalMs | no | 900000 | Cleanup frequency in milliseconds. | | enabled | no | true | Set to false to return the original Payload config unchanged; use S3_CACHE_STORAGE_ENABLED=true in the example above to enable it locally. |

Cache behavior

• Uploads go directly to S3 and do not warm the cache.
• S3_PREFIX is followed by the collection slug and filename: <S3_PREFIX>/<collection-slug>/<filename>. Each collection has an isolated cache namespace, so identically named files in different collections never share a cache entry.
• The first read downloads to a temporary file and atomically renames it into place.
• Fresh reads use the local file without another S3 request.
• TTL is age-based: entries expire one hour after download by default. Cleanup runs immediately at Payload startup and then every 15 minutes by default; expired files are refreshed under a per-filename in-memory lock.
• Replacements invalidate the local cache after a successful S3 write; deletes invalidate it after a successful S3 delete.
• An S3 failure does not fall back to expired content in v1.
• Browsers receive private, max-age=0, must-revalidate; the server-side cache owns freshness.

For a 24-hour cache lifetime, set ttlMs independently of the cleanup interval:

cache: {
  ttlMs: 1000 * 60 * 60 * 24,
  cleanupIntervalMs: 1000 * 60 * 15,
}

Deployment constraints

• Run exactly one long-lived Node.js instance for each cache directory.
• Use a persistent writable volume and monitor available disk space.
• No capacity-based or LRU cache eviction is implemented; provision disk space for the configured TTL and workload.
• Do not use this adapter in serverless or horizontally scaled deployments without redesigning locking and storage sharing.

HTTP errors

• 400: invalid or traversal-like filename.
• 404: S3 object does not exist.
• 502: S3, stream, or local cache I/O failure.

Payload performs collection read access control before calling the static handler.

Development

pnpm install
pnpm exec vitest run
pnpm build
pnpm lint

Releases

package.json is the version source. Once a new SemVer version reaches the Default-Branch, GitLab CI publishes it to the public npm Registry. The registry check prevents duplicate publication attempts and treats registry or authentication failures as errors.

Security

• Publishing uses short-lived GitLab-OIDC credentials through npm Trusted Publishing; no persistent npm publish token belongs in GitLab CI.
• Public package installation requires no npm or GitLab Package Registry token.
• S3 credentials can be omitted to use the AWS SDK credential chain.
• Request filenames are rejected before filesystem access if they contain traversal or separator syntax.

License

MIT