npm i @codebucket/files

Optional integrations:

• Azure Blob uploader requires a peer dependency:
  • npm i @azure/storage-blob

Quick start

This package provides:

• A simple Files facade (filesystem or S3-compatible)
• Standalone uploaders (FileSystemUploader, S3CompatibleUploader, AzureBlobUploader)
• A Multer storage engine helper (createMulterUploader) for plug-and-play uploads

For AI agents / code generation tools

• Import from the package root: @codebucket/files
• Prefer the exported typings instead of deep imports into src/*
• Files supports only filesystem and s3; use AzureBlobUploader directly for Azure Blob Storage
• Public root exports include Files, all three uploaders, createMulterUploader, FilesStorageEngine, and the related config/option types
• upload() takes a Buffer; uploadStream() takes a Node Readable
• download() and downloadZip() return a Buffer when no response object is passed, or stream to the response when res is provided
• getSignedUrl() / getSignedUploadUrl() produce time-limited URLs on S3 and Azure; they are not available on FileSystemUploader
• For Multer, set keepBuffer: false when you want true streaming behavior instead of preserving req.file.buffer
• A concise Codex-focused API reference lives in AGENTS.md

S3 / S3-compatible example

// utils/storage.js
const { Files } = require('@codebucket/files');

const fileStorage = new Files({
  type: 's3',
  publicBaseUrl: process.env.BASEURL + '/files/',
  bucketName: process.env.S3_BUCKET_NAME,
  endpoint: process.env.S3_ENDPOINT,
  s3Config: {
    region: process.env.S3_REGION,
    credentials: {
      accessKeyId: process.env.S3_ACCESS_KEY_ID,
      secretAccessKey: process.env.S3_SECRET_ACCESS_KEY,
    },
  },
});

module.exports = { fileStorage };

Upload

const fs = require('fs');
const path = require('path');
const { fileStorage } = require('./storage');

async function upload(files, body, key = 'document') {
  const fileName =
    new Date()
      .toISOString()
      .replace(/:/g, '-')
      .replace(/[^a-z0-9]/gi, '_')
      .toLowerCase() + path.extname(files[key].originalFilename);

  await fileStorage.upload(`/${body.userId}/${fileName}`, fs.readFileSync(files[key].filepath));

  return {
    name: fileName,
    url: fileStorage.getPublicUrl(`/${body.userId}/${fileName}`),
  };
}

Download (stream to response)

async function download(req, res) {
  let fullPath = req.path;
  fullPath = fullPath.replace('/files', '');
  await fileStorage.download(fullPath, res);
}

// Express route
app.get('/files/*', async (req, res) => {
  await download(req, res);
});

Download ZIP

async function downloadZip(res) {
  const filePaths = ['/folder/file1.txt', '/folder/file2.txt'];
  await fileStorage.downloadZip(filePaths, res);
}

Filesystem example

// utils/storage.js
const { Files } = require('@codebucket/files');

const fileStorage = new Files({
  type: 'filesystem',
  publicBaseUrl: process.env.BASEURL + '/',
  baseDir: './public',
});

module.exports = { fileStorage };

Upload/download usage is the same as the S3 example above.

Using standalone uploaders

You can instantiate uploaders directly (useful when you don’t want the Files facade).

FileSystemUploader

const { FileSystemUploader } = require('@codebucket/files');

const uploader = new FileSystemUploader('./public', process.env.BASEURL + '/');

const location = await uploader.upload('/avatars/a.png', Buffer.from('...'), {
  contentType: 'image/png',
  metadata: { scope: 'avatars' },
});
const publicUrl = uploader.getPublicUrl('/avatars/a.png');

S3CompatibleUploader

const { S3CompatibleUploader } = require('@codebucket/files');

const uploader = new S3CompatibleUploader(
  process.env.S3_BUCKET_NAME,
  process.env.S3_ENDPOINT,
  '/base-prefix', // optional baseDir
  process.env.BASEURL + '/files/', // optional publicBaseUrl
  {
    region: process.env.S3_REGION,
    credentials: {
      accessKeyId: process.env.S3_ACCESS_KEY_ID,
      secretAccessKey: process.env.S3_SECRET_ACCESS_KEY,
    },
  },
);

const location = await uploader.upload('/docs/readme.txt', Buffer.from('hello'), {
  contentType: 'text/plain',
});

AzureBlobUploader (Azure Blob Storage)

Azure is shipped as an optional peer dependency.

Install:

npm i @codebucket/files @azure/storage-blob

Use:

const { AzureBlobUploader } = require('@codebucket/files');

const uploader = new AzureBlobUploader(
  process.env.AZURE_CONTAINER_NAME,
  process.env.AZURE_STORAGE_CONNECTION_STRING,
  {
    baseDir: '/uploads', // optional
    publicBaseUrl: process.env.AZURE_PUBLIC_BASE_URL, // optional
  },
);

const location = await uploader.upload('/user-1/a.png', Buffer.from('...'), {
  contentType: 'image/png',
});
const url = uploader.getPublicUrl('/user-1/a.png');

Multer integration (Express / Nest / any Multer-based stack)

This library exposes a Multer storage engine so you can write incoming files using any IUploader.

What the storage engine supports

• Direct use with multer({ storage: createMulterUploader(...) })
• Buffer uploads for backwards compatibility
• Streamed uploads for uploaders that implement uploadStream(...)
• Upload metadata (contentType, contentLength, custom metadata)
• Cleanup via uploader.delete(key) if Multer aborts or post-upload hooks fail
• Richer req.file info:
  • req.file.key
  • req.file.location
  • req.file.mimetype
  • req.file.originalname

Example: Multer + S3CompatibleUploader

const express = require('express');
const multer = require('multer');
const path = require('path');
const {
  S3CompatibleUploader,
  createMulterUploader,
} = require('@codebucket/files');

const app = express();

const uploader = new S3CompatibleUploader(
  process.env.S3_BUCKET_NAME,
  process.env.S3_ENDPOINT,
  undefined,
  process.env.BASEURL + '/files/',
  {
    region: process.env.S3_REGION,
    credentials: {
      accessKeyId: process.env.S3_ACCESS_KEY_ID,
      secretAccessKey: process.env.S3_SECRET_ACCESS_KEY,
    },
  },
);

function sanitizeFilename(filename) {
  const ext = path.extname(filename).toLowerCase();
  const base = path.basename(filename, ext).replace(/[^a-z0-9-_]/gi, '-').replace(/-+/g, '-');
  return `${base || 'file'}${ext}`;
}

const upload = multer({
  storage: createMulterUploader(uploader, {
    key: (req, file) => `/uploads/${req.user.id}/${Date.now()}-${sanitizeFilename(file.originalname)}`,
    keepBuffer: false, // enables true streaming when the uploader supports uploadStream(...)
    buildUploadOptions: (req, file) => ({
      contentType: file.mimetype,
      metadata: {
        uploadedBy: String(req.user.id),
        field: file.fieldname,
      },
    }),
    onUploaded: async ({ key, location }) => {
      // optional hook: persist DB row, emit event, etc.
    },
  }),
  limits: {
    fileSize: 10 * 1024 * 1024, // 10 MB
  },
  fileFilter: (req, file, cb) => {
    if (!['image/png', 'image/jpeg', 'application/pdf'].includes(file.mimetype)) {
      return cb(new Error('Unsupported file type'));
    }
    cb(null, true);
  },
});

app.post('/upload', upload.single('file'), (req, res) => {
  // Multer sets req.file.path and req.file.location to the returned uploader location
  res.json({
    key: req.file.key,
    location: req.file.location,
    originalname: req.file.originalname,
    mimetype: req.file.mimetype,
    size: req.file.size,
  });
});

Example: keep compatibility with req.file.buffer

If your downstream code expects req.file.buffer, keep the default keepBuffer: true:

const upload = multer({
  storage: createMulterUploader(uploader, {
    key: (req, file) => `/uploads/${Date.now()}-${file.originalname}`,
  }),
});

That preserves the old behavior and still uploads through your configured uploader.

Example: multiple files

app.post('/photos', upload.array('photos', 5), (req, res) => {
  res.json(
    req.files.map((file) => ({
      key: file.key,
      location: file.location,
      size: file.size,
    })),
  );
});

Example: mixed fields

const mixedUpload = multer({
  storage: createMulterUploader(uploader, {
    key: (req, file) => `/${file.fieldname}/${Date.now()}-${sanitizeFilename(file.originalname)}`,
    keepBuffer: false,
  }),
});

app.post(
  '/profile',
  mixedUpload.fields([
    { name: 'avatar', maxCount: 1 },
    { name: 'resume', maxCount: 1 },
  ]),
  (req, res) => {
    res.json({
      avatar: req.files.avatar?.[0]?.location,
      resume: req.files.resume?.[0]?.location,
    });
  },
);

Example: custom storage metadata

const upload = multer({
  storage: createMulterUploader(uploader, {
    key: (req, file) => `/documents/${Date.now()}-${sanitizeFilename(file.originalname)}`,
    keepBuffer: false,
    buildUploadOptions: async (req, file) => ({
      contentType: file.mimetype,
      metadata: {
        tenantId: String(req.tenant.id),
        category: req.body.category || 'general',
      },
    }),
  }),
});

Using the storage engine in NestJS

import { Controller, Post, UploadedFile, UseInterceptors } from '@nestjs/common';
import { FileInterceptor } from '@nestjs/platform-express';
import multer from 'multer';
import { S3CompatibleUploader, createMulterUploader } from '@codebucket/files';

const uploader = new S3CompatibleUploader(
  process.env.S3_BUCKET_NAME!,
  process.env.S3_ENDPOINT!,
  undefined,
  process.env.BASEURL + '/files/',
  {
    region: process.env.S3_REGION,
    credentials: {
      accessKeyId: process.env.S3_ACCESS_KEY_ID!,
      secretAccessKey: process.env.S3_SECRET_ACCESS_KEY!,
    },
  },
);

const storage = createMulterUploader(uploader, {
  key: (req, file) => `/uploads/${Date.now()}-${file.originalname}`,
  keepBuffer: false,
});

@Controller('files')
export class FilesController {
  @Post('upload')
  @UseInterceptors(FileInterceptor('file', { storage }))
  upload(@UploadedFile() file: Express.Multer.File) {
    return {
      key: (file as any).key,
      location: (file as any).location,
      size: file.size,
    };
  }
}

Multer cleanup behavior

If Multer aborts an upload and calls _removeFile, the storage engine will best-effort call uploader.delete(key) (where key is the same value returned by your key() resolver).

Pre-signed URLs

Instead of proxying bytes through your server, hand the client a time-limited URL that talks to storage directly.

• getSignedUrl(filePath, options?) — a temporary download (GET) URL
• getSignedUploadUrl(filePath, options?) — a temporary upload (PUT) URL

Both are available on Files (when configured for s3), S3CompatibleUploader, and AzureBlobUploader. FileSystemUploader does not support them.

// Temporary download link, valid for 15 minutes
const url = await fileStorage.getSignedUrl('/user-1/invoice.pdf', { expiresIn: 900 });

// Force a browser download under a friendly name
const url = await fileStorage.getSignedUrl('/user-1/invoice.pdf', {
  expiresIn: 900,
  downloadAs: 'invoice-march.pdf',
});

// Let the browser upload straight to storage
const uploadUrl = await fileStorage.getSignedUploadUrl('/user-1/avatar.png', {
  expiresIn: 600,
  contentType: 'image/png',
});

// Client side
await fetch(uploadUrl, {
  method: 'PUT',
  headers: { 'Content-Type': 'image/png' }, // must match exactly
  body: file,
});

Options

getSignedUrl(filePath, options?)

| Option | Description | | --- | --- | | expiresIn | Seconds until expiry. Default 3600, max 604800 (7 days). | | downloadAs | Sets Content-Disposition: attachment with this filename. | | responseContentType | Overrides the Content-Type storage returns. | | responseContentDisposition | Sets Content-Disposition verbatim. |

getSignedUploadUrl(filePath, options?)

| Option | Description | | --- | --- | | expiresIn | Seconds until expiry. Default 3600, max 604800 (7 days). | | contentType | Signed. The client must send this exact Content-Type header. | | contentLength | Signed. The client must send exactly this many bytes. | | metadata | Travels in the query string; the client does not replay it. |

Anything signed is enforced by the storage provider, so a URL issued for image/png cannot be used to upload something else. Omit contentType / contentLength if you'd rather let the client decide — a mismatch is rejected as SignatureDoesNotMatch.

Change the default lifetime for every URL from one place:

const fileStorage = new Files({
  type: 's3',
  // ...
  s3Config: {
    // ...
    signedUrlExpiresIn: 900, // 15 minutes instead of 1 hour
  },
});

Backblaze B2 notes

Backblaze's S3-compatible API supports pre-signed GET and PUT, so both methods work there. Two things are worth knowing:

• Region must match the endpoint. B2 signs with SigV4, which folds the region into the signature. s3.us-west-004.backblazeb2.com must be paired with region us-west-004, or B2 answers SignatureDoesNotMatch. If you don't pass region, it is now derived from the endpoint hostname automatically. Passing region explicitly always wins.
• No POST-form uploads. B2 does not implement browser POST uploads to a pre-signed policy, which is why the upload flow here is PUT-based. A PUT from fetch or XMLHttpRequest works fine from the browser.

const fileStorage = new Files({
  type: 's3',
  bucketName: process.env.B2_BUCKET_NAME,
  endpoint: 'https://s3.us-west-004.backblazeb2.com',
  s3Config: {
    region: 'us-west-004', // optional — derived from the endpoint when omitted
    credentials: {
      accessKeyId: process.env.B2_KEY_ID,
      secretAccessKey: process.env.B2_APPLICATION_KEY,
    },
  },
});

Connection / concurrency tuning (S3)

The AWS SDK caps its HTTP agent at 50 sockets per host by default. Requests past that limit queue up inside the agent, and that queue wait is not covered by requestTimeout — so an undersized pool shows up as uploads that hang rather than uploads that fail. Streaming uploads make this arrive sooner than you'd expect: every multipart part in flight holds its own socket, so with the default uploadQueueSize of 4 you saturate the pool at roughly 12 concurrent uploads.

All of it is configurable through s3Config:

const fileStorage = new Files({
  type: 's3',
  bucketName: process.env.S3_BUCKET_NAME,
  endpoint: process.env.S3_ENDPOINT,
  s3Config: {
    region: process.env.S3_REGION,
    credentials: { /* ... */ },

    maxSockets: 250,          // concurrent sockets per host (default 50)
    uploadQueueSize: 4,       // multipart parts in flight per uploadStream() (default 4)
    uploadPartSize: 8 * 1024 * 1024, // bytes per part (default 5 MiB)
    downloadConcurrency: 20,  // files fetched in parallel by downloadZip()
    connectionTimeout: 5000,  // ms to establish the connection (default: none)
    requestTimeout: 60000,    // ms for a single request (default: none)
  },
});

| Option | Default | What it controls | | --- | --- | --- | | maxSockets | 50 | Concurrent sockets per host. Raise this first under load. | | maxTotalSockets | unset | Ceiling across all hosts. | | keepAlive | true | Socket reuse between requests. | | keepAliveMsecs | SDK default | Idle time before a kept-alive socket is probed. | | connectionTimeout | 0 (none) | Ms to establish the TCP/TLS connection. | | requestTimeout | 0 (none) | Ms a single request may take. | | uploadQueueSize | 4 | Multipart parts in flight per uploadStream() call. | | uploadPartSize | 5 MiB | Multipart part size in bytes. | | downloadConcurrency | maxSockets | Files fetched in parallel by downloadZip(). | | signedUrlExpiresIn | 3600 | Default signed-URL lifetime in seconds. | | clientConfig | unset | Escape hatch merged into the S3Client config last. |

Sizing rule of thumb: maxSockets should comfortably exceed concurrent uploads x uploadQueueSize, plus headroom for downloads and deletes sharing the same pool.

downloadZip() is bounded by downloadConcurrency so a large ZIP can never enqueue more requests than the pool can serve.

If you see socket usage at capacity warnings

@smithy/node-http-handler:WARN - socket usage at capacity=50 and 454 additional requests are enqueued.

A large enqueued count means sockets are being held, not just used. Downloads are the usual culprit, because a storage response only releases its socket once its body is fully read or explicitly destroyed. Two things used to strand them, both fixed here:

• A client disconnecting mid-download. Piping to a response does not tear down the source stream when the client goes away, so the body sat unread and its socket never returned to the pool. download() now destroys the source on disconnect, and resolves only once the bytes are actually delivered.
• A throw between receiving the response and reading it. Any error in that window leaked the socket. download() now destroys the body on every failure path.

If you still see the warning after upgrading, it is genuine load rather than a leak — raise maxSockets, and remember each streaming upload holds uploadQueueSize sockets of its own.

Note that download(path, res) and downloadZip(paths, res) now resolve when the transfer completes rather than when it starts. Awaiting them is what you want; a client that disconnects part-way resolves normally rather than throwing, since that is routine rather than an application error.

Delete semantics

All uploaders implement:

• delete(filePath: string): Promise<void>

Example:

await fileStorage.delete('/user-1/old-file.png');