nx-resources

S3 Resource Gateway SDK for Node.js - fetch text, images, videos, and binaries from S3-compatible storage.

Features

• 🚀 Unified API for fetching resources from S3-compatible storage
• 📦 Multiple access modes: buffer, stream, or pre-signed URLs
• 🎯 Smart classification: automatically detects text, images, videos, and binaries
• 🔄 Retry logic: built-in retry with exponential backoff
• ⚡ Streaming support: efficient handling of large files
• 🎬 Range requests: support for partial content (video streaming)
• 🔧 Configurable: supports nx-config2 or environment variables
• 📝 Fully typed: TypeScript with comprehensive type definitions

Installation

npm install nx-resources

Quick Start

Using Environment Variables (for testing)

import { createS3ResourceGateway } from 'nx-resources';

// Set these environment variables (all required):
// SPACE_REGION=nyc3
// SPACE_ACCESS_KEY=your-access-key
// SPACE_SECRET_KEY=your-secret-key
// SPACE_ENDPOINT=https://nyc3.digitaloceanspaces.com
// SPACE_BUCKET=your-bucket-name (REQUIRED - very important!)
// FORCE_PATH_STYLE=true (important for S3-compatible services like DigitalOcean Spaces)

const gateway = await createS3ResourceGateway('default');

// Get text file
const readme = await gateway.getText('docs/readme.txt');
console.log(readme.data); // string

// Get image as stream
const image = await gateway.getStream('images/logo.png', {
  expectedKind: 'image'
});
image.stream.pipe(response); // Express/Fastify response

// Get pre-signed URL
const videoUrl = await gateway.getSignedUrl('videos/intro.mp4', {
  expiresInSeconds: 3600
});
console.log(videoUrl.url);

Using nx-config2

import { createS3ResourceGateway } from 'nx-resources';

// Config loaded from nx-config2: resourceGateway.s3.default
const gateway = await createS3ResourceGateway('default');

const resource = await gateway.getResource('path/to/file.jpg', {
  mode: 'stream'
});

API Reference

Factory Functions

createS3ResourceGateway(profileName?, options?)

Creates a gateway instance from a configuration profile.

• profileName (string, default: 'default'): Configuration profile name
• options.strict (boolean): Throw error if profile not found
• options.logger (ResourceGatewayLogger): Optional logger for observability

Returns: Promise<S3ResourceGateway>

createS3ResourceGatewayFromConfig(config, logger?)

Creates a gateway instance from an explicit config object (useful for testing).

Gateway Methods

getResource(key, options?)

Fetch a resource with automatic mode selection.

const resource = await gateway.getResource('file.jpg', {
  mode: 'auto' | 'buffer' | 'stream' | 'url',
  expectedKind: 'image',
  encoding: 'utf8',
  range: { start: 0, end: 1023 }
});

getText(key, options?)

Always returns text. Throws if resource is not text-like.

const text = await gateway.getText('readme.txt', {
  encoding: 'utf8'
});
console.log(text.data); // string

getStream(key, options?)

Returns a stream for large media files.

const stream = await gateway.getStream('video.mp4', {
  range: { start: 0, end: 1023 },
  expectedKind: 'video'
});
stream.stream.pipe(response);

getSignedUrl(key, options?)

Generate a pre-signed URL without fetching content.

const urlResource = await gateway.getSignedUrl('file.jpg', {
  expiresInSeconds: 3600
});
console.log(urlResource.url);

headResource(key, options?)

Get metadata only (HEAD request).

const metadata = await gateway.headResource('file.jpg');
console.log(metadata.contentType);
console.log(metadata.contentLength);

Configuration

Environment Variables

For testing, you can use these environment variables (all are required except where noted):

• SPACE_REGION: AWS region (e.g., nyc3, eu-central-1) - Required
• SPACE_ACCESS_KEY: Access key ID - Required
• SPACE_SECRET_KEY: Secret access key - Required
• SPACE_BUCKET: Bucket name - Required - Very Important! (also accepts S3_BUCKET as fallback)
• SPACE_ENDPOINT: Custom endpoint (for S3-compatible storage) - Optional but recommended
• FORCE_PATH_STYLE: Boolean (true, 1, yes) - Important for S3-compatible services like DigitalOcean Spaces, MinIO, etc. Defaults to true if SPACE_ENDPOINT is set.

nx-config2 Configuration

{
  "resourceGateway": {
    "s3": {
      "default": {
        "region": "eu-central-1",
        "bucket": "my-bucket",
        "endpoint": "https://nyc3.digitaloceanspaces.com",
        "forcePathStyle": true,
        "credentialMode": "static",
        "accessKeyId": "your-key",
        "secretAccessKey": "your-secret",
        "defaultMode": "stream",
        "maxBufferSizeBytes": 10000000,
        "defaultSignedUrlExpiresInSeconds": 3600
      }
    }
  }
}

Resource Classification

Resources are automatically classified as:

• text: .txt, .md, .json, .csv, .log, etc.
• image: .png, .jpg, .jpeg, .gif, .webp, .svg, etc.
• video: .mp4, .mov, .webm, .mkv, .avi, etc.
• binary: everything else

Classification can be customized via configuration.

Access Modes

• auto (default): Automatically chooses buffer or stream based on size
• buffer: Loads entire file into memory
• stream: Returns a readable stream
• url: Returns a pre-signed URL

Error Handling

import {
  ResourceNotFoundError,
  ResourceTooLargeError,
  S3AccessError,
  ClassificationError
} from 'nx-resources';

try {
  const resource = await gateway.getResource('file.jpg');
} catch (error) {
  if (error instanceof ResourceNotFoundError) {
    // Handle 404
  } else if (error instanceof ResourceTooLargeError) {
    // Handle size limit
  }
}

Express/Fastify Example

import express from 'express';
import { createS3ResourceGateway, ResourceNotFoundError } from 'nx-resources';

const app = express();
const gateway = await createS3ResourceGateway('default');

app.get('/media/:key', async (req, res, next) => {
  try {
    const key = req.params.key;
    const resource = await gateway.getStream(key, {
      expectedKind: 'image'
    });

    res.setHeader('Content-Type', resource.contentType ?? 'image/jpeg');
    if (resource.contentLength != null) {
      res.setHeader('Content-Length', String(resource.contentLength));
    }

    resource.stream.on('error', next);
    resource.stream.pipe(res);
  } catch (err) {
    if (err instanceof ResourceNotFoundError) {
      res.status(404).end();
    } else {
      next(err);
    }
  }
});

Video Streaming with Range Support

app.get('/video/:key', async (req, res, next) => {
  try {
    const key = req.params.key;
    const rangeHeader = req.headers.range;
    
    // Parse range header (you can use a library or implement parsing)
    const range = parseHttpRangeHeader(rangeHeader);

    const video = await gateway.getStream(key, {
      expectedKind: 'video',
      range
    });

    res.setHeader('Content-Type', video.contentType ?? 'video/mp4');
    res.setHeader('Accept-Ranges', 'bytes');

    if (video.range?.partial) {
      const { start, end, total } = video.range.served;
      res.status(206);
      res.setHeader('Content-Range', `bytes ${start}-${end}/${total}`);
      res.setHeader('Content-Length', String(end - start + 1));
    } else if (video.contentLength != null) {
      res.setHeader('Content-Length', String(video.contentLength));
    }

    video.stream.pipe(res);
  } catch (err) {
    next(err);
  }
});

License

MIT

Repository

https://github.com/nx-intelligence/nx-resources