S3Strata

Object storage abstraction layer with dual-bucket tiered storage and visibility control for Bun/TypeScript.

Features

• Dual-Bucket Architecture: Abstracts HOT (SSD) and COLD (HDD) storage tiers
• Multi-Endpoint Support: Use different S3 providers/regions for each tier (e.g., AWS S3 for HOT, Backblaze B2 for COLD)
• Visibility Control: PUBLIC and PRIVATE file access with path-based separation
• Smart URL Generation: Direct URLs for public files, presigned URLs for private files
• Hot Storage Duration: Set time-based expiration for files in HOT storage with automatic archival to COLD
• Type-safe: Full TypeScript support
• Database Agnostic: Pluggable storage adapter interface - bring your own ORM/database
• Lateral Movement: Easy migration between storage tiers and visibility levels
• Automatic Archival: Housekeeping method to periodically move expired HOT files to COLD storage

Installation

npm install s3strata
# or
bun add s3strata

Architecture

Storage Tiers

• HOT: Fast SSD storage for frequently accessed files
• COLD: Cost-effective HDD storage for archival/infrequent access

File Visibility

• PUBLIC: Publicly accessible files with /public prefix
  • Returns direct URLs without presigning
  • Example: https://s3.example.com/hot-bucket/public/images/uuid.jpg
• PRIVATE: Protected files with /private prefix
  • Returns presigned URLs (default 4-hour expiration)
  • Example: Presigned URL to /hot-bucket/private/submissions/uuid.pdf

Hot Storage Duration

Files in HOT storage can be flagged with an expiration time (hot_until). When the expiration time is reached, files can be automatically archived to COLD storage using the housekeeping method.

• Permanent HOT: Files with hot_until = null stay in HOT storage indefinitely
• Temporary HOT: Files with hot_until = <date> are marked for archival after that date
• Immediate archival: Set duration = 0 to mark for immediate archival

Path Structure

{bucket}/{visibility}/{custom-path}
├── hot-bucket/
│   ├── public/       # Publicly accessible HOT files
│   └── private/      # Protected HOT files
└── cold-bucket/
    ├── public/       # Publicly accessible COLD files
    └── private/      # Protected COLD files

Quick Start

1. Implement a Storage Adapter

S3Strata is database-agnostic. You need to implement the StorageAdapter interface to connect to your database:

import type { StorageAdapter, PhysicalFile, StorageTier } from 's3strata';

class MyStorageAdapter implements StorageAdapter {
  async create(data: {
    storage_tier: StorageTier;
    filename: string;
    path: string;
    hot_until: Date | null;
  }): Promise<PhysicalFile> {
    // Implement: Create a new file record in your database
  }

  async findById(id: string | number): Promise<PhysicalFile | null> {
    // Implement: Find a file by ID
  }

  async update(
    id: string | number,
    data: {
      storage_tier?: StorageTier;
      path?: string;
      hot_until?: Date | null;
    }
  ): Promise<PhysicalFile> {
    // Implement: Update a file record
  }

  async delete(id: string | number): Promise<void> {
    // Implement: Delete a file record
  }

  async findExpiredHotFiles(): Promise<PhysicalFile[]> {
    // Implement: Find HOT files where hot_until <= now
  }
}

See examples/prisma-adapter.ts for a complete Prisma implementation example.

2. Initialize FileManager

S3Strata supports two configuration modes:

Option A: Shared Endpoint (same S3 provider for both tiers)

import { FileManager, StorageTier, FileVisibility } from 's3strata';
import type { S3StrataConfig } from 's3strata';

// Configure S3 connection
const config: S3StrataConfig = {
  endpoint: 's3.amazonaws.com',
  port: 443,
  useSSL: true,
  accessKey: 'YOUR_ACCESS_KEY',
  secretKey: 'YOUR_SECRET_KEY',
  hotBucket: 'my-hot-bucket',
  coldBucket: 'my-cold-bucket',
  // Advanced options (optional)
  advanced: {
    defaultPresignedUrlExpiration: 7200, // 2 hours (default: 14400 = 4 hours)
    maxFileSize: 500 * 1024 * 1024, // 500MB (default: Infinity = no limit)
    defaultStorageTier: StorageTier.HOT, // default: StorageTier.HOT
    defaultVisibility: FileVisibility.PRIVATE, // default: FileVisibility.PRIVATE
  },
};

// Create your storage adapter instance
const adapter = new MyStorageAdapter(/* ... */);

// Create FileManager instance
const fileManager = new FileManager(config, adapter);

Option B: Separate Endpoints (different S3 providers for each tier)

Perfect for cost optimization - use expensive fast storage for HOT, cheap archival for COLD:

const config: S3StrataConfig = {
  // HOT storage on AWS S3 (fast, expensive)
  hot: {
    endpoint: 's3.us-east-1.amazonaws.com',
    accessKey: 'AWS_ACCESS_KEY',
    secretKey: 'AWS_SECRET_KEY',
    bucket: 'my-hot-bucket',
  },
  // COLD storage on Backblaze B2 (slower, 77% cheaper)
  cold: {
    endpoint: 's3.us-west-004.backblazeb2.com',
    accessKey: 'B2_ACCESS_KEY',
    secretKey: 'B2_SECRET_KEY',
    bucket: 'my-archive-bucket',
  },
};

const fileManager = new FileManager(config, adapter);

See MULTI_ENDPOINT.md for detailed multi-endpoint configuration guide.

Advanced Configuration Options

S3Strata allows you to customize default behaviors through the advanced configuration object:

const config: S3StrataConfig = {
  // ... your S3 configuration ...
  advanced: {
    // Default expiration time for presigned URLs in seconds
    // Default: 14400 (4 hours)
    defaultPresignedUrlExpiration: 3600, // 1 hour

    // Maximum allowed file size in bytes
    // Default: Infinity (no limit)
    maxFileSize: 100 * 1024 * 1024, // 100MB

    // Default storage tier for new files
    // Default: StorageTier.HOT
    defaultStorageTier: StorageTier.HOT,

    // Default visibility for new files
    // Default: FileVisibility.PRIVATE
    defaultVisibility: FileVisibility.PRIVATE,
  },
};

When to use advanced options:

• Set maxFileSize to enforce upload limits and prevent abuse
• Customize defaultPresignedUrlExpiration based on your security requirements
• Change defaultStorageTier if you want new uploads to go to COLD storage by default
• Set defaultVisibility to PUBLIC if most of your files should be publicly accessible

All advanced options are optional and have sensible defaults. You can override defaults on a per-operation basis (e.g., specify expiresIn when calling getUrl()).

3. Use FileManager

// Upload a private file to HOT storage
const file = await fileManager.upload(buffer, {
  tier: StorageTier.HOT,
  visibility: FileVisibility.PRIVATE,
  filename: 'document.pdf',
});

// Get URL (returns presigned URL for private files)
const url = await fileManager.getUrl(file, {
  expiresIn: 14400, // 4 hours (default)
});

// Make file public (moves from /private to /public)
const updatedFile = await fileManager.setVisibility(file, {
  visibility: FileVisibility.PUBLIC,
});

// Get URL (now returns direct public URL)
const publicUrl = await fileManager.getUrl(updatedFile);

// Move to COLD storage (migrates to HDD bucket)
const coldFile = await fileManager.setTier(updatedFile, {
  tier: StorageTier.COLD,
});

// Delete file
await fileManager.delete(coldFile);

API Reference

FileManager

upload(data: Buffer | Blob, options?: UploadOptions): Promise<PhysicalFile>

Upload a file to object storage.

Options:

• tier?: StorageTier - Storage tier (HOT or COLD), defaults to HOT
• visibility?: FileVisibility - File visibility (PUBLIC or PRIVATE), defaults to PRIVATE
• filename?: string - Custom filename (optional, auto-generated if not provided)
• pathSuffix?: string - Custom path suffix (optional, auto-generated if not provided)
• hotDuration?: number - Duration in seconds to keep file in HOT storage (only for HOT tier)

Example:

const file = await fileManager.upload(buffer, {
  tier: StorageTier.HOT,
  visibility: FileVisibility.PRIVATE,
  filename: 'report.pdf',
  hotDuration: 7 * 24 * 60 * 60, // 7 days
});

getUrl(file: PhysicalFile, options?: GetUrlOptions): Promise<string>

Generate URL for a file. Returns direct URL for PUBLIC files, presigned URL for PRIVATE files.

Options:

• expiresIn?: number - Presign duration in seconds (default: 14400 = 4 hours), only used for PRIVATE files

Example:

const url = await fileManager.getUrl(file, { expiresIn: 3600 }); // 1 hour

setVisibility(file: PhysicalFile, options: SetVisibilityOptions): Promise<PhysicalFile>

Change file visibility (PUBLIC ↔ PRIVATE). Moves file between /public and /private prefixes.

Options:

• visibility: FileVisibility - New visibility level
• moveFile?: boolean - Whether to move the file immediately (default: true)

Example:

const publicFile = await fileManager.setVisibility(file, {
  visibility: FileVisibility.PUBLIC,
});

setTier(file: PhysicalFile, options: SetTierOptions): Promise<PhysicalFile>

Move file between storage tiers (HOT ↔ COLD).

Options:

• tier: StorageTier - New storage tier
• moveFile?: boolean - Whether to move the file immediately (default: true)
• hotDuration?: number - Duration in seconds to keep file in HOT storage (only when tier=HOT)

Example:

const coldFile = await fileManager.setTier(file, {
  tier: StorageTier.COLD,
});

setHotDuration(file: PhysicalFile, options: SetHotDurationOptions): Promise<PhysicalFile>

Set hot storage duration for a HOT tier file.

Options:

• duration: number | null - Duration in seconds (null = permanent, 0 = immediate archival)

Example:

// Set 30-day expiration
const updated = await fileManager.setHotDuration(file, {
  duration: 30 * 24 * 60 * 60,
});

// Make permanent (remove expiration)
const permanent = await fileManager.setHotDuration(file, {
  duration: null,
});

// Mark for immediate archival
const immediate = await fileManager.setHotDuration(file, {
  duration: 0,
});

delete(file: PhysicalFile): Promise<void>

Delete a file from both storage and database.

Example:

await fileManager.delete(file);

getById(id: string | number): Promise<PhysicalFile | null>

Get file record from database by ID.

Example:

const file = await fileManager.getById(123);

exists(file: PhysicalFile): Promise<boolean>

Check if file exists in object storage.

Example:

const exists = await fileManager.exists(file);

archiveExpiredHotFiles(): Promise<number>

Housekeeping method to move expired HOT files to COLD storage. Returns the number of files moved.

Example:

// Run periodically via cron job
const movedCount = await fileManager.archiveExpiredHotFiles();
console.log(`Archived ${movedCount} files from HOT to COLD`);

listFiles(): Promise<PhysicalFile[]>

List all file records from the database.

Example:

const allFiles = await fileManager.listFiles();
console.log(allFiles);

Storage Adapter Examples

Prisma Adapter

import { PrismaClient } from '@prisma/client';
import { PrismaStorageAdapter } from 's3strata';

const prisma = new PrismaClient();
const adapter = new PrismaStorageAdapter(prisma);
const fileManager = new FileManager(config, adapter);

Prisma Schema:

model PhysicalFile {
  id           Int          @id @default(autoincrement())
  storage_tier StorageTier
  filename     String
  path         String
  hot_until    DateTime?
  created_at   DateTime     @default(now())
  updated_at   DateTime     @updatedAt
}

enum StorageTier {
  HOT
  COLD
}

Custom Adapter Example

import type { StorageAdapter, PhysicalFile, StorageTier } from 's3strata';
import { Database } from 'some-database-library';

class CustomStorageAdapter implements StorageAdapter {
  constructor(private db: Database) {}

  async create(data: {
    storage_tier: StorageTier;
    filename: string;
    path: string;
    hot_until: Date | null;
  }): Promise<PhysicalFile> {
    const result = await this.db.insert('physical_files', data);
    return result;
  }

  async findById(id: string | number): Promise<PhysicalFile | null> {
    return await this.db.findOne('physical_files', { id });
  }

  async update(
    id: string | number,
    data: {
      storage_tier?: StorageTier;
      path?: string;
      hot_until?: Date | null;
    }
  ): Promise<PhysicalFile> {
    return await this.db.update('physical_files', { id }, data);
  }

  async delete(id: string | number): Promise<void> {
    await this.db.delete('physical_files', { id });
  }

  async findExpiredHotFiles(): Promise<PhysicalFile[]> {
    return await this.db.findMany('physical_files', {
      storage_tier: 'HOT',
      hot_until: { lte: new Date() },
    });
  }
}

Complete Usage Example

import { FileManager, StorageTier, FileVisibility, PrismaStorageAdapter } from 's3strata';
import { PrismaClient } from '@prisma/client';

// Initialize
const prisma = new PrismaClient();
const adapter = new PrismaStorageAdapter(prisma);
const config = {
  endpoint: 's3.amazonaws.com',
  accessKey: process.env.S3_ACCESS_KEY!,
  secretKey: process.env.S3_SECRET_KEY!,
  hotBucket: 'my-hot-bucket',
  coldBucket: 'my-cold-bucket',
};
const fileManager = new FileManager(config, adapter);

// Upload a file with 7-day hot storage
const file = await fileManager.upload(fileBuffer, {
  tier: StorageTier.HOT,
  visibility: FileVisibility.PRIVATE,
  filename: 'user-upload.pdf',
  hotDuration: 7 * 24 * 60 * 60, // 7 days
});

// Get presigned URL (4 hour expiration)
const url = await fileManager.getUrl(file);

// After processing, make it public
const publicFile = await fileManager.setVisibility(file, {
  visibility: FileVisibility.PUBLIC,
});

// Get public URL
const publicUrl = await fileManager.getUrl(publicFile);

// Move to cold storage for long-term archival
const archivedFile = await fileManager.setTier(publicFile, {
  tier: StorageTier.COLD,
});

// Run housekeeping (e.g., in a cron job)
setInterval(async () => {
  const count = await fileManager.archiveExpiredHotFiles();
  console.log(`Archived ${count} expired files`);
}, 60 * 60 * 1000); // Every hour

Environment Variables

You can use environment variables for configuration:

S3_ENDPOINT=s3.amazonaws.com
S3_PORT=443
S3_USE_SSL=true
S3_ACCESS_KEY=your-access-key
S3_SECRET_KEY=your-secret-key
S3_BUCKET_HOT=hot-bucket
S3_BUCKET_COLD=cold-bucket

TypeScript Support

S3Strata is written in TypeScript and provides full type definitions.

import type {
  PhysicalFile,
  StorageAdapter,
  S3StrataConfig,
  S3StrataAdvancedOptions,
  UploadOptions,
  GetUrlOptions,
  SetVisibilityOptions,
  SetTierOptions,
} from 's3strata';

License

MIT

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.