@geekmidas/storage

A comprehensive, type-safe storage client for cloud storage services with support for multiple providers and advanced features like versioning and presigned URLs.

Features

• Multi-provider support: AWS S3, with extensible interface for Google Cloud Storage and Azure Blob Storage
• Type-safe: Full TypeScript support with comprehensive type definitions
• Presigned URLs: Generate secure upload and download URLs without exposing credentials
• File versioning: Support for retrieving and managing file versions
• Direct uploads: Upload files directly to storage without intermediate servers
• Flexible configuration: Support for custom endpoints (useful for MinIO, LocalStack, etc.)
• Modern async/await API: Promise-based interface throughout
• URL caching: Built-in support for caching presigned URLs to reduce API calls and improve performance

Installation

npm install @geekmidas/storage

Peer Dependencies

For AWS S3 support, you'll need to install the AWS SDK v3 packages:

npm install @aws-sdk/client-s3 @aws-sdk/s3-presigned-post @aws-sdk/s3-request-presigner

Quick Start

AWS S3

import { AmazonStorageClient } from '@geekmidas/storage/aws';
import { InMemoryCache } from '@geekmidas/cache/memory';

// Create client with credentials
const storage = AmazonStorageClient.create({
  bucket: 'my-bucket',
  region: 'us-east-1',
  accessKeyId: process.env.AWS_ACCESS_KEY_ID,
  secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
});

// Create client with caching enabled
const cache = new InMemoryCache<string>();
const storageWithCache = AmazonStorageClient.create({
  bucket: 'my-bucket',
  region: 'us-east-1',
  accessKeyId: process.env.AWS_ACCESS_KEY_ID,
  secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
  cache,
});

// Upload a file directly
await storage.upload('documents/readme.txt', 'Hello, World!', 'text/plain');

// Generate a download URL
const downloadUrl = await storage.getDownloadURL({ 
  path: 'documents/readme.txt',
  name: 'README.txt' // Optional: sets Content-Disposition header
});

// Generate a presigned upload URL
const uploadUrl = await storage.getUploadURL({
  path: 'uploads/new-file.pdf',
  contentType: 'application/pdf',
  contentLength: 1024 * 1024, // 1MB
});

MinIO / LocalStack

import { AmazonStorageClient } from '@geekmidas/storage/aws';

// For local development with MinIO
const storage = AmazonStorageClient.create({
  bucket: 'test-bucket',
  region: 'us-east-1',
  accessKeyId: 'minioadmin',
  secretAccessKey: 'minioadmin',
  endpoint: 'http://localhost:9000',
});

API Reference

StorageClient Interface

The core interface that all storage providers implement:

interface StorageClient {
  readonly provider: StorageProvider;
  readonly cache?: Cache<string>;
  
  // Direct upload
  upload(key: string, data: string | Buffer, contentType: string): Promise<void>;
  
  // Download URLs
  getDownloadURL(file: File, expiresIn?: number): Promise<string>;
  
  // Upload URLs
  getUploadURL(params: GetUploadParams, expiresIn?: number): Promise<string>;
  getUpload(params: GetUploadParams, expiresIn?: number): Promise<GetUploadResponse>;
  
  // Versioning
  getVersions(key: string): Promise<DocumentVersion[]>;
  getVersionDownloadURL(file: File, versionId: string): Promise<string>;
}

AmazonStorageClient

Factory Method

AmazonStorageClient.create(options: AmazonStorageClientCreateOptions)

Options:

• bucket (required): S3 bucket name
• region: AWS region (default: uses AWS SDK default)
• accessKeyId: AWS access key ID
• secretAccessKey: AWS secret access key
• endpoint: Custom S3 endpoint (useful for MinIO, LocalStack)
• acl: Canned ACL for uploads (default: authenticated-read)
• cache: Optional cache implementation for storing presigned URLs
• forcePathStyle: Force path-style URLs (useful for MinIO)

Methods

upload(key: string, data: string | Buffer, contentType: string): Promise<void>

Upload data directly to storage.

// Upload text
await storage.upload('documents/hello.txt', 'Hello, World!', 'text/plain');

// Upload binary data
const buffer = Buffer.from('binary data');
await storage.upload('files/binary.dat', buffer, 'application/octet-stream');

getDownloadURL(file: File, expiresIn?: number): Promise<string>

Generate a presigned download URL. When a cache is configured, URLs will be cached based on the file path.

// Simple download URL
const url = await storage.getDownloadURL({ path: 'documents/file.pdf' });

// With custom filename in Content-Disposition
const url = await storage.getDownloadURL({ 
  path: 'documents/file.pdf',
  name: 'My Document.pdf'
});

// Custom expiration (in seconds)
const url = await storage.getDownloadURL({ path: 'documents/file.pdf' }, 3600);

Caching behavior:

• URLs are cached with key format: download-url:{file.path}
• Cache TTL is set to expiresIn - 60 seconds (with 1 minute buffer)
• URLs with expiration < 60 seconds are not cached
• Cached URLs are returned immediately without generating new presigned URLs

getUploadURL(params: GetUploadParams, expiresIn?: number): Promise<string>

Generate a presigned PUT upload URL.

const uploadUrl = await storage.getUploadURL({
  path: 'uploads/new-file.pdf',
  contentType: 'application/pdf',
  contentLength: 1024 * 1024,
});

// Use the URL to upload
const response = await fetch(uploadUrl, {
  method: 'PUT',
  headers: {
    'Content-Type': 'application/pdf',
    'Content-Length': '1048576',
  },
  body: fileData,
});

getUpload(params: GetUploadParams, expiresIn?: number): Promise<GetUploadResponse>

Generate a presigned POST upload with form fields.

const upload = await storage.getUpload({
  path: 'uploads/form-upload.jpg',
  contentType: 'image/jpeg',
  contentLength: 500000,
});

// Use with HTML form
const formData = new FormData();
upload.fields.forEach(({ key, value }) => {
  formData.append(key, value);
});
formData.append('file', fileInput.files[0]);

const response = await fetch(upload.url, {
  method: 'POST',
  body: formData,
});

getVersions(key: string): Promise<DocumentVersion[]>

Get all versions of a file (requires S3 versioning).

const versions = await storage.getVersions('documents/versioned-file.txt');
console.log(versions); // [{ id: 'version-1', createdAt: Date }, ...]

getVersionDownloadURL(file: File, versionId: string): Promise<string>

Generate download URL for a specific version.

const url = await storage.getVersionDownloadURL(
  { path: 'documents/file.txt' },
  'version-12345'
);

Types

File

interface File {
  path: string;
  name?: string; // Optional display name for Content-Disposition
}

GetUploadParams

interface GetUploadParams {
  path: string;
  contentType: string;
  contentLength: number;
}

DocumentVersion

interface DocumentVersion {
  id: string;
  createdAt: Date;
}

StorageProvider

enum StorageProvider {
  AWSS3 = 'geekimdas.toolbox.storage.aws.s3',
  GCP = 'geekimdas.toolbox.storage.gcp',
  AZURE = 'geekimdas.toolbox.storage.azure',
}

Advanced Usage

Custom S3 Client

import { S3Client } from '@aws-sdk/client-s3';
import { AmazonStorageClient, AmazonCannedAccessControlList } from '@geekmidas/storage/aws';
import { InMemoryCache } from '@geekmidas/cache/memory';

const s3Client = new S3Client({
  region: 'us-east-1',
  credentials: {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
  },
});

// Without cache
const storage = new AmazonStorageClient(
  s3Client,
  'my-bucket',
  AmazonCannedAccessControlList.PublicRead
);

// With cache
const cache = new InMemoryCache<string>();
const storageWithCache = new AmazonStorageClient(
  s3Client,
  'my-bucket',
  AmazonCannedAccessControlList.PublicRead,
  cache
);

Access Control Lists (ACLs)

import { AmazonCannedAccessControlList } from '@geekmidas/storage/aws';

const storage = AmazonStorageClient.create({
  bucket: 'my-bucket',
  acl: AmazonCannedAccessControlList.PublicRead, // Files will be publicly readable
});

Available ACLs:

• Private - Owner gets full control, no one else has access
• PublicRead - Owner gets full control, everyone else gets read access
• PublicReadWrite - Owner gets full control, everyone else gets read/write access
• AuthenticatedRead - Owner gets full control, authenticated users get read access
• BucketOwnerRead - Object owner gets full control, bucket owner gets read access
• BucketOwnerFullControl - Object and bucket owner get full control
• LogDeliveryWrite - Log delivery service gets write access
• AwsExecRead - Amazon EC2 gets read access for AMI bundles

Caching

Caching presigned URLs can significantly reduce the number of API calls to AWS S3 and improve performance.

import { AmazonStorageClient } from '@geekmidas/storage/aws';
import { InMemoryCache } from '@geekmidas/cache/memory';
import { UpstashCache } from '@geekmidas/cache/upstash';

// In-memory cache for development/testing
const memoryCache = new InMemoryCache<string>();
const storage = AmazonStorageClient.create({
  bucket: 'my-bucket',
  cache: memoryCache,
});

// Redis cache for production
const redisCache = new UpstashCache<string>({
  url: process.env.UPSTASH_REDIS_URL,
  token: process.env.UPSTASH_REDIS_TOKEN,
});
const productionStorage = AmazonStorageClient.create({
  bucket: 'my-bucket',
  cache: redisCache,
});

// Cache behavior example
const url1 = await storage.getDownloadURL({ path: 'file.pdf' }); // Generates new URL
const url2 = await storage.getDownloadURL({ path: 'file.pdf' }); // Returns cached URL
console.log(url1 === url2); // true

Cache keys and TTL:

• Download URLs are cached with key: download-url:{path}
• Cache TTL is automatically calculated as expiresIn - 60 seconds
• URLs expiring in less than 60 seconds are not cached
• Upload URLs are not cached (they are typically single-use)

Error Handling

try {
  await storage.upload('documents/file.txt', 'content', 'text/plain');
} catch (error) {
  if (error.name === 'NoSuchBucket') {
    console.error('Bucket does not exist');
  } else if (error.name === 'AccessDenied') {
    console.error('Access denied');
  } else {
    console.error('Upload failed:', error);
  }
}

Development

Running Tests

# Run all tests
npm test

# Run unit tests only
npm run test:unit

# Run integration tests only (requires MinIO)
npm run test:integration

# Run tests once
npm run test:once

Local Development with MinIO

1. Start MinIO using Docker Compose:

   docker-compose up -d minio

2. MinIO will be available at:

   • API: http://localhost:9000
   • Console: http://localhost:9001
   • Credentials: minioadmin/minioadmin

3. Run integration tests:

   npm run test:integration

Project Structure

src/
├── index.ts              # Main exports
├── aws.ts                # AWS-specific exports
├── StorageClient.ts      # Core interfaces and types
├── AmazonStorageClient.ts # AWS S3 implementation
└── __tests__/
    ├── StorageClient.spec.ts                    # Interface tests
    ├── AmazonStorageClient.spec.ts              # Unit tests
    └── AmazonStorageClient.integration.spec.ts  # Integration tests

Contributing

1. Follow the existing code style (2 spaces, single quotes, semicolons)
2. Add comprehensive tests for new features
3. Update documentation for API changes
4. Use the "Integration over Unit" testing philosophy - prefer real dependencies over mocks

License

MIT License - see the LICENSE file for details.