cloud-file-kit

A lightweight, provider-agnostic TypeScript SDK for uploading, deleting, and serving files across multiple cloud object storage backends — AWS S3, Cloudflare R2, Supabase Storage, Google Cloud Storage, and Azure Blob Storage — using a single, unified API.

Switch storage providers by changing one config field. Your application code stays the same.

Table of Contents

• Why cloud-file-kit?
• Features
• Supported Providers
• Installation
• Quick Start
• Configuration Reference
• Provider Setup Guides
• API Reference
• Switching Providers
• Enhanced Uploads & Performance
• Error Handling & Retries
• Express Integration Example
• Architecture
• Adding a Custom Provider
• Environment Variables
• Development
• Roadmap
• License

Why cloud-file-kit?

Object storage APIs are similar in concept but different in practice. Each provider has its own SDK, credential format, URL conventions, and edge cases. When you outgrow one provider — or want to use R2 in staging and S3 in production — you end up rewriting upload logic everywhere.

cloud-file-kit solves this with a thin abstraction layer:

| Without cloud-file-kit | With cloud-file-kit | |---|---| | Import @aws-sdk/client-s3 in one service, @google-cloud/storage in another | One FileCloud class everywhere | | Different method signatures per provider | upload(key, buffer) works for all providers | | Hard-coded bucket URLs in your app | getUrl(key) returns a provider-appropriate URL | | Vendor lock-in in business logic | Swap provider: 's3' → provider: 'r2' in config |

This library is ideal for:

• SaaS apps that let customers choose their storage backend
• Multi-environment setups (dev on Supabase, prod on S3)
• Migration projects moving from one cloud to another
• Prototypes where you want to start cheap (R2/Supabase) and scale later (S3/GCS)

Note: This SDK targets object storage (S3-compatible buckets, blob stores). Media transformation services like Cloudinary or ImageKit are not included today, but you can add them by implementing the CloudProvider interface. See Adding a Custom Provider.

Features

• Unified API — upload, delete, getUrl, and uploadMultiple work identically across providers
• TypeScript-first — Full type definitions for config, options, and results
• Provider factory — Instantiate the correct backend from a single config object
• Enhanced mode — Optional concurrency-limited upload queue and performance metrics
• Minimal dependencies — Uses official SDKs under the hood (@aws-sdk/client-s3, @google-cloud/storage, etc.)
• Extensible — Add new providers by implementing one interface

Supported Providers

| Provider | Config value | Status | Best for | |---|---|---|---| | Amazon S3 | s3 | ✅ Implemented | Production workloads, AWS ecosystem | | Cloudflare R2 | r2 | 🚧 Stub | S3-compatible storage with zero egress fees | | Supabase Storage | supabase | ✅ Implemented | Postgres + storage in one platform | | Google Cloud Storage | gcs | ✅ Implemented | GCP-native apps, global CDN | | Azure Blob Storage | azure | 🚧 Stub | Microsoft Azure deployments |

Implemented providers use real SDK calls. Stub providers return placeholder URLs and are ready for you to wire up — R2 uses the same AWS SDK as S3 with a custom endpoint; Azure uses @azure/storage-blob (already listed as a dependency).

Installation

npm install cloud-file-kit

Or with yarn:

yarn add cloud-file-kit

Peer requirements

• Node.js 18+ recommended
• Provider credentials for whichever backend you use (see Provider Setup Guides)

The SDK bundles these provider SDKs as dependencies:

| Provider | Underlying package | |---|---| | S3 / R2 | @aws-sdk/client-s3, @aws-sdk/s3-request-presigner | | Supabase | @supabase/supabase-js | | GCS | @google-cloud/storage | | Azure | @azure/storage-blob |

Quick Start

import { FileCloud } from 'cloud-file-kit';

// 1. Create a client — pick your provider
const cloud = new FileCloud({
  provider: 's3',
  accessKey: process.env.AWS_ACCESS_KEY_ID!,
  secretKey: process.env.AWS_SECRET_ACCESS_KEY!,
  bucket: 'my-app-uploads',
  region: 'us-east-1',
});

// 2. Upload a file from a Buffer
const buffer = Buffer.from('Hello, cloud!');
const result = await cloud.upload('documents/hello.txt', buffer, {
  contentType: 'text/plain',
  cacheControl: 'public, max-age=3600',
});

console.log(result.url);   // https://my-app-uploads.s3.us-east-1.amazonaws.com/documents/hello.txt
console.log(result.key);   // documents/hello.txt
console.log(result.size);  // 13

// 3. Get a signed URL (temporary access)
const signedUrl = await cloud.getUrl('documents/hello.txt');

// 4. Delete when done
await cloud.delete('documents/hello.txt');

Upload from a local file with Node.js:

import { readFileSync } from 'fs';
import { FileCloud } from 'cloud-file-kit';

const cloud = new FileCloud({ /* ... */ });
const buffer = readFileSync('./avatar.png');

await cloud.upload('users/42/avatar.png', buffer, {
  contentType: 'image/png',
  metadata: { userId: '42' },
});

Configuration Reference

All providers share a single FileCloudConfig object:

interface FileCloudConfig {
  provider: 's3' | 'r2' | 'supabase' | 'gcs' | 'azure';
  bucket: string;

  // AWS S3 / Cloudflare R2
  accessKey?: string;
  secretKey?: string;
  region?: string;      // Default: 'us-east-1' for S3
  endpoint?: string;    // Required for R2 (e.g. https://<account>.r2.cloudflarestorage.com)

  // Google Cloud Storage
  projectId?: string;
  // For GCS service accounts, accessKey = client_email, secretKey = private_key

  // Azure Blob Storage
  accountName?: string;

  // Supabase Storage
  url?: string;         // Supabase project URL
  token?: string;       // Supabase service role or anon key
}

Upload options

Pass these as the third argument to upload():

interface UploadOptions {
  contentType?: string;              // MIME type (e.g. 'image/jpeg')
  metadata?: Record<string, string>; // Custom key-value metadata
  public?: boolean;                    // Hint for public access (provider-dependent)
  cacheControl?: string;               // Cache-Control header value
  onProgress?: (progress: UploadProgress) => void;
}

interface UploadProgress {
  key: string;
  loaded: number;
  total: number;
  percentage: number;
}

Upload result

Every successful upload returns:

interface UploadResult {
  url: string;       // Public or constructed URL for the object
  key: string;       // Object key/path you uploaded to
  bucket: string;    // Bucket/container name
  etag?: string;     // Entity tag (S3/GCS)
  versionId?: string;// Version ID (versioned S3 buckets)
  size: number;      // Uploaded size in bytes
}

Provider Setup Guides

Amazon S3

1. Create an S3 bucket in the AWS Console.
2. Create an IAM user with s3:PutObject, s3:GetObject, and s3:DeleteObject on your bucket.
3. Configure the SDK:

const cloud = new FileCloud({
  provider: 's3',
  accessKey: process.env.AWS_ACCESS_KEY_ID!,
  secretKey: process.env.AWS_SECRET_ACCESS_KEY!,
  bucket: 'my-bucket',
  region: 'us-east-1',
});

Signed URLs: getUrl() returns a pre-signed URL valid for 1 hour.

Cloudflare R2

R2 is S3-compatible. Use the s3 provider logic with a custom endpoint, or the dedicated r2 provider once fully implemented:

const cloud = new FileCloud({
  provider: 's3',  // R2 works via S3-compatible API today
  accessKey: process.env.R2_ACCESS_KEY_ID!,
  secretKey: process.env.R2_SECRET_ACCESS_KEY!,
  bucket: 'my-r2-bucket',
  region: 'auto',
  endpoint: `https://${process.env.R2_ACCOUNT_ID}.r2.cloudflarestorage.com`,
});

Create R2 API tokens in the Cloudflare dashboard under R2 → Manage R2 API Tokens.

Supabase Storage

1. Create a Supabase project.
2. Create a public or private storage bucket under Storage.
3. Use your project URL and service role key:

const cloud = new FileCloud({
  provider: 'supabase',
  url: process.env.SUPABASE_URL!,           // https://xxxx.supabase.co
  token: process.env.SUPABASE_SERVICE_KEY!, // service_role key for server-side uploads
  bucket: 'avatars',
});

await cloud.upload('user-123.png', buffer, {
  contentType: 'image/png',
  cacheControl: '3600',
});

Supabase uploads use upsert: true — re-uploading the same key overwrites the existing file.

Google Cloud Storage

1. Create a GCS bucket in the GCP Console.
2. Create a service account with Storage Object Admin role.
3. Download the JSON key file:

const cloud = new FileCloud({
  provider: 'gcs',
  projectId: 'my-gcp-project',
  bucket: 'my-gcs-bucket',
  accessKey: process.env.GCS_CLIENT_EMAIL!,   // client_email from JSON key
  secretKey: process.env.GCS_PRIVATE_KEY!,    // private_key from JSON key (with \n newlines)
});

Signed URLs: getUrl() returns a signed read URL expiring in 1 hour.

Azure Blob Storage

The Azure provider is currently a stub. Wire it up using @azure/storage-blob or contribute a full implementation.

Expected configuration shape:

const cloud = new FileCloud({
  provider: 'azure',
  accountName: process.env.AZURE_STORAGE_ACCOUNT!,
  accessKey: process.env.AZURE_STORAGE_KEY!,  // account key or SAS token
  bucket: 'my-container',                       // container name
});

API Reference

FileCloud

The main entry point. Delegates all operations to the configured provider.

Constructor

new FileCloud(config: FileCloudConfig)

Methods

| Method | Signature | Description | |---|---|---| | upload | (key, buffer, options?) → Promise<UploadResult> | Upload a file buffer to the given object key | | delete | (key) → Promise<void> | Delete an object by key | | getUrl | (key) → Promise<string> | Get a URL for the object (public or signed, provider-dependent) | | uploadMultiple | (files[]) → Promise<UploadResult[]> | Upload several files in parallel |

uploadMultiple example:

const results = await cloud.uploadMultiple([
  { key: 'a.png', buffer: bufferA, options: { contentType: 'image/png' } },
  { key: 'b.png', buffer: bufferB, options: { contentType: 'image/png' } },
  { key: 'c.png', buffer: bufferC, options: { contentType: 'image/png' } },
]);

EnhancedFileCloud

Extends FileCloud with a concurrency-limited upload queue and built-in performance tracking. Use this when uploading many files or when you need upload metrics.

import { EnhancedFileCloud } from 'cloud-file-kit';

const cloud = new EnhancedFileCloud(
  { provider: 's3', /* ... */ },
  5  // max concurrent uploads (default: 5)
);

const result = await cloud.upload('large-file.zip', buffer, {
  contentType: 'application/zip',
  priority: 10,           // Higher priority uploads run first
  trackPerformance: true, // Default: true
});

// Inspect metrics
const stats = cloud.getPerformanceStats();
console.log(stats.averageSpeedBps, stats.averageDurationMs);

// Adjust concurrency at runtime
cloud.setMaxConcurrentUploads(10);

// Reset metrics
cloud.clearPerformanceData();

Additional options

interface EnhancedUploadOptions extends UploadOptions {
  trackPerformance?: boolean;  // Default: true
  priority?: number;             // Higher = processed first in the queue
}

uploadMultiple with sequential mode

// Parallel (default) — all files queued with concurrency limit
await cloud.uploadMultiple(files, true);

// Sequential — one file at a time, respecting priority
await cloud.uploadMultiple(files, false);

Performance reporting

import { PerformanceReporter } from 'cloud-file-kit';

const stats = cloud.getPerformanceStats();
const report = PerformanceReporter.generateReport(stats);
console.log(report);

ProviderFactory

Low-level factory if you need direct access to a provider instance:

import { ProviderFactory } from 'cloud-file-kit';

const provider = ProviderFactory.createProvider({
  provider: 'gcs',
  bucket: 'my-bucket',
  projectId: 'my-project',
});

await provider.upload('path/to/file', buffer);

Switching Providers

The core value of this SDK is zero business-logic changes when you swap backends.

Strategy 1: Environment-based config

import { FileCloud, FileCloudConfig } from 'cloud-file-kit';

function createCloudStorage(): FileCloud {
  const provider = process.env.STORAGE_PROVIDER as FileCloudConfig['provider'];

  const configs: Record<string, FileCloudConfig> = {
    s3: {
      provider: 's3',
      bucket: process.env.S3_BUCKET!,
      accessKey: process.env.AWS_ACCESS_KEY_ID!,
      secretKey: process.env.AWS_SECRET_ACCESS_KEY!,
      region: process.env.AWS_REGION ?? 'us-east-1',
    },
    r2: {
      provider: 's3',
      bucket: process.env.R2_BUCKET!,
      accessKey: process.env.R2_ACCESS_KEY_ID!,
      secretKey: process.env.R2_SECRET_ACCESS_KEY!,
      endpoint: process.env.R2_ENDPOINT!,
      region: 'auto',
    },
    supabase: {
      provider: 'supabase',
      bucket: process.env.SUPABASE_BUCKET!,
      url: process.env.SUPABASE_URL!,
      token: process.env.SUPABASE_SERVICE_KEY!,
    },
    gcs: {
      provider: 'gcs',
      bucket: process.env.GCS_BUCKET!,
      projectId: process.env.GCS_PROJECT_ID!,
      accessKey: process.env.GCS_CLIENT_EMAIL!,
      secretKey: process.env.GCS_PRIVATE_KEY!,
    },
  };

  const config = configs[provider];
  if (!config) throw new Error(`Unknown STORAGE_PROVIDER: ${provider}`);

  return new FileCloud(config);
}

// Usage — identical regardless of provider
const cloud = createCloudStorage();
await cloud.upload('uploads/photo.jpg', imageBuffer, { contentType: 'image/jpeg' });

Set STORAGE_PROVIDER=s3 in production and STORAGE_PROVIDER=supabase locally. No code changes.

Strategy 2: Dual-write during migration

const oldStorage = new FileCloud({ provider: 's3', /* ... */ });
const newStorage = new FileCloud({ provider: 'gcs', /* ... */ });

async function uploadWithMigration(key: string, buffer: Buffer, options?: UploadOptions) {
  const [s3Result] = await Promise.all([
    newStorage.upload(key, buffer, options),  // Write to new provider
    oldStorage.upload(key, buffer, options),    // Keep old provider in sync during migration
  ]);
  return s3Result;
}

Strategy 3: Read from new, fall back to old

async function getFileUrl(key: string): Promise<string> {
  try {
    return await newStorage.getUrl(key);
  } catch {
    return oldStorage.getUrl(key);
  }
}

Error Handling & Retries

The SDK includes utilities for resilient uploads:

import { FileCloud, FileCloudError, withRetry } from 'cloud-file-kit';

const cloud = new FileCloud({ /* ... */ });

try {
  const result = await withRetry(
    () => cloud.upload('critical-backup.zip', buffer),
    3  // max retries with exponential backoff (1s, 2s, 4s)
  );
} catch (error) {
  if (error instanceof FileCloudError) {
    console.error(`Provider ${error.provider} failed: ${error.message}`);
  }
  throw error;
}

FileCloudError wraps provider failures with the provider name prefixed to the message for easier debugging in multi-provider setups.

Express Integration Example

A common pattern: accept multipart uploads in an API route and store them in cloud storage.

import express from 'express';
import { FileCloud } from 'cloud-file-kit';

const app = express();
const cloud = new FileCloud({
  provider: 's3',
  accessKey: process.env.AWS_ACCESS_KEY_ID!,
  secretKey: process.env.AWS_SECRET_ACCESS_KEY!,
  bucket: 'user-uploads',
  region: 'us-east-1',
});

// Accept raw binary body (Content-Type: application/octet-stream)
app.put('/upload/:key', express.raw({ type: '*/*', limit: '10mb' }), async (req, res) => {
  try {
    const key = `uploads/${req.params.key}`;
    const result = await cloud.upload(key, req.body as Buffer, {
      contentType: req.headers['content-type'] as string,
    });
    res.json({ url: result.url, key: result.key, size: result.size });
  } catch (err) {
    res.status(500).json({ error: (err as Error).message });
  }
});

app.delete('/upload/:key', async (req, res) => {
  await cloud.delete(`uploads/${req.params.key}`);
  res.status(204).send();
});

app.listen(3000);

For multipart/form-data, pair this SDK with multer and pass req.file.buffer to cloud.upload().

Architecture

┌─────────────────────────────────────────────────────────┐
│                     Your Application                    │
└──────────────────────────┬──────────────────────────────┘
                           │
              ┌────────────┴────────────┐
              │       FileCloud         │  ← Unified API
              │  EnhancedFileCloud      │  ← + queue & metrics
              └────────────┬────────────┘
                           │
              ┌────────────┴────────────┐
              │    ProviderFactory     │  ← Reads config.provider
              └────────────┬────────────┘
                           │
     ┌─────────┬───────────┼───────────┬─────────┐
     │         │           │           │         │
  S3Provider R2Provider Supabase   GCSProvider Azure
     │         │        Provider       │      Provider
     │         │           │           │         │
     └─────────┴───────────┴───────────┴─────────┘
                           │
              ┌────────────┴────────────┐
              │   Official Provider SDKs │
              │  AWS / Supabase / GCP /  │
              │         Azure            │
              └──────────────────────────┘

Project structure

cloud-file-kit/
├── src/
│   ├── core/
│   │   ├── file-cloud.ts          # Main SDK class
│   │   ├── enhanced-file-cloud.ts # Queue + performance layer
│   │   └── provider-factory.ts    # Provider instantiation
│   ├── providers/
│   │   ├── s3-provider.ts
│   │   ├── r2-provider.ts
│   │   ├── supabase-provider.ts
│   │   ├── gcs-provider.ts
│   │   └── azure-provider.ts
│   ├── types/
│   │   ├── config.types.ts        # FileCloudConfig, UploadOptions, UploadResult
│   │   └── provider.types.ts      # CloudProvider interface
│   ├── utils/
│   │   ├── error-handler.ts       # FileCloudError, withRetry
│   │   └── performance/
│   │       ├── upload-queue.ts    # Concurrency-limited queue
│   │       └── monitor.ts         # Metrics & reporting
│   └── index.ts                   # Public exports
├── tests/
│   └── file-cloud.test.ts
├── package.json
└── tsconfig.json

Adding a Custom Provider

Implement the CloudProvider interface and register it in ProviderFactory:

// src/providers/cloudinary-provider.ts
import { CloudProvider, FileCloudConfig, UploadOptions, UploadResult } from '../types';

export class CloudinaryProvider implements CloudProvider {
  constructor(private config: FileCloudConfig) {}

  async upload(key: string, buffer: Buffer, options?: UploadOptions): Promise<UploadResult> {
    // Use cloudinary SDK or REST API
    throw new Error('Not implemented');
  }

  async delete(key: string): Promise<void> {
    // ...
  }

  async getUrl(key: string): Promise<string> {
    // Return transformation URL
    throw new Error('Not implemented');
  }
}

Register in provider-factory.ts:

case 'cloudinary':
  return new CloudinaryProvider(config);

Extend the config type:

// types/config.types.ts
provider: 's3' | 'r2' | 'supabase' | 'gcs' | 'azure' | 'cloudinary';
cloudName?: string;  // Cloudinary-specific
apiKey?: string;
apiSecret?: string;

Your application code does not change — only the config and factory.

Environment Variables

Example .env for a multi-provider setup:

# Active provider: s3 | r2 | supabase | gcs | azure
STORAGE_PROVIDER=s3

# AWS S3
AWS_ACCESS_KEY_ID=AKIA...
AWS_SECRET_ACCESS_KEY=...
AWS_REGION=us-east-1
S3_BUCKET=my-app-uploads

# Cloudflare R2 (S3-compatible)
R2_ACCOUNT_ID=abc123
R2_ACCESS_KEY_ID=...
R2_SECRET_ACCESS_KEY=...
R2_BUCKET=my-r2-bucket
R2_ENDPOINT=https://abc123.r2.cloudflarestorage.com

# Supabase
SUPABASE_URL=https://xxxx.supabase.co
SUPABASE_SERVICE_KEY=eyJ...
SUPABASE_BUCKET=uploads

# Google Cloud Storage
GCS_PROJECT_ID=my-project
GCS_BUCKET=my-gcs-bucket
GCS_CLIENT_EMAIL=service-account@my-project.iam.gserviceaccount.com
GCS_PRIVATE_KEY="-----BEGIN PRIVATE KEY-----\n..."

# Azure Blob Storage
AZURE_STORAGE_ACCOUNT=mystorageaccount
AZURE_STORAGE_KEY=...
AZURE_CONTAINER=uploads

Never commit .env files. Add .env to .gitignore.

Development

# Install dependencies
npm install

# Build TypeScript → dist/
npm run build

# Run tests
npm test

# Lint
npm run lint

# Watch mode (dev server)
npm run dev

Running tests

Tests use mocked S3 credentials. They verify the SDK wiring and error paths:

npm test

See tests/file-cloud.test.ts for examples.

Building for publish

npm run build   # Outputs to dist/
npm run prepublishOnly  # Runs automatically before npm publish

Import from the built package:

import { FileCloud } from 'cloud-file-kit';

Or during local development:

import { FileCloud } from './src/core/file-cloud';

Roadmap

• [ ] Complete Cloudflare R2 provider (S3-compatible endpoint wrapper)
• [ ] Complete Azure Blob Storage provider using @azure/storage-blob
• [ ] Cloudinary provider for image/video transformation use cases
• [ ] Stream uploads via uploadStream() on all providers
• [ ] Upload progress callbacks wired through to provider SDKs
• [ ] Presigned upload URLs (not just download)
• [ ] Multi-part upload support for large files (>5 GB on S3)
• [ ] Provider health checks and connection validation

Contributions welcome — especially for stub providers and Cloudinary support.

License

MIT