@plyaz/storage

Multi-provider file storage package with support for Cloudflare R2, Supabase Storage, and custom adapters. Includes virus scanning, metadata extraction, media processing, compliance management, document rendering (PDF/Excel/DOCX), and webhook tracking.

Installation

pnpm add @plyaz/storage

Requirements:

• Node.js >= 22.4.0
• pnpm >= 8.0.0

Optional Dependencies

The package uses optional dependencies for specific features. Install only what you need:

# Storage Adapters
pnpm add @aws-sdk/client-s3 @aws-sdk/s3-request-presigner  # For Cloudflare R2 / AWS S3
pnpm add @supabase/supabase-js                              # For Supabase Storage

# Image Processing
pnpm add sharp                                              # Image optimization & variants

# Video Processing
pnpm add @ffmpeg-installer/ffmpeg fluent-ffmpeg            # Video transcoding

# PDF Rendering
pnpm add pdfkit                                             # Lightweight PDF (~5MB)
pnpm add puppeteer                                          # Full HTML/CSS support (~200MB)

# Excel/Word Documents
pnpm add exceljs                                            # Excel spreadsheets
pnpm add docxtemplater pizzip                               # Word documents

See Confluence Documentation for complete installation scenarios and dependency matrix

Quick Start

Basic File Upload

import { StorageService, CloudflareR2Adapter } from '@plyaz/storage';
import { FileCategory, EntityType } from '@plyaz/types';

// Initialize with Cloudflare R2
const storage = new StorageService({
  adapters: [
    new CloudflareR2Adapter({
      name: 'cloudflare-r2',
      accountId: process.env.CLOUDFLARE_ACCOUNT_ID!,
      accessKeyId: process.env.R2_ACCESS_KEY_ID!,
      secretAccessKey: process.env.R2_SECRET_ACCESS_KEY!,
      bucket: 'my-uploads'
    })
  ],
  handlers: {
    onFileUploaded: async (payload) => {
      console.log('File uploaded:', payload.metadata?.fileId);
      // Save metadata to database
    }
  }
});

// Upload file
const result = await storage.uploadFile({
  file: fileBuffer,
  filename: 'document.pdf',
  mimeType: 'application/pdf',
  category: FileCategory.DOCUMENT,
  entityType: EntityType.USER,
  entityId: 'user-123'
});

console.log('File ID:', result.metadata.fileId);
console.log('Public URL:', result.metadata.publicUrl);

Multi-Provider with Failover

import {
  StorageService,
  CloudflareR2Adapter,
  SupabaseStorageAdapter,
  createProductionLogger
} from '@plyaz/storage';

const logger = createProductionLogger({ service: 'storage' });

const storage = new StorageService({
  adapters: [
    new CloudflareR2Adapter({
      name: 'r2-primary',
      priority: 100,  // Higher priority = preferred
      // ... config
    }),
    new SupabaseStorageAdapter({
      name: 'supabase-backup',
      priority: 50,   // Fallback adapter
      // ... config
    })
  ],
  logger
});

Image Processing with Variants

import { StorageService, SharpImagePlugin } from '@plyaz/storage';

const storage = new StorageService({
  adapters: [/* ... */],
  plugins: [
    new SharpImagePlugin({
      enableVariants: true,
      variants: [
        { name: 'thumbnail', width: 150, height: 150 },
        { name: 'medium', width: 800, height: 600 },
        { name: 'large', width: 1920, height: 1080 }
      ],
      formats: ['webp', 'jpeg'],
      quality: 85
    })
  ]
});

// Automatically generates: thumbnail.webp, medium.webp, large.webp variants
const result = await storage.uploadFile({
  file: imageBuffer,
  filename: 'photo.jpg',
  mimeType: 'image/jpeg',
  category: FileCategory.PROFILE_IMAGE,
  entityType: EntityType.USER,
  entityId: 'user-123'
});

console.log('Variants:', result.metadata.variants);

PDF Generation from Templates

import { StorageService, CloudflareR2Adapter } from '@plyaz/storage';
import { StorageRendererType, OutputFormat, FileCategory, EntityType } from '@plyaz/types';

// StorageService handles templates internally!
const storage = new StorageService({
  adapters: [r2Adapter],
  template: {
    templateBasePath: './templates',  // Path to your templates folder
    defaultLocale: 'en',
    renderers: [StorageRendererType.PDFKIT]  // Auto-registers PDFKit renderer
  }
});

// Generate and upload PDF from template
const result = await storage.generateFileToPath({
  templateId: 'invoice',
  templateData: {
    invoiceNumber: 'INV-001',
    customerName: 'John Doe',
    items: [{ name: 'Product', price: 99.99 }],
    total: 99.99
  },
  filename: 'invoice-001.pdf',
  format: OutputFormat.PDF,
  category: FileCategory.FINANCIAL_DOCUMENT,
  entityType: EntityType.ORGANIZATION,
  entityId: 'org-123'
});

console.log('Generated PDF:', result.metadata.publicUrl);

Features

Multi-Provider Storage

• ✅ Cloudflare R2 - S3-compatible, zero egress fees
• ✅ Supabase Storage - PostgreSQL-based object storage
• ✅ AWS S3 - Full S3 support via R2 adapter
• ✅ Mock Adapter - Testing and development

Automatic Failover

• Priority-based adapter selection
• Health monitoring with circuit breaker
• Automatic retry with exponential backoff
• Adapter degradation handling

Image Processing (Sharp)

• Automatic variant generation (thumbnail, medium, large)
• Format conversion (WebP, AVIF, JPEG, PNG)
• Quality optimization
• EXIF metadata extraction
• Responsive images for different devices

Video Processing (FFmpeg)

• Video transcoding
• Thumbnail generation
• Format conversion (MP4, WebM, HLS)
• Resolution variants
• Bitrate optimization

Document Rendering

• PDFKit - Lightweight programmatic PDFs (~5MB)
• Puppeteer - Complex HTML/CSS to PDF (~200MB)
• Playwright - Multi-browser rendering (~300MB)
• ExcelJS - Excel spreadsheets and reports
• DocxTemplater - Word document generation

Compliance & Security

• Virus Scanning - ClamAV, VirusTotal integration
• Metadata Extraction - EXIF, ID3, PDF metadata
• File Validation - Size, type, content validation
• Compliance Manager - Retention policies, immutability
• Signed URLs - Temporary download/upload links
• Idempotency - Prevent duplicate uploads

Template System

• Handlebars templates with layouts
• Markdown support
• Multi-language (i18n)
• YAML frontmatter configuration
• Dynamic data binding

CDN Integration

• Cloudflare CDN - Cache purging, zone management
• CloudFront - Invalidation, distribution management
• Fastly - Edge caching, instant purge

Event System

• Upload/download/delete lifecycle events
• Adapter failover notifications
• Plugin execution tracking
• Webhook delivery events

Advanced Features

• Chunked uploads for large files (>100MB)
• Upload progress tracking
• Multi-tenant bucket routing
• Automatic bucket creation
• Queue-based processing
• Database metadata integration

Storage Adapter Matrix

| Feature | Cloudflare R2 | Supabase | Mock | |---------|--------------|----------|------| | File Upload | ✅ Full | ✅ Full | ✅ Full | | Signed URLs | ✅ Yes | ✅ Yes | ✅ Yes | | Chunked Upload | ✅ Yes | ✅ Yes | ✅ Yes | | CDN Integration | ✅ Cloudflare | ❌ No | ❌ No | | Cost | Zero egress | PostgreSQL-based | Free | | Best For | Production media | Postgres-based apps | Testing |

Plugin Support Matrix

| Plugin | Dependencies | Size | Use Case | |--------|--------------|------|----------| | SharpImagePlugin | sharp, exifr | ~25MB | Image variants, optimization | | FFmpegVideoPlugin | ffmpeg, fluent-ffmpeg | ~100MB | Video transcoding | | MetadataExtractionPlugin | exifr, music-metadata, pdf-parse | ~5MB | File metadata extraction | | VirusScanPlugin | clamav or virustotal-api | Varies | Security scanning | | CloudflareCDNPlugin | None (HTTP API) | 0MB | Cache management | | CloudFrontCDNPlugin | None (HTTP API) | 0MB | AWS CloudFront cache invalidation | | FastlyCDNPlugin | None (HTTP API) | 0MB | Fastly CDN cache purging |

Idempotency Support

Prevent duplicate uploads and webhook processing:

| Adapter | Dependencies | Persistence | Use Case | |---------|--------------|-------------|----------| | InMemoryIdempotencyAdapter | None | In-memory (process lifetime) | Testing, development, serverless | | RedisIdempotencyAdapter | ioredis | Redis (distributed) | Production, multi-instance deployments |

Example Usage:

import {
  StorageService,
  WebhookManager,
  RedisIdempotencyAdapter
} from '@plyaz/storage';

const idempotencyStore = new RedisIdempotencyAdapter({
  redis: {
    host: 'localhost',
    port: 6379,
  },
  ttl: 86400, // 24 hours
});

const webhookManager = new WebhookManager({
  idempotencyAdapter: idempotencyStore,
});

// Duplicate webhooks are automatically detected and ignored
await webhookManager.processWebhook(payload, headers);

Development Commands

# Development
pnpm dev              # Watch mode development
pnpm build            # Build for production
pnpm clean            # Clean dist directory

# Testing
pnpm test             # Run all tests
pnpm test:watch       # Run tests in watch mode
pnpm test:coverage    # Run tests with coverage report
pnpm test:ui          # Open Vitest UI

# Code Quality
pnpm lint             # Run ESLint
pnpm lint:fix         # Auto-fix linting issues
pnpm format           # Format code with Prettier
pnpm format:check     # Check formatting
pnpm type:check       # TypeScript type checking

# Examples
pnpm example:basic                    # Basic upload/download
pnpm example:adapter:cloudflare-r2    # Cloudflare R2 setup
pnpm example:adapter:supabase         # Supabase setup
pnpm example:adapter:both             # Multi-provider routing

Package Dependencies

Per Plyaz monorepo architecture:

Internal Dependencies

• @plyaz/types - Type definitions and interfaces
• @plyaz/logger - Structured logging with PII redaction
• @plyaz/api - Global API configuration
• @plyaz/errors - Error handling with correlation IDs
• @plyaz/config - Configuration management

Core Dependencies (Always Installed)

• file-type - File type detection (~500KB)
• handlebars - Template engine (~500KB)
• marked - Markdown parsing (~100KB)
• gray-matter - YAML frontmatter (~100KB)

Total Core Size: ~2MB

Optional Dependencies

See the Confluence documentation for the complete list of feature-specific dependencies and installation scenarios.

Documentation

Comprehensive documentation organized into focused guides.

📚 Documentation Structure

Documentation Index - Start here for complete navigation

Quick Access:

• Quick Start Guide - Installation and first steps
• Architecture Overview - System architecture and data flow
• API Reference - Complete API documentation
• Storage Adapters - R2, Supabase, Mock adapters
• Plugin System - Virus scan, image/video processing, CDN
• Templates & Documents - PDF, Excel, Word generation
• Deployment Guide - Production deployment strategies
• Examples - Real-world usage examples

For Confluence: Complete documentation is available on Confluence. Each guide is self-contained and published as a separate page.

Architecture

StorageService
├── AdapterRegistry (multi-provider routing & failover)
├── PluginRegistry (lifecycle hooks & processing)
├── EventManager (event emission & subscriptions)
├── BucketRouter (content-based routing)
├── ComplianceManager (retention & immutability)
├── QueueProcessor (async operations)
└── WebhookManager (delivery tracking)
    ├── Storage Adapters (Cloudflare R2, Supabase, Mock)
    ├── Processing Plugins (Sharp, FFmpeg, Virus Scan)
    ├── CDN Plugins (Cloudflare, CloudFront, Fastly)
    ├── Renderers (PDFKit, Puppeteer, ExcelJS, DOCX)
    └── Idempotency (InMemory, Redis)

Error Handling

Uses Result type pattern for safe error handling:

const result = await storage.uploadFile({ /* ... */ });

if (result.success) {
  console.log('File uploaded:', result.metadata.fileId);
  console.log('URL:', result.metadata.publicUrl);
  console.log('Adapter used:', result.metadata.adapter);
} else {
  console.error('Upload failed:', result.error);
  // result.error contains: message, code, context
}

Common error codes:

import { STORAGE_ERROR_CODES } from '@plyaz/storage';

// FILE_TOO_LARGE, INVALID_FILE_TYPE, ADAPTER_OPERATION_FAILED,
// ADAPTER_TIMEOUT, ADAPTER_CONFIGURATION_INVALID, PLUGIN_EXECUTION_FAILED

Event System

const storage = new StorageService({
  adapters: [/* ... */],

  // Lifecycle events
  handlers: {
    onFileUploaded: async (event) => {
      console.log('Uploaded:', event.metadata?.fileId);
      // Save to database
      await db.files.create({
        id: event.metadata?.fileId,
        path: event.metadata?.path,
        adapter: event.metadata?.adapter
      });
    },

    onFileDeleted: async (event) => {
      console.log('Deleted:', event.metadata?.fileId);
      // Update database
      await db.files.delete(event.metadata?.fileId);
    },

    onAdapterFailed: async (event) => {
      console.warn('Adapter failed, using fallback:', event.data);
      // Log to monitoring service
      await monitoring.alert('storage-adapter-failure', event);
    },

    onFileUploadFailed: async (event) => {
      console.error('All adapters failed:', event.error);
      // Send alert
      await alerts.critical('storage-complete-failure', event);
    }
  }
});

Multi-Tenant Routing

Automatic routing based on content type and organization tier:

import { StorageService, BucketRouter } from '@plyaz/storage';
import { BucketPurpose } from '@plyaz/types';

const storage = new StorageService({
  adapters: [r2Adapter, supabaseAdapter],
  bucketRouter: new BucketRouter({
    availableAdapters: ['cloudflare-r2', 'supabase'],
    enableDefaultRules: true,
    adapterPreferences: {
      [BucketPurpose.COMPLIANCE]: 'cloudflare-r2',      // Financial docs → R2
      [BucketPurpose.MEDIA_IMAGES]: 'supabase',         // Images → Supabase
      [BucketPurpose.MEDIA_VIDEOS]: 'cloudflare-r2',    // Videos → R2
    }
  })
});

// Automatically routes to R2 (compliance documents)
await storage.uploadFile({
  file: invoiceBuffer,
  category: FileCategory.FINANCIAL_DOCUMENT,
  documentType: DocumentType.INVOICE,
  organizationTier: OrganizationTier.ENTERPRISE
});

// Automatically routes to Supabase (media)
await storage.uploadFile({
  file: imageBuffer,
  category: FileCategory.PROFILE_IMAGE,
  visibility: 'public'
});

Compliance Features

import { StorageService } from '@plyaz/storage';
import { FileCategory } from '@plyaz/types';

// StorageService handles compliance internally!
const storage = new StorageService({
  adapters: [r2Adapter],
  compliance: {
    enabled: true,
    strictMode: true,  // Enforce retention policies strictly
    retentionPolicies: {
      [FileCategory.FINANCIAL_DOCUMENT]: {
        retentionYears: 7,        // Keep for 7 years
        immutable: true,          // Cannot be deleted
        requiresAuditLog: true,
        softDelete: true
      }
    },
    defaultRetentionPolicy: {
      retentionYears: 1,
      gracePeriodDays: 30
    }
  }
});

// Compliance is automatically enforced on delete operations
try {
  await storage.deleteFile({
    fileId: 'invoice-123',
    category: FileCategory.FINANCIAL_DOCUMENT
  });
} catch (error) {
  console.log('Deletion blocked by compliance:', error.message);
  // "File is immutable and cannot be deleted"
}

Installation Scenarios

Scenario 1: Basic Storage (Minimal)

pnpm add @plyaz/storage @aws-sdk/client-s3
# Total: ~14MB
# Use case: Simple file uploads to R2/S3

Scenario 2: Image Hosting

pnpm add @plyaz/storage @aws-sdk/client-s3 sharp
# Total: ~37MB
# Use case: Photo galleries, user avatars with variants

Scenario 3: Document Management

pnpm add @plyaz/storage @supabase/supabase-js pdfkit exceljs
# Total: ~13MB
# Use case: Document generation and storage

Scenario 4: Full Media Platform

pnpm add @plyaz/storage @aws-sdk/client-s3 sharp @ffmpeg-installer/ffmpeg fluent-ffmpeg puppeteer
# Total: ~337MB
# Use case: Images, videos, PDFs with full processing

Scenario 5: Serverless Optimized

pnpm add @plyaz/storage @aws-sdk/client-s3 pdfkit
# Total: ~17MB
# Use case: AWS Lambda, Vercel, Netlify Functions

Common Use Cases

User Profile Picture with Variants

// Automatically generates thumbnail, medium, large variants
const result = await storage.uploadFile({
  file: avatarBuffer,
  filename: 'avatar.jpg',
  category: FileCategory.PROFILE_IMAGE,
  entityType: EntityType.USER,
  entityId: userId
});

// Use variants in frontend
<img src={result.metadata.variants?.thumbnail} alt="Avatar" />

Generate & Store Invoice PDF

import { OutputFormat, FileCategory, DocumentType, EntityType } from '@plyaz/types';

// StorageService handles PDF generation + upload in one call!
const result = await storage.generateFileToPath({
  templateId: 'invoice',
  templateData: {
    invoiceNumber: 'INV-001',
    customer: customerData,
    items: lineItems
  },
  filename: `invoice-${invoiceNumber}.pdf`,
  format: OutputFormat.PDF,
  category: FileCategory.FINANCIAL_DOCUMENT,
  documentType: DocumentType.INVOICE,
  entityType: EntityType.ORGANIZATION,
  entityId: organizationId
});

console.log('Invoice PDF:', result.metadata.publicUrl);
// Compliance is automatically enforced based on category

Video Upload with Virus Scan

const storage = new StorageService({
  adapters: [r2Adapter],
  plugins: [
    new VirusScanPlugin({
      provider: new ClamAVProvider({ host: 'clamav-service' })
    })
  ]
});

const result = await storage.uploadFile({
  file: videoBuffer,
  filename: 'tutorial.mp4',
  category: FileCategory.POST_VIDEO,
  entityType: EntityType.POST,
  entityId: postId
});
// Automatically scanned for viruses before upload

Using in Your Application

This is a library package - you install it in your application and configure it programmatically.

Configuration in Your App

import { StorageService, CloudflareR2Adapter } from '@plyaz/storage';

// Configure with your app's configuration system
const storage = new StorageService({
  adapters: [
    new CloudflareR2Adapter({
      name: 'cloudflare-r2',
      accountId: config.cloudflare.accountId,        // From your config
      accessKeyId: config.r2.accessKeyId,            // From your config
      secretAccessKey: config.r2.secretAccessKey,    // From your config
      bucket: config.r2.bucketName
    })
  ]
});

Note: The package itself does NOT use process.env. You pass configuration when instantiating adapters. How you load config (env vars, config files, secrets manager, etc.) is up to your application.

Docker Deployment (Your Application)

When deploying your application that uses this package:

FROM node:22-alpine

# Install system dependencies if using image/video processing
RUN apk add --no-cache \
    vips-dev \        # Required for Sharp (if using SharpImagePlugin)
    ffmpeg            # Required for FFmpeg (if using FFmpegVideoPlugin)

WORKDIR /app
COPY package.json pnpm-lock.yaml ./

# Install only the features you need
RUN pnpm add @plyaz/storage @aws-sdk/client-s3 sharp

COPY . .
RUN pnpm build

CMD ["node", "dist/index.js"]

Examples Directory

The examples/ directory contains .env.example for local testing only. These show how to configure the package but are NOT part of the package distribution.

# For running examples locally
cp examples/.env.example examples/.env
# Edit .env with your credentials
pnpm example:basic

Contributing

When adding new features:

1. Add types to @plyaz/types
2. Implement with full TypeScript support
3. Add comprehensive tests (aim for 90%+ coverage)
4. Update Confluence documentation
5. Add examples in examples/ directory if applicable
6. Update internal tracking documents

Testing

# Run all tests
pnpm test

# Current test coverage
# Test Files: 53 passed (53)
# Tests: 1,962 passed (1,962)
# Coverage: ~95% lines, ~90% branches

License

ISC © Plyaz