@x12i/xronox-storage-s3

S3-compatible storage adapter for Xronox ecosystem - supports AWS S3, DigitalOcean Spaces, MinIO, Cloudflare R2, and other S3-compatible services.

🚀 Env-Ready Component (ERC 2.0)

This component supports zero-config initialization via environment variables using @x12i/env.

Quick Start (Zero-Config Mode)

# 1. Install the package
npm install @x12i/xronox-storage-s3 @aws-sdk/client-s3 @aws-sdk/s3-request-presigner

# 2. Copy .env.example to .env (auto-generated in package root)
cp node_modules/@x12i/xronox-storage-s3/.env.example .env

# 3. Fill in required values in .env
# S3_ENDPOINT=https://nyc3.digitaloceanspaces.com
# S3_REGION=nyc3
# S3_ACCESS_KEY_ID=your_access_key
# S3_SECRET_ACCESS_KEY=your_secret_key

# 4. Use with zero config!
import { S3Storage } from '@x12i/xronox-storage-s3';
const storage = new S3Storage(); // Auto-discovers from process.env

// Use it!
await storage.putJSON('my-bucket', 'data/item.json', { id: '123', name: 'Test' });
const data = await storage.getJSON('my-bucket', 'data/item.json');

Advanced Mode (Programmatic Configuration)

import { S3Storage, S3Client } from '@x12i/xronox-storage-s3';
import { S3Client as AWSS3Client } from '@aws-sdk/client-s3';

// Option 1: Explicit config
const storage = new S3Storage({
  config: {
    endpoint: 'https://custom-endpoint.com',
    region: 'us-east-1',
    accessKeyId: 'key',
    secretAccessKey: 'secret'
  }
});

// Option 2: Explicit S3Client
const s3Client = new AWSS3Client({
  endpoint: 'https://custom-endpoint.com',
  region: 'us-east-1',
  credentials: {
    accessKeyId: 'key',
    secretAccessKey: 'secret'
  }
});
const storage = new S3Storage({ s3Client });

Environment Variables

See .env.example (auto-generated in package root) for the complete list of required and optional variables with descriptions.

Required Variables:

• S3_ENDPOINT - S3-compatible endpoint URL
• S3_REGION - AWS region or provider region
• S3_ACCESS_KEY_ID - S3 access key ID
• S3_SECRET_ACCESS_KEY - S3 secret access key

Optional Variables:

• S3_FORCE_PATH_STYLE - Force path-style URLs (auto-detected if not provided)
• S3_BUCKET - Default bucket name

ERC 2.0 Compliance

• ✅ Auto-discovers configuration from environment variables
• ✅ Type-safe with automatic coercion and validation
• ✅ All dependency requirements documented
• ✅ Transitive requirements automatically merged

Dependencies:

• ✅ @x12i/env (Configuration engine)
• ℹ️ @aws-sdk/client-s3 (peer dependency, non-ERC - must be installed separately)
• ℹ️ @aws-sdk/s3-request-presigner (peer dependency, non-ERC - must be installed separately)

Features

✅ Multi-Provider Support - Works with AWS S3, DigitalOcean Spaces, MinIO, Cloudflare R2
✅ JSON & Binary Storage - Automatic serialization for JSON, raw buffer support
✅ Data Integrity - SHA-256 verification for all uploads
✅ Presigned URLs - Generate time-limited access URLs
✅ Stream Handling - Safe conversion of streams to buffers
✅ Error Handling - Clear, actionable error messages
✅ Structured Logging - Built-in logging via @x12i/logxer
✅ Configuration Management - nxconfig integration with ENV token resolution
✅ TypeScript - Full type safety and IntelliSense support

Installation

npm install @x12i/xronox-storage-s3 @aws-sdk/client-s3 @aws-sdk/s3-request-presigner

Automatic Provider Detection

This package automatically detects S3-compatible providers and sets optimal configuration:

import { createS3Client, detectProvider } from '@x12i/xronox-storage-s3';

// Detect provider information
const provider = detectProvider('https://nyc3.digitaloceanspaces.com');
console.log(provider.provider); // 'digitalocean'
console.log(provider.forcePathStyle); // true

Supported Providers:

• ✅ AWS S3 - Virtual-hosted-style URLs
• ✅ DigitalOcean Spaces - Automatic path-style detection
• ✅ MinIO - Automatic path-style detection
• ✅ Cloudflare R2 - Automatic configuration

Usage Examples

Using ERC-Compliant S3Storage Class (Recommended)

import { S3Storage } from '@x12i/xronox-storage-s3';

// Zero-config initialization (reads from process.env)
const storage = new S3Storage();

// Store JSON data
await storage.putJSON('my-bucket', 'data/item.json', { id: '123', name: 'Test' });

// Retrieve JSON data
const data = await storage.getJSON('my-bucket', 'data/item.json');

// Store raw binary data
await storage.putRaw('my-bucket', 'file.bin', Buffer.from('binary data'));

// Get raw data
const buffer = await storage.getRaw('my-bucket', 'file.bin');

// Check if object exists
const exists = await storage.head('my-bucket', 'data/item.json');
console.log(exists.exists); // true/false

// List objects
const list = await storage.list('my-bucket', 'data/');
console.log(list.keys); // ['data/item.json', 'data/other.json']

// Generate presigned URL
const url = await storage.presignGet('my-bucket', 'data/item.json', 3600);

Using S3 Operations Directly

import { S3Client } from '@aws-sdk/client-s3';
import { putJSON, getJSON } from '@x12i/xronox-storage-s3';

// Create S3 client
const s3 = new S3Client({
  endpoint: 'https://nyc3.digitaloceanspaces.com',
  region: 'nyc3',
  credentials: {
    accessKeyId: process.env.SPACE_ACCESS_KEY!,
    secretAccessKey: process.env.SPACE_SECRET_KEY!
  },
  forcePathStyle: false
});

// Store JSON data
const data = { id: '123', name: 'Test Item' };
const result = await putJSON(s3, 'my-bucket', 'data/item.json', data);
console.log('Stored:', result.sha256, 'Size:', result.size);

// Retrieve JSON data
const retrieved = await getJSON(s3, 'my-bucket', 'data/item.json');
console.log('Retrieved:', retrieved);

Using with nxconfig

import { autoLoadConfig } from '@nx-intelligence/nxconfig';
import { S3Client } from '@aws-sdk/client-s3';
import { putJSON, getJSON } from '@x12i/xronox-storage-s3';

// Load configuration
const { config } = await autoLoadConfig();
const s3Config = config.spacesConnections.default;

// Create S3 client from config
const s3 = new S3Client({
  endpoint: s3Config.endpoint,
  region: s3Config.region,
  credentials: {
    accessKeyId: s3Config.accessKey,
    secretAccessKey: s3Config.secretKey
  },
  forcePathStyle: s3Config.forcePathStyle
});

// Use S3 operations
await putRecords(s3, 'bucket', 'key.json', { data: 'value' });

Configuration File (xronox-config.json)

{
  "xronox": {
    "spacesConnections": {
      "default": {
        "endpoint": "ENV.SPACE_ENDPOINT",
        "region": "ENV.SPACE_REGION",
        "accessKey": "ENV.SPACE_ACCESS_KEY",
        "secretKey": "ENV.SPACE_SECRET_KEY",
        "forcePathStyle": false
      }
    }
  }
}

Environment Variables (.env)

S3_ENDPOINT=https://nyc3.digitaloceanspaces.com
S3_REGION=nyc3
S3_ACCESS_KEY=your_access_key
S3_SECRET_KEY=your_secret_key

# Optional: Configure logging
XRONOX_STORAGE_LOG_LEVEL=info
XRONOX_STORAGE_LOG_FORMAT=json

API Reference

Records Operations

putRecords(s3, bucket, key, payload, contentType?)

Store records data in S3.

const result = await putRecords(s3, 'bucket', 'path/file.json', {
  id: '123',
  data: 'value'
});
// Returns: { size: number, sha256: string, etag?: string }

getRecords(s3, bucket, key)

Retrieve records data from S3.

const data = await getRecords(s3, 'bucket', 'path/file.json');
// Returns: Record<string, unknown>

Raw Data Operations

putRaw(s3, bucket, key, body, contentType?)

Store raw binary data in S3.

const buffer = Buffer.from('raw data');
const result = await putRaw(s3, 'bucket', 'path/file.bin', buffer, 'application/octet-stream');
// Returns: { size: number, sha256: string, etag?: string }

get(s3, bucket, key)

Retrieve raw data as Buffer.

const buffer = await get(s3, 'bucket', 'path/file.bin');
// Returns: Buffer

Metadata Operations

head(s3, bucket, key)

Get object metadata without downloading.

const metadata = await head(s3, 'bucket', 'path/file.json');
// Returns: { contentLength?: number, contentType?: string, lastModified?: Date, etag?: string, metadata?: Record<string, string> }

exists(s3, bucket, key)

Check if an object exists.

const fileExists = await exists(s3, 'bucket', 'path/file.json');
// Returns: boolean

List Operations

list(s3, bucket, prefix, options?)

List objects with a prefix (async generator).

for await (const key of list(s3, 'bucket', 'path/')) {
  console.log('Found:', key);
}
// Yields: string (object keys)

Management Operations

del(s3, bucket, key)

Delete an object.

await del(s3, 'bucket', 'path/file.json');

copy(s3, bucket, sourceKey, destKey)

Copy an object within the same bucket.

await copy(s3, 'bucket', 'source.json', 'destination.json');

URL Operations

presignGet(s3, bucket, key, ttlSeconds?)

Generate presigned URL for direct access.

const url = await presignGet(s3, 'bucket', 'path/file.json', 3600);
// Returns: string (presigned URL, default: 1 hour expiration)

Provider Configuration Examples

AWS S3

const s3 = new S3Client({
  region: 'us-east-1',
  credentials: {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!
  }
});

DigitalOcean Spaces

import { createS3Client } from '@x12i/xronox-storage-s3';

// Automatic provider detection - forcePathStyle is automatically set to true
const s3 = createS3Client({
  endpoint: 'https://nyc3.digitaloceanspaces.com',
  region: 'nyc3',
  credentials: {
    accessKeyId: process.env.DO_SPACES_KEY!,
    secretAccessKey: process.env.DO_SPACES_SECRET!
  }
  // forcePathStyle automatically detected and set to true
});

// Or explicitly set it
const s3 = new S3Client({
  endpoint: 'https://nyc3.digitaloceanspaces.com',
  region: 'nyc3',
  credentials: {
    accessKeyId: process.env.DO_SPACES_KEY!,
    secretAccessKey: process.env.DO_SPACES_SECRET!
  },
  forcePathStyle: true // Required for DigitalOcean Spaces compatibility
});

MinIO

import { createS3Client } from '@x12i/xronox-storage-s3';

// Automatic provider detection - forcePathStyle automatically detected
const s3 = createS3Client({
  endpoint: 'http://localhost:9000',
  region: 'us-east-1',
  credentials: {
    accessKeyId: process.env.MINIO_ACCESS_KEY!,
    secretAccessKey: process.env.MINIO_SECRET_KEY!
  }
  // forcePathStyle automatically detected and set to true
});

Cloudflare R2

const s3 = new S3Client({
  endpoint: `https://${accountId}.r2.cloudflarestorage.com`,
  region: 'auto',
  credentials: {
    accessKeyId: process.env.R2_ACCESS_KEY_ID!,
    secretAccessKey: process.env.R2_SECRET_ACCESS_KEY!
  }
});

Error Handling

All operations throw descriptive errors:

try {
  await getJSON(s3, 'bucket', 'missing.json');
} catch (error) {
  // Error: Object 'missing.json' not found in bucket 'bucket'.
  console.error(error.message);
}

Common errors:

• NoSuchBucket - Bucket doesn't exist
• NoSuchKey - Object not found
• AccessDenied - Insufficient permissions
• InvalidBucketName - Invalid bucket name format

Logging

Built-in structured logging via @x12i/logxer:

# Configure log level
export XRONOX_STORAGE_LOG_LEVEL=debug  # debug|info|warn|error

# Configure log format
export XRONOX_STORAGE_LOG_FORMAT=json  # text|json

# Enable file logging
export XRONOX_STORAGE_LOG_TO_FILE=true
export XRONOX_STORAGE_LOG_FILE=/var/log/xronox-storage.log

Example log output (JSON format):

{
  "timestamp": "2025-10-12T10:30:00.000Z",
  "package": "XRONOX_STORAGE_S3",
  "level": "INFO",
  "message": "putJSON completed successfully",
  "data": {
    "bucket": "my-bucket",
    "key": "data/item.json",
    "size": 1234,
    "sha256": "abc123...",
    "duration": 45
  }
}

Testing

This package includes comprehensive real-world testing in the peer demos package following Poiesis Rule 25 principles - no mocks, no JEST, only real S3 connections.

Running Demos

Demos for this package are maintained in the peer demos repository:

# Clone the demos repository
git clone https://github.com/xronoces/xronox-demos.git

# Navigate to storage-s3 demos
cd xronox-demos/packages/xronox-storage-s3

# Run demos
npm run demo:all

Demo Coverage

The demos package includes:

• S3 Operations (10 demos) - Direct S3 operations
• Adapter Operations (9 demos) - StorageAdapter interface
• Integration Workflows (3 demos) - Complete scenarios

Evidence Capture

Every demo captures real I/O evidence:

• io/input/ - What was sent to S3
• io/output/ - What was received from S3
• io/debug/demo.log - Complete execution log

See the demos repository for detailed testing documentation.

Development

# Install dependencies
npm install

# Build
npm run build

# Watch mode
npm run dev

# Clean
npm run clean

Package Boundaries

✅ This Package Does

• S3 operations (PUT, GET, DELETE, HEAD, LIST, COPY)
• JSON serialization/deserialization
• Stream to buffer conversion
• SHA-256 hash computation
• Presigned URL generation
• Multi-provider S3 compatibility

❌ This Package Does NOT

• Bucket management (create/delete buckets)
• Permission management (IAM policies)
• Data transformation (compression, encryption)
• Caching strategies
• Multi-part uploads
• Custom retry logic

Dependencies

• nx-config2 (^2.5.0) - Configuration management and ERC 2.0 compliance
• logs-gateway (^1.4.0) - Structured logging
• @aws-sdk/client-s3 (peer, ^3.450.0) - AWS S3 SDK (non-ERC, external dependency)
• @aws-sdk/s3-request-presigner (peer, ^3.450.0) - Presigned URLs (non-ERC, external dependency)

License

MIT

Contributing

This package follows Poiesis Development Methodology:

• One expertise per component
• Clear boundaries
• Real tests with I/O evidence
• Structured logging via logs-gateway

See docs/context.md for package context and boundaries.

Support

For issues and questions:

• GitHub Issues: Report a bug
• Documentation: See docs/ folder

Changelog

1.0.0 (2025-10-12)

• Initial release
• Full S3-compatible storage operations
• Multi-provider support (AWS, DigitalOcean, MinIO, Cloudflare R2)
• SHA-256 data integrity verification
• Structured logging via logs-gateway
• nxconfig integration