storage-sdk

@orbit-stream/storage is a reusable, extensible, enterprise-ready object storage SDK for Node.js.

The SDK provides a unified API for cloud, on-premises, and hybrid object storage systems.

Supported providers:

• AWS S3
• MinIO
• Local Storage

Designed for:

• Cloud-native applications
• Kubernetes deployments
• Edge computing
• Satellite telemetry ingestion
• Large file processing
• High-throughput streaming workloads

Features

• Object upload & download
• Streaming uploads
• Streaming downloads
• Multipart uploads
• Live streaming uploads
• Parallel uploads
• Upload progress tracking
• Signed URLs
• Retry support
• AWS S3 compatible
• MinIO compatible
• Local storage provider
• Enterprise telemetry ingestion ready

Installation

npm install @orbit-stream/storage

Environment Variables

AWS S3

STORAGE_PROVIDER=s3-compatible

S3_REGION=us-east-1

S3_BUCKET=my-bucket

S3_ACCESS_KEY=your-access-key

S3_SECRET_KEY=your-secret-key

MinIO

STORAGE_PROVIDER=s3-compatible

S3_REGION=us-east-1

S3_BUCKET=my-bucket

S3_ENDPOINT=http://localhost:9000

S3_FORCE_PATH_STYLE=true

S3_ACCESS_KEY=minioadmin

S3_SECRET_KEY=minioadmin

Create Storage Client

const StorageFactory = require("@orbit-stream/storage");

const storage = StorageFactory.create();

Upload Object

const fs = require("fs");

await storage.upload({
  key: "documents/test.zip",
  body: fs.createReadStream("./test.zip"),
  contentType: "application/zip",
});

Upload With Progress

const fs = require("fs");

let lastLogged = null;

await storage.upload({
  key: "documents/test.zip",
  body: fs.createReadStream("./test.zip"),
  contentType: "application/zip",

  onProgress: (progress) => {
    const uploaded = (progress.loaded / 1024 / 1024 / 1024).toFixed(1);

    if (uploaded !== lastLogged) {
      lastLogged = uploaded;

      console.log(`Uploaded ${uploaded} GB`);
    }
  },
});

Upload Readable Stream

Use this when you already have a Node.js Readable stream.

const fs = require("fs");

await storage.uploadStream({
  key: "documents/test.zip",
  stream: fs.createReadStream("./test.zip"),
  contentType: "application/zip",
});

Streaming Upload (Live Data)

Use this when data arrives continuously from Redpanda, Kafka, UDP, TCP, telemetry streams, sensors, or any other live source.

The SDK buffers incoming data and uploads it to S3/MinIO using Multipart Upload while creating a single object.

const uploader = await storage.createStreamingUpload({
  key: "telemetry/pass001.raw",
  contentType: "application/octet-stream",
  partSize: 100 * 1024 * 1024,
});

await uploader.write(packet1);

await uploader.write(packet2);

await uploader.write(packet3);

await uploader.close();

Example: Redpanda Consumer

const uploader = await storage.createStreamingUpload({
  key: "telemetry/pass001.raw",
});

consumer.on("message", async ({ message }) => {
  await uploader.write(message.value);
});

// Complete upload when processing finishes
await uploader.close();

Download Object

const fs = require("fs");

const stream = await storage.download("documents/test.zip");

stream.pipe(fs.createWriteStream("./downloaded.zip"));

Check Object Exists

const exists = await storage.exists("documents/test.zip");

console.log(exists);

Delete Object

await storage.delete("documents/test.zip");

List Objects

const files = await storage.list("documents/");

console.log(files);

Generate Upload Signed URL

const signedUrl = await storage.getUploadSignedUrl({
  key: "documents/test.zip",
  contentType: "application/zip",
  expiresIn: 3600,
});

console.log(signedUrl);

Upload Using Signed URL

const axios = require("axios");
const fs = require("fs");

await axios.put(signedUrl, fs.createReadStream("./test.zip"), {
  headers: {
    "Content-Type": "application/zip",
  },
});

Generate Download Signed URL

const downloadUrl = await storage.getDownloadSignedUrl(
  "documents/test.zip",
  3600,
);

console.log(downloadUrl);

Multipart Upload API

Create Multipart Upload

const multipart = await storage.createMultipartUpload({
  key: "documents/large.zip",
  contentType: "application/zip",
});

console.log(multipart.UploadId);

Generate Multipart Upload Signed URL

const signedUrl = await storage.getMultipartUploadSignedUrl({
  key: "documents/large.zip",
  uploadId: multipart.UploadId,
  partNumber: 1,
});

Complete Multipart Upload

await storage.completeMultipartUpload({
  key: "documents/large.zip",
  uploadId: multipart.UploadId,
  parts: [
    {
      PartNumber: 1,
      ETag: '"etag"',
    },
  ],
});

Abort Multipart Upload

await storage.abortMultipartUpload({
  key: "documents/large.zip",
  uploadId: multipart.UploadId,
});

Health Check

const health = await storage.health();

console.log(health);

Provider Capabilities

console.log(storage.capabilities());

Supported Providers

| Provider | Upload | Multipart | Signed URLs | Streaming Upload | | ------------- | :----: | :-------: | :---------: | :--------------: | | AWS S3 | ✅ | ✅ | ✅ | ✅ | | MinIO | ✅ | ✅ | ✅ | ✅ | | Local Storage | ✅ | ❌ | ❌ | ❌ |

Choosing the Right API

| API | Description | | ------------------------- | --------------------------------------------------------------------------- | | upload() | Upload a Buffer, String, or Readable stream. | | uploadStream() | Upload an existing Node.js Readable stream. | | createStreamingUpload() | Continuously upload packets/chunks received over time into a single object. | | createMultipartUpload() | Low-level multipart upload API for browser/mobile signed URL workflows. |

Recommended Upload Strategy

| Use Case | Recommendation | | --------------------------------------- | --------------------------- | | Files < 1 GB | upload() | | Files 1–10 GB | Multipart Upload | | Files > 10 GB | Multipart + Parallel Upload | | Live telemetry / Kafka / Redpanda / UDP | createStreamingUpload() |

Enterprise Features

• AWS S3 compatible
• MinIO compatible
• Local storage provider
• Multipart uploads
• Streaming uploads
• Streaming downloads
• Parallel uploads
• Upload progress tracking
• Signed URLs
• Automatic retry support
• High-throughput telemetry ingestion
• Kubernetes friendly
• Enterprise-ready architecture

License

MIT