@mdimranh/onestorage
v1.0.1
Published
Universal, zero-dependency object storage for Node.js. One typed API for local disk, Amazon S3, Cloudflare R2, Backblaze B2, MinIO, Azure Blob, Google Cloud Storage and Wasabi.
Maintainers
Readme
onestorage
One typed API for all your object storage.
Local disk, Amazon S3, Cloudflare R2, Backblaze B2, MinIO, Azure Blob, Google Cloud Storage, Wasabi — and anything else that speaks the S3 API. Write your upload code once, choose the backend from configuration, and change your mind later without touching a single call site.
- Zero runtime dependencies. Every signature (AWS SigV4, Azure Shared Key and SAS, Google Cloud V4) is implemented in this package against
node:crypto. There is no AWS SDK, no Azure SDK, noundici. - Dual ESM + CommonJS, with complete TypeScript declarations for both.
- Streaming everywhere. Uploads and downloads never buffer a whole object unless you ask them to.
- Not a thin wrapper. Retries with jitter, byte-range reads, multipart uploads, resumable uploads, pagination, batch operations, metrics, events and a unified error hierarchy ship in the box.
npm install @mdimranh/onestorageRequires Node.js 18.17 or newer.
Quick start
import { createStorage } from '@mdimranh/onestorage';
// One line to choose the backend. Everything else is the same for every driver.
const storage = createStorage({ driver: 'fs', root: './uploads' });
await storage.put('notes/hello.txt', 'hello world');
const text = await storage.getText('notes/hello.txt'); // 'hello world'
const size = (await storage.head('notes/hello.txt')).size;
await storage.delete('notes/hello.txt');Switch to S3 by changing the configuration — not the code:
const storage = createStorage({
driver: 's3',
provider: 'aws',
region: 'eu-west-1',
bucket: 'my-bucket',
credentials: {
accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
},
});Why not the vendor SDK?
Because you usually do not want one. A typical application stores files in one place, reads them back, occasionally lists a prefix and hands out a URL. The vendor SDKs are excellent for full-platform work, but they tie your application to one cloud, add tens of megabytes of transitive dependencies, and still leave you writing your own retry, pagination and error-mapping code — the parts that actually cause production incidents.
onestorage gives you the 5% of storage API surface that 95% of applications use, with the operational concerns handled properly.
Supported backends
| Driver | Backend | Config driver | Notes |
| --- | --- | --- | --- |
| fs | Local filesystem | 'fs' | Atomic writes, signed URLs for your own HTTP server |
| memory | In-process map | 'memory' | Tests and caching tiers |
| s3 (AWS) | Amazon S3 | 's3' + provider: 'aws' | All regions, virtual-host addressing |
| s3 (R2) | Cloudflare R2 | 's3' + provider: 'r2' | Endpoint derived from accountId |
| s3 (B2) | Backblaze B2 | 's3' + provider: 'b2' | Path-style addressing, region from cluster |
| s3 (MinIO) | MinIO | 's3' + provider: 'minio' | Any custom endpoint |
| s3 (Wasabi) | Wasabi | 's3' + provider: 'wasabi' | |
| s3 | DigitalOcean Spaces | 's3' + provider: 'digitalocean' | |
| s3 | Linode Object Storage | 's3' + provider: 'linode' | |
| s3 | Scaleway, Oracle, anything else | 's3' + provider: 'scaleway' \| 'oracle' \| 'custom' | Full endpoint control |
| azure | Azure Blob Storage | 'azure' | Shared Key, SAS, Entra ID, Azurite |
| gcs | Google Cloud Storage | 'gcs' | Service accounts, metadata server, emulators |
Local filesystem
const storage = createStorage({
driver: 'fs',
root: './uploads',
metadataSidecar: true, // persist content-type and custom metadata next to files
signSecret: process.env.SIGN_SECRET,
baseUrl: 'https://files.example.com',
});Writes go to a temporary file that is renamed into place, so a reader never sees a half-written object and a crash cannot leave a truncated file behind. Keys are resolved against root and proven to stay inside it, so a malicious key cannot escape.
Amazon S3
const storage = createStorage({
driver: 's3',
provider: 'aws',
region: 'us-east-1',
bucket: 'assets',
credentials: async () => ({
// Called per request, so rotating or temporary credentials just work.
...(await getCredentialsFromSecretsManager()),
}),
publicUrl: 'https://cdn.example.com',
multipart: { partSize: 16 * 1024 * 1024, concurrency: 8, threshold: 64 * 1024 * 1024 },
});Cloudflare R2
const storage = createStorage({
driver: 's3',
provider: 'r2',
accountId: process.env.R2_ACCOUNT_ID!,
bucket: 'assets',
// Region "auto" and the r2.cloudflarestorage.com endpoint are filled in for you.
credentials: {
accessKeyId: process.env.R2_ACCESS_KEY_ID!,
secretAccessKey: process.env.R2_SECRET_ACCESS_KEY!,
},
});Backblaze B2
const storage = createStorage({
driver: 's3',
provider: 'b2',
bucket: 'my-bucket',
// Find the cluster in the bucket details, e.g. us-west-004 → s3.us-west-004.backblazeb2.com
endpoint: 'https://s3.us-west-004.backblazeb2.com',
credentials: { accessKeyId: process.env.B2_KEY_ID!, secretAccessKey: process.env.B2_APP_KEY! },
});MinIO (and any S3-compatible server)
const storage = createStorage({
driver: 's3',
provider: 'minio',
endpoint: 'http://127.0.0.1:9000',
forcePathStyle: true,
bucket: 'dev',
credentials: { accessKeyId: 'minioadmin', secretAccessKey: 'minioadmin' },
skipBucketCheck: true,
});Wasabi
const storage = createStorage({
driver: 's3',
provider: 'wasabi',
region: 'eu-central-1',
bucket: 'archive',
credentials: { accessKeyId: process.env.WASABI_KEY!, secretAccessKey: process.env.WASABI_SECRET! },
});Azure Blob Storage
const storage = createStorage({
driver: 'azure',
// A connection string covers endpoint, account name and key in one value.
connectionString: process.env.AZURE_STORAGE_CONNECTION_STRING!,
container: 'uploads',
});Authentication options, in the order Azure itself prefers them:
// 1. Shared Key (required if you want this package to mint SAS URLs)
{ accountName: 'myaccount', accountKey: process.env.AZURE_KEY! }
// 2. A container SAS token
{ accountName: 'myaccount', sasToken: process.env.AZURE_SAS! }
// 3. Microsoft Entra ID (managed identity, workload identity, …)
{ accountName: 'myaccount', tokenProvider: () => getEntraToken() }
// 4. Azurite, for local development
{ connectionString: 'UseDevelopmentStorage=true' }Google Cloud Storage
const storage = createStorage({
driver: 'gcs',
bucket: 'my-bucket',
// An object, a JSON string, or a path to a key file.
credentials: process.env.GOOGLE_APPLICATION_CREDENTIALS,
});
// On GCE, Cloud Run or GKE — no key file required:
const workloadIdentity = createStorage({
driver: 'gcs',
bucket: 'my-bucket',
useMetadataServer: true,
});Uploads below the resumable threshold go out as a single multipart/related request; larger or streamed payloads use a resumable session with 8 MiB chunks.
One client, many backends
The real payoff for a migration or a tiered architecture: a single client that routes keys to different backends by prefix.
const storage = createStorage({
default: 'disk',
drivers: {
disk: { driver: 'fs', root: '/var/data' },
hot: {
driver: 's3',
provider: 'r2',
accountId: process.env.R2_ACCOUNT_ID!,
bucket: 'hot',
credentials: r2Credentials,
},
cold: {
driver: 's3',
provider: 'glacier-capable-vendor',
bucket: 'cold',
credentials: coldCredentials,
},
},
mounts: {
'uploads/': 'hot', // newest uploads land on R2
'archive/': 'cold', // long-term storage
},
});
await storage.put('uploads/photo.jpg', jpeg); // → R2
await storage.put('archive/2019/old.jpg', jpeg); // → cold
await storage.put('scratch.txt', data); // → local disk (the default)
// The longest matching prefix wins.
storage.resolveDriver('uploads/2026/photo.jpg'); // 'hot'Keys are passed to the driver unchanged — a mount chooses a backend, it does not rewrite your keys. media/a.png stays media/a.png.
Moving data between backends
copy and move work across drivers. When the source and destination are the same driver they use the backend's server-side copy; when they differ, the bytes stream through your process without being buffered:
// R2 → local disk, streamed, constant memory.
await storage.copy('uploads/photo.jpg', 'thumbnails/photo.jpg');
// Move it and remove the original atomically from the caller's perspective.
await storage.move('archive/2019/old.jpg', 'uploads/recent/old.jpg');API
Writing
await storage.put(key, input, options?); // any payload type
await storage.putJson(key, value, options?); // JSON with the right content type
await storage.uploadFile(key, filePath, options?); // streams from diskinput may be a Buffer, string, Uint8Array, ArrayBuffer, Blob, web ReadableStream, Node Readable, async or sync iterable, or a factory function returning any of those. Factories are special: they can be called again, which makes retries safe for large payloads.
await storage.put('a.txt', 'text');
await storage.put('a.bin', Buffer.from([1, 2, 3]));
await storage.put('a.txt', fs.createReadStream('./file.txt'), { size: bytes });
await storage.put('rows.csv', async function* () { for (const row of rows) yield row + '\n'; });
await storage.put('big.bin', () => fs.createReadStream('./big.bin'), { size: 5_000_000 });Options include contentType, metadata, cacheControl, acl, storageClass, tags, overwrite: false for conditional creation, ifMatch for optimistic concurrency, multipart tuning, onProgress, signal and per-call retry/timeoutMs.
Reading
const result = await storage.get(key, options?);
result.body; // ReadableStream<Uint8Array> — pipe it anywhere
result.stream(); // Node Readable, for pipe() and pipelines
await result.bytes(); // Buffer
await result.text(); // string
await result.json<T>();
await storage.getBuffer(key, options?);
await storage.getText(key, options?);
await storage.getJson<T>(key, options?);
await storage.downloadFile(key, localPath, options?); // streams to disk, atomicallyA download body is a one-shot stream. Read it once — through bytes(), text(), json(), stream(), or by piping body — and the connection is released. A second read throws rather than silently returning an empty value.
Ranged reads come in two flavours:
await storage.get(key, { range: { start: 0, end: 1023 } }); // inclusive end
await storage.get(key, { range: 'bytes=-500' }); // last 500 bytes
await storage.get(key, { head: true }); // metadata onlyInspecting
await storage.head(key); // ObjectMetadata, throws if missing
await storage.exists(key); // boolean, never throws
await storage.existsMany(keys); // Map<string, boolean>Listing
const page = await storage.list({ prefix: 'photos/', maxResults: 100 });
page.objects; // ObjectMetadata[]
page.prefixes; // 'folders' when a delimiter is used
page.continuationToken; // pass back to continue
page.truncated;
// Every backend caps a single listing at ~1,000 keys, so for real work use the
// stream, which follows pagination for you and holds one page in memory.
for await (const object of storage.listAll({ prefix: 'photos/' })) {
console.log(object.key, object.size);
}
await storage.list({ prefix: 'photos/', recursive: false }); // 'folder' viewDeleting
await storage.delete(key);
await storage.deletePrefix('tmp/'); // whole subtree
await storage.deleteMany(keys); // { succeeded, failed }, never throws per key
const removed = await storage.clear('staging/'); // count of deleted keysdeleteMany never aborts on the first failure: you get per-key outcomes, so a nightly cleanup can report exactly what it could not remove.
Copying and moving
await storage.copy(source, destination, options?);
await storage.move(source, destination, options?);URLs
Browser uploads with POST policies
A presigned PUT forces one fixed key and nothing else. A POST policy is the browser-native shape: your server signs a form contract, the browser POSTs a multipart/form-data to the backend directly, and the backend refuses anything outside the policy — wrong key, oversized file, unexpected content type. The secret key never reaches the browser.
const post = await storage.signedUploadPostUrl('uploads/${filename}', {
expiresIn: 600,
contentTypes: ['image/'], // any image, by prefix
contentLengthRange: { min: 1, max: 5 * 1024 * 1024 },
});
// Render the form for the browser:
const form = new FormData();
for (const [name, value] of Object.entries(post.fields)) form.append(name, value);
form.append('file', fileFromUser); // must be the LAST field appended
await fetch(post.url, { method: 'POST', body: form });${filename} inside the key lets users keep their own filenames; every other key is fixed. Use exactLength when the size is known up front, or fields to pin extra form fields — each one is folded into the signature, so a tampered field fails validation. post.expiresAt tells you when to mint a new one.
Supported by S3-compatible backends (S3, R2, B2, MinIO, Wasabi) and Google Cloud Storage, where the equivalent scheme signs with the service account key. Azure has no POST-policy equivalent — a SAS token signs one URL, not a form — so it rejects the call with NotSupportedError rather than accepting restrictions it cannot enforce.
const url = await storage.signedUrl(key, { expiresIn: 3600 });
const uploadUrl = await storage.signedUploadUrl(key, { expiresIn: 600, contentType: 'image/png' });
const pub = storage.publicUrl(key); // undefined when not configured
const best = await storage.url(key); // public URL, else a signed oneFor direct-from-browser uploads where the client should not choose the key or the size, prefer the POST policy above.
publicUrl returns undefined unless you configure one. url() is the convenient choice: it hands back the CDN URL when you have one and a signed URL when you do not.
For the filesystem driver, signedUrl produces a URL your own HTTP handler must verify:
import { verifyLocalRequest } from '@mdimranh/onestorage';
const { valid, expired } = verifyLocalRequest({
secret: process.env.SIGN_SECRET!,
key: request.params.key,
method: request.method,
expiresAt: Number(request.query.expires),
signature: String(request.query.signature),
});The HTTP method is part of the signed payload, so a URL minted for GET cannot be replayed as a DELETE.
Errors
Every failure is a StorageError with a stable code, so you never parse provider messages:
import { isNotFound, isRetryable, StorageError } from '@mdimranh/onestorage';
try {
await storage.get('missing.txt');
} catch (error) {
if (isNotFound(error)) return fallback();
if (isRetryable(error)) return retryLater();
throw error;
}| Code | Meaning |
| --- | --- |
| NOT_FOUND | Object, bucket or container does not exist |
| ALREADY_EXISTS | Conflict with an existing object |
| PRECONDITION_FAILED | A conditional write (overwrite: false, ifMatch) failed |
| PERMISSION_DENIED / UNAUTHENTICATED | Credentials rejected or insufficient |
| INVALID_ARGUMENT | The call or its payload cannot be accepted |
| CONFIGURATION_ERROR | The driver is misconfigured |
| NOT_SUPPORTED | This driver cannot do that |
| RATE_LIMITED | Backend throttled the request (retryable) |
| TIMEOUT / CONNECTION_ERROR | Transport failure (retryable) |
| PAYLOAD_TOO_LARGE | Exceeds a documented backend limit |
| CHECKSUM_MISMATCH | Integrity verification failed |
| CONFLICT | Concurrent modification |
| ABORTED | Cancelled through an AbortSignal |
Provider codes (NoSuchKey, BlobNotFound, notFound, …) and request ids are preserved on error.details and error.requestId for support tickets:
console.error(error.toJSON());
// { name: 'NotFoundError', code: 'NOT_FOUND', driver: 's3', key: 'a.txt',
// requestId: 'REQ123', details: { code: 'NoSuchKey' }, ... }Retries, timeouts and cancellation
Retrying is on by default: three attempts with exponential backoff and full jitter, honoring provider Retry-After headers. Full jitter matters — synchronized retries are how a brief blip becomes an outage.
const storage = createStorage({
driver: 's3',
bucket: 'b',
credentials,
retry: { maxAttempts: 5, baseDelayMs: 200, maxDelayMs: 20_000, jitter: 'full' },
timeoutMs: 30_000,
});
// Per call:
await storage.put(key, data, { retry: 1, timeoutMs: 5_000 }); // fail fast
await storage.put(key, data, { retry: false }); // never retryOnly genuinely transient failures are retried. A missing object is not requested three times. Writes are retried only when the payload can be replayed — a half-consumed stream is never re-sent as if it were complete.
Cancellation uses standard AbortSignals and aborts in-flight multipart parts too:
const controller = new AbortController();
const upload = storage.uploadFile('big.iso', './big.iso', { signal: controller.signal });
controller.abort();
await upload.catch((error) => {
if (error.code === 'ABORTED') console.log('cancelled cleanly');
});Progress reporting is throttled to 100 ms so it never competes with the transfer. It is also aggregated: a multipart or resumable upload is dozens of requests, but onProgress always describes the whole object. loaded accumulates across every part — including parts uploading in parallel — rather than restarting at each part boundary, so a progress bar moves smoothly to 100% and never jumps backwards. A final callback always reports the full size, even when the last tick was throttled:
await storage.uploadFile('big.iso', './big.iso', {
// Progress is already throttled to 100ms, so just render it.
onProgress: ({ loaded, total, ratio }) => {
const percent = ratio === undefined ? '?' : `${Math.round(ratio * 100)}%`;
console.log(`${percent} (${loaded}/${total ?? 'unknown'} bytes)`);
},
});total is undefined when the payload arrived as a stream with no declared size, and ratio is undefined whenever the total is unknown rather than being guessed. The same guarantee holds whichever backend is behind the call — the filesystem driver reports progress for a local write too.
Observability
Metrics are built in and memory-bounded: counters plus a reservoir-sampled latency distribution that costs a constant amount of memory no matter how many operations run.
const snapshot = storage.metrics();
// { since, retries, operations: { put: { count, errors, bytesOut, latency: { p50, p95, p99, ... } } } }Expose it on a health endpoint, or forward it to Prometheus.
app.get('/health/storage', (_, res) => res.json(storage.metrics()));Events give you a stream of everything that happens, across every driver:
storage.events.on('operation', ({ driver, name, operation, key, durationMs, bytesIn, bytesOut, ok }) => {
logger.info({ driver, name, operation, key, durationMs, bytesIn, bytesOut, ok }, 'storage');
});
storage.events.on('retry', ({ driver, operation, attempt, delayMs, error }) => {
metrics.increment('storage.retry', { driver, operation, code: error.code });
});driver is the implementation (s3) and name is the configured instance (archive), which is what you want when several buckets share one driver.
Hooks let you observe the raw HTTP traffic, for tracing or cost attribution:
createStorage({
driver: 's3',
bucket: 'b',
credentials,
hooks: {
onRequest: ({ method, url }) => tracer.start(method, url),
onResponse: ({ status, durationMs }) => histogram.observe(durationMs),
},
});URLs passed to hooks are redacted — a presigned URL is a bearer credential and never appears in logs or error messages. Hooks are observers, so a hook that throws is logged and ignored rather than failing the transfer.
Logging is opt-in and respects whatever logger you already use, including pino and winston:
createStorage({ driver: 'fs', root: './data', logger: pino() });Tuning throughput
Node's fetch already maintains a keep-alive connection pool per origin, so nothing extra is needed for good throughput. Three knobs matter when you push harder:
const storage = createStorage({
driver: 's3',
bucket: 'b',
credentials,
batchConcurrency: 16, // parallel deleteMany / existsMany
multipart: { partSize: 16 * 1024 * 1024, concurrency: 8 },
maxConcurrency: 32, // hard ceiling on requests in flight
});
// Bring your own connection pool if you want to tune it. `onestorage` does not
// depend on undici, so this is entirely your choice.
import { Agent } from 'undici';
createStorage({ driver: 's3', bucket: 'b', credentials,
dispatcher: new Agent({ connections: 128, pipelining: 1, keepAliveTimeout: 60_000 }) });Multipart uploads read sequentially but upload concurrently, holding only concurrency parts in memory. A 100 GB object costs the same heap as a 1 GB one.
The other two settings bound one operation each; maxConcurrency bounds the whole client. Every request takes a slot, including the parts of a chunked upload and requests started while another operation is still running, so a hundred concurrent gets cannot open a hundred sockets however they were started. A slot is held until its response has finished arriving — a download that is still streaming is still in flight — and is returned when the body is read, cancelled or fails. Nothing is capped unless you set it.
Because the limit is taken per request rather than per operation, an upload keeps all its parts and still finishes with maxConcurrency: 1. One exception is worth knowing: an operation that has to hold one request open while making another — a copy streaming from one backend into a different one — is granted a single slot and its own nested requests are exempt, so that transfer can put a download and an upload on the wire at once. Nested parallelism stays bounded by the driver's own multipart.concurrency.
A Semaphore may be passed instead of a number to share one budget between several clients:
import { Semaphore, createStorage } from '@mdimranh/onestorage';
const budget = new Semaphore(64);
const primary = createStorage({ driver: 's3', bucket: 'b', credentials, maxConcurrency: budget });
const backups = createStorage({ driver: 's3', bucket: 'backups', credentials, maxConcurrency: budget });The filesystem and memory drivers do not make HTTP requests, so the limit does not apply to them.
Custom drivers
Any backend can join the party and inherit the retry, metrics, event and key-handling behavior for free:
import { BaseStorageDriver, DEFAULT_CAPABILITIES, registerDriver, createStorage } from '@mdimranh/onestorage';
class PostgresDriver extends BaseStorageDriver {
readonly name = 'postgres';
readonly capabilities = { ...DEFAULT_CAPABILITIES, range: true, copy: false };
protected async doPut(key, body, options, context) { /* ... */ }
protected async doGet(key, options, context) { /* ... */ }
protected async doHead(key, options, context) { /* ... */ }
protected async doDelete(key, options, context) { /* ... */ }
protected async doList(options, context) { /* ... */ }
protected async doCopy(source, destination, options, context) { /* ... */ }
protected async doSignedUrl(key, options, context) { /* ... */ }
}
registerDriver('postgres', (config, context) => new PostgresDriver(config, context));
const storage = createStorage({ driver: 'postgres', connectionString: '...' });You implement the protocol; the base class handles retries, timeouts, metrics, events, prefix scoping, exists, move, deleteMany and paginated listAll.
If your driver chunks a large upload into many requests, hand each part's callback to createUploadProgress() and the caller's single onProgress still describes the whole object:
import { createUploadProgress } from '@mdimranh/onestorage';
const progress = createUploadProgress(body.size, options.onProgress);
parts.forEach((part, index) => { request.onUploadProgress = progress?.part(index); });
await commitParts(parts);
progress?.complete(); // guarantees a terminal 100%If your driver ever has to hold one request open while it makes another — streaming a response into a second request, say — wrap the pair in context.http.withPermit(). Requests inside the scope share the operation's single slot, which is what stops a client capped at one request at a time from waiting on itself.
Security notes
- Path traversal is rejected, not sanitized. Keys containing
..throw rather than being silently rewritten, so a bug in your key construction is loud instead of quietly writing somewhere unexpected. - Keys are normalized at the edge: leading and trailing slashes are stripped, repeated slashes collapsed, control characters and NUL bytes refused, and lengths checked against the backend's limit. If you have existing objects with keys like
a//b, address them through a driver whose prefix does not normalize them, or rename them. - Signed URLs are never logged. Errors and hooks only ever see a redacted URL — host and path, no query string.
- Signature comparison is constant-time for the filesystem driver's local URLs.
- Credentials are never stored in URLs except inside the signature itself, and session tokens are passed as headers.
- TLS is enforced by the endpoints you configure. Use
https://in production; the S3UNSIGNED-PAYLOADfallback is only safe over TLS, which is why buffered uploads send a real content hash whenever possible.
Limitations
- The S3 driver's server-side copy is limited to 5 GiB per call, which is an S3 API limit. Above that,
get+putstreams the object through your process — slower, but correct. - Uploading a live stream to S3 or Azure requires the byte length up front, because those APIs need
Content-Length. Passsize, hand over aBlob, or useuploadFile, which reads it from the filesystem. - GCS resumable uploads align chunk size to Google's required 256 KiB multiple.
- The filesystem driver's listings walk the directory tree. That is inherent to a filesystem;
maxScanKeysguards against pathological trees.
Testing
The suite runs on Node's built-in test runner — no test framework dependency — and covers signing vectors, error mapping, retry and concurrency behaviour, and full CRUD against the filesystem and memory drivers.
npm testThe S3, Azure and Google upload paths are exercised against mock transports, including a deliberately out-of-order Azure block upload and a resumable session that only accepts a final chunk carrying the total length. Progress reporting is covered end to end, so the reset-at-every-part regression cannot return unnoticed.
SigV4, the highest-risk code in the package, is not tested against hand-copied vectors. Every signing case is cross-checked against aws4, an independent implementation, so a canonicalization regression cannot hide: if the signatures diverge, the build fails. Azure and Google signing are verified by deriving the expected signature independently and, for Google, by verifying the RSA signature against the public key.
License
MIT
