S3 Killer — pg-lo

Stop renting object storage you already own. Your Postgres is a battle-tested, transactional blob store — pg-lo finally unlocks it: stream files of any size straight through your database with strict TypeScript, zero new services, zero new bills, zero new keys to leak. Ship it today, thank yourself forever.

• ⚡ Blast past the sidecar era — blobs live next to your rows, committed in the same ACID transaction. No eventual consistency, no orphan files, no cleanup cron.
• 🧘 Zero-drama ops — one CREATEd database you already back up, already monitor, already trust. Nothing else to deploy, scale, or pay for.
• 🌊 Stream everything — uploads and downloads flow as streams with range reads, HTTP-ready serving, and a self-maintaining filesystem cache.
• 🛡️ Typed to the teeth — ESM + strict TypeScript with descriptive errors that coach you (LoReadError, LoEmptyError, …) instead of silent undefineds.

Honest scope note: pg-lo slays the “yet another S3 bucket for app blobs” use case — uploads, images, documents, exports inside one database. It is not a multi-region CDN replacement. For single-app scale, though? Nothing touches this simplicity-to-power ratio.

Requires Node >=18 and Postgres >=14 (tested on postgres:18).

Install

npm install pg pg-lo

pg (>=6) is a peer dependency — bring the driver you already use.

Quick start

Feel the rush: upload a blob and stream it back in under a minute.

import { Readable } from "node:stream";
import { Client } from "pg";
import { createPgLo } from "pg-lo";

const client = new Client({ connectionString: process.env.DATABASE_URL });
await client.connect();
const lom = createPgLo({ client });

// Upload — inside a transaction.
await client.query("BEGIN");
const oid = await lom.createFromStream({ readable: Readable.from(Buffer.from("hello")) });
await client.query("COMMIT");

// Download — inside a transaction.
await client.query("BEGIN");
const readable = await lom.openReadableStream({ oid });
const chunks = [];
for await (const chunk of readable) chunks.push(chunk);
await client.query("COMMIT");
console.log(Buffer.concat(chunks).toString()); // hello

Using a Pool? Check out one client per transaction and run everything on it — large objects are bound to the connection that opened them.

Superpowers

• Low-level LargeObject — loread / lowrite / lo_lseek64 / lo_tell64 / lo_close with stream adapters and errors that tell you exactly what went wrong.
• High-level LargeObjectManager — createFromStream / openReadableStream / sizeOf / unlink. Ranges, caching, and handle juggling handled for you.
• Self-driving CacheManager — TTL, per-file and directory budgets, oldest-first eviction. Hot bytes serve straight from disk while the DB tail backfills silently.
• Range reads, HTTP-ready — inclusive start/end slicing composes perfectly with Content-Range responses (see examples/05-http-server.ts).
• Proven, not promised — Vitest unit (mocked) + e2e (real Postgres) coverage, currently 93/93 green.

API tour

See src/ for the exact shapes — CacheOptions in src/cache-manager.ts, PgLoOptions in src/index.ts, ReadOptions/WriteOptions in src/types.ts.

Import note: the package root exports createPgLo (plus the PgLoOptions type). Every module is also a first-class subpath — no deep dist/ spelunking:

import { createPgLo } from "pg-lo";
import { LargeObjectManager } from "pg-lo/lom.js";
import { LargeObject, SeekWhence } from "pg-lo/lo.js";
import { CacheManager, type CacheOptions } from "pg-lo/cache-manager.js";
import { LoEmptyError, LoReadError, isPgLoError } from "pg-lo/errors.js";
import type { PgLoClient, ReadOptions, WriteOptions } from "pg-lo/types.js";

Keep the .js suffix on subpaths — Node ESM performs no extension searching, so pg-lo/errors (extensionless) fails at runtime while pg-lo/errors.js resolves everywhere.

createPgLo({ client, cache })

Your launchpad. Pass any connected pg Client, PoolClient, or Pool, plus optional cache overrides (directory, TTL, per-file and directory budgets, exit cleanup, background-error hook). Defaults live next to the type.

LargeObjectManager

• create — mint a fresh oid via lo_create.
• open — lo_open a handle, reused per mode (r | w | rw); pass a callback and it closes that mode for you when the work is done.
• close — lo_close cached handles for an oid (one mode or all).
• createFromStream — pour a stream into a brand-new object; tees one branch to the DB and one to the cache in the background, then hands you the oid. Pass signal to make the write abortable: aborting rejects with the abort reason and unlinks the half-written object (database row plus cache file) in the background, so cancelled writes never leave orphans eating disk space. A pre-aborted signal throws before anything is created.
• openReadableStream — the star of the show. Serves cache hits directly, stitches a cached head to a DB tail on partial hits (appending the tail for next time), and clamps ranges to the real size. Empty ranges yield an empty stream.
• sizeOf — cache stat plus DB size ({ fsSize, isCached, loSize }).
• unlink — lo_unlink in the DB plus cache eviction. Clean break, no leftovers.

LargeObject

• read(length) — up to length bytes. Throws LoReadError when loread yields no data, so EOF bugs surface loudly instead of hiding as undefined.
• write(buffer) — append one chunk via lowrite.
• seek(offset, whence = SeekWhence.SET) / tell() — random access via lo_lseek64 / lo_tell64.
• size(oid) — total bytes via SUM(octet_length), memoized. Throws LoEmptyError for empty objects — you'll know, not guess.
• close() — idempotent lo_close; safe to call twice.
• getReadableStream({ chunkSize = 16_384, maxBytes = Infinity, signal }) — endless-safe byte stream (empty when maxBytes <= 0); ends via its byte cap.
• getWritableStream({ signal }) — a Writable for pipeline that forwards chunks to lowrite and closes on finish.

Errors that coach you

Every failure arrives as a PgLoError carrying message, cause, and a context object ({ oid }, { fd, length }, …) — logging gold. Narrow with isPgLoError().

| Error | Default message | When you'll meet it | |-------|-----------------|---------------------| | ClosedHandleError | LargeObject handle is closed. | Using a handle after close() | | LoCreateError | lo_create returned no rows. | create() got an empty result | | LoEmptyError | large object is empty. | Sizing (or streaming) an object with no pages | | LoOpenError | lo_open returned no rows. | open() got an empty result | | LoReadError | loread returned no rows. (loread returned null data. for NULL payloads) | Reading where the DB yields nothing | | LoSeekError | lo_lseek64 returned no rows. | seek() got an empty result | | LoSizeError | size lookup returned no rows. | Size lookup got an empty result | | LoTellError | lo_tell64 returned no rows. | tell() got an empty result |

Cache: your built-in CDN

One base file per oid (<temporaryDirectory>/<oid>) holds the full bytes. Range reads slice it; partial hits concatenate a cached head with a DB tail that is appended for next time. Writes and eviction run in the background and never reject — failures go to your onError hook, never into your request path.

| Option | Default (production) | Default (development) | |--------|---------------------|-----------------------| | temporaryDirectory | <cwd>/tmp/pg-lo | <cwd>/tmp/pg-lo | | timeToLive | 5 minutes | Infinity (no expiry) | | maxFileSize | 5 MB | Infinity | | maxTemporaryDirectorySize | Infinity | 1 GB | | deleteBeforeExit | off (opt in) | off (opt in) | | onError | rethrow-ish default | rethrow-ish default |

Eviction is oldest-first: expired files, then oversized files, then LRU down to budget. Set deleteBeforeExit for throwaway environments; leave it off when warm restarts make you fast.

Must-know notes

• ⚠️ Transactions are non-negotiable. Every lo_* call must run inside BEGIN/COMMIT on a single connection — Postgres closes large-object descriptors at transaction end. No transaction, no blobs.
• 🏊 Pool users: check out one client per transaction and do all LO work on it. Handles never cross connections.
• 🧹 Aborted writes clean up. Pass an AbortSignal to createFromStream and aborting removes the half-written object from the database and the cache. Cleanup runs in the background — still roll back your transaction as usual.
• 🔌 DATABASE_URL defaults to postgres://postgres:postgres@localhost:5432/db across examples and e2e tests — export it or rely on the default.
• ✂️ Ranges: start defaults to 0, end defaults to EOF (inclusive). Inverted or out-of-range requests yield empty streams, never errors.
• 🕳️ Empty objects throw: size()/sizeOf() raise LoEmptyError on 0-page objects. If you intentionally store empty blobs, catch it and celebrate — your code just documented an edge case.
• 📦 Root export: createPgLo (+ PgLoOptions) is the package entrypoint. Reach into src/ deep paths for the classes and errors.

Testing

make docker-run-postgres
npm run typecheck
npm run lint
npm run test:run     # unit + e2e
npm run test:unit
npm run test:e2e

npm run check runs knip, typecheck, lint, and tests. E2E needs Postgres at DATABASE_URL (default postgres://postgres:postgres@localhost:5432/db). Unit tests are fully mocked — fast, offline, fearless; e2e tests run against the real thing so you can trust every green run.

Examples

Runnable TypeScript samples in examples/ (see examples/README.md for the full tour). Each is self-contained and documented:

| File | Shows | |------|-------| | 01-basic.ts | Buffer upload/download, equals check, cache options | | 02-faker-image.ts | Faker SVG image upload/download round-trip | | 03-file-stream.ts | File upload via createReadStream, download via pipeline | | 04-transaction.ts | withTransaction helper, low-level LargeObject API | | 05-http-server.ts | HTTP upload/download streaming with Content-Length |

npx tsx examples/01-basic.ts
npm run example:http

Project structure

src/
  index.ts         — factory plus public exports
  lom.ts           — LargeObjectManager
  lo.ts            — LargeObject plus SeekWhence
  cache-manager.ts — CacheManager
  types.ts         — PgLoClient, ReadOptions, WriteOptions
  utilities.ts     — tee, concatReadables, sizes, ordering, ranges
  errors.ts        — typed errors plus isPgLoError
  constant.ts      — production flag
tests/
  unit/            — mocked unit tests
  e2e/             — real Postgres tests plus helpers
examples/          — runnable usage samples

Contribute

Found a rough edge? Open an issue or PR at https://github.com/rayenboussayed/pg-lo — bug reports with a failing test case get hero status. Run npm run check before pushing; keep it typed, tested, and fast.

License

ISC. Go build something unforgettable.