@fractional-ai/storage-connect
v0.2.0
Published
Storage connection library: a thin StorageAdapter interface, adapters, the storage profile/audit schema fragment and the profile-lifecycle engine for a host app. No UI, no framework assumption.
Maintainers
Readme
@fractional-ai/storage-connect
Public reference for the package. The design documents this README cites by section number live in the maintainers' private repository; where a section is cited, the README states the decision it records.
A storage connection library for a host app: one thin StorageAdapter
interface, adapters behind it, the storage_profile / storage_audit schema
fragment the host migrates itself, credential custody, and the
profile-lifecycle engine a host's own page calls. No HTML, no framework
import, no session handling — the host supplies the page and the actor.
Design: docs/design/storage-connection.md. The drop-in brief for a team
adding the feature to an app ships in this package as HANDOVER.md.
0.2.0 is a breaking release: storage profiles. 0.1.0 held one storage
configuration per deployment; 0.2.0 holds many named storage profiles,
exactly one of which is the default for new writes in each scope. A host
records which profile a file was written to, permanently, at upload; when
the default changes, files already written stay where they are and stay
readable through adapterFor(profileId). 0.1.0 published with no
dependents, so nothing carries its names forward — see the changelog at the
end for the 0.1.0 to 0.2.0 shape.
Install
pnpm add @fractional-ai/storage-connect drizzle-ormRequires Node 22 or later. drizzle-orm (>=0.45.2 <1.0.0) is a peer
dependency — the host project builds its own instance and migrates the
storage_profile / storage_audit tables through its own drizzle-kit
series (see "The ProfileStore port" below). The floor rose from
>=0.36.0 in 0.2.0: the partial unique index in the fragment is written
with uniqueIndex().on(sql).where(sql), which is the 0.45 API; the fragment
is built and its DDL generated against [email protected] with
[email protected], and no lower version has been tried.
Status: foundation, custody, profile engine, and both production
adapters. This package ships the interface, the local filesystem adapter,
the S3-compatible adapter (AWS S3, Cloudflare R2, Backblaze B2, custom
endpoints such as RustFS and MinIO) with its RustFS integration suite, the
Azure Blob adapter with its Azurite integration suite, the default adapter
factory covering both, the shared key rules, the shared canary check, the
Drizzle schema fragment with its Postgres integration suite, credential
custody, the credential-mode ordering, and the profile engine proven against
the local adapter and a test double. The Drizzle-backed ProfileStore is the
host's own (a reference implementation is in HANDOVER.md), and the
environment-seeding precedence rule is documented, not shipped as code.
Neither adapter is preferred over the other anywhere in this package: the
provider choice lives in the deployed app.
No workspace-protocol dependency. The eventual consumer is an external npm
project (vendored or published), so every dependency is a public package.
drizzle-orm is a peer dependency: the host's own instance must build the
tables it migrates.
Exports
.— everything below../schema— only the Drizzle fragment, for a host'sschema.tsto re-export into its owndrizzle-kitseries.
| Export | What it is |
|---|---|
| StorageAdapter and its types | The contract (design doc §3). put / get / delete / list / presign / check. |
| LocalFsAdapter | Objects on local disk. Test isolation and developer convenience; never a production store. |
| S3Adapter, createS3Adapter | The S3-compatible adapter for all four presets, and its AdapterFactory-shaped constructor (below). |
| AzureBlobAdapter, createAzureBlobAdapter, DELEGATOR_ROLE_MISSING | The Azure Blob adapter for all three credential modes, its constructor, and the exact reason string presign reports when the Delegator role is missing (below). |
| createDefaultAdapterFactory | The AdapterFactory covering both providers, with per-provider options (below). |
| assertValidKey, assertValidPrefix, keyProblem, prefixProblem, isValidKey | The shared key rules every adapter applies before touching its provider. |
| runCanaryCheck, CANARY_PREFIX, isPassingCheck | The put → get → delete → presign round-trip an adapter's check() delegates to, and the guard that narrows its result. |
| StorageError, StorageKeyError, ObjectNotFoundError, StorageEngineError, CustodyError | Typed failures with a code. |
| storageProfile, storageAudit, the six pg_enums | The schema fragment (design doc §5), verbatim. |
| CredentialCustody, deriveCredentialKey, credentialLastFour | Credential custody (design doc §10). |
| CREDENTIAL_MODE_ORDERING, credentialModesFor | The credential-mode ordering a host's page renders (design doc §6a). |
| createStorageEngine, normalizeAzureInput, normalizeS3Input, validateCandidate | The engine (design doc §6a). |
| RECOMMENDED_PROFILE_COUNT_WARNING, RETIRE_GUARD_LIST_LIMIT | The advisory profile-count threshold a host's page renders a banner past (10; never enforced), and the one-page bound on the retire guard's listing. |
| InMemoryProfileStore | The ProfileStore the base suite runs on; a host's tests can too. |
Host obligation — the attachment record
This section is an obligation on every host that integrates this library, not a suggestion. It is where the go-live conditions on content validation and malware scanning land (design doc §3a and §16, ROY-2026-09-14-19).
The library handles an opaque key: string and nothing else. It has no idea
what an object is, who owns it, or what it is for; put() is content-agnostic
by design, because the same adapter serves every host regardless of what that
host stores. So every host must keep its own attachment record, separate
from storage_profile / storage_audit, with at least:
| Field | Rule |
|---|---|
| storage_profile_id | New in 0.2.0, required. A foreign key to storage_profile.id, ON DELETE RESTRICT, in the host's own schema. Written once, at upload, from whichever profileId the adapter came from (adapterFor / defaultAdapter, below), and never rewritten however many times the scope's default changes afterwards. This FK is the first lock against a profile disappearing out from under a live file: the engine cannot see the host's attachment table, so its own retire guard is defence in depth on top of this, not a substitute for it. |
| tenant_id | New in 0.2.0, required. Whose file this is. Not derivable from the profile's own tenant_id: a profile can be shared across tenants via path_prefix, so a platform-scope profile (tenant_id IS NULL) can hold files belonging to several tenants at once, distinguishable only here. A single-tenant host holds one constant value, or NULL, same as on the profile. |
| Object key | The exact string handed to put / get / delete. Never a URL. A URL encodes a provider and a moment-in-time signing scheme, neither of which survives a profile ceasing to be the default. |
| SHA-256 | Of the uploaded bytes, computed by the host before or during put. |
| Byte size | As uploaded. |
| Declared content type | From the upload's Content-Type header or file extension. Untrusted. |
| Validated content type | From magic-byte sniffing at the host's write path, where the format allows it. Where a format has no reliable signature (DXF/DWG-class files are the named case), the host decides and records how it validated, and that decision is a go-live condition for the app, not a library setting. |
| Owning entity | Whatever domain object the file belongs to. The recommended shape is an owner_type / owner_id split, so one attachment table can serve more than one kind of owning entity (an invoice, a certificate, a drawing) without a table per type. That split is the host's choice, not a library requirement: the library only ever sees key: string and has no opinion on whether the owning reference is polymorphic or a single foreign key; a host with one kind of attachment loses nothing by using a plain owner_id. |
| Uploading actor | The same host-resolved identity the host passes to the engine's audit line. |
| Timestamp | When the upload was accepted. |
Magic-byte validation and malware scanning are the host's job, at the
host's own write path, before or immediately alongside its call to put().
This library does neither and never will: it never inspects content, it
never rejects an object for what it contains, and nothing in put() can be
configured to. A host going live with user-supplied uploads scans them at its
upload handler, with a scanner it chooses and operates. If your app has a
provenance discipline already (import batches keyed by SHA-256, values held
as observed until a named person promotes them), the attachment record is
that same discipline applied to uploaded files, not a new pattern.
One more obligation, from design doc §9: where the stored files are a client's compliance or traceability records, tell the client in writing what retiring the profile that holds them means — still there, still costing them storage, still theirs to delete with their provider if that is what they want; this library never deletes provider-side objects. Changing which profile is the default needs no such notice: it changes where new objects go and strands nothing, by construction (below). The engine refuses an unconfirmed retirement of a non-empty location; it does not copy.
Contract decisions every adapter follows
Keys are object keys, never paths. Forward-slash segments; no leading or
trailing slash; no empty, . or .. segments; no backslashes; no control
characters; at most 1024 UTF-8 bytes. Validation lives in keys.ts and every
adapter calls it first, so an adversarial key (../../etc/passwd, an absolute
path, a backslash path) throws StorageKeyError from every method before any
provider is touched. The local adapter additionally proves the resolved path
is inside its root, as a second line, not the first.
Listing is paginated and bounded. list(prefix, { limit, cursor })
returns at most limit objects (default and cap MAX_LIST_LIMIT, 1000) in
ascending key order, plus nextCursor when more remain. The cursor is opaque:
pass it back, never build one. The local adapter honours a small limit so
the cursor round-trip is exercised without a bucket.
"Cannot presign" is a typed result, not an error. presign() returns
{ supported: true, url, method, headers, expiresAt } or
{ supported: false, reason }; the union makes a caller discriminate before
reading url. A thrown error from presign() always means the adapter does
support presigning and the attempt failed (credentials, network, a missing
role). check() records an unsupported presign as a passing step with the
reason in detail; the typed truth is capabilities.presign.
delete() is idempotent. Deleting a key with no object resolves. get()
on a missing key throws ObjectNotFoundError.
check() never throws. Every step is recorded as ok or not, with a
detail line; a presigned URL is never written into a detail line. Its result
is a discriminated union on ok: if (result.ok) narrows to
PassingCheckResult, the only kind the engine will save.
Credential custody
A pasted credential is sealed with AES-256-GCM under a key derived
from APP_SECRET_KEY by HKDF-SHA256 with the context string
"storage-credential" (no salt, 32-byte output) — never the raw secret.
Each seal draws a fresh random 96-bit nonce; the stored text is
scc1.<nonce>.<ciphertext>.<tag>, base64url, with the scc1 label bound in
as additional authenticated data. Node's hkdfSync, createCipheriv and
createDecipheriv are called directly. APP_SECRET_KEY shorter than 32
bytes is refused outright. The library never reads process.env: the host
reads the variable and passes it to createStorageEngine.
credential_last_four is captured in clear before sealing: the last four
characters of the SAS token, the account key, or (for access_key mode) the
access key id, which is the identifier rather than the secret. Ambient
modes (managed_identity, iam_role) store no credential and no last four.
Each profile seals its own credential, with its own nonce and its own
tag; rotating one profile's credential cannot read, write or invalidate any
other profile's ciphertext (design doc §10). One deployment's
APP_SECRET_KEY therefore protects N sealed credentials rather than one,
and a retired profile keeps its credential indefinitely (its files must stay
readable), so that inventory only ever grows for the life of the deployment.
The custody sign-off behind this holds for as long as every profile in a
deployment belongs to the same single client; a tenant_id populated with
more than one real value is a different custody question this package does
not answer (design doc §10a).
Named, tracked gap (design doc §10, ROY-2026-09-14-10): the code path that
seals afresh on every create and every rotation exists; a rehearsed
end-to-end re-encryption under a rotated APP_SECRET_KEY does not. Under a
new key the engine fails closed — checkStanding and adapterFor throw
CustodyError, retireGuard reports "unknown" — rather than reading
anything.
Credential-mode ordering
CREDENTIAL_MODE_ORDERING is the order every host's page renders
(ROY-2026-09-14-11): managed_identity first for Azure, iam_role first for
S3, both tier: "primary"; every pasted-key mode is tier: "secondary" and
singlePointOfFailure: true, which a page names out loud
(ROY-2026-09-14-12). credentialModesFor(provider) returns one provider's
listings in that order. A host may reword the labels; it may not reorder.
assume_role is in the schema enum for a later hosted-model slice and is
deliberately absent from the ordering and from CredentialInput.
The engine
import { createStorageEngine, normalizeS3Input } from "@fractional-ai/storage-connect";
const engine = createStorageEngine({
appSecretKey: process.env.APP_SECRET_KEY!, // the host reads it; the library never does
store, // a ProfileStore over storage_profile / storage_audit
adapters, // an AdapterFactory: CandidateConfig -> StorageAdapter
tenantId: null, // the scope a single-tenant host sets once and never passes again
});
const { endpoint, region, bucket } = normalizeS3Input(pasted, "aws");
const candidate = { provider: "s3", preset: "aws", endpointOrAccount: endpoint, region, bucketOrContainer: bucket!, credential };
const check = await engine.testConnection(candidate);
if (check.ok) {
const profile = await engine.createProfile(candidate, check, principalId, { label: "Prod (S3)" });
await engine.setDefaultProfile(profile.id, principalId); // add-and-set-default: the ordinary path
}
// Later, for a real upload: record the profile id on the attachment row, next to the key.
const adapter = await engine.defaultAdapter();The functions, all plain and async, all provider-agnostic. Every function
that reads or writes a stored profile takes a tenantId (or a profileId);
omitted, tenantId falls back to the engine's own, so a single-tenant host
never passes it:
| Function | What it does |
|---|---|
| normalizeAzureInput(pasted) | Account name, account URL or connection string in; { accountUrl, accountName, container? } out. Drops any SAS or AccountKey it finds; never returns one. |
| normalizeS3Input(pasted, preset) | Region, endpoint or bucket URL in; { endpoint, region?, bucket?, accountId? } out, per preset (aws, r2, b2, custom). |
| probeCapabilities(candidate) | Runs the adapter's check() and returns its capabilities. Each adapter's probes are described below: addressing, multipart, versioning and object lock for S3; block staging, versioning, immutability and, under managed identity, the Delegator role for Azure. |
| testConnection(candidate) | The canary round-trip. Never throws for a connection problem; an adapter that cannot even be built is a CheckResult with every step failed. |
| createProfile(candidate, passingCheck, actor, { tenantId?, label? }?) | Persists a new profile, never the default. Purely additive: nothing points at a new profile yet, so nothing is guarded. |
| setDefaultProfile(profileId, actor, tenantId?) | Makes the profile the scope's default for new writes, unsetting the previous one in one store transaction. Metadata only: no check, no adapter call, no guard. Refuses an unknown or retired profile, or one from another scope (invalid_candidate). Already the default: a no-op, no audit row. |
| listProfiles(tenantId?) | Every profile in the scope, default and retired included, never the sealed credential. |
| retireGuard(profileId) | { safe, reason?, objectCount? } from one bounded listing of the profile's own location. Throws cannot_retire_default for the scope's current default. Writes nothing. |
| retireProfile(profileId, actor, confirmationText?) | Sets retired_at and nothing else. Refuses the current default outright (cannot_retire_default, no confirmation path); refuses a location the guard reports unsafe unless a non-blank confirmationText is given (retire_requires_confirmation), recorded verbatim in the audit row's note. |
| rotateCredential(profileId, newCredential, passingCheck, actor) | Replaces only that profile's credential. Allowed on a retired profile, whose files must stay readable. The only call that invalidates adapterFor's cache, for that one profile. |
| getStatus(tenantId?) | The scope's default profile: { state: "unconfigured" }, { state: "active", config } or { state: "error", detail, config? }. Never throws. Profiles that exist without a default read as unconfigured. |
| checkStanding(profileId?, tenantId?) | The standing --check against one profile, or the scope's default when omitted; records status and last_checked_at, no audit row. Throws unconfigured when there is no default to check, invalid_candidate for an unknown profileId. |
| adapterFor(profileId) | A working StorageAdapter for the profile, built from its own sealed credential through the factory and cached by profile id. Works for a retired profile. |
| defaultAdapter(tenantId?) | Resolves the scope's default profile fresh on every call, then adapterFor(id). Throws unconfigured when there is none. |
Nothing is saved until a check passes, enforced twice. The type:
createProfile and rotateCredential take a PassingCheckResult, so a
CheckResult that has not been narrowed on ok is a compile error. The
runtime: the object passed must be the very one testConnection returned,
in this process, for a candidate with the same provider, location and
credential (check_mismatch otherwise), and its steps are re-read
(check_not_passing). A page that tests in one request and saves in another
re-runs testConnection at save time; a stale pass is not evidence.
Every mutating call takes a host-supplied actor. The parameter is
required in the type; an empty string is refused at runtime
(invalid_actor). The library never resolves an identity. tenantId is
host-supplied in the same way: the library never infers one and never treats
it as a security boundary — it scopes the default-per-scope rule and nothing
else. The empty string is refused as a tenant id (invalid_candidate): it
is the sentinel the schema's own index coalesces the null scope to.
A profile's location is immutable once created. Provider, endpoint,
bucket and path prefix never change after createProfile returns; there is
deliberately no updateProfile. This is what makes adapterFor's cache safe
to hold indefinitely and an attachment's storage_profile_id a trustworthy
permanent pointer. The only things that change afterwards are the credential
(rotateCredential) and the lifecycle flags (setDefaultProfile,
retireProfile). A profile created with a typo, before anything was written
to it, is retired (trivially: it was never default and its bucket is empty)
and replaced with a fresh one.
Add-and-set-default is the ordinary path; retire is the guarded one.
Promoting a profile can orphan nothing, so setDefaultProfile has no guard.
Retiring is where a file could be stranded, so retireGuard lists one page
(up to RETIRE_GUARD_LIST_LIMIT) of that profile's own location through
adapterFor and reports unsafe when anything is there, and reports unsafe
when the listing throws or the stored credential cannot be opened — fail
closed, objectCount: "unknown", exactly as 0.1.0's switch guard did. Two
refusals, in order: the scope's current default is refused outright, before
any listing, because a scope must keep a default for new writes to go
somewhere (reassign it first); then an unsafe location is refused without a
confirmation string. Retiring never deletes the row, the sealed credential
or anything at the provider — retired_at is the one column it sets, and
no engine function deletes a storage_profile row at all. A retired profile
is never selected for new writes and can never be promoted again, but
adapterFor(retiredId) keeps working so its files stay readable. Migrating
an attachment's bytes between profiles is later, separate work; nothing in
this package moves an object.
adapterFor caches per profile id, and only rotateCredential
invalidates it — for that one profile. Nothing else can make an entry
stale: location is immutable, and a retired profile still needs its adapter.
A build that fails (a wrong APP_SECRET_KEY, a factory that throws) is not
cached, so the next call tries again. defaultAdapter caches nothing of its
own: it re-reads which profile is the scope's default on every call, so a
setDefaultProfile is visible to the very next defaultAdapter() call with
nothing for the host to invalidate. The host's one job is to record the
profileId the adapter came from on the attachment row at upload time; that
id, not "whatever is default now", is what reads or deletes the file later.
The factory owns the path prefix. The engine hands the factory the whole
candidate, pathPrefix included, and treats the adapter it gets back as
already rooted there. The engine never prefixes a key and lists a profile's
location with an empty prefix. A pathPrefix, when given, must be a valid
key prefix ending in /.
The ProfileStore port is six methods over the two tables:
loadProfile(id), loadDefaultProfile(tenantId), listProfiles(tenantId),
saveProfile(row, audit) (insert-or-replace by id plus one audit entry,
atomically where the store can), setDefault(profileId, tenantId, audit)
(unset the scope's current default, set the new one, write the audit entry,
in one transaction — two UPDATEs, old off then new on, so the partial
unique index, checked per statement, never sees two defaults in one scope),
and recordCheck(profileId, outcome) for a standing check's status and
timestamp. InMemoryProfileStore is the reference and holds the schema's two
invariants in memory the way Postgres does (a retired row is never the
default; one default per coalesce(tenant_id, '') scope, checked after each
statement-equivalent); a Drizzle-backed store is a straight mapping, written
out in HANDOVER.md. The row and audit types (StorageProfileRow,
StorageAuditInsert) are inferred from the fragment.
Audit rows carry before / after reduced to identity, lifecycle
flags, mode and last four — never the sealed credential, never a plaintext —
and are written against the profile acted on (profile_id). The action set
is the closed storage_audit_action enum: create, set_default,
retire, rotate_credential; seed is reserved for a host's own
bootstrap code and emitted by nothing here. set_default writes one row, on
the newly promoted profile, with before.previousDefaultProfileId naming
the profile it displaced (or null), so a query for every row about profile
X still finds the moment X stopped being default. retire carries the
guard's objectCount in before and the confirmation text in note. There
is no update action: nothing is ever re-saved in place.
The local filesystem adapter
new LocalFsAdapter({ root }) or await LocalFsAdapter.createTemporary().
Under root: objects/ holds the files, meta/ a JSON sidecar per object
(content type, size, SHA-256 etag), tmp/ in-flight uploads, renamed into
place only once fully written. Known limits, by design: no presign; a key
a and a key a/b cannot coexist (a file cannot also be a directory); the
characters < > : " | ? * are refused because not every filesystem can store
them, though buckets can.
The S3-compatible adapter
createS3Adapter(candidate, options?) takes an S3Candidate (the output of
normalizeS3Input plus bucket, preset and credential) and returns a working
S3Adapter. This is the AdapterFactory-shaped constructor for the s3
provider: a host's factory dispatches on candidate.provider and calls it —
const adapters: AdapterFactory = (candidate) =>
candidate.provider === "s3" ? createS3Adapter(candidate) : createAzureBlobAdapter(candidate);— or uses createDefaultAdapterFactory(), which is that dispatch with
per-provider options (see "The default adapter factory" below).
The client is ours, FlyDrive sits on top of it. The adapter constructs
the S3Client itself with requestChecksumCalculation: "WHEN_REQUIRED" and
responseChecksumValidation: "WHEN_REQUIRED" for every preset, aws
included (design doc §4: the January 2025 SDK default of attaching a CRC32
checksum to every upload broke R2 outright and is the same risk class for
every non-AWS endpoint), and hands that client to FlyDrive's s3 driver in
its { client, ...options } form. The setting is readable off
adapter.client.config, not inferred from an options object. With the SDK's
default in place a presigned PUT carries x-amz-sdk-checksum-algorithm and
x-amz-checksum-crc32 in its query string and a browser upload to R2 or
RustFS fails; with this setting neither parameter is present (asserted
offline in s3.test.ts).
FlyDrive's driver carries delete() and both presign directions. put(),
get() and list() go to the SDK commands on the same client, because the
driver's own surface discards what the contract returns: its put returns
void (the ETag is lost), its getStream drops the response headers (size,
etag, content type), and its listAll reads ContentLength off
ListObjectsV2 entries, which carry Size instead. Recovering each of
those through FlyDrive would cost a second round trip per call.
supportsACL is false for every preset: with it on, FlyDrive sends
x-amz-acl: private on every upload, which AWS buckets under the
bucket-owner-enforced ownership setting (the default for new buckets since
April 2023) reject with AccessControlListNotSupported, and which R2 and
RustFS-shaped endpoints do not implement. Objects are private by default
everywhere; visibility is the bucket policy's concern.
Addressing is probed, never asked. check() sends a HeadBucket under
virtual-hosted addressing first, then path-style, and records in
capabilities.provider.addressing the style the request actually went out
with, read off the built request — the SDK addresses an IP endpoint
path-style whatever forcePathStyle says, and the probe reports that
honestly (addressingProbe carries the attempt log). The adapter then adopts
the style that worked for every later call on that instance. Before
check() has run, an adapter uses path-style for the custom preset and
virtual-hosted for aws, r2 and b2; a host that has a persisted
capabilities_json can pin the probed style with the addressing option.
One seam worth knowing: the engine's AdapterFactory receives only the
candidate, so the adapter adapterFor(profileId) builds for a stored profile
— the one real uploads and retireGuard's listing run on — starts un-probed
under that default; for a custom endpoint that needs the other style the
retire guard's listing fails and the guard fails closed, which is the safe
direction, and a host pins the probed style through the factory (below).
Multipart. Buffers above multipartThresholdBytes (default 8 MiB, never
below the protocol's 5 MiB minimum part size) and every Readable body go
through @aws-sdk/lib-storage's Upload in parts of that size; smaller
buffers are one PutObject. FlyDrive has no multipart path at all — its
put and putStream are a single PutObjectCommand, which the SDK refuses
for a stream of unknown length — so the SDK's own multipart logic, which
lives only in lib-storage, is the one added dependency beyond the three the
design named. capabilities.multipart is probed by creating and immediately
aborting a multipart upload under the canary prefix; the integration suite
separately proves a real three-part round trip.
Credential modes. access_key passes the pasted pair to the client;
iam_role passes nothing and lets the SDK's default provider chain resolve
the host's own instance, task or profile credentials. Bucket-scoped
credentials — an IAM policy, an R2 API token or a B2 application key scoped
to one bucket — need no adapter code; check() records
bucketScopedCredentials: "credential-side" in the provider bag to say so.
Presign. presign(key, "put", { contentType }) returns headers:
{ "content-type": ... } when a content type was given: the uploader sends
it verbatim. GET URLs need no headers. Default expiry 900 seconds.
Listing. list() maps the cursor onto S3's ContinuationToken /
NextContinuationToken one to one, wrapped so a cursor from another adapter
is refused as invalid_cursor before any request. Keys come back in the
provider's ascending order, with the candidate's pathPrefix stripped.
Probed capabilities, all read-only and each swallowed into
capabilities rather than thrown: multipart true when a multipart upload
can be created; versioning true when versioning is enabled on this
bucket, false when the endpoint answered and it is not, "unknown" when
the call was refused (a bucket-scoped key commonly lacks
s3:GetBucketVersioning); objectLock the same for object lock. The raw
finding for each is in capabilities.provider.
Content type. Stored as given. When omitted, a single-request upload gets the type FlyDrive infers from the key's extension and a multipart upload gets the provider's default; pass it explicitly — the host's attachment record holds the declared and validated types in any case.
The Azure Blob adapter
createAzureBlobAdapter(candidate, options?) takes an AzureCandidate (the
accountUrl from normalizeAzureInput plus container and credential) and
returns a working AzureBlobAdapter: the AdapterFactory-shaped
constructor for the azure_blob provider, the same shape as
createS3Adapter. It is hand-written against @azure/storage-blob and
@azure/identity (design doc §4: no credible licensed driver exists to
wrap). The account name and the addressing style are read back off the
account URL through normalizeAzureInput: https://<account>.blob.<suffix>
is virtual-hosted, an emulator's http://host:port/<account> is path-style;
Azure has no second style to fall back to, so this is recorded, not probed.
Credential modes, all three the ordering allows:
| Mode | Requests | presign() | Recorded as presignMechanism |
|---|---|---|---|
| account_key | signed with StorageSharedKeyCredential | a per-blob service SAS signed with the key: read-only for GET, create+write for PUT, scoped to that one blob | service-sas |
| sas_token | the pasted SAS on every request (AnonymousCredential + the SAS in the URL) | the blob URL carrying the configured SAS as-is: the adapter holds no key and cannot mint a narrower one, so the URL is exactly as broad and lives exactly as long as the SAS the person pasted; expiresAt is clamped to the SAS's own se, and an already-expired SAS is refused rather than handed out | configured-sas |
| managed_identity | bearer token from DefaultAzureCredential (or the tokenCredential option, for a user-assigned identity) | a user-delegation SAS: getUserDelegationKey on the service, then the same per-blob SAS math under that key; needs the identity to hold the Storage Blob Delegator role | user-delegation-sas |
The sas_token row is worth a host's page naming next to
singlePointOfFailure: a presigned URL under that mode is not scoped to one
object, it is the pasted SAS. A deployment that needs per-object URLs uses an
account key or a managed identity.
The Delegator role, detected by name. Under managed_identity,
check() asks the service for a user delegation key as part of the probe.
If that call is refused with a 403 (the identity authenticated, since a bad
token is a 401, and was refused the one operation the Storage Blob Delegator
role grants) the probe records delegatorRole: "missing" and the reason in
capabilities.provider.delegatorRoleProbe, capabilities.presign is
false, and every presign() returns { supported: false, reason } where
reason starts with the exported constant DELEGATOR_ROLE_MISSING
(identity lacks Storage Blob Delegator role) and names the grant to make.
The check still passes: uploads and downloads work without the role, and a
page shows why presigning does not. Any other failure of that call (no
ambient identity, network, a 401) is thrown as what it is, never rewritten
into the role message, and never rewritten into a generic one either. A
delegation key is requested for at least an hour and reused across
presign() calls while it covers the requested expiry.
Presign headers. A presigned PUT needs x-ms-blob-type: BlockBlob (the
service refuses a Put Blob without it), plus content-type when one was
given; both come back in headers for the uploader to send verbatim. GET
URLs need none. Default expiry 900 seconds; SAS start times are backdated
five minutes for clock skew.
Uploads and block staging. A buffer up to blockSizeBytes (default
8 MiB, clamped to 64 KiB to 256 MiB, the SDK's own single-shot ceiling) is
one Put Blob; a larger buffer is staged in blocks of that size (grown only
if 50,000 blocks would not be enough) and committed; every Readable is
staged through uploadStream in blocks of that size, with the bytes counted
on the way through for PutResult.size. Downloads stream. delete() goes
through deleteIfExists with snapshots included, so a missing blob resolves
and a missing container still throws. Content type is stored as given; the
service records application/octet-stream when none is given, and get()
reports what the service recorded.
Listing. list() maps the cursor onto listBlobsFlat's continuation
token one to one, wrapped (azb1.) so a cursor from another adapter is
refused as invalid_cursor before any request. Keys come back in the
service's ascending order with the candidate's pathPrefix stripped.
Probed capabilities, each swallowed into capabilities rather than
thrown: multipart true when a block can be staged (the probe stages one
block under the canary prefix, commits an empty block list, which is Azure's
way of discarding staged blocks, and deletes the probe blob); versioning
true when that commit came back with a version id, which the service returns
only when blob versioning is enabled, false when it did not; objectLock
true when the container reports an immutability policy, a legal hold or
immutable storage with versioning, false when it answered with none,
"unknown" when reading container properties was refused (a container SAS
is refused this on Azurite). The raw finding for each is in
capabilities.provider, alongside sasExpiresOn under sas_token and
containerScopedCredentials: "credential-side": a container-level SAS or a
container-scoped role assignment needs no adapter code.
Nothing secret leaves the adapter. Every public operation passes
through a guard that blanks the account key, the SAS and any sig=
parameter out of an error's message before it propagates, on the same error
object so its status and error code survive; the shared canary check writes
an error's message into a detail line and has no way to redact it itself.
Presigned URLs are returned, never logged. Under pipelineOptions a host can
pass the SDK's retry, proxy or httpClient settings to every client the
adapter builds; the unit suite uses that seam to stand in for the service.
Not proven by this package: managed identity and the user-delegation
SAS. Azurite cannot emulate them (see the Azurite suite below). The
managed-identity code path (constructing the client under a
TokenCredential, calling getUserDelegationKey, signing the SAS with the
key it returns, and the Delegator-role classification of a 403) is
exercised in azure-blob.test.ts against a fake token and a fake service
answering a crafted 403 and a crafted key. That proves the adapter does the
right thing with those answers; it proves nothing about a real storage
account. a maintainer's live click-through against a real storage account is what
proves it.
The default adapter factory
createDefaultAdapterFactory(options?) returns the AdapterFactory the
engine takes: "s3" goes to createS3Adapter, "azure_blob" to
createAzureBlobAdapter, anything else is invalid_candidate. Options are
per provider, and each may be a static object or a function of the candidate
(sync or async):
const adapters = createDefaultAdapterFactory({
s3: async (candidate) => ({ addressing: await persistedAddressingFor(candidate) }),
azure: { blockSizeBytes: 16 * 1024 * 1024 },
});
const engine = createStorageEngine({ appSecretKey, store, adapters });The addressing seam, and where it sits. The engine's AdapterFactory
contract is unchanged: it hands the factory the candidate and nothing else,
so the adapter adapterFor(profileId) builds from a stored S3 profile — the
instance real uploads and retireGuard's listing run on — has not probed
addressing. The engine does not pass the persisted capabilities_json
through, and changing that is an engine contract change, not this package's
to make in passing. What the factory gives a host is the place to close the
gap itself: the s3 function above receives the candidate the engine built
from the stored row, so a host that keeps the persisted
capabilities.provider.addressing can hand it back and the profile's
adapter runs under the style that worked from its first call. Without that,
the retire guard fails closed on a custom endpoint that needs the other
style, which is the safe direction. Azure has no such seam: its addressing
is a property of the account URL.
Tests
The base suite needs no database and no container:
pnpm --filter @fractional-ai/storage-connect build
pnpm --filter @fractional-ai/storage-connect test # or, from the repo root: pnpm vitest run tools/storage-connectThe build is part of the proof: src/engine/engine.types.test.ts holds
@ts-expect-error lines for every call the engine's types must refuse, and
tsc -b fails if any of them compiles.
The engine suite runs the put → get → delete round-trip, the
empty-versus-non-empty retire guard, and the profile lifecycle for real
against LocalFsAdapter: a file written under profile A is read back
through adapterFor(A.id) after profile B becomes the default; retiring the
current default is refused outright; a retired profile's row, sealed
credential and adapter are unchanged by retirement; rotating A's credential
leaves B's ciphertext and B's cached adapter alone; setDefaultProfile
writes exactly one audit row naming the previous default. Two branches the
local adapter cannot reach — a presign that succeeds, and a list() that
throws — run against src/testing/fake-adapter.ts, a full StorageAdapter
over an in-memory map with injectable canned responses. It is a test double
in the test tree, not an export, and it proves nothing about any provider.
InMemoryProfileStore's emulation of the schema's two invariants is tested
in the same file; what Postgres itself does with them is the schema
integration suite below.
src/adapters/s3.test.ts runs offline: presigning is signature arithmetic on
the client, so the FlyDrive wrap, the checksum setting, the addressing style
on the URL and the key rules are all proven without a bucket.
src/adapters/azure-blob.test.ts runs offline too: service-SAS presigning is
signature arithmetic, SAS reuse is string work, and the managed-identity path
runs against a fake TokenCredential and a fake IHttpClient handed in
through pipelineOptions: a crafted 403 drives the Delegator-role path, a
crafted delegation key drives the granted path, and a service that echoes
secrets back proves the redaction guard. src/adapters/factory.test.ts
proves the default factory's dispatch, its options hook, and one engine
round trip over a transport that refuses to send.
The RustFS integration suite
pnpm --filter @fractional-ai/storage-connect test:integration:s3A separate command with its own config (vitest.integration.s3.config.ts):
the file is s3.integration.ts, not *.test.ts, so a bare pnpm test at
the package or the repo root never loads it. The config reads
tools/storage-connect/.env.integration (gitignored, UTF-8 without a BOM —
process.loadEnvFile keeps a BOM as part of the first key name) if it
exists, and the suite skips itself with a visible warning when
STORAGE_TEST_S3_ENDPOINT, STORAGE_TEST_S3_ACCESS_KEY and
STORAGE_TEST_S3_SECRET_KEY are not all set (STORAGE_TEST_S3_REGION is
optional, default us-east-1). Variables already in the environment win
over the file.
Every run creates one bucket named
storage-connect-it-<epoch-ms>-<6 hex> through a bare client, runs
everything inside it, and deletes it — objects, in-progress multipart uploads,
then the bucket — in afterAll, on pass and fail alike. It never lists or
touches any other bucket, and the test credential should be scoped to that
name pattern. Neither a credential nor a presigned URL is ever written to the
output; the one assertion that inspects output for them is boolean, so a
failure cannot print the value it was looking for.
What it proves against a live RustFS: the adapter's own check() passes
every canary step (the same code path production uses); a presigned PUT and a
presigned GET executed for real with fetch; list() paginating across
three real pages with the cursor round-tripped; a three-part multipart upload
read back byte-for-byte, and a Readable of unknown length streamed in two
parts; the adversarial-key set refused with zero requests sent; the path
prefix rooting keys and stripping them from listings. Last run green from
a maintainer's workstation through an SSH tunnel to the maintainers' build host
(127.0.0.1:9000), RustFS 1.0.0-rc.6, 15 September 2026.
What it cannot prove: anything about AWS S3, R2 or B2 (no live account in
scope — a maintainer's own R2 click-through is what tests a managed provider);
versioning and object lock on RustFS, because the scoped test key is refused
GetBucketVersioning and GetObjectLockConfiguration and the probe reports
them "unknown"; virtual-hosted addressing against RustFS, because through
an IP endpoint the SDK sends path-style whatever is requested, so the probe
recorded path as the style RustFS accepted and the fallback branch was
never needed.
Feature matrix, Custom (RustFS/MinIO-shaped) column, as observed
Design doc §12 is the master; these are the cells this suite moved, with the evidence, for that table to carry.
| Capability | Before | After |
|---|---|---|
| Presign GET/PUT | assumed | verified — both executed with fetch against RustFS 1.0.0-rc.6, 200 each |
| Multipart upload | assumed, not verified against RustFS specifically | verified — create/abort probe accepted; 3-part upload via lib-storage read back byte-for-byte |
| Checksum default trap | assumed same risk class, mitigated the same way | verified — RustFS accepts the WHEN_REQUIRED setting on PutObject, multipart and presigned PUT |
| Path-style addressing | commonly required, probed | verified — path-style is what RustFS answered to through the tunnel; the virtual-hosted branch could not be exercised over an IP endpoint |
| Versioning | assumed absent/partial by default | unchanged: probe refused (AccessDenied) under the scoped test key |
| Object lock / WORM | assumed absent | unchanged: probe refused (AccessDenied) under the scoped test key |
| Bucket-scoped credentials | depends on implementation, assumed | unchanged in kind; the fixture's operator reports the test key is scoped to storage-connect-it-*, and the AccessDenied on the bucket-level reads is consistent with that — reported, not independently verified by the suite |
The Azurite integration suite
pnpm --filter @fractional-ai/storage-connect test:integration:azureA separate command with its own config (vitest.integration.azure.config.ts),
the same shape as the RustFS one: the file is azure-blob.integration.ts, not
*.test.ts, so a bare pnpm test never loads it, and each integration
config selects only its own file, so test:integration:s3 never loads this
one either. The config reads tools/storage-connect/.env.integration
(gitignored, UTF-8 without a BOM) if it exists, and the suite skips itself
with a visible warning when STORAGE_TEST_AZURITE_CONNECTION_STRING is not
set to a connection string carrying an AccountKey. The value may be
Azurite's own UseDevelopmentStorage=true shorthand: the suite expands it to
the published development account (devstoreaccount1 and its well-known
key, from the Azurite README, a public constant that every Azurite install
shares and nothing outside an emulator accepts; the one line carrying it is
allowlisted for gitleaks by citation). The account URL and name come from
normalizeAzureInput on the expanded string, the same path a pasted
connection string takes in the engine.
Every run creates one container named
storage-connect-it-<epoch-ms>-<6 hex> through a bare client, runs
everything inside it, and deletes it in afterAll, on pass and fail alike
(confirmed by forcing a failure mid-run and checking the container was
gone). It never lists or touches any other container. A container SAS for
the sas_token tests is minted per run by the suite from the account key
and is never printed. Neither a credential, a SAS, nor a presigned URL is
ever written to the output; the assertions that inspect output for them are
boolean, so a failure cannot print the value it was looking for.
What it proves against a live Azurite 3.37.0 (blob service only): the
adapter's own check() passes every canary step under account_key and
under sas_token (a container SAS with racwdl); a presigned PUT and a
presigned GET executed for real with fetch under each of those two modes
(201 and 200), with the service-SAS URL confirmed scoped, since the same URL
is refused a PUT to a sibling key; list() paginating across three real
pages with the continuation token round-tripped; a buffer over the block
size committed from three staged blocks and a Readable of unknown length
from two, read back from the service's own block list rather than inferred,
and a small buffer landing as a single Put Blob with no block list; the
adversarial-key set refused with zero requests attempted (a second adapter
for the same container runs over a transport that refuses to send) and
nothing bad in the container afterwards; the path prefix rooting keys and
stripping them from listings; and the canary prefix left clean by check().
Last run green from a maintainer's workstation through an SSH tunnel to
the maintainers' build host (127.0.0.1:10000), 15 September 2026, 11/11.
What it cannot prove, stated plainly: managed identity and the
user-delegation SAS, including the Delegator-role probe's specific error.
Azurite's OAuth support (--oauth basic) validates a token's issuer,
audience and expiry and, by Azurite's own documentation, does not check the
token's signature or permission. A managed-identity run against it would
report the Delegator role as present whatever the truth, which is worse than
not testing it, so that plumbing is deliberately not built. Those paths are
exercised at the unit level against a crafted 403 and a crafted key
(azure-blob.test.ts) and are proven only by a maintainer's live click-through
against a real storage account, later. Also not proven here: blob versioning
and immutability (Azurite reports neither, so the probes recorded false
for a plain container and the enabled branches never ran); virtual-hosted
addressing (Azurite is path-style; the adapter's handling of a
<account>.blob.core.windows.net URL is proven offline on the presigned
URL's shape only); and anything about a real storage account's limits,
throttling or geo-redundancy.
Feature matrix, Azure Blob column, as observed
Design doc §12 is the master; these are the cells this suite moved, with the evidence, for that table to carry.
| Capability | Before | After |
|---|---|---|
| Presign GET/PUT | documented (SAS; user-delegation SAS needs Delegator role) | verified for account_key and sas_token only: both executed with fetch against Azurite 3.37.0, 201 and 200, under each mode; managed-identity presign (user-delegation SAS) and the Delegator-role probe: unproven by this suite, unit-level against a crafted 403 only |
| Multipart upload | different mechanism (block staging + commit), documented | verified: stage/empty-commit probe accepted; 3-block buffer upload and 2-block stream upload committed and read back byte-for-byte, block lists read from the service |
| Checksum default trap | n/a (different protocol) | unchanged: n/a |
| Versioning | documented (blob versioning) | unchanged in kind: the probe (version id on commit) answered not enabled on Azurite; the enabled branch is unexercised |
| Object lock / WORM | documented (immutability policies) | unchanged in kind: the container-properties probe answered not enabled under the account key on Azurite; "unknown" under a container SAS (properties read refused); the enabled branch is unexercised |
| Path-style addressing | n/a | unchanged: n/a for a real account; the emulator's path-style form is handled and was the form under test |
| Bucket-scoped credentials | container-level SAS, documented | verified in shape: a container SAS (racwdl) drove the whole sas_token run; the adapter records containerScopedCredentials: "credential-side"; nothing about a real account's RBAC scoping was tested |
| Local emulator for tests | Azurite, verified | unchanged: Azurite 3.37.0, blob service only, pinned tag |
The Postgres schema integration suite
pnpm --filter @fractional-ai/storage-connect test:integration:schemaA separate command with its own config (vitest.integration.schema.config.ts),
the same shape as the other two: the file is schema.integration.ts, not
*.test.ts, so a bare pnpm test never loads it. The config reads
tools/storage-connect/.env.integration (gitignored, UTF-8 without a BOM)
if it exists, and the suite skips itself with a visible warning when
STORAGE_TEST_PG_URL is not set. The role behind the URL needs CREATEDB.
drizzle-kit and postgres are dev dependencies for this suite alone;
nothing in the shipped package imports either.
Every run creates one database named
storage_connect_it_<epoch-ms>_<6 hex> on the server, generates the
fragment's DDL through drizzle-kit/api (generateDrizzleJson +
generateMigration, the same statements a host's drizzle-kit generate
produces), applies every statement, runs everything inside that database,
and drops it in afterAll, on pass and fail alike. It never reads or
touches any other database. The connection string is never written to the
output.
What it proves against a real Postgres, rather than against
InMemoryProfileStore's emulation: the DDL applies cleanly and the server
holds storage_profile_default_per_scope as a unique partial index on
COALESCE(tenant_id, '') where is_default; two rows with tenant_id NULL
and is_default true collide (23505, that index), which a plain index on
tenant_id would not catch; two real tenant ids each hold their own default
while a second in the same tenant collides; a retired default is refused on
insert and on update (23514, storage_profile_retired_default_ck);
setDefault's two UPDATEs in one transaction succeed in the order
old-off-then-new-on and are refused and rolled back in the reverse order;
storage_audit.action refuses a value outside the enum (22P02) and an
orphan profile_id (23503); and an audit row cascades from a purged
profile.
Changelog
0.2.0 — storage profiles (breaking)
0.1.0 published with no dependents, so this release carries no
compatibility shim: no deprecated alias for saveConfig, no storageConfig
re-export next to storageProfile.
- Schema.
storage_configbecomesstorage_profile, many rows per deployment, withtenant_id,label,is_defaultandretired_at; the status enum isstorage_profile_status; a new checkstorage_profile_retired_default_ckand a new partial unique indexstorage_profile_default_per_scopeoncoalesce(tenant_id, '')whereis_default.storage_audit.config_idbecomesprofile_id, andstorage_audit.actionbecomes the closed enumstorage_audit_action(create,rotate_credential,set_default,retire,seed) — there is noupdateaction any more. - Engine.
saveConfig/switchGuard/confirmSwitchare replaced bycreateProfile+setDefaultProfile(add-and-set-default, never guarded) andretireGuard/retireProfile(the one guarded operation).rotateCredential,getStatusandcheckStandingtake aprofileIdortenantId. New:listProfiles,adapterFor(profileId)(cached),defaultAdapter(tenantId?),RECOMMENDED_PROFILE_COUNT_WARNING,StorageEngineOptions.tenantId. There is noupdateProfile: a profile's location is immutable. Error codes:switch_requires_confirmationis gone;cannot_retire_defaultandretire_requires_confirmationare new.SWITCH_GUARD_LIST_LIMITisRETIRE_GUARD_LIST_LIMIT. - Port.
ConfigStore(three methods, one row) becomesProfileStore(six methods, many rows, one default per scope);InMemoryConfigStorebecomesInMemoryProfileStore;StorageConfigRow/ConfigRecordbecomeStorageProfileRow/ProfileRecord;SwitchGuardResultbecomesRetireGuardResult. - Package.
drizzle-ormpeer floor>=0.45.2 <1.0.0.HANDOVER.mdships in the package, copied at build time from the maintainers' handover brief.
Migrating a 0.1.0 database, should one exist. No host is known to have
shipped 0.1.0; this is the shape a Postgres host's own drizzle-kit
generate should produce, written down so the decision inside it is made
explicitly rather than mechanically:
ALTER TABLE storage_config RENAME TO storage_profile;
ALTER TYPE storage_status RENAME TO storage_profile_status;
ALTER TABLE storage_profile RENAME CONSTRAINT storage_config_preset_ck TO storage_profile_preset_ck;
ALTER TABLE storage_profile RENAME CONSTRAINT storage_config_db_credential_ck TO storage_profile_db_credential_ck;
ALTER TABLE storage_profile
ADD COLUMN tenant_id text,
ADD COLUMN label text,
ADD COLUMN is_default boolean NOT NULL DEFAULT false,
ADD COLUMN retired_at timestamptz;
UPDATE storage_profile SET is_default = true; -- the one row 0.1.0 could ever have held becomes the scope's default
ALTER TABLE storage_profile
ADD CONSTRAINT storage_profile_retired_default_ck CHECK (retired_at IS NULL OR is_default = false);
CREATE UNIQUE INDEX storage_profile_default_per_scope ON storage_profile (coalesce(tenant_id, '')) WHERE is_default;
ALTER TABLE storage_audit RENAME COLUMN config_id TO profile_id;
CREATE TYPE storage_audit_action AS ENUM ('create', 'rotate_credential', 'set_default', 'retire', 'seed');
-- Every 0.1.0 'create' and 'rotate_credential' row maps unchanged. A historical 'update' row
-- ("the same row, re-saved", including a confirmed switch) has no equivalent in the closed 0.2.0
-- set; it is mapped to 'rotate_credential' as the nearest surviving concept rather than
-- discarded, and this comment is the record of that judgement. Its `note` (a switch
-- confirmation) and `before.outgoingObjectCount` survive as written.
UPDATE storage_audit SET action = 'rotate_credential' WHERE action = 'update';
ALTER TABLE storage_audit
ALTER COLUMN action TYPE storage_audit_action USING action::storage_audit_action;The foreign key storage_audit.profile_id -> storage_profile.id keeps its
ON DELETE CASCADE; a host's own attachment table gains
storage_profile_id (ON DELETE RESTRICT) and tenant_id, and every
existing attachment row is backfilled with the one profile id the table now
holds.
0.1.0
First public release: the StorageAdapter contract, the local filesystem,
S3-compatible and Azure Blob adapters, the default factory, key rules, the
canary check, credential custody, the credential-mode ordering, the
single-row storage_config / storage_audit fragment and the wizard engine
over it.
