@xyo-network/archivist-s3

An XYO framework archivist for S3-compatible object stores. The same S3Archivist serves AWS S3, compatible services, and Auto Drive. Runtime clients and credentials stay outside the serialized module config.

import { S3Client } from '@aws-sdk/client-s3'
import { S3Archivist, S3ArchivistConfigSchema } from '@xyo-network/archivist-s3'
import { asSchema, PayloadBuilder } from '@xyo-network/sdk-protocol'

const client = new S3Client({ region: 'us-east-1' })
const archivist = await S3Archivist.create({
  account: 'random',
  client,
  config: {
    schema: S3ArchivistConfigSchema,
    bucket: 'my-payloads',
    keyPrefix: 'application/tenant',
  },
})
const payload = { schema: asSchema('com.example.reading', true), value: 42 }
await archivist.insert([payload])
const hash = await PayloadBuilder.hash(payload)
const retrieved = await archivist.get([hash])
await archivist.stop()
client.destroy() // Borrowed clients remain caller owned.

Auto Drive

The /auto-drive entry point creates an ordinary S3Archivist configured with Auto Drive's endpoint, individual logical deletion, credential-safe errors, and a required admission policy. API keys must remain on the server.

import { createAutoDriveArchivist } from '@xyo-network/archivist-s3/auto-drive'
import { asSchema, PayloadBuilder } from '@xyo-network/sdk-protocol'

const archivist = await createAutoDriveArchivist({
  apiKey: process.env.AUTODRIVE_API_KEY!,
  bucket: 'my-payloads',
  keyPrefix: 'application/selected',
  admission: {
    allowedSchemas: ['com.example.reading'],
    maxPayloadBytes: 1024,
    schemaMaxPayloadBytes: { 'com.example.reading': 512 },
  },
})
const payload = { schema: asSchema('com.example.reading', true), value: 42 }
await archivist.insert([payload])
const hash = await PayloadBuilder.hash(payload)
const receipt = await archivist.getObjectReceipt(hash)
await archivist.stop() // The factory owns and destroys its client.

Admission rejects an entire mixed batch before any provider access. An empty allowlist admits nothing. Limits apply to UTF-8 JSON after storage metadata is removed; client metadata remains included. Per-schema limits only tighten the global limit. Direct insertion, signed insert queries, and parent-read caching all pass through the shared framework insertion boundary. Applications may filter batches before calling the archivist when intentional exclusion is their preferred behavior.

For multiple namespaces, createAutoDriveS3Client({ apiKey }) allows one shared client. Pass it to S3Archivist.create with provider: 'auto-drive', deleteMode: 'individual', and a required admission config. The caller then owns the client. An archivist whose owned client has stopped must be recreated.

Storage and recovery

Within keyPrefix, primary JSON objects live at by-hash/{hash}, sequence keys at by-seq/{sequence}-{hash}, and data-hash aliases at by-data/{dataHash}. Bodies retain client metadata and exclude _hash, _dataHash, and _sequence; identity and ordering metadata travel in object metadata. Retrieval verifies canonical XYO hashes. A CID is a provider locator, not an XYO identity or evidence of completed network archival.

Repeated insertion checks exact local primary objects. It repairs missing indexes before reporting duplicates, without rewriting an existing primary body. A metadata-bearing payload's data-hash view does not establish that its data-only payload is stored as an independent primary object. Full enumeration and commit read primary objects, so incomplete indexes cannot hide persisted payloads.

Local storage mutations serialize per archivist instance. Parent calls and lifecycle events run after the local lock is released, so listeners may await further mutations. Separate instances or processes writing the same prefix require external coordination. S3 operations are not a transaction: a failed operation can have persisted part of its work. Auto Drive uses one transport attempt because retrying an ambiguous upload can create additional permanent storage. Reconcile by reinserting the same payload identity; do not assume a timeout means nothing was stored.

commit() captures a consistent local snapshot and copies it to each configured commit parent and verifies complete acknowledgment, including read-back of previously stored duplicates. It never clears local data, including after successful copies. With no commit parents it returns an empty receipt list. Parent failure rejects the commit and leaves local records available. This differs deliberately from transient archivists that flush and clear on commit.

Deletion on Auto Drive only removes logical provider access. It does not erase permanent network data. capabilities reports this distinction and whether bulk or individual delete operations are used. Costs, durable budgets, structural schema validation, and network archival monitoring belong to the integrating application and are not implied by a successful insert or returned CID.

Tests

Offline framework and provider qualification lives under src/spec. It covers canonical identities, direct and signed operations, admission, provider error redaction, client ownership, index recovery, concurrency, and commit behavior. Live Auto Drive qualification is kept in the consuming Aries application's separately invoked pnpm test:autodrive suite; ordinary SDK tests do not load an API key or upload permanent data.