lore-s3

Track S3 file history and lineage across workflow steps. Record what happened to every file, link derived files back to their sources, and replay failed steps — all stored directly in your S3 bucket.

Install

npm install lore-s3
# or
pnpm add lore-s3

Quick start

import { LoreS3 } from 'lore-s3';

const lore-s3 = new LoreS3({
  bucket: 'my-bucket',
  region: 'us-east-1',
});

// Run a tracked step
await lore-s3.run('uploads/report.pdf', {
  step: 'validate',
  input: { maxSizeBytes: 10_000_000 },
  handler: async (ctx) => {
    const file = await downloadFromS3(ctx.fileKey);
    if (file.size > ctx.input.maxSizeBytes) throw new Error('File too large');
    return { sizeBytes: file.size };
  },
});

// Run a step that produces a new derived file
await lore-s3.run('uploads/report.pdf', {
  step: 'convert-to-images',
  produces: ['renders/report-page-1.png', 'renders/report-page-2.png'],
  handler: async (ctx) => {
    const pages = await convertPdfToImages(ctx.fileKey);
    await Promise.all(pages.map((p) => uploadToS3(p.key, p.data)));
    return { pageCount: pages.length };
  },
});

// Get history
const history = await lore-s3.getHistory('uploads/report.pdf');
console.log(history?.entries);

// Traverse the full lineage of a derived file
const lineage = await lore-s3.getLineage('renders/report-page-1.png');
// lineage.parents[0].fileKey === 'uploads/report.pdf'

How it works

History is stored as JSON files in your S3 bucket under a .lore-s3/ prefix (configurable):

.lore-s3/
  uploads/report.pdf.json      ← history for report.pdf
  renders/report-page-1.png.json  ← history for derived file, with derivedFrom link

Each history file records:

• Step entries — every step run against the file, with status, input, output, timing, and errors
• Lineage refs — which upstream files and step entries produced this file

API

new LoreS3(options)

| Option | Type | Default | Description | |---|---|---|---| | bucket | string | required | S3 bucket name | | region | string | 'us-east-1' | AWS region | | historyPrefix | string | '.lore-s3' | Prefix where history files are stored | | s3Client | S3Client | — | Bring your own S3Client (useful for mocking) |

lore-s3.run(fileKey, options)

Run a handler and record the result as a history entry.

const result = await lore-s3.run('uploads/photo.jpg', {
  step: 'resize',
  description: 'Resize to 800px wide',
  input: { width: 800 },
  produces: 'processed/photo-800w.jpg', // or an array
  skipIfSucceeded: true,                // idempotent re-runs
  handler: async (ctx) => {
    // ctx.fileKey, ctx.bucket, ctx.step, ctx.entryId, ctx.input
    return { outputKey: 'processed/photo-800w.jpg' };
  },
});

// result.status: 'success' | 'failed' | 'skipped'
// result.entryId: string
// result.output: whatever your handler returned
// result.error: set when status === 'failed'

When produces is set and the step succeeds, lore-s3 automatically initialises history for each produced file and records that it was derived from the source file + this step entry.

lore-s3.getHistory(fileKey)

Returns the full FileHistory for a file, or null if not yet tracked.

lore-s3.getLineage(fileKey)

Recursively resolves the full ancestor tree of a file.

const node = await lore-s3.getLineage('renders/page-1.png');
// node.parents[0] → { fileKey: 'uploads/report.pdf', history, parents: [] }

lore-s3.getDescendants(fileKey)

Returns the file keys of all files directly derived from the given file.

lore-s3.getFailedSteps(fileKey)

Returns all HistoryEntry objects with status === 'failed' for a file.

lore-s3.replay(fileKey, entryId, handler)

Re-runs a specific step entry with a new handler. The replay is recorded as a new history entry linked back to the original.

const failed = await lore-s3.getFailedSteps('uploads/photo.jpg');
await lore-s3.replay('uploads/photo.jpg', failed[0].id, async (ctx) => {
  // same input as the original attempt is in ctx.input
  return { outputKey: 'processed/photo-800w.jpg' };
});

lore-s3.registerDerivedFile(derivedKey, sourceKey, sourceEntryId)

Manually link a file as derived from another. Called automatically by run() when produces is set, but available for files created outside of LoreS3.

lore-s3.clearHistory(fileKey)

Wipes all step entries for a file while preserving its lineage refs.

lore-s3.listTrackedFiles(prefix?)

Returns the file keys of all files lore-s3 has history for, optionally filtered by prefix.

lore-s3.listAllHistories()

Returns every FileHistory in the bucket. Use with care on large buckets.

Types

interface FileHistory {
  fileKey: string;
  bucket: string;
  derivedFrom: LineageRef[];   // upstream files this was derived from
  entries: HistoryEntry[];
  createdAt: string;
  updatedAt: string;
}

interface LineageRef {
  fileKey: string;   // source file
  entryId: string;   // step entry that produced this file
}

interface HistoryEntry {
  id: string;
  step: string;
  description?: string;
  status: 'running' | 'success' | 'failed' | 'skipped';
  input?: unknown;
  output?: unknown;
  error?: { message: string; name?: string; stack?: string };
  produces?: string[];
  startedAt: string;
  completedAt?: string;
  durationMs?: number;
  metadata?: Record<string, unknown>;
}

interface LineageNode {
  fileKey: string;
  history: FileHistory;
  parents: LineageNode[];   // recursively resolved
}

IAM permissions

LoreS3 needs the following S3 permissions on your bucket:

{
  "Effect": "Allow",
  "Action": [
    "s3:GetObject",
    "s3:PutObject",
    "s3:ListBucket"
  ],
  "Resource": [
    "arn:aws:s3:::my-bucket",
    "arn:aws:s3:::my-bucket/.lore-s3/*"
  ]
}

License

MIT