@speles7172/backup-bridge

Setting this up end to end — tables, infrastructure, schedule, querying, restore — is docs/BACKUP.md.

The two Lambda handlers behind @speles7172/backup-client, deployed by infra/modules/backup.

EventBridge ──hourly──▶ runner ──▶ Postgres ──▶ S3 (Parquet)
                          │
                          └──▶ athena bridge ──▶ Glue / Athena

| Entry | Handler | What it is | |---|---|---| | ./runner | dist/runner-handler.handler | Decides whether a backup is due, takes it, registers the tables, prunes, reaps | | ./athena | dist/athena-handler.handler | Issues the catalog DDL from outside the VPC |

Why there are two

The runner has to be in the VPC to reach the database. A Lambda in private subnets cannot reach Athena without a NAT gateway or an interface endpoint — so the DDL goes through a second function that is not in the VPC. If your database is publicly reachable, leave ATHENA_BRIDGE_FUNCTION_NAME unset and the runner talks to Athena directly.

Configuration

Deployment shape is environment; the schedule is not. It lives in backup_settings so an operator can change it from @speles7172/backup-console — a schedule that needs a deploy to change is a schedule nobody can change at 2am.

| Variable | | |---|---| | BACKUP_BUCKET | Required. Where backups are written | | BACKUP_ROOT | Key prefix. Empty is a real answer — one application per bucket | | DB_SECRET_ARN | Secret holding the database credentials | | DB_HOST, DB_PORT, DB_NAME | Where the database is, when the secret does not say. An RDS-managed master password produces a secret of only {"username","password"}, so these are required with one | | BACKUP_TABLES | Comma-separated. Empty discovers every base table in public | | APP_FILES_BUCKET | The files half. Absent means database runs only | | GLUE_DATABASE | Absent skips catalog registration entirely | | ATHENA_BRIDGE_FUNCTION_NAME | Absent talks to Athena directly | | ATHENA_WORKGROUP, ATHENA_OUTPUT_LOCATION | Unless the workgroup sets its own |

Invoking it

{ "scheduled": true }        // the hourly tick — usually decides to do nothing
{ "store": "db" }            // a manual run from the console
{ "store": "s3" }

Things that will come up

The tick is gated on the last successful run, per store. A run that failed or is still going occupies the slot without producing a backup, so gating on the newest run of any status is how a failing backup silently stops retrying. Each store is asked about separately for the same reason: db succeeding says nothing about whether the files were copied, and one answer for both leaves the store that failed unretried until some later slot.

A run that could not export every table is an error, not a success. backup() resolves rather than throwing when individual tables fail — it writes the manifest and marks it error — so the manifest, not the absence of an exception, decides. A partial run must not claim the slot; the gap is otherwise invisible until a restore needs the missing table.

A snapshot whose Athena registration failed is still a snapshot. The data is in S3 and a restore from it works, so the run is recorded success and carries a catalog-pending note instead. Later ticks re-apply the catalog from that run's manifest — two DDL statements — and clear the note. Marking the run error would hide a usable backup from the console and make the next tick take a whole fresh dump to fix a CREATE TABLE.

Reconciliation runs after the backups, and is skipped only when the tick itself issued the DDL — not merely when a db run was scheduled. Those differ exactly when the scheduled run fails: it registers nothing, so gating on the schedule would strand the older pending snapshot for as long as the backup keeps failing, which is when an operator most needs to query it.

The run row is written before the backup starts. Recording only on success makes a killed runner indistinguishable from a run that never happened — and a killed runner is the common failure, because it never reaches its own error handler.

The reaper runs before the gate, on every tick, including when backups are switched off. A manual run whose Lambda died still needs marking, and it should not wait for somebody to re-enable the schedule. Its threshold is 20 minutes against a 15-minute function ceiling, so a run finishing near the limit is never reaped out from under itself.

Retention prunes after the backup, never before. Pruning first deletes the oldest run to make room for one that then fails, leaving fewer backups than there were when the tick started. It is also clamped to what the bucket's lifecycle will honour — a longer setting is a promise S3 will not keep.

One store failing does not stop the other. They are separate runs for separate reasons; a slow file copy should not cost a database backup that already succeeded.

The Athena bridge reports failures in band as { error } rather than as a Lambda FunctionError. A statement Athena rejected and a bridge that is down are different problems, and a caller that cannot tell them apart will retry the first one for ever.

Nothing is created at module load. A Lambda that throws while initialising fails every future invocation on the same container with an error naming the module rather than the cause — and this function spends most of its life deciding not to run.

What it does not do

• Restore. createRestorer is in backup-client; wiring it to an endpoint is the application's decision, because it overwrites a production database.
• Provision anything. See infra/modules/backup.
• Create its own tables. BACKUP_TABLES_SQL in backup-client is the DDL; where it lives and who may read it is the application's call.