jobs

One scheduled GitHub Actions run executes every recurring job the wtfalch org has, instead of each repo paying its own.

Install

pnpm add @wtfalch/jobs

An app uses three pieces of the package: defineJobs() to declare its recurring schedule, createReceiver() to verify and run an incoming tick, and syncJobs() (deploy time, not shown) to push the declared schedule to the scheduler. Full recipe: .plans/2026-09-15-jobs-recipe.md.

// jobs.config.ts -- committed, reviewed, synced on deploy
import { defineJobs } from '@wtfalch/jobs';

export default defineJobs({
  'nightly-digest': { every: '0 4 * * *', run: '/api/jobs/digest' },
});

// the receiving route this schedule POSTs to
import { createReceiver } from '@wtfalch/jobs';

const receive = createReceiver({
  secrets: [process.env.JOBS_SIGNING_SECRET!], // old + new, mid-rotation
  store: myIdempotencyStore, // alreadyHandled(id) / remember(id)
  async handle(delivery) {
    if (delivery.schedule === 'nightly-digest') await runDigest();
  },
});

export const POST = (req: Request) => receive(req);

createReceiver verifies the Jobs-Signature header, checks deliveryId against your store, and runs handle at most once per delivery that store has not already seen -- delivery itself is at-least-once, so handle must be safe to run twice for the same id. A project's baseUrl, syncKey and signing secret come from scripts/register-project.ts, run once against the scheduler (see "Registering a project" below).

GitHub bills Actions per job, rounded up to the whole minute. A poller that does five seconds of work still costs a minute, and twenty such jobs cost twenty minutes per tick. Run them all inside one job and the tick costs one minute however many jobs it carries -- the bill stops scaling with the number of jobs.

Running a tick

GITHUB_TOKEN="$(gh auth token)" node src/tick.ts

No install, no build. Node runs the TypeScript directly, so a tick is checkout plus node. typescript and vitest are devDependencies for developing and testing; the tick never sees them.

Writing a job

A job is an object with a name and a run that throws when the work failed. It takes fetch, now and log from its context instead of the globals, so a test can drive it with no network and no clock.

export const sweep: Job = {
  name: 'sweep',
  description: 'Deletes preview dirs whose PR is closed',
  timeoutMs: 30_000,
  async run(ctx) {
    ctx.log('...');
    if (bad) throw new Error('why');
  },
};

List it in src/jobs/index.ts. A job runs when, and only when, it is listed there.

Rules a job must keep:

• No runtime dependency. fetch and the Node standard library only. A job that needs a package belongs in its own workflow, not in the tick.
• Never sleep. Sleep is billed. More frequent checks come from a shorter cron, never from a loop with a wait in it.
• Throw to fail. The runner catches it, pings the failure heartbeat and keeps the other jobs going.

Isolation

Jobs run concurrently, each with its own timeout and its own catch. One job throwing or hanging cannot stop the rest. A job that passes its timeout is abandoned rather than cancelled -- Node cannot kill work it started -- so tick.ts ends with an explicit process.exit and the runner never bills minutes for a job that will not finish.

Heartbeats

One heartbeat for twenty jobs hides which one died, so every job gets its own check. sweep reads its URL from HEARTBEAT_SWEEP, actions-budget from HEARTBEAT_ACTIONS_BUDGET. A job that succeeded pings its URL; a job that failed pings <url>/fail, so the monitor alerts now rather than after a grace period. A job with no heartbeat variable set says so and runs anyway.

An unreachable monitor never fails a job. The monitor going quiet is already the alert.

The scheduler

Separate from the tick above, and the thing the design in .plans/2026-09-15-jobs-recipe.md is actually about. src/scheduler/ holds the three tables and the claim loop: SELECT ... FOR UPDATE SKIP LOCKED with a fenced lease, which reporting, valet and activity each arrived at independently.

Its tests need a real Postgres, because SKIP LOCKED and lease expiry mean nothing against a fake. pnpm test starts one in Docker and passes its URL in.

pnpm test

The two pieces are not alternatives. The tick suits repo chores that need a git checkout on an hourly or daily cadence. The scheduler suits application jobs, because GitHub bills a whole minute per tick however many jobs it carries, so a per-minute cadence is unaffordable there at any job count.

Delivery history

Every attempt to deliver a schedule writes one row to the deliveries table: when it started and finished, what happened (ok, retry, gave_up or abandoned), the HTTP status if there was one, and a truncated error if not. This is how a project tells its own schedule stopped firing without asking us (.plans/2026-09-15-jobs-recipe.md, "Watching the watcher").

A project reads its own history, and only its own, from the scheduler's GET /deliveries, authenticated with its sync key exactly like POST /sync and POST /timers:

GET /deliveries?name=digest&limit=50&before=2026-09-20T00:00:00Z
Authorization: Bearer <syncKey>

name (exact match), limit (default 50, max 500) and before (an ISO timestamp, for paging backwards) are all optional. Results come back newest first. From @wtfalch/jobs:

const entries = await jobs.history({ name: 'digest', limit: 20 });

History is kept 30 days, then purged. The same sweep also deletes a finished one-shot timer (source = 'timer', state = 'done') once it is 30 days past its last run — a declared schedule is never touched by age, only by dropping it from jobs.config.ts. The scheduler process runs this at most once an hour; a failed sweep is logged and never fatal.

A cross-project health view, for ops

GET /deliveries above is scoped to one project's own sync key on purpose — a project reads its own history, and only its own. That is the wrong shape for operator, which wants one pane of glass over which tenant apps' schedules are failing across the whole scheduler, not a poll per tenant with a key it would have to hold for every one of them. GET /admin/health answers that instead, authenticated with a separate operator credential, not any project's sync key:

GET /admin/health
Authorization: Bearer <SCHEDULER_ADMIN_TOKEN>

{ "projects": [
  { "name": "lokessmie", "alerting": false, "alertFailingSince": null,
    "pendingSchedules": 4, "runningSchedules": 0, "failingSchedules": 0 },
  { "name": "baxter", "alerting": true, "alertFailingSince": "2026-09-22T10:00:00Z",
    "pendingSchedules": 2, "runningSchedules": 1, "failingSchedules": 1 }
] }

Every project appears, including one with no schedules at all, ordered by name. alerting/alertFailingSince mirror the project-scoped alert (docs/decisions.md #9); failingSchedules counts rows that exhausted their attempts and are not coming back on their own — a one-shot's permanent end state, or a cron schedule between a give-up and its next occurrence's own claim, ordinarily near-instant. Set SCHEDULER_ADMIN_TOKEN on the scheduler process to enable this route; unset, it refuses every request with 401, exactly like a wrong token, rather than a distinct "not configured" error.

A schedule's own delivery timeout

Every delivery has a timeout: the scheduler aborts the fetch if the receiver hasn't answered within it, and treats that abort like any other failure — retried, then given up on, same as a non-2xx status. The default is 30 seconds, too short for a receiver doing real work — a nightly export, a bulk delete. timeoutMs raises it per schedule:

defineJobs({
  export: { every: '0 2 * * *', run: '/api/jobs/export', timeoutMs: 300_000 },
});

await jobs.at(when, '/api/jobs/export', payload, { timeoutMs: 300_000 });

timeoutMs is an integer number of milliseconds, 1_000..900_000 (1 second to 15 minutes). An out-of-range value is refused at defineJobs() or jobs.at() call time, and again by POST /sync/POST /timers for a caller that talks to the scheduler directly instead. Omitted, a schedule keeps today's 30-second default.

The claim lease that holds a schedule while it is being delivered stretches automatically to cover whatever timeout the row declares, plus a margin for the settle write that follows — nothing needs configuring for this, and a long-running delivery is never re-claimed by a second worker mid-flight.

A long timeout does not extend the shutdown grace. This repo does not configure Coolify's (or any platform's) container stop grace period. On SIGTERM, main.ts drains the claim loop and aborts whatever delivery is already in flight regardless of that delivery's own timeoutMs — a rolling deploy cuts a 15-minute delivery short exactly the same way it already cuts a 30-second one short, and the receiver sees an ordinary aborted attempt, retried on the schedule's next occurrence like any other failure.

Giving up, and alerting on it

A schedule's receiver gets DEFAULT_MAX_ATTEMPTS (5) tries, backing off 1s/2s/4s/8s between them, before the scheduler gives up on that occurrence. What happens next depends on the kind of schedule:

• A one-shot (a run_at schedule, or a timer filed by jobs.at()) ends failing for good. There is no later occurrence to come back at; redeclare it (change its run_at and re-sync, or file a fresh timer) to run it again.
• A cron schedule goes back to pending, attempts reset to zero, at the cron's next occurrence after the give-up — it is never lost. Before this existed, a cron schedule gave up exactly the same way a one-shot does, and the only way back was a resync that happened to change its cron or run_at; an app whose declared schedule never changes never re-syncs it, so a receiver down for about 15 seconds when its daily job was due lost that schedule silently, forever. A give-up is still recorded in delivery history with outcome gave_up, exactly as before — GET /deliveries shows it either way.

Either kind of give-up can also alert. Give scripts/register-project.ts a third argument, or set one later with scripts/set-alert-url.ts, and that project gets a POST <url>/fail the moment any of its schedules gives up, and a POST <url> (ok) the next time any of its schedules delivers successfully afterwards — never on an ordinary success with nothing to recover from. This is src/heartbeat.ts's own ping(), the same shape the scheduler process's own heartbeat uses, fire-and-forget with a 5-second timeout: a slow or unreachable alert endpoint never delays a settle, and can never turn a real success or give-up into something else.

Using a Healthchecks.io check (or similar) as the alert URL: this only ever receives a /fail ping (on a give-up) and an ok ping (on the next recovery) — never one ping per successful delivery, the way a schedule's own period would suggest. A Healthchecks check that receives no ping for period + grace reports itself down on the absence of a ping, which this integration does not produce on a healthy day. Set the check's period to something at least as long as the project's longest schedule interval (so an ordinary healthy run never trips the absence timeout on its own) and rely on the explicit /fail ping for the actual signal; or, simpler, set the period to its maximum and treat the check as fail-only, accepting that it will not also notice "the scheduler process itself vanished and never got the chance to call /fail at all" — the scheduler's own process heartbeat (HEARTBEAT_SCHEDULER, above) is what already covers that failure mode.

Cancelling a timer

jobs.at() now resolves to { name }, the full name the scheduler stored the timer under. Keep it if the timer might need cancelling later:

const { name } = await jobs.at(when, '/api/jobs/remind', { userId });
// ...later, the user cancels their own reminder:
const cancelled = await jobs.cancel(name);

jobs.cancel(name) DELETEs the scheduler's DELETE /timers/:name (the name URL-encoded into the path), authenticated with the sync key exactly like the other routes:

DELETE /timers/one-shot%3A9c2c...
Authorization: Bearer <syncKey>

It resolves true only when this call is the one that removed the row, and false for every reason it might not have — the name is unknown to this project, it names a declared schedule rather than a timer, it is already running (claimed by the claim loop), or it already finished or gave up. Those cases are deliberately not told apart: on the wire, all of "no such timer," "that's someone else's timer," and "that's a declared schedule's name" come back as a 404, and "it's running" or "it already settled" both come back as a 409, so a client can never use this route to probe which names exist. jobs.cancel() only ever throws for a wrong sync key, the scheduler being unreachable, or an unexpected response — never for a timer that simply wasn't cancellable.

Running locally without a scheduler

jobs.at(), jobs.cancel(), jobs.history() and syncJobs() all need a scheduler to talk to — but a developer running the app locally, or a CI job running its test suite, usually doesn't have one. With no scheduler base URL configured at all (no options.baseUrl and no JOBS_SCHEDULER_URL) and NODE_ENV is not exactly the string "production" — unset, "development", "test", anything else at all counts — all four calls skip the network: each logs one line at console.warn saying what it would have done (never the payload's contents, never the sync key) and resolves as if it had succeeded —

• jobs.at() still mints and returns { name }, so code that stores it for a later jobs.cancel() keeps working.
• jobs.cancel() resolves false.
• jobs.history() resolves [].
• syncJobs() resolves { applied: true, skipped: true }.

In production, a missing URL is still a hard configuration error and every one of these throws exactly as it always has. And if a URL is configured but the sync key is missing, that is unchanged too — it still throws, in every environment; the local-dev rule only ever looks at whether a URL is configured, never at the key.

Because the check is NODE_ENV !== 'production' rather than an allowlist of known-dev values, a real deploy that simply forgot to set JOBS_SCHEDULER_URL looks exactly like a healthy local run unless something downstream treats a skip as a failure. A deploy step calling syncJobs() should do exactly that — fail loudly instead of silently doing nothing:

const r = await syncJobs(defined, { baseUrl, syncKey, generation });
if (r.skipped) throw new Error('JOBS_SCHEDULER_URL missing');

Deploying

The scheduler applies schema.sql to its own database itself, at startup, before it starts accepting sync requests or claiming schedules (T8, docs/decisions.md) — a Coolify post-deployment command to apply it is no longer needed and should not be configured. It is safe for two containers of a rolling deploy to both start against the same database at once: the apply runs under a Postgres advisory lock, so only one of them does real DDL and the other finds the schema already there.

Registering a project

The database is private to the Coolify docker network the scheduler runs on, so scripts/register-project.ts cannot run from a laptop. Open a Coolify terminal on the jobs-scheduler container and run it there instead:

node scripts/register-project.ts <name> <base-url> [alert-url]

DATABASE_URL is already in the container's environment. The command prints the project's signing secret and sync key once; copy them into the app's own env now, neither is recoverable later in this readable form.

alert-url is optional — see "Giving up, and alerting on it" above. Add, change or remove it later without re-registering:

node scripts/set-alert-url.ts <name> <url|none>

Rotating a project's signing secret

A project has one signing_secret, and the scheduler signs every delivery with it. createReceiver's secrets option (src/client/receive.ts) already takes a list rather than one value, precisely so an app can accept two at once while a rotation is in progress. scripts/rotate-signing-secret.ts is the operator side of that: a leaked or ageing secret gets replaced with no downtime and no window where a delivery fails verification. Run it on the jobs-scheduler container, the same as register-project.ts.

1. Stage a new secret — minted the same way register-project.ts mints one. The scheduler keeps signing with the current secret; nothing about delivery changes yet.

   node scripts/rotate-signing-secret.ts <name> stage

   Prints the new secret once — copy it now, it is not shown again. Refuses, and prints how to cancel, if this project already has a staged secret: a second stage must never silently overwrite one an earlier stage's secret may already be deployed with.

2. Deploy the app accepting both secrets. This deploy is purely additive — the scheduler still signs only with the old one:

   const receive = createReceiver({
     secrets: [CURRENT_SECRET, NEW_SECRET],
     store: myIdempotencyStore,
     async handle(delivery) { /* ... */ },
   });

3. Commit. From this moment every delivery signs with the new secret:

   node scripts/rotate-signing-secret.ts <name> commit

   A delivery already in flight when this runs keeps the signature it was sent with. That's fine: the app deployed in step 2 still accepts both secrets until step 5 removes the old one, so that in-flight delivery still verifies.

4. Or cancel instead of committing, to back out with nothing changed:

   node scripts/rotate-signing-secret.ts <name> cancel

5. Remove the old secret from the app once the commit has been live long enough that nothing still in flight could have been signed with it, and redeploy with secrets: [NEW_SECRET].

stage, commit and cancel all refuse an unknown project name, and commit/cancel both refuse when nothing is staged — every refusal exits non-zero and prints no secret.

Registered jobs

| job | what it does | | ---------------- | --------------------------------------------------------- | | actions-budget | Fails when the org's Actions minutes this month cross 80% of the 3,000-minute allowance, and names the repos spending them. |

Not scheduled yet

There is no cron. The tick runs by hand until a second job is registered -- see docs/decisions.md for what is settled and what is still open, including where a job's code should live when it belongs to another repo.