@archival/carrier

TypeScript types for archival carriers — server side functions deployed alongside your site, one per directory under carriers/.

cd carriers/my-carrier
npm install --save-dev @archival/carrier

Typing a carrier

A carrier default-exports a function of (params, body, objects, site), and says so in its package.json:

{
  "type": "module",
  "archival": { "carrier": 2 },
  "devDependencies": { "@archival/carrier": "^1.2.0" }
}

import type { Carrier } from "@archival/carrier";

const carrier: Carrier = async (params, body, objects, site) => ({
  hello: params.get("name"),
  site: site.url,
});

export default carrier;

params is the request's query string and body its parsed body. objects is your site's objects, and site is what archival gives the carrier besides: the site's url, its uploads and its email.

Return an object to send JSON, or a string to send text/plain. A string prefixed with redirect: becomes a 302 to the rest of the string. Return a Response to send anything else.

Carrier API versions

What archival hands a carrier is versioned, so it can change without breaking a carrier written against an earlier version. "archival": { "carrier": 2 } in a carrier's package.json is what makes it version 2, which this package describes. A carrier that names no version is version 1 and always will be, so a carrier that never names one keeps working as it was written. A version that does not exist fails the deploy with a message naming the file.

Upgrading from version 1

Version 1 carriers are described by @archival/carrier@0. They receive (params, body, objects), with SITE_URL, UPLOADS and EMAIL merged into objects over any object of the same name. To move one to version 2:

1. Add "archival": { "carrier": 2 } to its package.json and install @archival/carrier@1.
2. Take a fourth argument, site, and read site.url, site.uploads and site.email where you read objects.SITE_URL, objects.UPLOADS and objects.EMAIL.
3. Run npx archival-carrier-types again.

objects then holds only your site's objects, so an object named SITE_URL, UPLOADS or EMAIL is no longer hidden.

What a carrier can import

A carrier is built the same way everywhere it runs: your site's deploy, and the preview in the Archival app. The build installs what your lockfile names, strips TypeScript's types, and links your files and your dependencies as the ES modules they are. Nothing is bundled and no build script is run.

That gives a carrier four rules:

• ES modules only. Your files and every package you depend on must use import and export. A CommonJS file (require, module.exports, .cjs) is refused, and so is a package that only publishes CommonJS.
• Web APIs only. fetch, crypto.subtle, URL, TextEncoder and the rest of the web platform are there. Node built-ins (node:fs, crypto, Buffer) are not, and importing one is refused.
• Dependencies need a lockfile. List them under dependencies in the carrier's package.json and commit its package-lock.json. Each package is downloaded from the tarball the lockfile records and checked against its hash. devDependencies are never installed, so that is where @archival/carrier and typescript belong. Install scripts do not run.
• TypeScript that erases. Types are removed, not compiled, which is the rule Node applies to .ts files: no enum, no namespace, no constructor parameter properties, and an import used only as a type must say import type. Setting erasableSyntaxOnly and verbatimModuleSyntax in your tsconfig.json makes your editor hold you to it.

Imports of your own files may name the file with or without its extension (./sheet, ./sheet.ts), and a .json file can be imported as its value.

When a carrier breaks one of these rules the deploy fails with a message that names the file or package, and the preview answers the request with the same message.

Reading uploads

site.uploads reads the files uploaded to your site. Reads are scoped to your site, and there is no way to write.

list() gives every upload, as the name it was uploaded with and the content hash it is stored under:

await site.uploads.list();
// [{ sha: "0c1f…", filename: "menu.pdf" }, …]

get() reads one, by name:

const upload = await site.uploads.get("menu.pdf");

It resolves to null when nothing matches. Names are not unique — the same name can be uploaded more than once, under different hashes — so looking one up by name alone throws when it is ambiguous. Pass the hash to say which you meant, or pass a file straight off objects, which carries its own:

await site.uploads.get("menu.pdf", "0c1f…");
await site.uploads.get(objects.settings.menu);

Serving one back is a Response, which is what lets a carrier put a file behind a check the site itself can't make:

const carrier: Carrier = async (params, body, objects, site) => {
  if (params.get("password") !== objects.settings.download_password) {
    return "redirect:/login";
  }
  const upload = await site.uploads.get("private-menu.pdf");
  if (!upload) {
    return "redirect:/404";
  }
  const headers = new Headers();
  upload.writeHttpMetadata(headers);
  headers.set("etag", upload.httpEtag);
  return new Response(upload.body, { headers });
};

Sending email

site.email.send sends mail from one of your site's own email addresses. It needs a plan that includes email and at least one address on one of your site's domains; add those in the site's email settings first.

const carrier: Carrier = async (params, body, objects, site) => {
  await site.email.send({
    to: body.email,
    subject: "Thanks for getting in touch",
    text: `We got your message and will reply within a day.`,
  });
  return "redirect:/thanks";
};

from defaults to the first address on your site's primary domain; pass one of your other addresses to send from that instead. A copy of every message lands in that address's Sent folder. to takes a single address or up to ten; text, html (at least one) and replyTo do what they say.

send resolves once archival has accepted the message, with an id for its logs and the from it resolved. Delivery happens afterwards - if it fails, the site's owner is emailed about it. It rejects when the message is not accepted: the site has no email, from is not one of its addresses, a recipient is malformed, or the site is over its hourly sending limit (100 messages).

Reading the fediverse

A site that publishes to the fediverse (an [activitypub] section in its archival_editor.toml) has followers there, and its posts get likes, boosts and replies. site.activitypub reads them:

const carrier: Carrier = async (params, body, objects, site) => {
  const post = objects.post.find((p) => p.path === params.get("post"));
  if (!post) {
    return "redirect:/404";
  }
  const [counts] = await site.activitypub.counts([post]);
  const replies = await site.activitypub.replies(post);
  return { counts, replies: replies.items };
};

A post is one of your objects, or its path ("post/hello-world"). counts takes up to 100 posts and answers in the same order. followers, likes, boosts and replies read a page at a time - pass the page's next as cursor for the following one - and account describes the site's own account, with its handle and follower count.

Replies are only the public ones. Their content is HTML reduced to paragraphs, line breaks, emphasis, lists, quotes, code and links, which is what Mastodon keeps of a post from another server; names and content warnings are plain text, so escape them.

A post's like count on the fediverse can include the likes it has on your site itself. Keep that count wherever your own like button stores it, and pass the whole of it to setLocalLikes each time it changes:

await site.activitypub.setLocalLikes(post, likes);

Other servers show the post's fediverse likes plus yours the next time they fetch it. counts and likes still read only the fediverse's.

A site that does not publish to the fediverse, and every preview and local run, reads as having no account and nothing on any list, and keeps no local likes.

Reading mentions from other sites

A site with a [webmention] section in its archival_editor.toml receives Webmentions: other sites tell it when one of their pages links to one of its own, and each link is verified before it is kept. site.webmentions reads them:

const carrier: Carrier = async (params, body, objects, site) => {
  const post = objects.post.find((p) => p.path === params.get("post"));
  if (!post) {
    return "redirect:/404";
  }
  const [counts] = await site.webmentions.counts([post]);
  const replies = await site.webmentions.list(post, { kind: "reply" });
  return { counts, replies: replies.items };
};

A page is one of your objects, or a site path ("post/hello-world", "about", and "" or "/" for the home page). counts takes up to 100 pages and answers in the same order with each page's replies, likes, reposts, bookmarks and plain mentions. list reads a page at a time, newest first, of one kind or every kind; pass the page's next as cursor for the following one.

A mention carries the page that links to yours (source), the entry's own url, its author's name, url and photo when the page says, its name, its content reduced the way a fediverse reply's is, and when it was published. Names are plain text, so escape them.

A site that does not receive, and every preview and local run, reads as one no mention has reached.

Storing data

site.sql is your site's own SQLite database, on the Pro plan. Every carrier on the site shares it, and it keeps what it holds between requests and deploys. On a plan without it, every call rejects.

const carrier: Carrier = async (params, body, objects, site) => {
  await site.sql.exec(
    "CREATE TABLE IF NOT EXISTS signups (email TEXT PRIMARY KEY, at TEXT NOT NULL)",
  );
  await site.sql.exec(
    "INSERT INTO signups (email, at) VALUES (?, ?) ON CONFLICT DO NOTHING",
    body.email,
    new Date().toISOString(),
  );
  const [{ count }] = await site.sql.exec<{ count: number }>(
    "SELECT count(*) AS count FROM signups",
  );
  return { count };
};

exec(sql, ...params) runs one statement and resolves to its rows, each keyed by column name, binding params to the statement's ? placeholders in order. transaction(statements) runs a list of [sql, ...params] statements in order and resolves to each one's rows:

await site.sql.transaction([
  ["UPDATE stock SET count = count - 1 WHERE sku = ?", sku],
  ["INSERT INTO orders (sku, email) VALUES (?, ?)", sku, body.email],
]);

Every call is one transaction: if a statement fails, the call rejects with SQLite's error and nothing it did is kept. Strings, numbers and null are stored as they are, booleans as 1 and 0, and bytes (Uint8Array, ArrayBuffer) as blobs, which read back as Uint8Array.

The Pro plan includes, each billing period:

| Usage | Included | Past it, at cost | | ------------ | ----------- | ------------------------ | | Calls | 360,000 | $0.45 per 1,000,000 | | Rows read | 360,000,000 | $0.01 per 10,000,000 | | Rows written | 360,000 | $0.01 per 10,000 | | Storage | 50 MB | $0.02 per 100 MB a month |

Usage past an allowance is added to the site's next invoice. With "Bill me for usage beyond my plan" turned off in the site's settings, the database refuses instead once an allowance is spent, until the period resets: every call for calls and rows read, writes for rows written, and growth for storage. A scan reads every row it passes, so index the columns your queries filter on.

Whatever the billing, these stop a runaway carrier, and a call that would go past one rejects:

| Ceiling | | | ------------ | ----------------------------------------------------------------------------------------------- | | Calls | 5,000 an hour. | | Rows read | 5,000,000 an hour. | | Rows written | 5,000 an hour. A call that would pass it is rolled back. | | Size | 500 MB. A call that would grow the database past it is rolled back; deleting rows always works. | | One call | 100 statements, and 5 MB of rows back. |

site.sql works in a deployed carrier. In the preview in the Archival app and under archival run, it rejects.

Running on a schedule

A carrier can also run on its own, on a schedule. Name it as a cron expression in the carrier's package.json, beside its carrier API version:

{
  "type": "module",
  "archival": { "carrier": 2, "schedule": "*/15 * * * *" }
}

and export scheduled from its main file:

import type { CarrierScheduled } from "@archival/carrier";

export const scheduled: CarrierScheduled = async (objects, site, event) => {
  const due = await site.sql.exec<{ email: string }>(
    "SELECT email FROM reminders WHERE at <= ?",
    event.scheduledTime,
  );
  for (const { email } of due) {
    await site.email.send({
      to: email,
      subject: "Your reminder",
      text: "It's time.",
    });
  }
};

scheduled receives your site's objects, the same site a request gets, and event: the cron it is running on and the scheduledTime it was due, in milliseconds since the epoch. A carrier can export a default function as well, to answer requests; one that exports only scheduled answers them with a 404.

A schedule has the five fields of a standard crontab - minute, hour, day of the month, month and day of the week - and is read in UTC. @hourly, @daily and the other nicknames work too. Five fields means nothing runs more often than once a minute: a sixth field, for seconds, fails the deploy.

• Each run counts as a request to your site, against the same allowance your visitors use. A site that has used its allowance with "Bill me for usage beyond my plan" turned off stops running its schedules too, until the period resets.
• A run happens at most once. A run that is missed, or that fails, is not retried; the next one happens on schedule. One that throws is logged with the carrier's other logs, in the site's dashboard.
• Only the deployed site runs schedules. The preview in the Archival app and archival run never call scheduled.

The deploy fails when a schedule is not a five-field cron expression, never runs (0 0 30 2 *), or belongs to a carrier on version 1 of the carrier API, and when a carrier names a schedule but does not export scheduled.

Typing your site's objects

objects is your site's archival objects. Every site's objects are different, so the types for them are generated from your archival_objects.toml:

npx archival-carrier-types

That runs archival types and writes an archival-objects.d.ts next to each carrier, for the carrier API version that carrier names. It refuses a carrier whose installed @archival/carrier describes a different version than the one its package.json names. Commit those files — they contain no object values, only your schema, and committing them means your editor and CI work on a fresh clone with no extra setup.

After generating, objects is fully typed:

const carrier: Carrier = async (params, body, objects) => ({
  titles: objects.posts.map((post) => post.title), // (string | null)[]
  contact: objects.settings.contact,
  hero: objects.posts[0].hero?.url,
});

Objects backed by a directory (objects/posts/*.toml) are arrays; objects backed by a single file (objects/settings.toml) are not. Unset fields are null. Child objects default to []. Every object read from its own file also carries path and order.

Until you generate, reading anything off objects is a compile error — deliberately, so a carrier can't silently read an object that isn't there.

You need the archival binary on your PATH or in your node_modules (npm install --save-dev archival, or cargo install archival).

Options

| Flag | | | ----------------------- | -------------------------------------------------------------- | | --check | Don't write; exit non-zero if anything is out of date. For CI. | | --carrier <name> | Only generate for one carrier. | | --carriers-dir <path> | Carriers directory, if not carriers. | | --site <path> | Site root, if not an ancestor of the working directory. | | --out <path> | Write a single file here instead of one per carrier. | | --archival <path> | Path to the archival binary. |

To keep generated types honest in CI:

npx archival-carrier-types --check

Note on secret fields

Fields declared secret are hidden from templates, but carriers run on the server and do receive their values. They are typed as string | null like any other string.

A secret published from the editor is stored encrypted in your site's repository, under a key that belongs to the site. Your carriers are deployed with the decrypted value, so objects holds what you typed, never the stored form.