o2h

Introduction

o2h stands for [JavaScript] Object to (2) HTML, which then needs to be hydrated in the browser's main thread ("H2O", get it?).

Careful performance measures show that HTML is the fastest format for a web page to display content. It is the native format the browser knows how to render right away. The ability to display the content as it streams in is something the Web achieved way back in 1994 and was instrumental to the web's success. With the advent of new api's designed to stream HTML fragments into portions of a web page via declarative partial updates, looking at ways we can architect applications to make use of these new techniques is of paramount importance. Particularly if the desire is to optimize the end user's experience on low-end mobile phones out in the field with spotty connections.

The biggest roadblock is that in today's environment, backend API's that provide HTML output is quite rare. JSON response is now the norm. Rewriting or enhancing these API's to provide HTML could be costly to maintain if each such write is done "manually".

So o2h provides a core solution to this problem -- it tries to reduce the cost by providing a predictable, easy, reusable, declarative pattern for transforming JSON API's into HTML API's with as much flexibility as possible, while adhering to data-driven configuration. It includes a strategy for hydrating back into objects in the web browser, that benefits from adopting a common declarative approach uniformly to all APIs.

o2h converts a JavaScript object (typically a JSON API response) into HTML, as described by a builtIns.tree transform configuration: the same JSON the client-side handler takes.

o2h will typically run on the server (e.g. in a Cloudflare Worker) or in a service worker, to generate HTML from an API in a form the browser can digest directly. It uses only assign-gingerly's pure (DOM-free) exports, and the same structural decisions builtIns.tree makes, so the server renders what assign-gingerly's Tree handler would render on the client.

import { o2h } from 'o2h';

const { html, unrendered } = o2h(config, apiResponse, { pipes, managers });

• html has the elements, attributes and text, plus Tree's <!--?start--> / <!--?end--> markers.
• Pending marks: each element that lost something HTML can't carry (properties without an attribute, event bindings, itemscope managers) is marked data-pending: "hide this until the client restores it" (see style below).
• unrendered lists keys o2h couldn't render, rather than guessing.

Where the first paint comes from

o2h is for the second and third rows; builtIns.tree hydrates in both, and is all you need for the first.

| Approach | Who renders the first paint | What the main thread does | Section | |---|---|---|---| | Main thread only | builtIns.tree, in the page | Everything: fetch, parse the JSON, render | builtIns.tree | | Server first paint | o2h on a server or edge worker, in the page's HTML | Hydrate (adopt) | Finishing on the client: hydration | | Service worker first paint | o2h in a service worker, for the page's own fetches | Assign the HTML, then hydrate | below |

Finishing on the client: hydration

builtIns.tree, with adopt: true and the same config and data, takes the server's elements over instead of rebuilding them:

await assignFromAsync(container, { '?. =>': { do: 'builtIns.tree', adopt: true, ...config } }, { from: data, pipes, registry });

It needs the config and the data on the client, e.g. embedded by o2h's embed option:

const data = JSON.parse(container.querySelector('script[type="application/json"][data-tree="tree"]').textContent);

Afterwards everything is live: events and properties are bound, managers are attached and re-render on ctx.rebind(), and the data-pending marks are removed. If the markup doesn't match the config, Tree rebuilds it instead.

How it works:

1. The walk: Tree walks the config and the existing elements together, level by level. It pairs each level's slots (its elements, and the <!--?switch--> placeholders of switches rendering nothing) with the specs in order. The pairing uses the same functions o2h used to decide what to render: which map keys, which repeat items (by key), and which switch case. Text and whitespace are skipped.
2. The bind: the ordinary bind then runs on the adopted elements. Text that's already right isn't rewritten.
3. The fallback: any mismatch (another tag, another number of children or items) rebuilds the whole tree instead.

Details: assign-gingerly's tree.md, "How the walk pairs spec and markup".

adopt: 'pending' (experimental) adopts the same way. On the first run, it binds only the elements marked data-pending (with their contents and ancestors, and nested trees' hosts), and trusts the rest of o2h's HTML as is. That halves hydration time or better on mostly static pages (see the benchmark below). The trade-off: anything the client would render differently from the server (newer data, another locale) is only corrected on a later run. See tree.md, "adopt: 'pending'".

Service worker first paint

A service worker renders a page's JSON fetches with o2h, off the main thread. The page gets HTML (with the JSON embedded after it), assigns it, and hydrates.

import { whenO2HWorkerReady, prepareO2HTransform, storeO2HTransform, O2H_HEADER } from 'o2h/worker/client.js';

await whenO2HWorkerReady('/o2h-sw.js');                          // registered as a module; resolves once it controls the page
const { key, transform, lost } = await prepareO2HTransform(config, { pipes, managers });   // the client-side config and options
await storeO2HTransform(key, transform);                               // key: a hash of the content

const response = await fetch('/api/launch', { headers: { [O2H_HEADER]: key } });
container.innerHTML = await response.text();                     // HTML, then <script type="application/json" data-tree="tree">
const data = JSON.parse(container.querySelector('script[data-tree="tree"]').textContent);
await assignFromAsync(container, { '?. =>': { do: 'builtIns.tree', adopt: 'pending', ...config } }, { from: data, pipes, registry });

How much the worker renders depends on the config:

• Data-only configs: the worker renders everything, and the page only binds what HTML can't carry (events, managers).
• Configs with code: the worker renders all it can, and leaves exactly the parts needing the page's code (pipes, managers) for the page to finish.

Data-only configs

The worker itself:

• Marked requests only: the worker handles fetches with an X-O2H-Transform header, and lets everything else through. It forwards the request without that header, so a cross-origin API never sees it, and it causes no CORS preflight.
• The transform ({ config, options }, in IndexedDB) is keyed by the header's value, so one worker serves very different API calls. The page writes transforms; the worker only reads them.
• The response: o2h(config, json, { ...options, embed: true }) as text/html; charset=utf-8, with Vary: X-O2H-Transform.
• Falling back: when the API's response isn't OK, or isn't JSON, it's returned as is. When the transform is missing or rendering fails, the original JSON is returned, with an X-O2H-Error header saying why.
• The worker file: workers can't use import maps, so o2h-sw.js is a bundle, built from worker/serviceWorker.ts with npm run build:worker (esbuild). Serve it from a path whose scope covers your pages, e.g. the site root.
• Shared with other workers: worker/core.js (handleO2HRequest(request, lookup)) uses only the Fetch API and o2h, so a Cloudflare Worker can use it with its own lookup: o2h-edge does.

A config and options with no functions or classes (tree notation, aka, markerName, snippets, style, and data pipes) are stored as they are, and the worker renders all of it.

Configs with code: stand-ins and pending parts

Transforms live in IndexedDB, which can't hold functions or classes. prepareO2HTransform(config, options) copies the client-side config and options:

• Stand-ins: each pipe, protocol, manager class or other piece of code becomes a stand-in ({ $o2hLost, name }).
• What was lost comes back as lost, with paths.
• The key is derived from the content: the same config gets the same key, and a changed one gets a new key.

In the worker, stand-in pipes and protocols throw a LostReference when used, so o2h leaves exactly what needs them to the page:

• A pattern key that needs one isn't rendered, and its element is marked data-pending.
• A switch whose case needs one becomes <!--?switch pending-->.
• A repeat whose items need one is left empty, with its container marked data-pending.
• A manager whose class is lost renders from the raw value, marked data-pending.

The page then hydrates with the real code, with adopt: 'pending' (or true). builtIns.tree adopts everything else, binds the pending parts, and renders the pending switches and repeats fresh. The result is the same DOM as a main-thread-only render; more of it is just finished by the page.

Data pipes survive

Data pipes (assign-gingerly's dataPipes: Intl formatters, lookups, built-ins, chains and aggregates, described as data) aren't lost. Their descriptors are stored, and the worker runs them. So the more of a config's pipes are data pipes, the more of the page the worker renders.

What travels is the descriptor, not a function. A descriptor such as { "sum": "?.quantity" } is plain data, so it can sit in a JSON file, and a pipes option may hold descriptors, functions, or both, everywhere. So an app's JSON holds descriptors under pipes, and neither the app nor o2h has to convert them:

{ "options": { "pipes": { "usd": { "intl": "NumberFormat", "locale": "en-US", "options": { "style": "currency", "currency": "USD" } } } } }

Here's what happens to usd on its way through a page and a worker:

| Step | Where | usd is | | |---|---|---|---| | The app | app.json | a descriptor | Plain JSON | | Preparing the transform | prepareO2HTransform (worker/transform.ts) | a descriptor | Plain objects are copied as they are, so nothing is lost | | Stored | the worker's KV, or IndexedDB | a descriptor | | | Rendering in the worker | assign-gingerly's path resolution | a function | Made when a path first needs it. worker/core.ts imports assign-gingerly's pipes/dataPipes.js so that it's ready: o2h() is synchronous. | | Hydrating on the page | assign-gingerly's assignFromAsync | a function | Made the same way. assignFromAsync loads the module when it sees a descriptor. |

The worker and the page each make their own functions from the same data. The functions themselves never travel. How the conversion works, and what it costs, is in assign-gingerly's Data pipes.

A page that writes its pipes in JavaScript can pass descriptors the same way, or call dataPipes itself. Each function that returns carries its descriptor (under a registered symbol), and prepareO2HTransform copies that descriptor in place of the function, so from there the table above applies:

const pipes = dataPipes({ usd: { intl: 'NumberFormat', locale: 'en-US', options: { style: 'currency', currency: 'USD' } } });

Only a pipe that's really code stops the trip. A function without a descriptor, such as s => s.trim() or an app's { "import": "./pipes.js" } once it's loaded, becomes a stand-in, and what uses it is left to the page. Both kinds can share one pipes object. The stubs a worker uses for lost pipes are functions too, so they sit beside the descriptors, and a chain reaching one leaves its part to the page.

Calling o2h() directly with descriptors needs import 'assign-gingerly/pipes/dataPipes.js' first, since it's synchronous and can't load the module itself.

Edge first paint: o2h-edge

o2h-edge runs the same worker core as a Cloudflare Worker. The page stores a transform there, and asks it to call an API and render the JSON. renderViaEdge does that whole round trip:

import { renderViaEdge } from 'o2h/worker/edge.js';

const options = { pipes, style: true };   // one object: the transform is prepared once per config and options
const { by, data, elapsed, pending, lost, error } = await renderViaEdge(container, {
    worker: 'https://o2h-edge.<your-subdomain>.workers.dev',
    api: 'https://fakestoreapi.com/products',
    config,
    options,
    hydrate: { registry },                 // more assignFrom options for hydrating, if needed
});

• Stores the transform (prepareO2HTransform) with the worker, once per worker. If the worker answers that it has no such transform (for example, a restarted wrangler dev), it stores it again and retries once.
• Renders and hydrates: the worker's HTML goes into the container, and builtIns.tree hydrates it with adopt: 'pending' and the page's real pipes, finishing what the worker left pending (pending counts those elements).
• Falls back: when the worker passes the JSON back (X-O2H-Error), the page renders it itself, and by is 'page', with the worker's reason in error.
• Throws when the worker can't be reached, refuses the request (its guards, its rate limit), or the API fails.

The steps after the worker answers are exported on their own too, for pages that fetch the HTML some other way:

• embeddedO2HData(container, config): the data embedded after the HTML;
• hydrateO2H(container, config, data, options?, hydrate?): hydrates it (adopt: 'pending'), returning { pending };
• renderO2HOnPage(container, config, data, options?, hydrate?): renders the data on the page instead.

o2hRenderer(hydrate?) bundles the three, plus prepareO2HTransform, in the shape navigate-gingerly's mirror takes. mirror routes a whole app through o2h-edge, path by path:

import { mirror } from 'navigate-gingerly/mirror.js';
import { o2hRenderer } from 'o2h/worker/edge.js';

mirror({ worker, app, into: document.querySelector('main'), renderer: o2hRenderer() });

Options

| Option | | |---|---| | pipes | Named unary functions usable as path segments ('?.date_utc?.parseDate?.longUtc'), or descriptors of them | | aka | Aliases, under the config's own aka | | protocols | Synchronous protocol handlers (name://key values) | | snippets | Static HTML for { snippet: 'name' } specifications, by name: written as is (trimmed), with the data-hole elements the spec's holes name rendered like elements (needs well-formed hole markup). The same option builtIns.tree takes, which also checks it on the client (see tree.md, Snippets). | | managers | Itemscope manager classes by registered name. With one, a { tag, ish: 'name' } element's subtree renders from the manager's getters. Either way, the element gets itemscope="name", so the client can attach the manager. | | style | Prepend the default style, which hides pending parts, only when scripts run (so crawlers and no-JS readers see everything): <style>@media (scripting: enabled) { [data-pending] { visibility: hidden; } }</style>. Also exported as PENDING_STYLE. | | embed | Append the data the HTML was rendered from, as JSON, right after the tree: <script type="application/json" data-tree="<markerName>">, with < escaped so the data can't end the element. The client hydrates from it, so the data is always the snapshot the HTML came from. By default, values JSON drops or changes (functions, Dates, Maps and Sets, class instances, NaN, BigInts) are listed in the result's notSerializable, with their paths; embed: { check: false } skips that check, which costs about 3× a plain JSON.stringify. The data goes after the content, so it doesn't delay the first paint of what comes before it when the page streams. |

Types: types/o2h/types.d.ts.

Demo

demo/index.html renders demo/treeDefinition.js, a copy of assign-gingerly's tree-demo definition, against demo/data/launch.json. It shows o2h's HTML next to what builtIns.tree builds in the browser from the same definition, checks that the two are the same DOM, and finishes the server HTML with Hydrate (builtIns.tree adopts the server's elements; then the rocket card's "Increase by one" re-renders it). With style: true, the pending parts (the links bar, the rocket card) stay hidden until then.

demo/benchmark/adopt.html times hydration with adopt: true against the experimental adopt: 'pending' (bind only what's marked), on the rocket definition, a mostly static company profile (companyDefinition.js), and a directory of 100 profiles. It also checks that both give the same DOM and that the managers stay live.

Viewing Locally

1. Install git and node.js.
2. Fork/clone this repo, and open a command window in its folder.
3. git submodule update --init --recursive
4. npm install
5. npm run serve
6. Open http://localhost:8000/demo/ in a modern browser.

Within the beGingerly workspace

When this repo sits in a folder next to assign-gingerly (as in beGingerly/), the workspace folder can serve both, so the demo and tests use assign-gingerly's current source instead of the published package:

1. In the workspace folder: npm install, then npm run serve (port 8000; or set PORT).
2. Open http://localhost:8000/o2h/demo/

Pages get their import map from <!-- #include virtual="/imports.html" --> (spa-ssi). Served from this repo, that's imports.html (published packages). Served from the workspace, it's the workspace's imports.html (sibling repos).

Building and testing

• Build: npm run build (tsc), or npm run watch. The service worker bundle: npm run build:worker. In VS Code, Ctrl+Shift+B offers tsc: watch - tsconfig.json.
• Test: npm test (Playwright).
  • Unit tests run o2h in Node, with no DOM, as on a server.
  • The equivalence test checks, in Chromium, Firefox and WebKit, that o2h's HTML matches what builtIns.tree builds from the demo definition (apart from the data-pending marks and the managers' itemscope).
  • The hydration test checks that builtIns.tree (adopt: true) adopts o2h's HTML: every element reused, the same DOM as Tree's own, and the rocket card re-rendering on "Increase by one".
  • Inside the workspace, the browser tests are served from the workspace folder (port 8090), so they run against the sibling assign-gingerly. Otherwise they use the published package.

The previous implementation is kept in legacy/ for reference.