@orchestral/patterns

Part of the Orchestral monorepo — see the repo README for how the packages fit together.

The first-party Pattern catalog for Orchestral — the patterns you register and dispatch through a runtime.

It ships two tiers, all building on @orchestral/core:

• Atomic patterns — one per capability: text-to-image, image-to-image, image-to-text, text-to-video, image-to-video, video-to-video, text-to-speech, text-to-audio, automatic-speech-recognition, text-generation, and more.
• Meta pipelines — multi-step compositions with inlined prompts: best-of-N image selection, storyboarding, script-to-video, four short-form deliverables (product ad, UGC testimonial, explainer short, product photo pack), the caption → re-render image-edit fallback, and meta_plan — the one-shot plan interpreter that runs an LLM-authored step list (a JSON DAG) as one job. The interpreter itself is @orchestral/plan; this package depends on it to register meta_plan, and re-exports the createPlanMeta factory its own manifest names. To build or preflight a plan of your own — planToMeta, validatePlan, preflightPlan — import @orchestral/plan directly.

That is the whole catalog — nine metas, on purpose. The long-form novel → video pipeline (script planning, prose chunking, novel-to-events, event-to-script, idea-to-video, and the director agent that drove them) is not API: it is kept runnable, with its tests, as examples/long-form-video, a host that registers those six from its own source next to this catalog.

Agent-kind patterns are not here: they live in the optional @orchestral/agent package, which ships the pattern declarations only — the AgentRunImpl that drives their tool loop is yours to inject, with a reference wiring in examples/agent-hello-world. Agent support is an opt-in extension; this catalog is atomic + meta.

import { PatternRegistry } from '@orchestral/core'
import { createTextToImagePattern, createStoryboardMeta } from '@orchestral/patterns'

const registry = new PatternRegistry()
registry.register(createTextToImagePattern())
registry.register(createStoryboardMeta())

See @orchestral/core for how patterns, the router, and a runtime fit together, plus an end-to-end example.

Catalog

Every pattern this package exports, generated from the built package by pnpm docs:catalog (scripts/gen-pattern-catalog.mjs).

| Pattern | Kind | What it does | Input slots | Output | Host ops required | Alternatives | | --- | --- | --- | --- | --- | --- | --- | | automatic-speech-recognition | atomic | Transcribe spoken audio into text. | source:audio req | text | — | — | | image-to-image | atomic | Edit an existing image guided by a text prompt. | source:image[] reqmask:image | image, assets[] | — | → meta_image-to-image-via-caption | | image-to-text | atomic | Read one or more images and produce text about them. | source:image[] req | text | — | — | | image-to-video | atomic | Animate a still image: the source image is the starting frame and the model invents motion forward,… | startFrame:image reqendFrame:imagereference:image[]referenceVideo:video[]referenceAudio:audio[] | video, assets[] | — | — | | text-generation | atomic | Generate text from a single prompt. | — | text | — | — | | text-to-audio | atomic | Generate music, sound effects, or ambient audio from a text prompt. | — | audio, assets[] | — | — | | text-to-image | atomic | Generate an image from a text prompt. | reference:image[]control:image | image, assets[] | — | — | | text-to-speech | atomic | Synthesize speech audio from text using a provider voice. | voiceClone:audio | audio, assets[] | — | — | | text-to-video | atomic | Generate a short video clip from a text prompt. | reference:image[]endFrame:image | video, assets[] | — | — | | video-to-video | atomic | Transform an existing video: reframe its aspect ratio, upscale its resolution, restyle its look from… | source:video reqreference:image[] | video, assets[] | — | — | | meta_explainer-short | meta | Generate a short explainer video from a topic: write a typed scene breakdown, let the user review… | — | assets[] | concatVideosstillToVideo | — | | meta_image-best-of-n | meta | Render multiple image candidates and pick the best one via VLM quality judging. | — | assets[] | — | — | | meta_image-to-image-via-caption | meta | Edit an image without a native image-to-image model by chaining caption → text-to-image. | source:image[] req | image, assets[] | — | — | | meta_plan | meta | Run a fixed pipeline of registered patterns as one job: you write the steps as data and the runtime… | — | assets[] | getPattern | — | | meta_product-ad-short | meta | Generate a short product ad clip via a pick-then-animate flow. | — | assets[] | addBackgroundAudiorecordSessionAsset | — | | meta_product-photo-pack | meta | Generate a product photo pack (multiple e-commerce shots) from a product brief. | — | assets[] | — | — | | meta_script2video | meta | Generate a video from a scene script. | — | assets[] | concatVideos | — | | meta_storyboard | meta | Generate a multi-panel storyboard from a scene and character reference sheets, keeping each… | — | assets[] | — | — | | meta_ugc-testimonial | meta | Generate a UGC product testimonial video from a product description and optional persona. | — | assets[] | concatVideosaddBackgroundAudioaddSubtitlescreateSubtitleAsset | — |

19 Patterns.

• Input slots — the assetNeeds an author declared; the LLM fills them through input.references.<slot>. [] marks a multi-asset slot, req a required one. A Pattern with no slots takes text input only.
• Output — the same projection find_pattern shows the LLM: the outputs schema's modality literal, and assets[] when it returns produced assets. Every shipped meta that produces media returns it through assets[] (with a role label per element); a pattern that produces no media reads as — here.
• Host ops required — the operations the factory takes as constructor deps: MetaCommonDeps picks for the deliverable metas, meta_plan's getPattern registry read for the one-shot. This package specifies them but does not implement them, so the host must (see Deliverable metas).
• Alternatives — fallback paths the factory mounts by default; the runtime cascades to them when the primary path is unsatisfiable or fails.

Authoring a pattern

Atomic — a pattern is pure metadata; no provider code lives here. The factory returns an AtomicPattern with: id (the capability literal), searchHint + namespace (retrieval), description (host-engineer prose), primary.tool (the LLM-facing description + zod inputs), outputs (zod), and optional assetNeeds (named asset slots the LLM fills via input.references). Model-specific parameters do NOT belong on the input schema — they arrive through the host-derived providerOptions lift. Outputs follow the shared envelope convention: produced assets in assets[] ({ assetId, modality, url?, cost? }) plus cost / latencyMs / model / provider.

Meta — a compose(params, ctx) function that orchestrates sub-patterns with ctx.step (idempotent, retryable sub-dispatch), ctx.compute (idempotent local work), and ctx.askUser (human-in-the-loop). Host-side operations (ffmpeg concat, subtitle burn-in, …) are injected through the factory's deps object — see MetaCommonDeps for the shared op signatures. For compile-time-typed sub-pattern calls, wrap ids with createPatternFn from @orchestral/core.

Prompts are module constants colocated with each meta (prompts.ts). The raw constants stay internal; what a consumer gets is the frozen *_DEFAULT_PROMPTS object each meta exports from the main entry point (STORYBOARD_DEFAULT_PROMPTS, UGC_TESTIMONIAL_DEFAULT_PROMPTS, …), keyed by the same names the factory's prompts override map takes — so you can retune one step and spread the rest instead of forking the package. The *_DEFAULT_PROMPTS objects are marked @alpha: under 0.x their keys and wording may change with the metas they drive, so treat a prompt override map as something to re-check on each minor.

Deliverable metas

A deliverable meta is a multi-step, cost-gated meta that produces a finished user-facing artifact (a video, an image set, an audio track). Copy from the four exemplars: src/meta/product-ad-short/, src/meta/product-photo-pack/, src/meta/ugc-testimonial/, src/meta/explainer-short/.

Every helper they import from ../_shared/meta-utils (firstAsset, firstAssetId, labelAsset, labelledAssetShape, assetIdByLabel, parseJsonWithSchema, resolvePrompts, styleTag, sumCosts, toJsonSchemaCached, and the MetaCommonDeps / LabelledAsset types) is re-exported from the package root, so a copied exemplar compiles once you rewrite that one relative import to @orchestral/patterns. examples/long-form-video is that procedure applied to five metas and an agent that used to live in this repo's packages.

Be aware of what these metas assume of you: MetaCommonDeps declares six media operations — concatVideos, stillToVideo, addBackgroundAudio, addSubtitles, createSubtitleAsset, recordSessionAsset — that this package specifies but does not implement. Any meta that Picks one of them only runs on a host that brings its own multimedia backend (ffmpeg or equivalent) plus asset storage. The JSDoc on each op is the contract to implement against.

Conventions they all follow:

1. Never index assets[0] directly — read a sub-step's produced asset through firstAsset(out, label) / firstAssetId(out, label).

2. Return produced media through one flat top-level assets[] of z.object(labelledAssetShape(modality)) elements and nowhere else — no videoAssetId / imageAssetIds fields, top-level or nested. The model-facing projection rebuilds assets[] from the handle whitelist and passes every other field through untouched, so an id anywhere else reaches the model verbatim. The role an asset played (final-video, hero, scene-2-vo, winner) rides on the element as its label, which the projection keeps; stamp it with labelAsset(el, modality, label) and read another meta's deliverable back with assetIdByLabel(out, label, errLabel).

3. Bound every LLM-JSON array schema with .min(1) and clamp to the input cap (slice(0, cap)) so the paid-gen count can't exceed the bounded input.

4. Parse LLM JSON with parseJsonWithSchema(text, schema, label) — labeled error on malformed JSON, Zod errors propagate as-is.

5. Guard any indexOf-based pick mapping (idx < 0 → throw a labeled error).

6. Type ctx.step results with the real atomic Output types (TextToImageOutput, ImageToVideoOutput, …), never inline structural types. No any in tests — Record<string, unknown> + as unknown as T.

7. Every paid multi-gen sits behind a ctx.askUser checkpoint (cost gate), or is bounded and confirmed up front.

8. Every string in an outputs schema carries an explicit bound from @orchestral/core's vocabulary — boundedText(n), assetIdField(), urlField(), opaqueToken() — never a bare z.string(). The registry audits every outputs schema at registration and warns OUTPUTS_UNBOUNDED_FIELDS for what slipped through; the shipped catalog registers with zero warnings, and registry-outputs-bounded.test.ts pins that. Leave input schemas alone — the bound is about what reaches a model's context, and inputs do not. The bounds in use, in one place so they can be retuned together:

   | Output field | Bound | Sized for | | --- | --- | --- | | text-generation → text | 64 KiB | ~16k output tokens, a single completion's ceiling | | image-to-text → text | 16 KiB | a caption, a judge answer, an extract-style sheet | | automatic-speech-recognition → text | 256 KiB | ~3 h of speech | | automatic-speech-recognition → segments[].text | 4 KiB | one sentence / subtitle cue | | automatic-speech-recognition → words[].text | 256 | one spoken token | | automatic-speech-recognition → language | 64 | a BCP-47 tag (≤ 35 octets) | | meta_image-best-of-n → reason | 2 KiB | the judge's rationale | | meta_storyboard → panels[].visualDesc | 4 KiB | one shot's composition | | meta_storyboard → panels[].audioDesc | 2 KiB | a cue or a few lines of dialogue | | meta_storyboard → panels[].characterNames[] | 128 each | a character name | | meta_explainer-short → scenes[].narration | 2 KiB | one scene's voiced narration | | meta_image-to-image-via-caption → requestedSize | 32 | a WIDTHxHEIGHT pair | | every assets[].label | 64 | a role label (labelledAssetShape) | | every assets[].assetId / url | 128 / 2 KiB | core's producedAssetShape | | every model / provider | 256 / 128 | core's dispatchEnvelopeShape |

   What the audit cannot bound is array length, so the arrays say on their .describe() what sizes them: segments[] / words[] by the audio's length; panels[] by maxShots, which the design pass is told and refused past; characterNames[] by the input registry, since an unknown name fails closed; assets[] by the generations a dispatch makes (n ≤ 8 for best-of-n).

Picking the ctx.askUser method:

• confirm — plain yes/no cost gate before paid work (src/meta/product-photo-pack/index.ts).
• choose — single pick from a short, fixed text-label list, no images (src/meta/product-ad-short/index.ts, the no-session fallback).
• form — multi-field review/edit before generation (src/meta/explainer-short/index.ts, narration review).
• custom — bespoke widget payload with a hand-built answerSchema, e.g. image-thumbnail picking via kind: 'choice' (src/meta/product-ad-short/index.ts, the session/thumbnail path).

Versioning (0.x SemVer)

Pre-1.0: minor releases may include breaking changes. Pin "~0.1" and check the CHANGELOG.md.

License

Apache-2.0. See LICENSE.