deck-ir-vlm

The VLM layer for deck-ir: classify & cluster the slides of a .pptx into a library of editable HTML template layouts.

deck-ir faithfully renders a .pptx into HTML — deterministically, in pure JS. deck-ir-vlm builds on top of it: it renders each slide, shows the screenshots to a vision LLM, and turns the deck into a small set of reusable layout templates whose text is exposed as {{token}} slots you can re-fill.

What it does

Given a .pptx, deck-ir-vlm:

1. Renders every slide to HTML with deck-ir, then to a PNG (via a pluggable RenderProvider).
2. Classifies each slide with a vision LLM (via a pluggable VlmProvider): slide type, a Chinese layout label, and a per-text-element category (title / body / footer / logo / decoration …).
3. Clusters slides that share a layout into groups, picking one representative per group.
4. Emits an editable skeleton for each representative: editable text (title/subtitle/body) becomes a {{token}} slot tagged data-editable="true"; everything else (lines, images, fixed brand text, decoration) is rendered faithfully by deck-ir.

The result is an EditableTemplate: a slideTypeLibrary of layouts + a textCatalog of the original slot text.

.pptx ──deck-ir──▶ semantic slides ──render──▶ PNGs ──VLM──▶ classifications
                          │                                        │
                          └──────────── cluster ◀──────────────────┘
                                          │
                                          ▼
                          editable HTML templates ({{token}} slots)

Pluggable providers

The core has no hard dependency on a browser or any specific LLM SDK — you inject two small functions:

interface RenderProvider { (html: string, size: { w: number; h: number }): Promise<Uint8Array>; }
interface VlmProvider {
  (input: { images: Uint8Array[]; text: string }, opts: { schema?: object; model?: string }): Promise<string>;
}

Optional default implementations ship as separate entry points so their deps stay optional:

• deck-ir-vlm/render-puppeteer → puppeteerRenderer() (needs the optional puppeteer peer dep)
• deck-ir-vlm/vlm-openai → openaiVlm({ baseURL, apiKey, model }) (any OpenAI/Gemini-compatible vision /chat/completions endpoint; uses global fetch)

Bring your own by implementing the interfaces directly (e.g. Playwright renderer, a different LLM).

Install

npm i deck-ir-vlm
# optional — only if you use the bundled screenshot provider:
npm i puppeteer

deck-ir-vlm depends on deck-ir, which npm pulls in automatically. A vision LLM API key is required at runtime (see Usage). puppeteer is an optional peer dependency, only needed for the bundled puppeteerRenderer.

Usage

import { pptxToTemplate } from "deck-ir-vlm";
import { puppeteerRenderer } from "deck-ir-vlm/render-puppeteer";
import { openaiVlm } from "deck-ir-vlm/vlm-openai";

const template = await pptxToTemplate(buffer, {
  render: puppeteerRenderer(),
  vlm: openaiVlm({
    baseURL: process.env.OPENAI_BASE_URL,   // OpenAI / Gemini-compatible endpoint
    apiKey: process.env.OPENAI_API_KEY,
    model: "gpt-4o",                         // any multimodal model
  }),
});

// template.slideTypeLibrary : Array<{ type, label, htmlSkeleton, description, usageCount, sourceSlideIndex }>
// template.textCatalog       : { [token]: <original text body> }   // for the editable slots
// template.metadata          : { totalSlideCount, selectedSlideCount, parserVersion: "deck-ir-vlm/0.1" }

buffer is the raw bytes of a .pptx (Uint8Array or ArrayBuffer).

Output shape

Each entry of slideTypeLibrary is one layout:

| Field | Meaning | | --- | --- | | type | layout-<n>-<labelZh> stable id | | label | Chinese layout label from the VLM (封面 / 正文 / 章节标题 …) | | htmlSkeleton | the <section> HTML with editable text as {{token}} slots | | description | the VLM's 50–200 char layout description | | usageCount | how many slides used this layout | | sourceSlideIndex | the representative slide's index |

Advanced exports

The pipeline stages are exported individually for custom flows:

import { buildVlmSlide, classify, clusterByLabelZh, emitEditableSlide } from "deck-ir-vlm";

Example / CLI

OPENAI_BASE_URL=https://api.openai.com/v1 \
OPENAI_API_KEY=sk-... \
MODEL=gpt-4o \
node examples/extract.mjs deck.pptx template.json

It prints the discovered layout library and writes the full EditableTemplate JSON.

Requires a vision LLM

Classification quality depends entirely on the model you inject. deck-ir-vlm does not ship or call any model on its own — you provide the endpoint + key. Any OpenAI/Gemini-compatible multimodal chat endpoint works.

Known limitations

• Classification is non-deterministic. The same deck can yield slightly different labels/categories across runs. The reconcile step keeps output well-formed (missing tokens → decoration, hallucinated tokens → dropped, missing slides → other), but the labels themselves are the model's judgment.
• Synonym layout labels are not merged. Clustering is by trimmed layoutLabelZh only; two labels that mean the same thing stay separate (planned).
• Editability is heuristic. Only title/subtitle/body become slots; everything else is treated as fixed.
• Everything deck-ir can't render (charts/tables without cached drawings, animations, embedded fonts) carries over here — see deck-ir's limitations.

Testing

pnpm test

Unit tests cover the deterministic parts — classify reconcile (missing/hallucinated tokens, missing slides, input ordering), clustering, and the editable emitter — using stub providers, so they need no browser or API key. The provider implementations (puppeteer / OpenAI) and end-to-end extraction are exercised manually via examples/extract.mjs (needs Chromium + an API key).

Roadmap

• Synonym/semantic layout-label merging
• Confidence scores + a "needs review" flag per slide
• More built-in providers (Playwright renderer, native SDK adapters)
• Round-trip: fill a template's {{token}} slots → rendered slide

Want the whole product? (flash-deck)

deck-ir-vlm is the open VLM-templating layer. The full product — AI one-line → full PPT, an editable-template editor, presenter mode, export to editable PPTX, and mermaid diagrams — lives at flash-deck (flashdeck.cn).

License

deck-ir-vlm is licensed under AGPL-3.0-only.

The AGPL covers any use, including network / SaaS use: if you build on deck-ir-vlm and offer it over a network, your derivative must also be open-sourced under AGPL-3.0.

For closed-source or commercial use without AGPL obligations, a commercial dual-license is available — see COMMERCIAL.md or contact [email protected].

Copyright (C) 2026 Kelvin Gao <[email protected]>.