English | 中文

yume-dsl-shiki-highlight

▶ Try the live playground — type DSL, see tokens instantly

Edit tags in real time, toggle handlers on/off, watch the token tree update as you type.

Shiki code-highlighting plugin

A small syntax-highlighting library for yume-dsl-rich-text.

Core API is stable. Future updates will prioritize backward compatibility; breaking changes, if any, will land in major versions with explicit migration notes.

Two modes:

• Programmatic — tokenizeRichText returns colored token arrays you can render anywhere (terminal, canvas, custom UI).
• Shiki / TextMate — createRichTextGrammar generates a grammar you can feed to Shiki or any TextMate-compatible editor.

Table of Contents

• Ecosystem
• Install
• Quick Start
  • With createParser (recommended)
  • Standalone
  • Shiki integration
• API — Tokenizer
  • createTokenizerFromParser
  • createTokenizer
  • tokenizeRichText / tokenizeRichTextLines
  • renderStructuralTree
• API — Shiki Grammar
  • createRichTextGrammar
  • RICH_TEXT_TOKEN_COLORS
  • RICH_TEXT_SCOPE_NAME
• API — Utilities
  • escapeRegex
  • colorizeEscapes
  • splitTokensByLineBreak
  • pushToken
• Colors
• Types
• Relationship with parseStructural
• Changelog
• License

Ecosystem

text ──▶ yume-dsl-rich-text (parse) ──▶ TextToken[]  ──▶ yume-dsl-token-walker (interpret) ──▶ TNode[]
  │                  │
  │                  ├── parseStructural ──▶ StructuralNode[]
  │                  │                            │
  └──────────────────┴── yume-dsl-shiki-highlight ┘ ──▶ HighlightToken[] / Shiki grammar

| Package | Role | |------------------------------------------------------------------------------------|-----------------------------------------------------------------| | yume-dsl-rich-text | Parser — text to token tree | | yume-dsl-token-walker | Interpreter — token tree to output nodes | | yume-dsl-shiki-highlight | Syntax highlighting — tokens or TextMate grammar (this package) | | yume-dsl-markdown-it | markdown-it plugin — DSL tags inside Markdown |

Install

npm install yume-dsl-shiki-highlight
# or
pnpm add yume-dsl-shiki-highlight

yume-dsl-rich-text is a direct dependency and installed automatically.

Quick Start

With createParser (recommended)

Use one config object for both parsing and highlighting.

import { createParser, createSimpleInlineHandlers } from "yume-dsl-rich-text";
import { createTokenizerFromParser } from "yume-dsl-shiki-highlight";

const parserOpts = {
  handlers: createSimpleInlineHandlers(["bold", "code", "link"]),
};

const dsl = createParser(parserOpts);
const hl = createTokenizerFromParser(parserOpts, { tagName: "#0550AE" });

dsl.parse(text);        // TextToken[]
hl.tokenize(text);      // HighlightToken[]
hl.tokenizeLines(text); // same, split by line → HighlightToken[][]

Standalone

Highlight all tags without a parser — useful for editor previews, playgrounds, or documentation.

import { tokenizeRichText, tokenizeRichTextLines } from "yume-dsl-shiki-highlight";

// Single-line → flat token array
const tokens = tokenizeRichText("$$bold(hello)$$ world");

// Multi-line (handles cross-line raw/block tags)
const lines = tokenizeRichTextLines("$$code(ts)%\nconst x = 1;\n%end$$");

When using custom syntax, prefer createEasySyntax(...) from yume-dsl-rich-text and pass the result explicitly in options.syntax:

import { createEasySyntax } from "yume-dsl-rich-text";

const syntax = createEasySyntax({
  tagPrefix: "@@",
  tagOpen: "<<",
  tagClose: ">>",
  tagDivider: "||",
  escapeChar: "~",
});

const tokens = tokenizeRichText("@@bold<<hello>>@@", { syntax });

Or bind defaults once:

import { createTokenizer } from "yume-dsl-shiki-highlight";

const hl = createTokenizer({
  handlers,
  allowForms: ["inline"],
  colors: { tagName: "#FF0000" },
});

hl.tokenize(text);

Shiki integration

import { createHighlighterCore } from "shiki/core";
import { createOnigurumaEngine } from "shiki/engine/oniguruma";
import baseTheme from "shiki/themes/github-light-high-contrast.mjs";
import { createEasySyntax } from "yume-dsl-rich-text";
import { createRichTextGrammar, RICH_TEXT_TOKEN_COLORS } from "yume-dsl-shiki-highlight";

const syntax = createEasySyntax({
    tagPrefix: "@@",
    tagOpen: "<<",
    tagClose: ">>",
    tagDivider: "||",
    escapeChar: "~",
});

const grammar = createRichTextGrammar({syntax});  // match any tag with custom delimiters
// or restrict to known tags:
// const grammar = createRichTextGrammar({
//   allTags: ["bold", "code", "link", "info"],
//   rawTags: ["code"],
//   blockTags: ["info", "collapse"],
//   syntax,
// });

const theme = {
    ...baseTheme,
    tokenColors: [...(baseTheme.tokenColors ?? []), ...RICH_TEXT_TOKEN_COLORS],
};

const highlighter = await createHighlighterCore({
    themes: [theme],
    langs: [grammar],
    engine: await createOnigurumaEngine(() => import("shiki/wasm")),
});

const html = highlighter.codeToHtml(dslSource, {
    lang: "yume-rich-text-dsl",
    theme: theme.name,
});

Note: createRichTextGrammar({ syntax }) is available since v1.0.3.

API — Tokenizer

createTokenizerFromParser(parserOptions, colors?)

Recommended entry point. Create a tokenizer from the same defaults object you use for parsing.

function createTokenizerFromParser(
  parserOptions: TokenizeOptions,
  colors?: Partial<HighlightColors>,
): Tokenizer

interface Tokenizer {
  tokenize: (text: string, overrides?: TokenizeOptions) => HighlightToken[];
  tokenizeLines: (text: string, overrides?: TokenizeOptions) => HighlightToken[][];
}

parserOptions can include the same structural settings used by parseStructural: handlers, allowForms, implicitInlineShorthand, depthLimit, syntax, and tagName.

For consistent parser/highlight behavior with implicit inline shorthand (name(...) inside inline args), pass the same implicitInlineShorthand value you use in your parser config.

The optional colors argument is applied on top of those parser-derived defaults.

createTokenizer(defaults?)

Create a standalone tokenizer with bound defaults. Parser-related fields are forwarded to parseStructural.

function createTokenizer(defaults?: TokenizeOptions): Tokenizer

tokenizeRichText(text, options?) / tokenizeRichTextLines(text, options?)

Stateless one-shot functions. TokenizeOptions extends ParserBaseOptions, so you can pass parser gating options directly (including implicitInlineShorthand).

function tokenizeRichText(text: string, options?: TokenizeOptions): HighlightToken[]
function tokenizeRichTextLines(text: string, options?: TokenizeOptions): HighlightToken[][]

interface TokenizeOptions extends ParserBaseOptions {
  colors?: Partial<HighlightColors>;
}

renderStructuralTree(nodes, colors, syntax, textColor?)

Low-level renderer: converts a StructuralNode[] tree (from parseStructural) into colored tokens.

Use this when you want to insert your own logic between structural parsing and rendering. Pass the same syntax you used for parsing. Use createSyntax() for default syntax.

function renderStructuralTree(
  nodes: StructuralNode[],
  colors: HighlightColors,
  syntax: SyntaxConfig,
  textColor?: string,
): HighlightToken[]

API — Shiki Grammar

createRichTextGrammar(tagConfig?)

Generate a Shiki-compatible TextMate grammar for the rich-text DSL. The returned object can be passed directly to Shiki's langs array.

• No tagConfig: match any valid tag name
• With tagConfig: restrict matching to listed tag names
• tagConfig.syntax lets the grammar follow custom parser delimiters (available since v1.0.3)

function createRichTextGrammar(tagConfig?: GrammarTagConfig): LanguageRegistration

interface GrammarTagConfig {
  allTags: readonly string[];       // inline matching
  rawTags: readonly string[];       // $$tag(…)% … %end$$
  blockTags: readonly string[];     // $$tag(…)* … *end$$
  syntax?: Partial<SyntaxInput>;    // optional custom syntax tokens
  tagName?: Partial<TagNameConfig>; // validation rules
  anyTagPattern?: string;           // fallback regex for unrestricted matching
}

When syntax is omitted, the grammar uses the default yume-dsl-rich-text delimiters. If your parser uses createEasySyntax(...) or explicit custom tokens, pass the same syntax here.

RICH_TEXT_TOKEN_COLORS

Pre-built tokenColors array for Shiki themes.

const theme = {
    ...baseTheme,
    tokenColors: [...(baseTheme.tokenColors ?? []), ...RICH_TEXT_TOKEN_COLORS],
};

RICH_TEXT_SCOPE_NAME

The TextMate scopeName used by the generated grammar: "source.yume-rich-text-dsl".

API — Utilities

escapeRegex(value)

Escape all regex metacharacters (|, \, {, }, (, ), [, ], ^, $, +, *, ?, ., -) in a string so it can be safely embedded in a regex pattern as a literal match.

This is the same function used internally by createRichTextGrammar. Useful when:

• Building custom TextMate grammar patterns from user-configurable syntax tokens
• Embedding DSL delimiters (like $$, )*, %end$$) in regex strings
• Composing your own Shiki grammar rules that reference DSL syntax

function escapeRegex(value: string): string

import { escapeRegex } from "yume-dsl-shiki-highlight";

escapeRegex("hello");    // "hello"       — no metacharacters
escapeRegex("$$");       // "\\$\\$"      — both $ escaped
escapeRegex("*end$$");   // "\\*end\\$\\$"
escapeRegex("ns.tag");   // "ns\\.tag"    — dot escaped (not wildcard)

colorizeEscapes(text, valueColor, escapeColor, syntax)

Scan a string for DSL escape sequences (\(, \), \|, etc.), returning tokens where escapes are colored separately.

function colorizeEscapes(text: string, valueColor: string | undefined, escapeColor: string, syntax: SyntaxConfig): HighlightToken[]

splitTokensByLineBreak(tokens)

Split a flat HighlightToken[] into one array per line at \n boundaries.

function splitTokensByLineBreak(tokens: HighlightToken[]): HighlightToken[][]

pushToken(tokens, content, color?, fontStyle?)

Append a token to an array, automatically skipping empty content.

function pushToken(tokens: HighlightToken[], content: string, color?: string, fontStyle?: string): void

Colors

DEFAULT_COLORS

| Key | Hex | Role | |---------------|-----------|-------------------------------| | tagName | #0550AE | Tag name (bold, code, …) | | punct | #CF222E | $$ prefix/suffix | | bracket | #6639BA | ( ) argument brackets | | separator | #953800 | \| pipe divider | | operator | #1A7F37 | % * form markers | | end | #8250DF | end keyword | | escape | #116329 | Escape sequences | | argText | #0A3069 | Text inside arguments | | contentText | #0A7EA4 | Text inside raw/block content |

resolveColors(overrides?)

Merge partial overrides with DEFAULT_COLORS:

const colors = resolveColors({ tagName: "#FF0000" });

Types

interface HighlightToken {
    content: string;
    color?: string;
    fontStyle?: string;
}

interface TokenizeOptions extends ParserBaseOptions {
    colors?: Partial<HighlightColors>;
    // Inherited from ParserBaseOptions:
    // handlers?, allowForms?, depthLimit?, syntax?, tagName?
}

interface Tokenizer {
    tokenize: (text: string, overrides?: TokenizeOptions) => HighlightToken[];
    tokenizeLines: (text: string, overrides?: TokenizeOptions) => HighlightToken[][];
}

interface GrammarTagConfig {
    allTags: readonly string[];
    rawTags: readonly string[];
    blockTags: readonly string[];
    tagName?: Partial<TagNameConfig>;
    anyTagPattern?: string;
}

StructuralNode, StructuralParseOptions, ParserBaseOptions, and TagNameConfig are re-exported from yume-dsl-rich-text.

Relationship with parseStructural

TokenizeOptions extends ParserBaseOptions, so handlers, allowForms, implicitInlineShorthand, syntax, tagName, and depthLimit flow through to parseStructural without extra adapter code.

When handlers is provided, tag recognition and form gating are identical to parseRichText. When omitted, all tags and forms are accepted.

Differences from parseRichText (features, not bugs):

| | parseRichText | parseStructural | |--------------------------|-----------------------------------------|-------------------------------------------| | Tag recognition | Same (ParserBaseOptions) | Same (ParserBaseOptions) | | Form gating | Same | Same | | Line-break normalization | mode: "render" strips | Always preserves | | Escape representation | Unescaped at root, raw inside tags | Structural escape nodes | | Pipe \| | Plain text (post-processed by handlers) | separator nodes in args; text elsewhere | | Error reporting | onError callback | Silent degradation | | Output type | TextToken[] | StructuralNode[] |

Changelog

See also CHANGELOG.

License

MIT