@av-pi-studio/highlight

Server-side syntax-highlighting helper for Pi-Studio. A pure-JavaScript regex tokeniser — no external highlighting library, and (deliberately) no runtime dependencies at all.

Install

npm install @av-pi-studio/highlight

What it does

Turns source text into a stable, serializable token stream. This token stream is what the daemon sends to clients for file preview and git-diff rendering — the client then applies its own theme/styling to the token types, rather than the server shipping pre-styled HTML.

Two properties the tokenizer guarantees:

• Lossless — concatenating every token's value in order reproduces the original source string exactly. No character is ever dropped, transformed, or reordered.
• Graceful fallback — an unrecognized language (or any tokenizing edge case) falls back to a single plaintext token rather than throwing.

Usage

import { highlight, detectLanguage } from "@av-pi-studio/highlight";

const { language, tokens } = highlight(sourceText, "example.ts");
// language → "typescript"
// tokens   → [{ type: "keyword", value: "const" }, { type: "text", value: " " }, …]

detectLanguage("foo.tsx");   // → "typescript"
detectLanguage("foo.json");  // → "json"
detectLanguage("README.md"); // → "plaintext" (unrecognized extension)

Types

type TokenType = "keyword" | "string" | "comment" | "number" | "punctuation" | "identifier" | "text";
type Language = "typescript" | "javascript" | "json" | "plaintext";

interface HighlightToken {
  type: TokenType;
  value: string;
}

interface HighlightResult {
  language: Language;
  tokens: HighlightToken[];
}

detectLanguage(pathOrHint?: string): Language

Infers a language from a file path/extension, or accepts a bare hint string directly ("typescript", "ts", "javascript", "js", "json").

| Extension | Language | |---|---| | .ts, .tsx, .mts, .cts | typescript | | .js, .jsx, .mjs, .cjs | javascript | | .json | json | | anything else / undefined | plaintext |

highlight(source: string, hint?: string): HighlightResult

Tokenizes source, detecting the language via detectLanguage(hint). plaintext input returns a single text token wrapping the whole source (or an empty array for empty input).

Supported languages

• TypeScript / JavaScript — comments (//, /* */), string literals ("…", '…', `…`, escape-aware), numeric literals (hex, decimal, fractional, exponent), identifiers (promoted to keyword against a fixed keyword set — const, function, interface, async, typeof, etc.), and punctuation.
• JSON — string/number literals, true/false/null as keywords, and structural punctuation.
• Plaintext — everything else; the whole input becomes one text token.

Whitespace is always preserved as text tokens (this is what keeps the tokenizer lossless), and any character no matcher recognizes is emitted one-at-a-time as text rather than dropped.

Extending with a new language

1. Add the extension → language mapping to EXTENSION_LANGUAGE.
2. Add the language to the Language type.
3. Write a matcher table ({ type, re }[]) and wire it into tokenize().
4. Add the corresponding branch to detectLanguage().
5. Add tests in highlight.test.ts.

Development

npm run build       # tsc -b
npx vitest run packages/highlight

No dependencies in package.json — keep it that way; this package must stay usable anywhere plain JavaScript runs.