@dharmax/text-compiler

Compile natural-language instructions into deterministic, suspendable JavaScript state machines.

This package is for hosts that want an LLM to act like a compiler, not an executor. The model produces workflow code, but execution stays inside a runtime you control through explicit services, context access, and state-machine rules.

Status

Early foundation. The core compiler and state-machine runtime exist, package/test scaffolding is in place, and suspend/resume snapshots are covered by tests. Provider integration and higher-level examples are still incomplete.

What It Is

text-compiler takes a prompt such as "greet the user, look up context, then wait for confirmation" and compiles it into a workflow that:

• runs through an AnnotatedStateMachine
• calls only the services you expose
• stores cross-state data in serializable memory
• can suspend and resume through explicit snapshots
• can synthesize missing helper services through the compiler toolkit

The intended stack is:

1. You provide an AiRouter, or use LlmUtilsRouter to route compiler jobs through @dharmax/llm-utils.
2. You provide a runtime context backed by @dharmax/context-manager plus your runtime services.
3. compileText(...) generates a workflow.
4. The returned workflow executes inside your runtime.

Core API

The main exports are:

• compileText(...)
• CompilerToolkit
• LlmUtilsRouter
• AnnotatedStateMachine
• SuspendToken

Most users should start with CompilerToolkit plus compileText(...).

Installation

npm install @dharmax/text-compiler

Quick Start

import {
  AnnotatedStateMachine,
  CompilerToolkit,
  LlmUtilsRouter,
  DEFAULT_COMPILER_TASK_TYPES,
  compileText,
} from '@dharmax/text-compiler';

import { HeuristicContextManager } from '@dharmax/context-manager';

const router = new LlmUtilsRouter({
  providers: [{ id: 'ollama', host: 'http://127.0.0.1:11434' }],
  taskTypes: DEFAULT_COMPILER_TASK_TYPES,
  availableModels: [
    {
      id: 'qwen2.5-coder:7b',
      providerId: 'ollama',
      capabilities: { logic: 0.95, strategy: 0.7, data: 0.5 },
      local: true,
    },
  ],
});

const toolkit = new CompilerToolkit(router);

const workflow = await compileText(
  'Greet Ada',
  toolkit,
  [
    {
      name: 'greeter',
      description: 'Return a greeting',
      schema: {
        input: { name: 'string' },
        output: { message: 'string' },
      },
      async execute(args: { name: string }) {
        return { message: `Hello, ${args.name}!` };
      },
    },
  ],
);

const ctx = {
  sessionId: 'demo-session',
  sessionState: {},
  history: [],
  contextStore: {
    async query() { return []; },
    async add() {},
  },
  contextManager: new HeuristicContextManager({
    async query() { return []; },
    async add() {},
  }),
  async queryGraph() { return []; },
  async mutateGraph() {},
};

const runtimeTk = {
  sm: new AnnotatedStateMachine(),
  services: {},
  log() {},
  trigger() {},
  scheduleResume() {},
  async compileFunction() {
    throw new Error('compileFunction is injected by compileText');
  },
};

const result = await workflow.execute(ctx, runtimeTk);

console.log(result.status); // "completed"
console.log(result.output); // "Hello, Ada!"

Suspend And Resume

Suspension is explicit. A state returns tk.sm.suspend(...), and the machine returns a serializable snapshot you can persist.

const sm = new AnnotatedStateMachine();

sm.state(
  'start',
  'Capture state and suspend',
  async (_ctx, tk) => {
    tk.sm.memory.step = 'captured';
    return tk.sm.suspend('need-input', 'resume', { question: 'name' });
  },
  { resume: 'resume' },
);

sm.state(
  'resume',
  'Continue from a restored snapshot',
  async (_ctx, tk, input) => {
    return `${tk.sm.memory.step}:${input.name}`;
  },
  {},
);

const firstRun = await sm.run('start', {}, ctx, runtimeTk);
const snapshot = firstRun.snapshot;

const resumed = new AnnotatedStateMachine();
resumed.state('start', async () => null, { resume: 'resume' });
resumed.state('resume', async (_ctx, tk, input) => `${tk.sm.memory.step}:${input.name}`, {});
resumed.restoreSnapshot(snapshot);

const secondRun = await resumed.run(snapshot.resumeState, { name: 'Ada' }, ctx, runtimeTk);

Design constraints:

• tk.sm.memory must stay serializable
• snapshots contain cloned memory and trace
• non-serializable values are rejected before suspension

Mental Model

CompilerToolkit

Owns the LLM-facing compiler loop.

• analyzes whether the requested workflow needs additional primitive services
• generates workflow code
• optionally runs a critic pass
• validates generated code with acorn

compileText(...)

This is the high-level entrypoint.

• resolves missing services
• generates state-machine code
• returns a compiled workflow with an execute(...) method

AnnotatedStateMachine

This is the deterministic runtime surface.

• states are registered explicitly
• cross-state data lives in tk.sm.memory
• terminal states have an empty transition map
• suspension returns a snapshot you can restore later

Current Guarantees

Based on the current tests:

• state machines complete terminal states correctly
• suspension rejects non-serializable memory
• snapshots can be restored into a fresh machine instance
• synthesized service names are deterministic
• compiled workflows hydrate runtime services and execute successfully

Verification

npm test

This runs the TypeScript build first, then the Node test suite.

What Is Not Done Yet

• polished provider integration around @dharmax/llm-utils
• more complete host examples combining @dharmax/llm-utils and @dharmax/context-manager
• production examples beyond the minimal host wiring
• stronger validation around generated workflow semantics
• richer APIs for packaging reusable runtime adapters

Development Direction

The current development priorities are:

1. solidify determinism and suspend/resume behavior
2. integrate the LLM provider layer cleanly
3. improve input/output handling
4. formalize the context database abstraction

Repository

GitHub: https://github.com/dharmax/TextCompiler