@ogentx/ast

OgentDoc Markdown Parser and Transformer.

Overview

@ogentx/ast is a pure parsing layer that converts OgentDoc Markdown files into a structured JSON Abstract Syntax Tree (AST). It handles specification validation and structure extraction but does not perform any database operations.

Key Features

Section-Frontmatter Extension

OgentDoc extends standard Markdown with section-level frontmatter, allowing each section to define its own YAML configuration. This is a powerful feature that enables:

• Service Configuration: Define inheritance, dependencies, and server configurations
• Skill Parameters: Configure model, temperature, and other implementation parameters
• Implementer Definitions: Map skill parameters to dependency services

Document Structure

---
spec: v1.0.0
org: my-org
package: my-package
version: 1.0.0
---

# Service Name @+service-slug

```yaml
inherits: '@org/[email protected]'
depends:
  llm: '@ogent/llm'
  mailer: '@ogent/mailer'
servers:
  - https://api.example.com/v1/
```

Service description here.

Skills

Skill Name @+skill-slug

model: gpt-4
temperature: 0.7
max_tokens: 1000

Skill description.

#Chat

temperature: 0.7

system

You are a helpful assistant.

user

content:
  - type: text
    text: 'Hello'

User message.

#### How Section-Frontmatter Works

1. **Document Frontmatter**: YAML block at the top of the document, enclosed by `---` delimiters
2. **Section Frontmatter**: YAML code block (```yaml) immediately following a section header
3. **Nesting**: Section frontmatter is scoped to its section and does not affect parent or sibling sections

#### Visibility Modifiers

Section titles can include visibility modifiers in the `@` identifier:

| Modifier | Visibility | Description |
|----------|------------|-------------|
| `@+slug` | Public | Visible to all external users |
| `@slug` | Protected | Visible within package and inherited services |
| `@#slug` | Private | Visible only within the service itself |
| `@!slug` | Final | Cannot be overridden in inherited services |

## Architecture

This package is designed for portability and can run in any environment (Node.js, Browser, Edge). It is used by both the CLI (for local validation) and `@ogentx/store` (for persistence).

### Core Modules

| Module | Description |
|--------|-------------|
| `parser.ts` | Markdown parsing with section-frontmatter support |
| `extractor/service.ts` | Extracts ServiceAST from parsed document |
| `extractor/skill.ts` | Extracts SkillAST from skill sections |
| `extractor/schema.ts` | Extracts SchemaAST from schema definitions |
| `ogentdoc.ts` | Main entry point for parsing OgentDoc files |
| `validator.ts` | Validates AST against specification rules |
| `inheritance/resolver.ts` | Resolves service inheritance references |

## API Reference

### parseMarkdown

Parses raw Markdown content into a structured document.

```typescript
import { parseMarkdown } from '@ogentx/ast';

const doc = parseMarkdown(content, {
  extractSlugs: true,    // Extract URL-safe slugs from @ identifiers
  preserveRaw: false     // Include raw content in result
});

// Document-level frontmatter
console.log(doc.frontmatter.version);  // "1.0.0"

// Section hierarchy
console.log(doc.sections[0].title);    // "Service Name @+service-slug"
console.log(doc.sections[0].slug);     // "service-slug"
console.log(doc.sections[0].frontmatter);  // { inherits: "...", depends: {...} }

parseOgentDoc

High-level API for parsing OgentDoc files into PackageAST.

import { parseOgentDoc } from '@ogentx/ast';

const { ast, errors } = parseOgentDoc(markdownString);

if (errors.length > 0) {
  console.error('Validation failed:', errors);
} else {
  console.log('Parsed Package:', ast.package);
  console.log('Services:', ast.services);
}

extractSectionFrontmatter

Utility for extracting frontmatter from section content.

import { extractSectionFrontmatter } from '@ogentx/ast';

const result = extractSectionFrontmatter(`
\`\`\`yaml
model: gpt-4
\`\`\`

This is the remaining content.
`);

console.log(result.frontmatter); // { model: 'gpt-4' }
console.log(result.content); // 'This is the remaining content.'

Data Structures

ParsedDocument

interface ParsedDocument {
  frontmatter: Record<string, unknown>; // Document-level metadata
  sections: ParsedSection[]; // Top-level sections
  raw: string; // Original content (if preserved)
}

ParsedSection

interface ParsedSection {
  level: number; // Header level (1-6)
  title: string; // Raw title text
  slug: string; // URL-safe identifier
  frontmatter?: Record<string, unknown>; // Section-level YAML config
  content: string; // Text content (excluding subsections)
  subsections: ParsedSection[]; // Nested child sections
  lineStart: number; // Start line number
  lineEnd: number; // End line number
}

Responsibilities

1. Markdown Parsing: Converts Markdown source into a generic ParsedDocument.
2. AST Extraction: Extracts PackageAST, ServiceAST, and SkillAST from the parsed document.
3. Validation: Validates the extracted AST against the @ogentx/spec and JSON Schema rules.
4. Transformation: Provides logic for merging and resolving pure AST structures (e.g., base inheritance logic without DB fetching).

Related Packages

• @ogentx/spec: Defines the grammar and rules.
• @ogentx/types: TypeScript type definitions for AST structures.
• @ogentx/store: High-level persistence layer that uses this parser.

Testing

Run tests with:

pnpm vitest run

For specific test files:

pnpm vitest run test/parser.test.ts