@natl/agent

CI agent that turns a git diff into suggested NATL YAML tests via an OpenAI-compatible LLM Gateway, then gates output with JSON Schema + parseNatlDocument (same bar as natl validate).

Install

npm install -D @natl/agent @natl/cli

Config (natl-agent.yml)

llm:
  provider: openai   # openai | ollama | azure | custom
  endpoint: https://api.openai.com/v1
  api_key: ${LLM_API_KEY}
  model: gpt-4o-mini
  parameters:
    temperature: 0    # always forced to 0
    max_tokens: 2048
    top_p: 0.9
    seed: 42
mode: comment
test_roots: [tests, examples]

Examples in examples/:

| File | Use case | |------|----------| | natl-agent.openai.yml | Cloud OpenAI | | natl-agent.ollama.yml | Local Ollama (offline) | | natl-agent.azure.yml | Azure OpenAI | | natl-agent.custom.yml | vLLM / NIM / LM Studio / LocalAI |

CLI

# Cloud
export LLM_API_KEY=sk-...
natl-agent --config examples/natl-agent.openai.yml --mode stdout

# Local Ollama (no API key)
ollama serve && ollama pull llama3.2
natl-agent --config examples/natl-agent.ollama.yml --mode stdout

# Flags override file/env
natl-agent --provider ollama --base-url http://127.0.0.1:11434/v1 --model llama3.2

| Flag / env | Meaning | |------------|---------| | --config | Path to natl-agent.yml | | LLM_API_KEY / OPENAI_API_KEY | Cloud key (optional for ollama) | | OPENAI_BASE_URL / NATL_AGENT_ENDPOINT | Base URL …/v1 | | --provider / NATL_AGENT_PROVIDER | openai | ollama | azure | custom | | --mode | comment | stdout | commit | | --cli-validate | Extra gate via npx @natl/cli validate |

Determinism: every request sends temperature: 0 (non-zero config values are ignored with a warning).

GitHub Actions

- uses: actions/checkout@v4
  with:
    fetch-depth: 0
- uses: ./agent
  with:
    openai_api_key: ${{ secrets.OPENAI_API_KEY }}
    openai_base_url: ${{ vars.OPENAI_BASE_URL }}
    openai_model: ${{ vars.OPENAI_MODEL }}
    base_ref: origin/${{ github.base_ref }}
    config: natl-agent.yml   # optional

Pipeline

1. Diff (simple-git)
2. Semantic UIR — Tree-sitter WASM (Py/JS/TS/Java/Go/C#) with heuristic fallback
3. Few-shot context from existing YAML (+ import hints)
4. LLM Gateway → OpenAI-compatible chat/completions
5. Validate — YAML → AJV (natl.test.schema.json) → parseNatlDocument
6. Publish PR comment / stdout / commit

Programmatic API

import { runAgent, loadAgentConfig, createLlmClient } from '@natl/agent';

const config = loadAgentConfig({ cwd: process.cwd() });
const result = await runAgent({ config });

Offline / private

Point provider: ollama (or custom) at a local server. No outbound LLM traffic; only local git + filesystem are required.

Tiny model via npm (Transformers.js)

# once — creates gitignored ./local-llm/
node scripts/local-llm/setup.mjs
cd local-llm && npm start
# → http://127.0.0.1:8787/v1

Config: examples/natl-agent.local-transformers.yml or tests/agent-release/natl-agent.local.yml.

Pre-release suite: tests/agent-release/ (pnpm test).

Docker

cd agent
docker build -t natl-agent .
docker run --rm -e LLM_API_KEY -v "$PWD/..:/work" -w /work natl-agent --mode stdout --base origin/main

Optional Compose (Ollama + agent profile): see docker-compose.yml.

Eval (DoD ≥70% validate gate)

cd agent
pnpm eval
# writes eval/report.json — synthetic 100 candidates, no live LLM

Troubleshooting

| Symptom | Fix | |---------|-----| | Missing API key | Set LLM_API_KEY or use provider: ollama | | Empty UIR / no tests | Diff has no supported language files; check --base | | Validation exit 2 | Model YAML failed AJV/parseNatlDocument — see PR comment error | | Tree-sitter load fail | Heuristic fallback kicks in automatically |