llm-retry-validate

Validate LLM output against a Zod schema and automatically retry with targeted error feedback when validation fails.

Instead of resending the full prompt on failure, it sends back only the invalid JSON and the specific field errors — minimising token cost per retry.

Install

npm install llm-retry-validate zod

Quick Start

import { z } from 'zod';
import { validateWithRetry } from 'llm-retry-validate';

const schema = z.object({
  name: z.string(),
  age: z.number().int().positive(),
  email: z.string().email(),
});

const result = await validateWithRetry({
  schema,
  prompt: 'Generate a user profile as JSON with fields: name (string), age (positive integer), email (valid email).',
  generate: async (prompt) => {
    // Your LLM call here — see examples below
    return myLLM.generate(prompt);
  },
  maxRetries: 3,
  onRetry: (attempt, error, retryPrompt) => {
    console.log(`[attempt ${attempt}] retrying — ${error.message}`);
  },
});

if (result.success) {
  console.log(result.data);    // typed as { name: string; age: number; email: string }
  console.log(result.attempts);
} else {
  console.error(result.error);       // ZodError | Error
  console.log(result.lastRawOutput); // last raw text from LLM
}

Usage with Anthropic SDK

import Anthropic from '@anthropic-ai/sdk';
import { z } from 'zod';
import { validateWithRetry } from 'llm-retry-validate';

const client = new Anthropic();

const schema = z.object({
  title: z.string().min(1),
  summary: z.string().max(500),
  tags: z.array(z.string()).min(1).max(5),
  sentiment: z.enum(['positive', 'neutral', 'negative']),
});

const result = await validateWithRetry({
  schema,
  prompt: `Analyze the following article and return a JSON object with:
- title: article title (string)
- summary: brief summary under 500 characters (string)
- tags: 1–5 relevant topic tags (array of strings)
- sentiment: overall sentiment — must be exactly "positive", "neutral", or "negative"

Article: ${articleText}`,
  generate: async (prompt) => {
    const message = await client.messages.create({
      model: 'claude-sonnet-4-6',
      max_tokens: 1024,
      messages: [{ role: 'user', content: prompt }],
    });
    const block = message.content[0];
    return block.type === 'text' ? block.text : '';
  },
  maxRetries: 3,
});

Usage with OpenAI SDK

import OpenAI from 'openai';
import { z } from 'zod';
import { validateWithRetry } from 'llm-retry-validate';

const openai = new OpenAI();

const schema = z.object({
  items: z.array(
    z.object({
      name: z.string(),
      price: z.number().positive(),
      quantity: z.number().int().nonnegative(),
    }),
  ),
  total: z.number().positive(),
});

const result = await validateWithRetry({
  schema,
  prompt: 'Extract the order items from this receipt and return them as JSON...',
  generate: async (prompt) => {
    const completion = await openai.chat.completions.create({
      model: 'gpt-4o',
      messages: [{ role: 'user', content: prompt }],
    });
    return completion.choices[0].message.content ?? '';
  },
  maxRetries: 2,
});

API

validateWithRetry(options)

| Option | Type | Default | Description | |--------|------|---------|-------------| | schema | ZodType<T> | required | Zod schema to validate against | | generate | (prompt: string) => Promise<string> | required | Calls your LLM, returns raw text | | prompt | string | required | Initial prompt | | maxRetries | number | 3 | Retries after the first attempt | | onRetry | (attempt, error, retryPrompt) => void | — | Called before each retry |

Returns: Promise<ValidateResult<T>>

type ValidateResult<T> =
  | { success: true;  data: T;                 attempts: number }
  | { success: false; error: ZodError | Error; attempts: number; lastRawOutput: string }

Total LLM calls = maxRetries + 1 (1 initial + up to maxRetries retries).

extractJSON(text: string): unknown

Extracts and repairs JSON from LLM output. Handles:

| Input | Behaviour | |-------|-----------| | Plain JSON | Parsed directly | | ```json ... ``` or ``` ... ``` | Extracted from fence | | JSON embedded in surrounding text | Located by { / [ position | | Trailing commas ({"a": 1,}) | Stripped and re-parsed | | Unclosed brackets ({"a": [1, 2) | Closed and re-parsed |

Throws Error if no valid JSON can be extracted or repaired.

Retry Prompt Format

When validation fails, the retry prompt looks like:

Your previous JSON output has errors. Fix them and return the corrected JSON.

INVALID OUTPUT:
```json
{ "name": "John", "age": "thirty" }

ERRORS TO FIX:

• Field "age": Expected number, received string [code: invalid_type]

INSTRUCTIONS:

• Fix only the fields listed above
• Return the complete corrected JSON (include all fields, not just the changed ones)
• Return only raw JSON with no explanation, no markdown fences, no additional text

## Scripts

```bash
npm run build       # bundle to dist/ with tsup
npm run typecheck   # tsc --noEmit
npm test            # vitest run
npm run test:watch  # vitest watch mode

License

MIT