prompt-check

Open source CLI tool to validate AI outputs against expected outputs using rules and concept matching.

Quick Start

Install and Use

# Install globally
npm install -g prompt-check

# Or use with npx (no installation needed)
npx prompt-check init
npx prompt-check run

Setup

1. Initialize the data folder (creates structure and prompts for API key):

prompt-check init

2. Add your test data - Edit the JSON files in prompt-check-data/test-cases/:

   • Fill in referenceOutput (what you expect)
   • Fill in outputs (what your AI produced)

3. Run tests:

prompt-check run

Configuration

API Key Setup

Option A: Environment variable (recommended for CI/CD):

export OPENAI_API_KEY=your-api-key

Option B: Local config (saved during init): The API key is stored in prompt-check-data/config.json

Folder Structure

After running init, you'll have:

prompt-check-data/
  ├── config.json           # OpenAI API key (not committed to git)
  ├── rules.json            # Rules for validation
  └── test-cases/
      ├── example-test-case.json
      └── example-with-concepts.json

File Formats

rules.json

{
  "rules": [
    "be short and concise",
    "be friendly",
    "use clear and professional language"
  ]
}

Test Case File

Basic test case:

{
  "id": "my-test-case",
  "referenceOutput": "The expected output from your AI.",
  "outputs": "The actual output from your AI system."
}

With concept validation:

{
  "id": "support-response",
  "referenceOutput": "A customer support response should acknowledge the issue.",
  "outputs": "I understand this is frustrating. Your ticket is being reviewed.",
  "expectedConcepts": [
    "acknowledgment",
    "ticket reference",
    "timeline"
  ]
}

CLI Commands

Initialize

prompt-check init

Creates the data folder structure with example files.

Run Tests

# Run all test cases
prompt-check run

# Run a specific test case
prompt-check run --file example-test-case.json

# Use custom rules file
prompt-check run --rules custom-rules.json

# Use custom data directory
prompt-check run --dir ./my-data-folder

How It Works

1. Rules Validation: Checks if outputs follow the rules defined in rules.json
2. Concept Matching: Validates that expected concepts are present (if expectedConcepts is specified)
3. Scoring: Returns a percentage score with PASS/FAIL for each rule

Results show:

• Overall score percentage
• Per-rule breakdown with explanations
• Concept validation results

License

MIT