notdiamond-cli

Command-line interface for NotDiamond's prompt adaptation service.

Installation

npm install -g notdiamond-cli

Or use directly with npx:

npx notdiamond-cli

Quick Start

1. Configure your API key

notdiamond-cli auth

Enter your NotDiamond API key when prompted. The key will be securely stored in ~/.notdiamond/config.json.

2. Adapt a prompt

notdiamond-cli adapt \
  --origin "openai/gpt-4o-2024-11-20" \
  --target "anthropic/claude-3-5-sonnet-20241022" \
  --system "You are a helpful assistant" \
  --template "Answer this question: {question}" \
  --goldens '[{"question": "What is 2+2?", "answer": "4"}]'

3. Check adaptation status

notdiamond-cli status <run-id>

4. View results

notdiamond-cli results <run-id>

Commands

notdiamond-cli auth

Configure your NotDiamond API key.

notdiamond-cli config [setting] [value]

View or update configuration settings.

Available settings:

• min-goldens: Minimum number of golden examples (default: 25)
• auto-generate: Auto-generate sample goldens if needed (default: true)
• api-url: API endpoint URL

Examples:

notdiamond-cli config                    # View all settings
notdiamond-cli config min-goldens 10     # Set minimum goldens to 10
notdiamond-cli config auto-generate false # Disable auto-generation

notdiamond-cli adapt

Adapt a prompt for optimal performance across models.

Options:

• -o, --origin <model> - Origin model
• -t, --target <models...> - Target models to optimize for
• -s, --system <prompt> - System prompt
• --template <template> - User message template
• -f, --fields <fields...> - Template fields
• -g, --goldens <json> - Golden examples (JSON array)
• -m, --metric <metric> - Evaluation metric
• --min-goldens <number> - Minimum number of golden examples (default: 25)
• --auto-generate - Auto-generate sample goldens if needed (default: true)
• --no-auto-generate - Disable auto-generation of sample goldens

notdiamond-cli status <runId>

Check the status of an adaptation run.

notdiamond-cli results <runId>

Get the results of a completed adaptation run.

Supported Models

Models should be specified in provider/model format:

OpenAI

• openai/gpt-4o-2024-11-20
• openai/gpt-4o-2024-08-06
• openai/gpt-4o-mini-2024-07-18
• openai/gpt-4.1-2025-04-14
• openai/gpt-4.1-mini-2025-04-14
• openai/gpt-4.1-nano-2025-04-14

Anthropic

• anthropic/claude-3-5-sonnet-20241022
• anthropic/claude-3-7-sonnet-20250219
• anthropic/claude-sonnet-4-20250514
• anthropic/claude-opus-4-20250514

Google

• google/gemini-1.5-pro-latest
• google/gemini-2.5-flash
• google/gemini-2.5-pro

Mistral

• mistral/mistral-large-2411

Meta Llama (via OpenRouter)

• openrouter/llama-3.1-8b-instruct
• openrouter/llama-3.1-70b-instruct
• openrouter/llama-3.1-405b-instruct

Qwen (via OpenRouter)

• openrouter/qwen3-14b
• openrouter/qwen3-32b
• openrouter/qwen3-235b-a22b

Evaluation Metrics

• LLMaaJ:SQL - SQL query evaluation
• LLMaaJ:Sem_Sim_1 - Semantic similarity (top-1)
• LLMaaJ:Sem_Sim_3 - Semantic similarity (top-3)
• LLMaaJ:Sem_Sim_10 - Semantic similarity (top-10)
• LLMaaJ:Unsupervised_Correctness_1 - Unsupervised correctness evaluation
• JSON_Match - JSON output matching

Golden Examples

The API requires at least 25 golden examples by default. You can:

1. Provide your own goldens - Recommended for best results
2. Auto-generate sample goldens - The CLI will generate sample data if needed (enabled by default)
3. Reduce the minimum - Use --min-goldens flag or configure globally

Customizing Golden Requirements

# Via command flags
notdiamond-cli adapt --min-goldens 5 --no-auto-generate

# Via configuration
notdiamond-cli config min-goldens 10
notdiamond-cli config auto-generate false

# Via environment variables
export NOTDIAMOND_MIN_GOLDENS=5
export NOTDIAMOND_AUTO_GENERATE_GOLDENS=false

Examples

Basic adaptation

notdiamond-cli adapt \
  --origin "openai/gpt-4o-2024-11-20" \
  --target "anthropic/claude-3-5-sonnet-20241022" \
  --system "You are an expert Python developer" \
  --template "Write a function that {task}" \
  --fields "task" \
  --goldens '[
    {"task": "reverses a string", "answer": "def reverse_string(s): return s[::-1]"},
    {"task": "checks if a number is prime", "answer": "def is_prime(n): return n > 1 and all(n % i != 0 for i in range(2, int(n**0.5) + 1))"}
  ]' \
  --metric "LLMaaJ:Sem_Sim_1"

Multiple target models

notdiamond-cli adapt \
  --origin "openai/gpt-4o-2024-11-20" \
  --target "anthropic/claude-3-5-sonnet-20241022" \
           "google/gemini-1.5-pro-latest" \
           "mistral/mistral-large-2411" \
  --system "You are a helpful assistant" \
  --template "Explain {concept} in simple terms" \
  --goldens '[{"concept": "quantum computing", "answer": "..."}]'

Environment Variables

• NOTDIAMOND_API_KEY - Your NotDiamond API key
• NOTDIAMOND_API_URL - API endpoint (default: https://api.notdiamond.ai)
• NOTDIAMOND_MIN_GOLDENS - Minimum golden examples required
• NOTDIAMOND_AUTO_GENERATE_GOLDENS - Enable/disable auto-generation (true/false)

License

MIT