@artemiskit/core

Core runtime, evaluators, and storage for the ArtemisKit LLM evaluation toolkit.

Installation

npm install @artemiskit/core
# or
bun add @artemiskit/core

Overview

This package provides the foundational components for ArtemisKit:

• Runner - Execute test scenarios against LLM providers
• Evaluators - Built-in expectation matchers (contains, exact, regex, fuzzy, llm_grader, json_schema)
• Storage - Local filesystem and Supabase storage backends
• Redaction - PII/sensitive data filtering with built-in patterns
• Schema - YAML scenario parsing and validation

Usage

Most users should use the @artemiskit/cli package for running evaluations. This package is primarily for programmatic usage or building custom integrations.

import { 
  parseScenarioFile, 
  runScenario, 
  createAdapter 
} from '@artemiskit/core';

// Parse a scenario file
const scenario = await parseScenarioFile('./my-test.yaml');

// Create an adapter (requires adapter package to be registered)
const client = await createAdapter({
  provider: 'openai',
  apiKey: process.env.OPENAI_API_KEY,
});

// Run the scenario
const result = await runScenario({
  scenario,
  client,
  project: 'my-project',
  onCaseComplete: (caseResult, index, total) => {
    console.log(`Completed ${index + 1}/${total}: ${caseResult.passed ? 'PASS' : 'FAIL'}`);
  },
});

console.log(`Success: ${result.success}`);
console.log(`Passed: ${result.manifest.metrics.passed_cases}/${result.manifest.metrics.total_cases}`);

Note: The createAdapter function requires adapter packages (like @artemiskit/adapter-openai) to be installed and registered. For standalone usage, import adapters directly:

import { OpenAIAdapter } from '@artemiskit/adapter-openai';

const client = new OpenAIAdapter({
  provider: 'openai',
  apiKey: process.env.OPENAI_API_KEY,
});

Evaluators

Built-in expectation types:

| Type | Description | |------|-------------| | contains | Check if response contains text | | exact | Exact string match | | regex | Regular expression match | | fuzzy | Fuzzy string similarity (Levenshtein) | | llm_grader | LLM-based response grading | | json_schema | Validate JSON structure |

Redaction

Built-in patterns for sensitive data:

• email - Email addresses
• phone - Phone numbers
• ssn - Social Security Numbers
• credit_card - Credit card numbers
• api_key - API keys (sk_, pk_, etc.)
• jwt - JWT tokens
• aws_key - AWS access keys
• ipv4 - IPv4 addresses
• secrets - Password/secret assignments

import { Redactor, BUILTIN_PATTERNS } from '@artemiskit/core';

const redactor = new Redactor({
  enabled: true,
  patterns: [BUILTIN_PATTERNS.EMAIL, BUILTIN_PATTERNS.PHONE],
});

const result = redactor.redact('Contact: [email protected]');
// result.text = 'Contact: [REDACTED]'

Related Packages

• @artemiskit/cli - Command-line interface
• @artemiskit/adapter-openai - OpenAI/Azure adapter
• @artemiskit/adapter-vercel-ai - Vercel AI SDK adapter (20+ providers)
• @artemiskit/adapter-anthropic - Anthropic Claude adapter
• @artemiskit/redteam - Security testing
• @artemiskit/reports - HTML report generation

License

Apache-2.0