@artemiskit/cli
v0.7.1
Published
Command-line interface for ArtemisKit LLM evaluation toolkit
Maintainers
Readme
@artemiskit/cli
Command-line interface for ArtemisKit - the LLM evaluation toolkit.
Installation
npm install -g @artemiskit/cli
# or
bun add -g @artemiskit/cliQuick Start
# Initialize configuration
artemiskit init
# Run a test scenario
artemiskit run my-scenario.yaml
# Run red team security tests
artemiskit redteam my-scenario.yaml
# Run stress tests
artemiskit stress my-scenario.yaml --iterations 100 --concurrency 10Commands
Controlled workflows
akit init agent-workflow
akit tools list
akit scenario validate workflow.yaml
akit workflow preflight workflow.yaml --config artemis.config.yaml --output probe.json
akit workflow run workflow.yaml --config artemis.config.yaml --output execution.jsonValidation is offline. Preflight makes a bounded structured-tool round trip to the configured
provider; run executes the declared workflow in a fresh simulated or disposable Docker environment.
The workflow controls provider/model identity and tool authority. --output saves bounded metadata;
--state-output explicitly exports sensitive working state to a separate new file with mode 0600.
SIGINT/SIGTERM preserves an interrupted record when output was requested.
The 0.6.3 recovery interface adds --checkpoint-dir and --pause-after-actions to a run,
and akit workflow resume <file> --checkpoint-dir <directory> for a compatible private checkpoint.
A confirmed pause exits 10; budgets, tool-call cursor, configuration identity and the original
deadline survive restart. Operational checkpoints contain sensitive working data and stay separate
from saved public evidence. See the fault and recovery guide
for complete CLI/SDK examples, supported crash boundaries and qualification status.
For 0.6.2 workflow runs, exit 0 means verified task success, 8 means a valid task failure,
and 9 means invalid/unavailable outcome evaluation. Runtime/policy failures retain their existing
exits. Successful preflight still returns 0 without assessing the task. Optional semantic criteria
require --judge-config <file> with an explicit independent provider/model and bounded limits. See the
outcome guide and
execution guide
for the full exit map, budgets, sandbox requirements, and SDK equivalents. The CLI requires Bun.
artemiskit run <scenario>
Execute scenario-based evaluations against LLM providers.
artemiskit run tests/auth-flow.yaml --provider openai --model gpt-4oOptions:
--provider <name>- LLM provider (openai, azure-openai, anthropic)--model <name>- Model to use--redact- Enable PII/sensitive data redaction--redact-patterns <patterns...>- Custom redaction patterns--config <path>- Path to config file
artemiskit redteam <scenario>
Run adversarial security tests including prompt injection, jailbreak attempts, data extraction probes, and agent-specific attacks.
# Basic red team testing
artemiskit redteam tests/chatbot.yaml --count 5
# OWASP LLM Top 10 testing
artemiskit redteam tests/chatbot.yaml --owasp LLM01,LLM05
artemiskit redteam tests/chatbot.yaml --owasp-full
# Agent-specific testing
artemiskit redteam tests/agent.yaml --mutations agent-confusion,tool-abuse --agent-detection traceOptions:
-c, --count <n>- Number of mutated prompts per case (default: 5)--mutations <types...>- Mutations to apply:- Basic:
typo,role-spoof,instruction-flip,cot-injection - Encoding:
encoding(base64, ROT13, hex, unicode) - Multi-turn:
multi_turn(gradual escalation, context switching) - Agent:
agent-confusion,tool-abuse,memory-poisoning,chain-manipulation - OWASP:
bad-likert-judge,crescendo,deceptive-delight,output-injection,excessive-agency,system-extraction,hallucination-trap
- Basic:
--owasp <categories>- Test by OWASP LLM category (e.g.,LLM01,LLM05)--owasp-full- Full OWASP LLM Top 10 compliance scan--agent-detection <mode>- Detection mode for agent mutations (traceorresponse)--min-severity <level>- Filter attacks by severity (low, medium, high, critical)--custom-attacks <path>- Load custom attack patterns from YAML--redact- Enable PII/sensitive data redaction
artemiskit stress <scenario>
Perform load and stress testing with detailed latency metrics.
artemiskit stress tests/api.yaml --requests 100 --concurrency 10Options:
-n, --requests <n>- Total number of requests to make-c, --concurrency <n>- Number of concurrent requests (default: 10)-d, --duration <seconds>- Duration to run the test (default: 30)--ramp-up <seconds>- Ramp-up time (default: 5)--redact- Enable PII/sensitive data redaction
artemiskit report <manifest>
Regenerate HTML reports from saved run manifests.
artemiskit report artemis-runs/my-project/abc123.jsonartemiskit history
View past test runs.
artemiskit history --limit 10artemiskit compare <run1> <run2>
Compare results between two test runs.
artemiskit compare abc123 def456artemiskit init
Initialize ArtemisKit configuration in your project.
artemiskit initScenario File Format
name: my-test-scenario
description: Test user authentication flow
config:
provider: openai
model: gpt-4o
cases:
- id: login-success
prompt: "How do I log in to my account?"
expect:
- type: contains
value: "password"
- type: contains
value: "username"
- id: password-reset
prompt: "I forgot my password"
expect:
- type: contains
value: "reset"Configuration
Create artemis.config.yaml in your project root:
project: my-project
provider: openai
model: gpt-4o
providers:
openai:
apiKey: ${OPENAI_API_KEY}
azure-openai:
apiKey: ${AZURE_OPENAI_API_KEY}
resourceName: ${AZURE_OPENAI_RESOURCE}
deploymentName: ${AZURE_OPENAI_DEPLOYMENT}
apiVersion: "2024-02-15-preview"
storage:
type: local
basePath: ./artemis-runsEnvironment Variables
OPENAI_API_KEY- OpenAI API keyAZURE_OPENAI_API_KEY- Azure OpenAI API keyAZURE_OPENAI_RESOURCE- Azure resource nameAZURE_OPENAI_DEPLOYMENT- Azure deployment nameANTHROPIC_API_KEY- Anthropic API key
Aliases
The CLI is also available as akit:
akit run my-scenario.yamlRelated Packages
@artemiskit/core- Core runtime and evaluators@artemiskit/adapter-openai- OpenAI/Azure adapter@artemiskit/adapter-vercel-ai- Vercel AI SDK adapter (20+ providers)@artemiskit/adapter-anthropic- Anthropic Claude adapter@artemiskit/redteam- Security testing@artemiskit/reports- HTML report generation
Offline workflow reports
akit workflow report evidence.json --output assessment.html
akit workflow report evidence.json --format markdown --view executive --output executive.mdTechnical, executive and comprehensive views use the same saved V1/V2/V3 evidence; comprehensive
is the default. Multiple input files are supported with duplicate/resumed-attempt accounting.
No provider credentials, model calls or Docker are needed. Output files must be new paths; omission
of --output writes only the report to stdout. Exit 0 means export success, not a passed task.
See the workflow report guide for limits and provenance.
License
Apache-2.0
