npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2026 – Pkg Stats / Ryan Hefner

@artemiskit/cli

v0.7.1

Published

Command-line interface for ArtemisKit LLM evaluation toolkit

Readme

@artemiskit/cli

Command-line interface for ArtemisKit - the LLM evaluation toolkit.

Installation

npm install -g @artemiskit/cli
# or
bun add -g @artemiskit/cli

Quick Start

# Initialize configuration
artemiskit init

# Run a test scenario
artemiskit run my-scenario.yaml

# Run red team security tests
artemiskit redteam my-scenario.yaml

# Run stress tests
artemiskit stress my-scenario.yaml --iterations 100 --concurrency 10

Commands

Controlled workflows

akit init agent-workflow
akit tools list
akit scenario validate workflow.yaml
akit workflow preflight workflow.yaml --config artemis.config.yaml --output probe.json
akit workflow run workflow.yaml --config artemis.config.yaml --output execution.json

Validation is offline. Preflight makes a bounded structured-tool round trip to the configured provider; run executes the declared workflow in a fresh simulated or disposable Docker environment. The workflow controls provider/model identity and tool authority. --output saves bounded metadata; --state-output explicitly exports sensitive working state to a separate new file with mode 0600. SIGINT/SIGTERM preserves an interrupted record when output was requested.

The 0.6.3 recovery interface adds --checkpoint-dir and --pause-after-actions to a run, and akit workflow resume <file> --checkpoint-dir <directory> for a compatible private checkpoint. A confirmed pause exits 10; budgets, tool-call cursor, configuration identity and the original deadline survive restart. Operational checkpoints contain sensitive working data and stay separate from saved public evidence. See the fault and recovery guide for complete CLI/SDK examples, supported crash boundaries and qualification status.

For 0.6.2 workflow runs, exit 0 means verified task success, 8 means a valid task failure, and 9 means invalid/unavailable outcome evaluation. Runtime/policy failures retain their existing exits. Successful preflight still returns 0 without assessing the task. Optional semantic criteria require --judge-config <file> with an explicit independent provider/model and bounded limits. See the outcome guide and execution guide for the full exit map, budgets, sandbox requirements, and SDK equivalents. The CLI requires Bun.

artemiskit run <scenario>

Execute scenario-based evaluations against LLM providers.

artemiskit run tests/auth-flow.yaml --provider openai --model gpt-4o

Options:

  • --provider <name> - LLM provider (openai, azure-openai, anthropic)
  • --model <name> - Model to use
  • --redact - Enable PII/sensitive data redaction
  • --redact-patterns <patterns...> - Custom redaction patterns
  • --config <path> - Path to config file

artemiskit redteam <scenario>

Run adversarial security tests including prompt injection, jailbreak attempts, data extraction probes, and agent-specific attacks.

# Basic red team testing
artemiskit redteam tests/chatbot.yaml --count 5

# OWASP LLM Top 10 testing
artemiskit redteam tests/chatbot.yaml --owasp LLM01,LLM05
artemiskit redteam tests/chatbot.yaml --owasp-full

# Agent-specific testing
artemiskit redteam tests/agent.yaml --mutations agent-confusion,tool-abuse --agent-detection trace

Options:

  • -c, --count <n> - Number of mutated prompts per case (default: 5)
  • --mutations <types...> - Mutations to apply:
    • Basic: typo, role-spoof, instruction-flip, cot-injection
    • Encoding: encoding (base64, ROT13, hex, unicode)
    • Multi-turn: multi_turn (gradual escalation, context switching)
    • Agent: agent-confusion, tool-abuse, memory-poisoning, chain-manipulation
    • OWASP: bad-likert-judge, crescendo, deceptive-delight, output-injection, excessive-agency, system-extraction, hallucination-trap
  • --owasp <categories> - Test by OWASP LLM category (e.g., LLM01,LLM05)
  • --owasp-full - Full OWASP LLM Top 10 compliance scan
  • --agent-detection <mode> - Detection mode for agent mutations (trace or response)
  • --min-severity <level> - Filter attacks by severity (low, medium, high, critical)
  • --custom-attacks <path> - Load custom attack patterns from YAML
  • --redact - Enable PII/sensitive data redaction

artemiskit stress <scenario>

Perform load and stress testing with detailed latency metrics.

artemiskit stress tests/api.yaml --requests 100 --concurrency 10

Options:

  • -n, --requests <n> - Total number of requests to make
  • -c, --concurrency <n> - Number of concurrent requests (default: 10)
  • -d, --duration <seconds> - Duration to run the test (default: 30)
  • --ramp-up <seconds> - Ramp-up time (default: 5)
  • --redact - Enable PII/sensitive data redaction

artemiskit report <manifest>

Regenerate HTML reports from saved run manifests.

artemiskit report artemis-runs/my-project/abc123.json

artemiskit history

View past test runs.

artemiskit history --limit 10

artemiskit compare <run1> <run2>

Compare results between two test runs.

artemiskit compare abc123 def456

artemiskit init

Initialize ArtemisKit configuration in your project.

artemiskit init

Scenario File Format

name: my-test-scenario
description: Test user authentication flow

config:
  provider: openai
  model: gpt-4o

cases:
  - id: login-success
    prompt: "How do I log in to my account?"
    expect:
      - type: contains
        value: "password"
      - type: contains
        value: "username"

  - id: password-reset
    prompt: "I forgot my password"
    expect:
      - type: contains
        value: "reset"

Configuration

Create artemis.config.yaml in your project root:

project: my-project

provider: openai
model: gpt-4o

providers:
  openai:
    apiKey: ${OPENAI_API_KEY}
  
  azure-openai:
    apiKey: ${AZURE_OPENAI_API_KEY}
    resourceName: ${AZURE_OPENAI_RESOURCE}
    deploymentName: ${AZURE_OPENAI_DEPLOYMENT}
    apiVersion: "2024-02-15-preview"

storage:
  type: local
  basePath: ./artemis-runs

Environment Variables

  • OPENAI_API_KEY - OpenAI API key
  • AZURE_OPENAI_API_KEY - Azure OpenAI API key
  • AZURE_OPENAI_RESOURCE - Azure resource name
  • AZURE_OPENAI_DEPLOYMENT - Azure deployment name
  • ANTHROPIC_API_KEY - Anthropic API key

Aliases

The CLI is also available as akit:

akit run my-scenario.yaml

Related Packages

Offline workflow reports

akit workflow report evidence.json --output assessment.html
akit workflow report evidence.json --format markdown --view executive --output executive.md

Technical, executive and comprehensive views use the same saved V1/V2/V3 evidence; comprehensive is the default. Multiple input files are supported with duplicate/resumed-attempt accounting. No provider credentials, model calls or Docker are needed. Output files must be new paths; omission of --output writes only the report to stdout. Exit 0 means export success, not a passed task. See the workflow report guide for limits and provenance.

License

Apache-2.0