Autoverify

The official CLI tool for interacting with the VERIFY (Vulnerability, Evaluation, and Robustness Integrity) Framework.

Autoverify allows you to interactively configure your agent interaction points and connect your local agents to the Streaming Adversarial Evaluation Framework.

Features

• Automated Codebase Scanning: Automatically finds Python entry points in your agent codebase.
• Interactive Configuration: Select the agent, configure parameters, and set safety rules.
• WebSocket Integration: Connects to the EvalFramework server to seamlessly run the adversarial loop.
• YAML Configuration: Saves all setup to a standard verify.yaml file for reproducibility.

Installation

You can install the CLI globally using npm:

npm install -g autoverify

## Usage

1. Start the EvalFramework server (usually `python3 main.py` in the EvalFramework directory).
2. Navigate to your agent's codebase.
3. Run the CLI:

```bash
autoverify

The CLI will interactively guide you through:

• Locating your agent's entry point file.
• Selecting the function where to call the agent.
• Adding a description of the test agent.
• Configuring the rules you want to evaluate.
• Connecting to the test agent websocket at ws://localhost:8765.

Configuration (verify.yaml)

The CLI automatically persists your settings into a verify.yaml file in the current directory. An example verify.yaml:

file: examples/python/ai_agent.py
function: handle_query
description: retail customer service agent
parameters:
  - name: prompt
    type: str
rules:
  - "The agent must never disclose internal warehouse stock quantities"

License

ISC