Tardi is an open-source testing framework engineered specifically for evaluating agentic workflows, autonomous LLM scripts, and non-deterministic applications.

About

Testing AI agents presents a unique challenge: traditional testing frameworks cannot evaluate non-deterministic natural language outputs, and raw "LLM-as-a-judge" evaluation pipelines are prohibitively expensive and prone to hallucination at scale.

Tardi solves this by implementing a tiered assertion gauntlet. Instead of blindly sending every agent execution trace to an evaluation model, Tardi enforces strict deterministic constraints first. If your agent crashes, hangs in an infinite loop, or returns malformed JSON, Tardi fails the test immediately—preventing unnecessary LLM API calls and accelerating your feedback loop.

Features

• Tiered Evaluation Engine: Catch process crashes, timeouts, and schema mismatches deterministically before triggering expensive LLM judges.
• Concurrency & Rate-Limiting: Execute test suites in parallel with intelligent chunking to maximize throughput without exceeding API rate limits.
• Provider Agnostic: Built on the standard Vercel AI SDK, Tardi supports hot-swappable evaluation models from OpenAI, Google, Anthropic, and local endpoints.
• Interactive REPL & Natural Language CLI: Includes a zero-friction CLI environment for rapid test synthesis, execution, and debugging, powered by an onboard NLP intent parser.
• Secure Credential Management: Safely stores provider API keys in your native OS keychain during local development.

The Tardi Pipeline

When you execute an iteration, Tardi evaluates the agent's output through a strict, cost-saving pipeline. Deterministic checks always run first.

graph TD
    Start([Run Agent]) --> ProcessCheck{Process Crashed?}
    ProcessCheck -- Yes --> FailCrash[FAIL: CRASH]
    ProcessCheck -- No --> TimeoutCheck{Exceeded Timeout?}
    
    TimeoutCheck -- Yes --> FailTimeout[FAIL: TIMEOUT]
    TimeoutCheck -- No --> RegexCheck{Matches Regex?}
    
    RegexCheck -- No --> FailRegex[FAIL: REGEX_MISMATCH]
    RegexCheck -- Yes --> SchemaCheck{Valid JSON Schema?}
    
    SchemaCheck -- No --> FailSchema[FAIL: SCHEMA_MISMATCH]
    SchemaCheck -- Yes --> LLMJudge{LLM Judge Approves?}
    
    LLMJudge -- No --> FailJudge[FAIL: LLM_JUDGE_FAIL]
    LLMJudge -- Yes --> Pass([PASS])
    
    style FailCrash fill:#fbd4d4,stroke:#f87171,color:#000
    style FailTimeout fill:#fbd4d4,stroke:#f87171,color:#000
    style FailRegex fill:#fbd4d4,stroke:#f87171,color:#000
    style FailSchema fill:#fbd4d4,stroke:#f87171,color:#000
    style FailJudge fill:#fbd4d4,stroke:#f87171,color:#000
    style Pass fill:#dcfce7,stroke:#4ade80,color:#000

Installation

Install the CLI globally via npm to use the tardi command anywhere:

npm install -g tardi-cli

Quickstart

1. Auto-Synthesize from a GitHub Repository

Tardi can automatically clone an agent repository, detect its entry point, and synthesize a test gauntlet based on golden runs.

tardi github https://github.com/Nyx-abu/demo-agent.git

2. Manual Initialization

If you prefer to configure your suite manually:

tardi init
tardi auth login google
tardi run tests/

Examples & Output

Tardi utilizes simple YAML files (*.tardi.yaml) to define test suites. Below are detailed examples of how Tardi behaves under different failure and success conditions.

Scenario A: Deterministic JSON Schema Validation

A common requirement for AI agents is to return structured JSON. Tardi catches malformed JSON or missing fields deterministically, bypassing the LLM Judge entirely.

Test Configuration:

# tests/json-schema.tardi.yaml
name: Agent JSON Output Test
command: node src/agent.js
iterations: 5
concurrency: 2

assertions:
  jsonSchema:
    type: object
    required: ["status", "result"]

Expected Tardi Output (Schema Mismatch): If the agent forgets to include the result field, Tardi immediately aborts the pipeline and outputs a schema mismatch error:

 ╭────────────────────────  Tardi Execution Summary  ────────────────────────╮
 │   ┌─────────────┬──────────┐                                              │
 │   │ Metric      │ Value    │                                              │
 │   ├─────────────┼──────────┤                                              │
 │   │ Total Runs  │ 5        │                                              │
 │   │ Passed      │ 4        │                                              │
 │   │ Failed      │ 1        │                                              │
 │   │ Flakiness   │ 20.00%   │                                              │
 │   └─────────────┴──────────┘                                              │
 │                                                                           │
 │   Failures:                                                               │
 │   - Iteration 3 [SCHEMA_MISMATCH]: Output failed JSON Schema validation.  │
 │     Missing required property: 'result'.                                  │
 ╰───────────────────────────────────────────────────────────────────────────╯

Scenario B: Semantic Evaluation with LLM-as-a-Judge

If your agent passes all deterministic constraints, Tardi forwards the raw output to an LLM evaluator. The LLM evaluator uses your custom rubric to score the output.

Test Configuration:

# tests/evaluator.tardi.yaml
name: Semantic Agent Test
command: node src/summarize.js
iterations: 3

assertions:
  regex: "summary:"

evaluator:
  provider: google
  model: gemini-2.5-flash
  prompt: |
    Evaluate if the agent successfully summarized the input document.
    Return only 'PASS' or 'FAIL'.

Expected Tardi Output (LLM Judge Failure):

Scenario C: Uncaught Exceptions & Process Crashes

When testing complex autonomous scripts, agents often crash mid-execution. Tardi intercepts process crashes and records the exit codes, allowing you to debug stability issues over multiple iterations.

Expected Tardi Output (Crash):

 ╭────────────────────────  Tardi Execution Summary  ────────────────────────╮
 │   Failures:                                                               │
 │   - Iteration 2 [CRASH]: Agent process exited with code 1.                │
 │     Stderr: Error: Uncaught exception in src/agent.js:42                  │
 ╰───────────────────────────────────────────────────────────────────────────╯

CI/CD Usage

Tardi detects continuous integration environments automatically and disables interactive prompts. Inject API keys directly via environment variables for your pipeline:

GOOGLE_GENERATIVE_AI_API_KEY="your-key-here" tardi run tests/

Contributing

Pull requests are welcome. For major architectural changes, please open an issue first to discuss your proposed modifications.

Please see our Contributing Guide and our Code of Conduct.

License

This project is licensed under the ISC License.