@manet/agent-snapshot

A snapshot-based agent test framework for @manet/agent based Agents.

Overview

@manet/agent-snapshot provides a comprehensive testing framework for AI agents built with @manet/agent. It enables you to:

• Generate snapshots of real agent interactions with LLMs
• Replay tests using captured snapshots for deterministic, fast testing
• Verify behavior by comparing expected vs actual responses, tool calls, and event streams
• Normalize dynamic data to ignore timestamps, IDs, and other volatile values

Installation

npm install @manet/agent-snapshot

Quick Start

Basic Usage

import { AgentSnapshot } from '@manet/agent-snapshot';
import { Agent } from '@manet/agent';

// Create your agent instance
const agent = new Agent({
  // ... your agent configuration
});

// Create snapshot instance
const agentSnapshot = new AgentSnapshot(agent, {
  snapshotPath: './fixtures/my-test-case',
  snapshotName: 'my-test-case'
});

// Generate snapshots from real LLM interactions
await agentSnapshot.generate({
  input: 'Hello, how can you help me?'
});

// Run tests using captured snapshots
const result = await agentSnapshot.replay({
  input: 'Hello, how can you help me?'
});

Using as a Transparent Wrapper

// AgentSnapshot can be used as a drop-in replacement for Agent
const response = await agentSnapshot.run({
  input: 'What is the weather today?',
  stream: false
});

// Or with streaming
for await (const event of await agentSnapshot.run({
  input: 'Tell me a story',
  stream: true
})) {
  console.log(event);
}

Core Concepts

Snapshots

Snapshots capture the complete state of agent interactions:

• LLM Requests: Prompts sent to the language model
• LLM Responses: Model responses (including streaming chunks)
• Tool Calls: Function calls executed by the agent
• Event Streams: Complete agent event timeline

Directory Structure

fixtures/
└── my-test-case/
    ├── initial/
    │   └── event-stream.jsonl
    ├── loop-1/
    │   ├── llm-request.jsonl
    │   ├── llm-response.jsonl
    │   ├── tool-calls.jsonl
    │   └── event-stream.jsonl
    ├── loop-2/
    │   └── ...
    └── event-stream.jsonl

API Reference

AgentSnapshot

The main class for managing agent snapshots and testing.

Constructor

new AgentSnapshot(agent: Agent, options: AgentSnapshotOptions)

Options:

• snapshotPath: string - Directory to store snapshots
• snapshotName?: string - Name for the snapshot
• updateSnapshots?: boolean - Whether to update existing snapshots
• normalizerConfig?: AgentNormalizerConfig - Configuration for data normalization
• verification?: VerificationOptions - What to verify during tests

Methods

generate(runOptions)

Generate snapshots by executing the agent with real LLM calls.

const result = await agentSnapshot.generate({
  input: 'Your test input here'
});

Returns: SnapshotGenerationResult

replay(runOptions, config?)

Run tests using previously captured snapshots.

const result = await agentSnapshot.replay({
  input: 'Your test input here'
}, {
  updateSnapshots: false,
  verification: {
    verifyLLMRequests: true,
    verifyEventStreams: true,
    verifyToolCalls: true
  }
});

Returns: SnapshotRunResult

run(runOptions)

Transparent wrapper around the agent's run method that generates snapshots.

// Non-streaming
const response = await agentSnapshot.run({
  input: 'Your input',
  stream: false
});

// Streaming
for await (const event of await agentSnapshot.run({
  input: 'Your input',
  stream: true
})) {
  console.log(event);
}

Configuration

AgentNormalizerConfig

Configure how dynamic data is normalized during comparisons.

const normalizerConfig: AgentNormalizerConfig = {
  fieldsToNormalize: [
    { pattern: /id$/, replacement: '<<ID>>' },
    { pattern: 'timestamp', replacement: '<<TIMESTAMP>>' },
    { pattern: /Time$/, replacement: '<<TIMESTAMP>>' }
  ],
  fieldsToIgnore: [
    'debugInfo',
    /internal_/
  ],
  customNormalizers: [
    {
      pattern: 'customField',
      normalizer: (value, path) => normalizeCustomValue(value)
    }
  ]
};

VerificationOptions

Control what aspects of agent behavior are verified.

const verification: VerificationOptions = {
  verifyLLMRequests: true,    // Verify LLM prompts match
  verifyEventStreams: true,   // Verify event streams match
  verifyToolCalls: true       // Verify tool calls match
};

Advanced Usage

Custom Normalization

Handle dynamic data that shouldn't affect test results:

const agentSnapshot = new AgentSnapshot(agent, {
  snapshotPath: './fixtures/test',
  normalizerConfig: {
    fieldsToNormalize: [
      { pattern: 'sessionId', replacement: '<<SESSION_ID>>' },
      { pattern: 'elapsedMs', replacement: '<<ELAPSED_MS>>' },
      { pattern: /_at$/, replacement: '<<TIMESTAMP>>' }
    ],
    customNormalizers: [
      {
        pattern: 'userMessage',
        normalizer: (value, path) => {
          // Normalize user IDs in messages
          if (typeof value === 'string') {
            return value.replace(/user-\d+/g, 'user-<<ID>>');
          }
          return value;
        }
      }
    ]
  }
});

Selective Verification

Choose what to verify based on your test needs:

// Only verify LLM requests, ignore timing differences
const result = await agentSnapshot.replay(runOptions, {
  verification: {
    verifyLLMRequests: true,
    verifyEventStreams: false,  // Skip event stream verification
    verifyToolCalls: false      // Skip tool call verification
  }
});

Updating Snapshots

Update snapshots when behavior changes intentionally:

// Update mode - skips verification, updates snapshots
const result = await agentSnapshot.replay(runOptions, {
  updateSnapshots: true
});

Batch Testing with AgentSnapshotRunner

For testing multiple scenarios:

import { AgentSnapshotRunner } from '@manet/agent-snapshot';

const runner = new AgentSnapshotRunner([
  {
    name: 'basic-conversation',
    path: './test-cases/basic-conversation.js',
    snapshotPath: './fixtures/basic-conversation',
    vitestSnapshotPath: './tests/snapshots/basic-conversation'
  },
  {
    name: 'tool-usage',
    path: './test-cases/tool-usage.js',
    snapshotPath: './fixtures/tool-usage',
    vitestSnapshotPath: './tests/snapshots/tool-usage'
  }
]);

// Generate all snapshots
await runner.generateAll();

// Run all tests
await runner.replayAll();

Testing Framework Integration

Vitest Integration

import { test, expect } from 'vitest';
import { AgentSnapshot } from '@manet/agent-snapshot';

test('agent handles basic conversation', async () => {
  const agent = createTestAgent();
  const snapshot = new AgentSnapshot(agent, {
    snapshotPath: './fixtures/basic-conversation'
  });

  const result = await snapshot.replay({
    input: 'Hello, how are you?'
  });

  expect(result.events).toHaveLength(expectedEventCount);
  expect(result.meta.loopCount).toBe(expectedLoopCount);
});

CI/CD Integration

# Generate snapshots (only when needed)
npm run test:snapshots:generate

# Run tests against snapshots
npm run test:snapshots

# Update snapshots (when behavior changes)
npm run test:snapshots:update

Best Practices

1. Organize Tests Logically

fixtures/
├── user-interactions/
│   ├── basic-chat/
│   ├── complex-queries/
│   └── error-handling/
├── tool-usage/
│   ├── single-tools/
│   ├── multi-tools/
│   └── tool-errors/
└── edge-cases/
    ├── time-outs/
    ├── rate-limits/
    └── network-errors/

2. Use Descriptive Names

// Good
const snapshot = new AgentSnapshot(agent, {
  snapshotPath: './fixtures/user-interactions/basic-chat/greeting-response'
});

// Avoid
const snapshot = new AgentSnapshot(agent, {
  snapshotPath: './fixtures/test1'
});

3. Normalize Volatile Data

Always normalize timestamps, IDs, and other dynamic values:

const normalizerConfig = {
  fieldsToNormalize: [
    { pattern: /id$/, replacement: '<<ID>>' },
    { pattern: 'timestamp', replacement: '<<TIMESTAMP>>' },
    { pattern: 'requestId', replacement: '<<REQUEST_ID>>' }
  ]
};

4. Version Control Snapshots

Commit snapshots to version control to track behavior changes:

# Add snapshots to git
git add fixtures/
git commit -m "feat: add snapshots for user greeting flow"

# Update snapshots when behavior changes
npm run test:snapshots:update
git add fixtures/
git commit -m "feat: update snapshots for improved response format"

Troubleshooting

Common Issues

"No snapshot found" Error

Ensure snapshots are generated before running tests:

# Generate snapshots first
await agentSnapshot.generate(runOptions);

# Then run tests
await agentSnapshot.replay(runOptions);

"Loop count mismatch" Error

This indicates the agent behaved differently during test vs snapshot generation:

// Check loop counts in generation result
const genResult = await agentSnapshot.generate(runOptions);
console.log('Generated loops:', genResult.loopCount);

// Compare with test result
const testResult = await agentSnapshot.replay(runOptions);
console.log('Test loops:', testResult.meta.loopCount);

"Event stream doesn't match" Error

Use normalizer to ignore expected differences:

const snapshot = new AgentSnapshot(agent, {
  snapshotPath: './fixtures/test',
  normalizerConfig: {
    fieldsToIgnore: ['timing', 'debugInfo'],
    fieldsToNormalize: [
      { pattern: /elapsedMs$/, replacement: '<<ELAPSED_MS>>' }
    ]
  }
});

Debug Mode

Enable detailed logging:

import { logger } from '@manet/agent-snapshot';

// Enable debug logging
logger.setLevel('debug');

Contributing

1. Fork the repository
2. Create a feature branch
3. Add tests for new functionality
4. Ensure all tests pass
5. Submit a pull request

License

Apache-2.0 © Bytedance, Inc.