FireGlobe Agent Testing SDK

Universal testing framework for blockchain agents using AI-generated personalities

Overview

FireGlobe SDK is a comprehensive testing framework designed to rigorously evaluate AI agents using diverse AI-generated personas powered by ASI:One. The framework generates multiple distinct test personalities that simulate realistic user behaviors and edge cases, providing thorough testing beyond traditional evaluation methods.

Features

• Universal Agent Interface: Test any agent regardless of framework (LangChain, OpenAI SDK, Custom, etc.)
• AI-Powered Personalities: Generate unique test personalities tailored to your agent
• Automated Conversations: Each personality engages in natural conversations with your agent
• AI Evaluation: Advanced AI evaluation using ASI:One for comprehensive performance assessment
• Real-Time Logging: Watch conversations unfold in real-time
• Comprehensive Reports: Get detailed HTML and JSON reports
• Framework Agnostic: Works with any agent that implements the simple interface
• Blockchain Transaction Analysis: Integrates with BlockScout MCP for transaction analysis

Installation

npm install fireglobe-sdk-client

Quick Start

1. Install the SDK

npm install fireglobe-sdk-client

2. Implement the Universal Agent Interface

Any agent must implement the IUniversalAgent interface:

import { IUniversalAgent } from "fireglobe-sdk-client";

class MyAgent implements IUniversalAgent {
  async sendMessage(message: string): Promise<string> {
    // Your agent logic here
    return "Agent response";
  }

  async reset(): Promise<void> {
    // Reset conversation state
  }

  getMetadata?() {
    return {
      name: "My Agent",
      description: "Description of what your agent does",
      framework: "YourFramework",
      version: "1.0.0"
    };
  }
}

3. Get Your Access Token

Sign up at fireglobe.vercel.app to get your access token and add it to your .env

4. Run Tests

import { AgentTester } from "fireglobe-sdk-client";

const tester = new AgentTester({
  accessToken: process.env.ACCESS_TOKEN!, // Get from fireglobe.vercel.app
  agentDescription: "A DeFi agent that helps users with lending and borrowing"
});

// Add event listeners for real-time updates
tester.on((event) => {
  console.log(event);
});

// Run tests
const results = await tester.runTests(myAgent);

console.log(`Overall Score: ${results.overallScore}/100`);

LangChain Integration

We provide a ready-made adapter for LangChain-based agents:

import { AgentTester, LangChainAdapter } from "fireglobe-sdk-client";
import { createReactAgent } from "@langchain/langgraph/prebuilt";

// Your existing LangChain setup
const agent = createReactAgent({ llm, tools, checkpointSaver: memory });
const config = { configurable: { thread_id: "test" } };

// Wrap with adapter
const adapter = new LangChainAdapter({
  agent,
  config,
  metadata: {
    name: "My FireGlobe Agent",
    description: "DeFi operations on Base",
    version: "1.0.0"
  }
});

// Run tests
const tester = new AgentTester({
  agentDescription: "A DeFi agent built with FireGlobe"
});

const results = await tester.runTests(adapter);

API Reference

AgentTester

Main class for orchestrating tests.

Constructor Options

interface TestConfig {
  accessToken: string;                 
  agentDescription: string;           // Description of your agent
  maxMessagesPerConversation?: number; // Default: 10
  saveConversations?: boolean;         // Default: true
  conversationOutputPath?: string;     // Default: "./conversations"
  realTimeLogging?: boolean;           // Default: true
}

Methods

• runTests(agent: IUniversalAgent): Promise<TestResults> - Run complete test suite
• on(listener: EventListener): void - Add event listener for real-time updates

IUniversalAgent Interface

interface IUniversalAgent {
  sendMessage(message: string): Promise<string>;
  reset(): Promise<void>;
  getMetadata?(): {
    name: string;
    description: string;
    framework: string;
    version: string;
  };
  cleanup?(): Promise<void>;
}

LangChainAdapter

Adapter for LangChain-based agents.

const adapter = new LangChainAdapter({
  agent: langchainAgent,
  config: agentConfig,
  metadata: { /* ... */ }
});

Test Results

interface TestResults {
  testId: string;
  agentDescription: string;
  personalities: Personality[];
  conversations: Conversation[];
  evaluations: EvaluationResult[];
  startTime: Date;
  endTime: Date;
  overallScore: number;  // 0-100
  summary: {
    totalConversations: number;
    successfulConversations: number;
    failedConversations: number;
    averageScore: number;
    topStrengths: string[];
    topWeaknesses: string[];
  };
}

Events

The SDK emits real-time events during testing:

type TestEvent =
  | { type: "test_started"; testId: string; timestamp: Date }
  | { type: "personalities_generated"; count: number; personalities: Personality[] }
  | { type: "conversation_started"; conversationId: string; personalityName: string }
  | { type: "message_sent"; conversationId: string; role: "user" | "agent"; content: string }
  | { type: "conversation_completed"; conversationId: string }
  | { type: "evaluation_completed"; conversationId: string; score: number }
  | { type: "test_completed"; results: TestResults }
  | { type: "error"; error: string; context?: string };

Output Formats

JSON Results

Saved to: ./test-results/test_results_TIMESTAMP.json

Contains complete test data including all conversations and evaluations.

HTML Report

Interactive HTML report with:

• Overall score and summary
• All conversations with each personality
• Individual evaluation scores
• Strengths and weaknesses analysis
• Color-coded messages

Real-Time Logs

Each conversation is logged in real-time to: ./test-results/conversation_ID.jsonl

How ASI:One is Used

ASI:One (Artificial Superintelligence Alliance) serves as the core AI reasoning engine that powers FireGlobe's intelligent testing capabilities:

Personality Generation

ASI:One generates diverse, contextually appropriate test personas that challenge agents with realistic DeFi scenarios. Each personality is tailored to test specific aspects of your agent's capabilities:

• Trading Knowledge Seekers: Test market analysis and trading strategy capabilities
• Risk-Averse Users: Evaluate safety measures and conservative approaches
• DeFi Enthusiasts: Test advanced protocol interactions and complex operations
• Novice Users: Assess educational capabilities and user guidance

Conversation Evaluation

ASI:One provides sophisticated evaluation of agent performance across multiple dimensions:

• Tool Usage Assessment: Evaluates how effectively agents use their available tools
• DeFi Capability Analysis: Assesses knowledge of blockchain and DeFi protocols
• Response Quality: Measures relevance, accuracy, and helpfulness of responses
• Context Awareness: Evaluates how well agents understand conversation context

AI-Powered Insights

The system generates comprehensive insights including:

• Strengths Identification: Highlights areas where your agent excels
• Weakness Analysis: Identifies specific areas for improvement
• Performance Scoring: Provides detailed scoring across multiple criteria
• Recommendations: Offers actionable suggestions for enhancement

Integration Architecture

ASI:One integrates seamlessly through the FireGlobe backend:

// The SDK automatically handles ASI:One integration
const tester = new AgentTester({
  accessToken: process.env.ACCESS_TOKEN!,
  agentDescription: "Your agent description"
});

// ASI:One powers personality generation and evaluation behind the scenes
const results = await tester.runTests(myAgent);

How BlockScout is Used

BlockScout MCP (Model Context Protocol) provides comprehensive blockchain transaction analysis capabilities:

Transaction Data Retrieval

BlockScout MCP enables real-time access to detailed blockchain transaction data:

• Multi-Chain Support: Access data across Ethereum, Base, Optimism, Arbitrum, Polygon, and other networks
• Comprehensive Transaction Details: Gas usage, contract interactions, token transfers, and execution traces
• Real-Time Processing: Immediate analysis of transactions as they occur on-chain
• Historical Data Access: Retrieval of past transaction data for analysis and comparison

Transaction Analysis Features

When your agent performs blockchain operations, BlockScout provides:

• Gas Analysis: Detailed gas usage patterns and efficiency recommendations
• Contract Interaction Analysis: Explains smart contract calls and their purposes
• Token Transfer Detection: Identifies and explains token movements
• Risk Assessment: Evaluates potential security concerns or anomalies
• Transaction Classification: Identifies transaction types (transfers, swaps, approvals, etc.)

AI-Enhanced Analysis

BlockScout data is processed through ASI:One to provide intelligent insights:

• Context-Aware Analysis: Links transaction analysis to conversation context
• Personality-Based Insights: Tailors analysis to user personality and goals
• Educational Content: Provides learning opportunities about blockchain concepts
• Actionable Recommendations: Offers specific suggestions for optimization

Integration Flow

The BlockScout integration works automatically when your agent performs blockchain operations:

// When your agent executes a blockchain transaction
const response = await myAgent.sendMessage("Transfer 0.1 ETH to 0x...");

// BlockScout automatically analyzes the transaction
// Analysis is included in the test results
const results = await tester.runTests(myAgent);
console.log(results.transactionAnalyses); // BlockScout analysis results

Supported Networks

BlockScout MCP provides comprehensive coverage across major blockchain networks:

• Ethereum Mainnet (Chain ID: 1): Primary Ethereum network
• Base Mainnet (Chain ID: 8453): Coinbase's Layer 2 solution
• Base Sepolia (Chain ID: 84532): Base testnet for development
• Optimism (Chain ID: 10): Optimistic rollup solution
• Arbitrum (Chain ID: 42161): Arbitrum Layer 2 solution
• Polygon (Chain ID: 137): Polygon network

Technical Documentation

For detailed technical implementation documentation:

• ASI:One Integration: ASI Integration Documentation
• BlockScout MCP Integration: BlockScout Documentation

License

MIT

Vision

Our goal is to become the standard for on-chain agent testing, providing comprehensive evaluation tools that work across all major agent frameworks. We believe that robust testing is essential for the future of autonomous on-chain agents.

Adapter Contributions Welcome

We encourage the community to contribute adapters for their favorite agent frameworks! See our CONTRIBUTING.md for guidelines on how to contribute.

Contributing

We welcome contributions to FireGlobe! Here's how you can help:

Types of Contributions

1. Agent Framework Adapters: Create adapters for new agent frameworks
2. Testing Improvements: Enhance testing capabilities and metrics
3. Documentation: Improve guides, examples, and API documentation
4. Bug Fixes: Report and fix issues
5. Feature Requests: Suggest new features and improvements

Getting Started

1. Fork the repository
2. Create a feature branch: git checkout -b feature/amazing-feature
3. Make your changes and add tests if applicable
4. Commit your changes: git commit -m 'Add amazing feature'
5. Push to the branch: git push origin feature/amazing-feature
6. Open a Pull Request

Development Setup

1. Clone the repository:

   git clone https://github.com/Marshal-AM/fireglobe.git
   cd fireglobe

2. Install dependencies:

   # SDK
   cd SDK/cdp-agent-tester-sdk
   npm install
      
   # Backend
   cd ../backend
   pip install -r requirements.txt
      
   # Database Server
   cd ../db-server
   npm install

3. Set up environment variables (see individual component READMEs)

4. Run tests:

   # SDK tests
   cd SDK/cdp-agent-tester-sdk
   npm test
      
   # Backend tests
   cd ../backend
   python test.py

Creating an Agent Framework Adapter

To create an adapter for a new agent framework:

1. Study existing adapters in /SDK/cdp-agent-tester-sdk/src/adapters/

2. Implement the IUniversalAgent interface:

   export class YourFrameworkAdapter implements IUniversalAgent {
     async sendMessage(message: string): Promise<string> {
       // Your framework's message handling
     }
        
     async reset(): Promise<void> {
       // Reset conversation state
     }
        
     getMetadata?() {
       return {
         name: "Your Agent",
         description: "Agent description",
         framework: "YourFramework",
         version: "1.0.0"
       };
     }
   }

3. Add tests for your adapter

4. Update documentation with usage examples

5. Submit a Pull Request

Code Style

• TypeScript: Use strict typing and follow ESLint rules
• Python: Follow PEP 8 guidelines
• JavaScript: Use Prettier for formatting
• Documentation: Update README files and add JSDoc comments

Questions?

• GitHub Issues: For bug reports and feature requests
• Discussions: For questions and community discussions
• Discord: Join our community server (link coming soon)

Support

• Documentation: fireglobe.vercel.app
• Issues: GitHub Issues
• Discussions: GitHub Discussions