@memberjunction/testing-engine

Core test execution and evaluation engine for the MemberJunction Testing Framework. Provides the driver/oracle architecture for running tests, evaluating results with multiple oracle strategies, and recording outcomes.

Architecture

graph TD
    subgraph "@memberjunction/testing-engine"
        A[TestEngine] --> B[Test Drivers]
        A --> C[Oracles]
        A --> D[Utilities]

        B --> E[BaseTestDriver]
        E --> F[AgentEvalDriver]

        C --> G[ExactMatchOracle]
        C --> H[LLMJudgeOracle]
        C --> I[SchemaValidatorOracle]
        C --> J[SQLValidatorOracle]
        C --> K[TraceValidatorOracle]

        D --> L[Variable Resolver]
        D --> M[Scoring Calculator]
        D --> N[Cost Calculator]
        D --> O[Result Formatter]
        D --> P[Execution Context]
    end

    subgraph "Base Layer"
        Q["TestEngineBase<br/>(from engine-base)"]
    end

    A -->|extends| Q

    style A fill:#2d6a9f,stroke:#1a4971,color:#fff
    style E fill:#2d8659,stroke:#1a5c3a,color:#fff
    style F fill:#b8762f,stroke:#8a5722,color:#fff
    style G fill:#7c5295,stroke:#563a6b,color:#fff
    style H fill:#7c5295,stroke:#563a6b,color:#fff
    style I fill:#7c5295,stroke:#563a6b,color:#fff
    style J fill:#7c5295,stroke:#563a6b,color:#fff
    style K fill:#7c5295,stroke:#563a6b,color:#fff
    style Q fill:#2d6a9f,stroke:#1a4971,color:#fff

Overview

This is the server-side execution engine for MemberJunction tests. It implements a plugin-based architecture where:

• Drivers handle test execution (running agents, calling APIs, etc.)
• Oracles evaluate test outputs against expected outcomes
• Utilities handle variable resolution, scoring, cost tracking, and result formatting

Key capabilities:

• Execute individual tests or full test suites with parallel/sequential modes
• Multiple oracle strategies for evaluating test results
• Variable resolution system with priority cascading (run > suite > test > type)
• Automatic cost and duration tracking
• Progress callbacks for real-time execution monitoring
• Result persistence to MJ: Test Runs and MJ: Test Suite Runs entities

Installation

npm install @memberjunction/testing-engine

Core Concepts

Driver/Oracle Pattern

graph LR
    A[Test Definition] --> B[Driver]
    B -->|Executes| C[Target System]
    C -->|Output| D[Oracle 1]
    C -->|Output| E[Oracle 2]
    C -->|Output| F[Oracle N]
    D --> G[Combined Score]
    E --> G
    F --> G

    style A fill:#2d6a9f,stroke:#1a4971,color:#fff
    style B fill:#2d8659,stroke:#1a5c3a,color:#fff
    style D fill:#7c5295,stroke:#563a6b,color:#fff
    style E fill:#7c5295,stroke:#563a6b,color:#fff
    style F fill:#7c5295,stroke:#563a6b,color:#fff
    style G fill:#b8762f,stroke:#8a5722,color:#fff

1. The driver executes the test target (e.g., runs an AI agent)
2. Multiple oracles independently evaluate the output
3. Scores are combined using configurable weights

Test Drivers

| Driver | Description | |--------|-------------| | BaseTestDriver | Abstract base class for all test drivers | | AgentEvalDriver | Executes AI agents and captures their outputs |

Oracles

| Oracle | Description | |--------|-------------| | ExactMatchOracle | Compares output against an expected string | | LLMJudgeOracle | Uses an LLM to evaluate output quality with rubrics | | SchemaValidatorOracle | Validates output against a JSON schema | | SQLValidatorOracle | Validates output by running SQL queries | | TraceValidatorOracle | Validates execution trace/steps of an agent run |

Variable Resolution

Variables cascade through four levels with the highest priority winning:

Run Variables > Suite Variables > Test Variables > Type Variables

The resolver tracks the source of each resolved value for auditing.

Usage

Running a Single Test

import { TestEngine } from '@memberjunction/testing-engine';

const engine = TestEngine.Instance;
await engine.Config(false, contextUser);

const result = await engine.RunTest(testId, contextUser, {
    verbose: true,
    variables: { AIConfiguration: 'gpt-4o' },
    progressCallback: (p) => console.log(`${p.percentage}%: ${p.message}`)
});

console.log(`Status: ${result.status}, Score: ${result.score}`);

Running a Test Suite

const suiteResult = await engine.RunSuite(suiteId, contextUser, {
    parallel: true,
    maxParallel: 5,
    failFast: false,
    variables: { Temperature: 0.3 }
});

console.log(`Passed: ${suiteResult.passedTests}/${suiteResult.totalTests}`);
console.log(`Average Score: ${suiteResult.averageScore}`);

Utilities

| Utility | Description | |---------|-------------| | variable-resolver | Resolves variables through the priority cascade | | scoring | Combines oracle scores using weighted averages | | cost-calculator | Tracks and sums execution costs | | result-formatter | Formats test results for storage and display | | execution-context | Captures environment details (OS, Node.js version, CI/CD info) |

Testing

npm test
npm run test:coverage

Dependencies

| Package | Purpose | |---------|---------| | @memberjunction/testing-engine-base | Base engine, metadata caching, shared types | | @memberjunction/ai | AI model access for LLM oracles | | @memberjunction/ai-agents | Agent execution for AgentEvalDriver | | @memberjunction/ai-prompts | Prompt execution | | @memberjunction/core | Metadata, RunView, UserInfo | | @memberjunction/core-entities | Test entity types | | @memberjunction/global | Class factory | | zod | Schema validation | | rxjs | Observable patterns |

License

ISC