Using the PromptExperiment Decorator

PromptLens provides a powerful decorator for running A/B tests on your prompts. This allows you to automatically test different prompt variants without modifying your existing code.

Setup

To use the experimental decorators in TypeScript, make sure your tsconfig.json includes:

{
  "compilerOptions": {
    "experimentalDecorators": true,
    "emitDecoratorMetadata": true
  }
}

Basic Usage

Here's how to use the @PromptExperiment decorator with OpenAI:

import { PromptExperiment, MetricsCollector } from "promptlens";
import OpenAI from "openai";

class MyOpenAIService {
  private openai: OpenAI;
  public metricsCollector: MetricsCollector;

  constructor(apiKey: string) {
    this.openai = new OpenAI({ apiKey });
    this.metricsCollector = new MetricsCollector(
      "your-promptlens-api-key",
      "https://api.promptlens.dev"
    );
  }

  @PromptExperiment({
    id: "my-experiment",
    promptVariants: [
      "Explain like I'm 5: {{topic}}",
      "Technical explanation: {{topic}}",
      "Real-world examples of {{topic}}",
    ],
    distribution: "round-robin",
    trackMetrics: true,
  })
  async createChatCompletion(params: OpenAI.Chat.ChatCompletionCreateParams) {
    return this.openai.chat.completions.create(params);
  }
}

// Usage
const service = new MyOpenAIService("your-openai-api-key");
const result = await service.createChatCompletion({
  model: "gpt-4",
  messages: [
    { role: "system", content: "You are a helpful assistant." },
    { role: "user", content: "Explain recursion" },
  ],
});

console.log("Response:", result.response.choices[0].message.content);
console.log("Experiment details:", result.experiment);

Alternative: Apply Decorator Manually

If you're experiencing issues with the decorator syntax, you can also apply it manually:

// Define the method normally
async function createChatCompletion(params) {
  return this.openai.chat.completions.create(params);
}

// Apply the decorator to the prototype
PromptExperiment({
  promptVariants: ["Variant A", "Variant B"],
})(
  MyOpenAIService.prototype,
  "createChatCompletion",
  Object.getOwnPropertyDescriptor(
    MyOpenAIService.prototype,
    "createChatCompletion"
  )
);

Configuration Options

The PromptExperiment decorator accepts the following options:

| Option | Type | Description | | ---------------- | -------- | ------------------------------------------------------------------ | | id | string | Unique identifier for the experiment (optional) | | promptVariants | string[] | Array of prompt variants to test | | distribution | string | How to distribute variants: 'round-robin', 'random', or 'weighted' | | weights | number[] | Weights for variants when using 'weighted' distribution | | trackMetrics | boolean | Whether to track metrics for this experiment |

Experiment Results

The decorated method returns an ExperimentResult object with:

{
  response: OriginalResponse,
  experiment: {
    id: string,
    variantIndex: number,
    promptVariant: string,
    metrics: {
      latency_ms: number,
      // ...other metrics
    }
  }
}