A/B Test Calculator MCP Server

Statistical significance calculator for A/B tests with confidence intervals, power analysis, and sample size recommendations.

Features

• Statistical Significance Testing - Two-proportion z-test to determine if observed differences are statistically significant
• Confidence Intervals - Wald confidence intervals for both control and variant conversion rates
• Power Analysis - Calculate current statistical power and required sample sizes for 80% power
• Relative Uplift Calculation - Quantify the percentage improvement from variant over control
• P-Value Computation - Get precise p-values to assess test results
• Flexible Confidence Levels - Support for custom confidence levels (default 95%)

Installation

npm install @gongrzhe/server-ab-test-calculator

Usage

As a CLI

npx @gongrzhe/server-ab-test-calculator

Claude Desktop Configuration

Add this to your claude_desktop_config.json:

{
  "mcpServers": {
    "ab-test-calculator": {
      "command": "npx",
      "args": ["-y", "@gongrzhe/server-ab-test-calculator"]
    }
  }
}

Claude Code Configuration

claude mcp add ab-test-calculator -- npx -y @gongrzhe/server-ab-test-calculator

Tools

calculate-ab-test

Computes statistical significance for an A/B test including conversion rates, confidence intervals, z-score, p-value, and required sample size.

Parameters:

• controlVisitors - Number of visitors in the control group
• controlConversions - Number of conversions in the control group
• variantVisitors - Number of visitors in the variant group
• variantConversions - Number of conversions in the variant group
• confidenceLevel - Confidence level (0-1), defaults to 0.95

Returns:

• Conversion rates and confidence intervals for both groups
• Relative uplift percentage
• Z-score and p-value
• Statistical significance at the specified confidence level
• Required sample size per group for 80% statistical power
• Current power analysis

Example Prompt

Calculate significance for my A/B test: Control had 5000 visitors with 150 conversions, Variant had 5000 visitors with 195 conversions. Use 95% confidence level.

License

MIT