@postman/aiei

AI Effectiveness Index - CLI

Repo: github.com/postmanlabs/AIEI-Framework

AIEI measures whether your organization's APIs, codebase, and delivery systems are ready for AI agents, then scores AI-assisted engineering outcomes across quality, lead time, reliability, cost, and security from connected tools.

Quick Start

npx @postman/aiei

AIEI auto-detects credentials in your environment and runs a live assessment with whatever tools it finds. Set env vars for any combination of connectors (e.g. GITHUB_TOKEN + GITHUB_ORG, DD_API_KEY + DD_APP_KEY, POSTMAN_API_KEY + POSTMAN_WORKSPACE_ID) and run - no config file needed.

If no credentials are detected, AIEI shows a demo-data assessment so a customer can see the scorecard shape immediately. If partial credentials are found (e.g. GITHUB_TOKEN set but GITHUB_ORG missing), it tells you what to add.

What You Get

  POSTMAN AIEI - AI Effectiveness Index
  demo-org  2026-04-30

  Score: 68 / 100   Grade: C   Stage: Assessed
  ███████████████████████████░░░░░░░░░░░░░

  demo-org scores 68/100 (Grade C), placing it in Assessed.
  5/5 dimensions measured. Cost Efficiency is the primary improvement area.

  VALUE SNAPSHOT
    225 hrs/mo recovered  |  $202,500/yr value recovered
    79 hrs/mo available   |  $71,100/yr available value  |  ROI 1.3x

  DIMENSIONS

  Quality (Q)  74  C  Agent-Ready
  Is AI making our code better or just faster?
    Defect Rate (AI)             7.0%
    Test Pass Rate               91.0%
    Rework Rate                  16.0%
    Collection Quality           82
    Sources: github, postman, fern

  Reliability (R)  71  C  Agent-Ready
  Lead Time (L)    62  C  Agent-Ready
  Cost (C)         55  D  Agent-Ready
  Security (S)     73  C  supplemental

  CONNECTOR INSIGHTS

  Quality (Q)
    github and postman and fern show Defect Rate at 7.0%, Test Pass Rate at 91.0%.
    Postman: Enabling Spectral rules and adding response examples would push
    Quality higher - better specs mean better AI code generation with fewer retries.

  RECOMMENDATIONS
    1. Cut 11 hours of queue time to 4                    +14 index points
    2. Optimize AI spend per unit of value                +5 index points
    3. Close the 4% SLO gap                               +4 index points

  ASSESSMENT COVERAGE
    0 live signals, 22 demo estimates, 5 not measured
    4/4 Agent Experience areas measured  |  Full assessment  |  Reliability-focused profile

Five output profiles: terminal (glance scorecard, Postman orange/purple ANSI), HTML (executive dashboard), JSON (programmatic), agent (plain-text handoff for Claude/Codex/Cursor), and harness (machine-readable gates for CI/evals).

Terminal report (default)

Action-first layout: demo/live trust banner, failure modes, readiness gates, AX pillar row, compact dimensions. Full connector essays require --verbose.

| Flag | Purpose | |------|---------| | --verbose | Full dimension narratives, connector insights, all recommendations | | --ci | Exit 1 if no live evidence, critical failure modes, failed gates, or evidence stage below Agent-Ready (demo exempt) | | NO_COLOR=1 | Plain text (no ANSI); set AIEI_FORCE_COLOR=1 to keep brand colors in pipes/logs |

aiei demo
aiei --config ./.aiei.json score --verbose
aiei --config ./.aiei.json score --ci

Agent and harness output

Use these formats when the consumer is an autonomous coding agent, eval harness, CI policy runner, or OS-style orchestration layer.

| Format | Command | Contract | |--------|---------|----------| | agent | aiei --config ./.aiei.json score --format agent | Stable plain text with verdict, top failure modes, guardrails, evidence gaps, and next commands. | | harness | aiei --config ./.aiei.json score --format harness | JSON object with schema aiei.harness.v1, checks, policy, evidence, and remediation actions. | | json | aiei --config ./.aiei.json score --json | Full JSON report plus agentSummary (aiei.agent_summary.v1) and harness. |

The agent contract includes explicit doNotClaim guardrails. AI-specific quality, lead-time, reliability, or cost claims require attributionConfidence.tier === "explicit"; otherwise the output is readiness/evidence guidance, not a causal AI performance claim.

Installation

npm install -g @postman/aiei

Or run directly with npx:

npx @postman/aiei

Run a Live Assessment

export GITHUB_TOKEN="ghp_..." GITHUB_ORG="your-org"
npx @postman/aiei                                          # auto-detect, terminal scorecard
npx @postman/aiei --format html --output report.html       # executive HTML report
npx @postman/aiei --config ./aiei.json                     # use a prepared config
npx @postman/aiei --config ./aiei.json --format html       # config + HTML

Advanced Commands

These commands are for delivery teams, CI, and implementation troubleshooting. They are not the customer-facing starting surface.

| Command | Purpose | |---------|---------| | score | Run assessment with explicit advanced options (--signals, --industry, --sync) | | readiness --signals | Agent readiness diagnostic | | bva --signals | Business value estimate | | validate | Check config, credentials, or signal payloads | | init | Generate .aiei.json config template | | demo | Force the sample scorecard | | connectors | List available connectors and signals | | formula | Show scoring formulas | | signals | Show signal reference | | benchmark | Inspect sample benchmark environments and example benchmark output | | telemetry | Show or change anonymous CLI telemetry controls |

Benchmark examples:

aiei benchmark --json
aiei benchmark --example --format agent
aiei benchmark --example --format harness
aiei benchmark export --world payments-core --output ./payments-core

For npx usage, export the sample environment first so the OpenAPI, Postman collection, and policy files are available locally:

npx @postman/aiei benchmark export --world payments-core --output ./payments-core

Configuration

Use a prepared .aiei.json config:

{
  "orgName": "your-org",
  "weightProfile": "reliability_first",
  "connectors": {
    "github": { "token": "$GITHUB_TOKEN", "org": "your-org", "repos": ["api-service"] },
    "github_copilot": { "token": "$GITHUB_TOKEN", "org": "your-org" },
    "postman": { "apiKey": "$POSTMAN_API_KEY", "workspaceId": "your-workspace-id" },
    "datadog": { "apiKey": "$DD_API_KEY", "appKey": "$DD_APP_KEY" }
  }
}

Credential fields must use $ENV_VAR references. Plaintext connector secrets are rejected before connector calls run.

What It Measures

| Dimension | Question | |-----------|----------| | Q Quality | Is AI-assisted code better or worse? | | L Lead Time | Is AI making delivery faster? | | R Reliability | Are AI changes breaking production? | | C Cost Efficiency | Is the AI investment paying off? | | S Security | Are AI-generated APIs secure? |

Attribution Confidence

AIEI separates API/system readiness from AI-specific attribution. Every JSON score includes attributionConfidence:

| Tier | Meaning | |------|---------| | explicit | Connected tools distinguish AI from non-AI work and enough attributed signals are present. | | heuristic | Labels, branch patterns, or partial AI signals exist, but claims are directional. | | absent | AIEI can score API and delivery readiness, but cannot make AI-vs-human performance claims. |

For stronger AI usage evidence, configure github_copilot with a GitHub token that can access Copilot metrics reports.

Weight Profiles

| Profile | Q | L | R | C | Use When | |---------|---|---|---|---|----------| | reliability_first | 20% | 20% | 45% | 15% | Default. Production stability is primary risk. | | balanced | 25% | 25% | 25% | 25% | Initial assessments. | | quality_first | 35% | 20% | 35% | 10% | Regulated environments. | | speed_first | 20% | 40% | 25% | 15% | Competitive pressure on delivery. | | cost_first | 25% | 20% | 20% | 35% | Executive ROI reporting. |

Custom Q/L/R/C weights are supported in config when they sum to 1.0. Security remains parallel.

Telemetry

Anonymous CLI telemetry is disabled by default and never sends scores, org names, credentials, raw signals, environment variables, or local file paths. Score history sync is explicit opt-in via aiei score --sync. Run aiei telemetry to see controls.

Related Packages

| Package | Purpose | |---------|---------| | @postman/aiei-engine | Scoring engine for programmatic use | | MCP | npx @postman/aiei aiei-mcp --stdio (same package, second bin) | | @postman/aiei-templates | HTML report templates | | GitHub Action | uses: postmanlabs/aiei-framework/action@v1 (not published to npm) |

Reference

• CONNECTOR_SCOPES.md - Minimum permissions per connector
• SIGNALS.md - Signal definitions
• SCORING.md - Formula specifications

License

MIT

Built by Postman. Formulas are open, auditable, and versioned. Run aiei formula to see them.