@magentic-platform/postman-playwright

v0.7.0

Published

a month ago

Migrate Postman collections to a fully typed, runnable Playwright API test monorepo

0High
0Medium
0Low

karthikrana

postman playwright api-testing migration cli typescript llm

PostmanPlaywright

Point at a folder of Postman collections — get back a fully typed, runnable Playwright API test monorepo. Run again for the next module — it detects existing structure and slots in cleanly.

CLI-driven · TypeScript-strict · pnpm workspaces · Foundation model agnostic · Open source · MIT

Quick Start

npx @magentic-platform/postman-playwright

No flags needed on first run. The CLI walks you through setup interactively — choose your foundation model provider, enter credentials, point it at your collections folder:

  PostmanPlaywright — Postman → Playwright Migration Orchestrator
  ─────────────────────────────────────────────────────────────────

  No config found. Let's get you set up.

  ? Foundation model provider:
  ❯ Anthropic Claude     (API key)
    OpenAI               (API key — GPT-4o, o1, …)
    Azure OpenAI         (API key + endpoint)
    AWS Bedrock          (IAM role / access key — Claude, Llama, Titan, …)
    Google Vertex AI     (service account — Gemini Pro, …)
    Mistral AI           (API key — mistral-large, codestral, …)
    Ollama (local)       (no key — Llama 3, Mistral, Phi-3, Gemma 2, …)

  ? Postman collections folder: ./postman-collections
  ? Output folder for generated tests: ./playwright-api-tests

  ✓ Config saved to ~/.postman-playwright.json
  ✓ API key saved to ~/.postman-playwright.env

Subsequent runs load config automatically — just pick the next module:

npx @magentic-platform/postman-playwright migrate --module payments

End-to-End Workflow

# ── Step 1: One-time setup ──────────────────────────────────────────────────
npx @magentic-platform/postman-playwright setup
# Choose provider, enter API key, set input/output folders
# Saves: .postman-playwright.json + .env

# ── Step 2: Migrate a module ────────────────────────────────────────────────
npx @magentic-platform/postman-playwright migrate --module transforms
# [1/6] Scaffold monorepo (first run only)
# [2/6] Parse Postman collections + environment files → CollectionIR
#        Auto-detects baseUrl from Postman env → populates .env
# [3/6] Generate TypeScript types (models/)
# [4/6] Generate fixtures + typed API client
# [5/6] LLM → Playwright spec files (tests/transforms/)
#        Negative test detection · Postman status enforcement · example body values
# [6/6] tsc check → auto-fix assertion aliases → LLM repair method mismatches → ESLint

# ── Step 3: Run the generated tests ─────────────────────────────────────────
cd playwright-api-tests
# .env is auto-populated with baseUrl from Postman environment if available
# Only edit .env manually if no Postman environment file was provided
pnpm install
npx playwright install --with-deps chromium
pnpm test:smoke                 # fast feedback — happy path only
pnpm test                       # full suite

# ── Step 4: Validate parity (optional but recommended) ──────────────────────
npx @magentic-platform/postman-playwright validate \
  --module transforms \
  --env    staging \
  --delta  0.05
# Runs Newman against the original collections + Playwright against staging
# Reports pass-rate delta and per-endpoint mismatches

# ── Step 5: Add the next module — additive, never overwrites ─────────────────
npx @magentic-platform/postman-playwright migrate --module auth
npx @magentic-platform/postman-playwright migrate --module payments

What Gets Generated

Given a folder of Postman collection JSON files, the tool outputs a complete, runnable Playwright API test monorepo:

playwright-api-tests/
├── package.json                    ← scripts: test, test:smoke, typecheck, lint
├── playwright.config.ts            ← projects: smoke · regression · contract · security · staging
├── tsconfig.json                   ← strict: true, ESNext + bundler module resolution
├── pnpm-workspace.yaml
├── .env.example                    ← required env vars documented
├── .gitignore
├── global-setup.ts                 ← health check, env var validation, date var computation;
│                                      auto-generated seed blocks for pm.environment.set() captures
│
├── api/
│   ├── base.api.ts                 ← abstract BaseApi: retry, rate-limit, auth headers
│   └── {module}.api.ts             ← typed methods per request, path params resolved
│
├── config/
│   └── env.ts                      ← lazy env var getters (no crash at import time)
│
├── fixtures/
│   ├── auth.fixture.ts             ← JWT / OAuth2 / API key / Basic / None strategies
│   ├── base.fixture.ts             ← root fixture composing auth + all API clients
│   └── {module}.fixture.ts         ← per-module API client fixture
│
├── utils/
│   └── assertions.ts               ← assertStatus, assertArray, assertHasProperty,
│                                      assertHeaderExists, assertInclusion, assertEnumIncludes
│
├── models/                         ← TypeScript interfaces inferred from response shapes
│
├── data/
│   ├── factories/                  ← entity factory scaffolds (populated manually per project)
│   └── seeds/
│
├── tests/
│   └── {module}/
│       ├── list-*.spec.ts          ← @smoke · @contract · @security tags per spec
│       ├── create-*.spec.ts
│       └── poll-*.spec.ts
│
└── .github/
    └── workflows/
        └── ci.yml                  ← smoke suite on every PR, artifact upload

CLI Commands

# Interactive first run — setup + migrate
npx @magentic-platform/postman-playwright

# Migrate a specific module (flags override saved config)
npx @magentic-platform/postman-playwright migrate \
  --input  ./postman-collections \
  --output ./playwright-api-tests \
  --module auth

# Subsequent modules — additive, never overwrites existing
npx @magentic-platform/postman-playwright migrate --module payments
npx @magentic-platform/postman-playwright migrate --module exports --force  # overwrite if exists

# Migrate ALL modules at once (each subdirectory of --input = one module)
npx @magentic-platform/postman-playwright migrate --all --input ./postman-collections

# Use an OpenAPI spec alongside Postman collections (dual-source mode)
npx @magentic-platform/postman-playwright migrate --module payments --openapi ./openapi.yaml

# Migrate + immediately validate parity in one command
npx @magentic-platform/postman-playwright migrate --module auth --validate

# Reconfigure provider (e.g. switch from Anthropic → Azure OpenAI)
npx @magentic-platform/postman-playwright setup

# Show migration status across all modules
npx @magentic-platform/postman-playwright status --output ./playwright-api-tests

# Validate parity — Newman vs Playwright (standalone, run any time after migration)
npx @magentic-platform/postman-playwright validate \
  --module auth \
  --env    staging \
  --delta  0.02     # acceptable pass-rate delta (default: 0.02 = 2%)

Validate output example

  Validating auth → staging  (Δ threshold: 2%)

  Newman
    Requests : 8/10 passed
    Failed   : 2

  Playwright
    Tests    : 14/16 passed
    Failed   : 2
    Skipped  : 0

  Parity delta
    3.1%  (threshold: 2%)

  Endpoint mismatches
    GET /auth/me                             ✓ Newman  ✗ Playwright
    POST /auth/refresh                       ✗ Newman  ✓ Playwright

  ✗ Parity check failed — fix failing specs then re-run validate.

Supported Foundation Models

All 7 providers are wired and selectable via the setup wizard. API keys are stored in .env — never in the config JSON.

| Provider | Setup | Model examples | |---|---|---| | Anthropic Claude | ANTHROPIC_API_KEY | claude-opus-4-6, claude-sonnet-4-6 — optional baseUrl for proxy | | OpenAI | OPENAI_API_KEY | gpt-4o, o1, o3-mini | | OpenAI-compatible endpoint | OPENAI_API_KEY + baseUrl | Any hub (Azure AI Foundry, etc.) | | Azure OpenAI | AZURE_OPENAI_API_KEY + endpoint + deployment | Any GPT-4 deployment | | AWS Bedrock | IAM credentials + AWS_REGION | Claude, Llama 3, Titan | | Google Vertex AI | GCP service account + project | Gemini 1.5 Pro/Flash | | Mistral AI | MISTRAL_API_KEY | mistral-large, codestral | | Ollama (local) | No key — http://localhost:11434/v1 | Llama 3.1, Phi-3, Gemma 2, CodeLlama |

Idempotent Run Model

The tool inspects the output folder before every run and behaves accordingly:

| State | What happens | |---|---| | empty | Scaffolds full monorepo (config, fixtures, utils, CI) then migrates the module | | initialised | Reads existing structure, adds only the new module, merges base.fixture.ts | | has-module | Dry-run by default — prompts to skip or overwrite. --force skips the prompt |

This means you can safely run it on every CI push — it will never clobber a module already migrated.

Migration Pipeline

Postman JSON files (./postman-collections/<module>/)
  │
  ├─ [1/6] Scaffold         pnpm-workspace · playwright.config · base.api · utils · CI · eslint.config.mjs
  ├─ [2/6] Parse            Postman JSON + *.environment.json → CollectionIR
  │                          (method, path, headers, body, auth strategy,
  │                          pre-request scripts, pm.test() assertions,
  │                          setNextRequest chains, env var usage)
  │                         Reads Postman environment files — auto-sets API_URL in .env
  │                          from baseUrl variable if present (no manual editing needed)
  │                         URL env key resolution: {{apiTargetUrl}} → API_TARGET_URL, {{graphqlUrl}} → GRAPHQL_URL
  │                         Smart env merge: env file only overrides {{template}} placeholders
  ├─ [2b] OpenAPI parse     auto-detected openapi.yaml/yml/json or swagger.* in inputDir
  │   (optional)             explicit override via --openapi <path>
  │                          dual-source mode: deterministic types + per-request CONTRACT in LLM prompt
  │                          graceful fallback to Postman-only if not found
  ├─ [2c] Env capture       pm.environment.set() calls in test scripts → fetch-based seed blocks
  │                          injected into global-setup.ts at migration time — populates dynamic
  │                          IDs (userId, productId, etc.) as process.env vars before tests run
  ├─ [3/6] Type generation  response body shapes → models/*.ts TypeScript interfaces
  ├─ [4/6] Fixture + API    auth strategy → auth.fixture.ts
  │   client generation      CollectionIR requests → api/{module}.api.ts typed methods
  │                          (path params: {{variable}} and :param both resolved — camelCased)
  │                          GraphQL requests → dedicated graphqlRaw() wrappers with embedded operations
  ├─ [5/6] Test generation  Foundation Model → tests/{module}/*.spec.ts
  │                          Positive requests → happy path @smoke + validation @validation + auth @security
  │                          Negative requests (4xx) → detected by name/assertions, tested at expected error status
  │                          Postman status codes extracted and enforced — LLM cannot guess/change them
  │                          Postman example body values injected into prompt (not just types)
  │                          CONTRACT section per request in dual-source mode (required fields, params)
  └─ [6/6] Quality checks   tsc --noEmit → structured violations per file + line
                             Auto-fix pass 1: near-miss assertion names → aliases in utils/assertions.ts
                             Auto-fix pass 2: LLM repair — up to 3 passes, snapshot/restore if worse
                               codes: TS2339/TS2304/TS2352/TS2345/TS2554/TS2769/TS2322/TS18046/TS7006/TS2571+
                             Auto-fix 4a–4x: 24 deterministic hallucination fixes (markdown fences,
                               wrong imports, env fixture, tv4→ajv, {{VAR}} placeholders, $schema strip,
                               test.use() in body, RESPONSE_TIME, assertResponseTimeBelow 1-arg,
                               wrong assertions path, extra fixture exports, number key arrays,
                               sync missing assertion imports, Record cast widening for optional chains,
                               numeric-key array index swap, unknown validator param property access,
                               array type index-access casts, placeholder UUID → process.env)
                             Step 8: env var stub scan — appends missing SCREAMING_CASE= stubs to .env
                             ESLint pass (warns, never blocks migration)
                             Per-module parity matrix generated in HTML + JSON reports

Running Generated Tests

cd playwright-api-tests

# 1. Install dependencies
pnpm install
npx playwright install --with-deps chromium

# 2. Configure environment
# .env is created automatically. If your Postman environment file contains a
# baseUrl variable, API_URL is populated automatically — no manual editing needed.
# Falls back to localhost:8080 if no Postman baseUrl is found.
# API_USER / API_PASS are only needed if your API has an auth layer

# 3. Run
pnpm test:smoke        # happy path only — fast feedback
pnpm test              # full suite
pnpm test:contract     # schema/contract tests
pnpm test:security     # auth/401 tests
pnpm test:staging      # against staging URL
pnpm typecheck         # tsc --noEmit — catch spec issues

Configuration File

.postman-playwright.json is saved in your home directory (~/.postman-playwright.json) so it works from any folder. Sensitive values use ${ENV_VAR} references — the actual keys live in ~/.postman-playwright.env only.

{
  "provider": "openai",
  "inputDir": "./postman-collections",
  "outputDir": "./playwright-api-tests",
  "version": "0.5.0",
  "credentials": {
    "type": "openai",
    "apiKey": "${OPENAI_API_KEY}",
    "model": "gpt-4o",
    "baseUrl": "https://your-hub.example.com/v1"
  }
}

Run npx @magentic-platform/postman-playwright setup at any time to switch provider or update credentials.

Known Limitations

| Limitation | Notes | |---|---| | npx playwright install not auto-run | You need to install Playwright browsers manually after migration | | Type inference is heuristic-only | Complex nested response shapes may need manual type refinement | | Chained request specs | setNextRequest chains are detected but not yet woven into sequential test specs |

See the full roadmap for details.

Contributing

See the contributing guide for local development setup, architecture, and versioning.