@magentic-platform/postman-playwright

v0.2.0

Published

a day ago

Migrate Postman collections to a fully typed, runnable Playwright API test monorepo

Downloads

438

0High
0Medium
0Low

karthikrana

postman playwright api-testing migration cli typescript llm

PostmanPlaywright

Point at a folder of Postman collections — get back a fully typed, runnable Playwright API test monorepo. Run again for the next module — it detects existing structure and slots in cleanly.

CLI-driven · TypeScript-strict · pnpm workspaces · Foundation model agnostic · Open source · MIT

Quick Start

npx @magentic-platform/postman-playwright

No flags needed on first run. The CLI walks you through setup interactively — choose your foundation model provider, enter credentials, point it at your collections folder:

  PostmanPlaywright — Postman → Playwright Migration Orchestrator
  ─────────────────────────────────────────────────────────────────

  No config found. Let's get you set up.

  ? Foundation model provider:
  ❯ Anthropic Claude     (API key)
    OpenAI               (API key — GPT-4o, o1, …)
    Azure OpenAI         (API key + endpoint)
    AWS Bedrock          (IAM role / access key — Claude, Llama, Titan, …)
    Google Vertex AI     (service account — Gemini Pro, …)
    Mistral AI           (API key — mistral-large, codestral, …)
    Ollama (local)       (no key — Llama 3, Mistral, Phi-3, Gemma 2, …)

  ? Postman collections folder: ./postman-collections
  ? Output folder for generated tests: ./playwright-api-tests

  ✓ Config saved to .postman-playwright.json
  ✓ API key saved to .env

Subsequent runs load config automatically — just pick the next module:

npx @magentic-platform/postman-playwright migrate --module payments

End-to-End Workflow

# ── Step 1: One-time setup ──────────────────────────────────────────────────
npx @magentic-platform/postman-playwright setup
# Choose provider, enter API key, set input/output folders
# Saves: .postman-playwright.json + .env

# ── Step 2: Migrate a module ────────────────────────────────────────────────
npx @magentic-platform/postman-playwright migrate --module transforms
# [1/6] Scaffold monorepo (first run only)
# [2/6] Parse Postman collections + environment files → CollectionIR
#        Auto-detects baseUrl from Postman env → populates .env
# [3/6] Generate TypeScript types (models/)
# [4/6] Generate fixtures + typed API client
# [5/6] LLM → Playwright spec files (tests/transforms/)
#        Negative test detection · Postman status enforcement · example body values
# [6/6] tsc check → auto-fix assertion aliases → LLM repair method mismatches → ESLint

# ── Step 3: Run the generated tests ─────────────────────────────────────────
cd playwright-api-tests
# .env is auto-populated with baseUrl from Postman environment if available
# Only edit .env manually if no Postman environment file was provided
pnpm install
npx playwright install --with-deps chromium
pnpm test:smoke                 # fast feedback — happy path only
pnpm test                       # full suite

# ── Step 4: Validate parity (optional but recommended) ──────────────────────
npx @magentic-platform/postman-playwright validate \
  --module transforms \
  --env    staging \
  --delta  0.05
# Runs Newman against the original collections + Playwright against staging
# Reports pass-rate delta and per-endpoint mismatches

# ── Step 5: Add the next module — additive, never overwrites ─────────────────
npx @magentic-platform/postman-playwright migrate --module auth
npx @magentic-platform/postman-playwright migrate --module payments

Local Development (contributing / running from source)

# Clone and install
git clone https://github.com/magentic-platform/postman-playwright
cd postman-playwright
pnpm install

# Build all packages in dependency order
pnpm -r build

# Link the CLI globally so you can run it like the published version
cd apps/cli
npm link

# Now use it from anywhere
postman-playwright setup
postman-playwright migrate --module transforms

# Or run directly without linking
node apps/cli/dist/index.js migrate --module transforms

# Rebuild after making changes
pnpm -r build                                        # rebuild everything
pnpm --filter @magentic/quality-agent build          # rebuild one package
pnpm --filter @magentic-platform/postman-playwright build  # rebuild CLI only

# Type check without building
pnpm --filter @magentic-platform/postman-playwright typecheck

What Gets Generated

Given a folder of Postman collection JSON files, the tool outputs a complete, runnable Playwright API test monorepo:

playwright-api-tests/
├── package.json                    ← scripts: test, test:smoke, typecheck, lint
├── playwright.config.ts            ← projects: smoke · regression · contract · security · staging
├── tsconfig.json                   ← strict: true, ESNext + bundler module resolution
├── pnpm-workspace.yaml
├── .env.example                    ← required env vars documented
├── .gitignore
├── global-setup.ts                 ← health check + env var validation on startup
│
├── api/
│   ├── base.api.ts                 ← abstract BaseApi: retry, rate-limit, auth headers
│   └── {module}.api.ts             ← typed methods per request, path params resolved
│
├── config/
│   └── env.ts                      ← lazy env var getters (no crash at import time)
│
├── fixtures/
│   ├── auth.fixture.ts             ← JWT / OAuth2 / API key / Basic / None strategies
│   ├── base.fixture.ts             ← root fixture composing auth + all API clients
│   └── {module}.fixture.ts         ← per-module API client fixture
│
├── utils/
│   └── assertions.ts               ← assertStatus, assertArray, assertHasProperty,
│                                      assertHeaderExists, assertInclusion, assertEnumIncludes
│
├── models/                         ← TypeScript interfaces inferred from response shapes
│
├── tests/
│   └── {module}/
│       ├── list-*.spec.ts          ← @smoke · @contract · @security tags per spec
│       ├── create-*.spec.ts
│       └── poll-*.spec.ts
│
└── .github/
    └── workflows/
        └── ci.yml                  ← smoke suite on every PR, artifact upload

CLI Commands

# Interactive first run — setup + migrate
npx @magentic-platform/postman-playwright

# Migrate a specific module (flags override saved config)
npx @magentic-platform/postman-playwright migrate \
  --input  ./postman-collections \
  --output ./playwright-api-tests \
  --module auth

# Subsequent modules — additive, never overwrites existing
npx @magentic-platform/postman-playwright migrate --module payments
npx @magentic-platform/postman-playwright migrate --module exports --force  # overwrite if exists

# Migrate + immediately validate parity in one command
npx @magentic-platform/postman-playwright migrate --module auth --validate

# Reconfigure provider (e.g. switch from Anthropic → Azure OpenAI)
npx @magentic-platform/postman-playwright setup

# Show migration status across all modules
npx @magentic-platform/postman-playwright status --output ./playwright-api-tests

# Validate parity — Newman vs Playwright (standalone, run any time after migration)
npx @magentic-platform/postman-playwright validate \
  --module auth \
  --env    staging \
  --delta  0.02     # acceptable pass-rate delta (default: 0.02 = 2%)

Validate output example

  Validating auth → staging  (Δ threshold: 2%)

  Newman
    Requests : 8/10 passed
    Failed   : 2

  Playwright
    Tests    : 14/16 passed
    Failed   : 2
    Skipped  : 0

  Parity delta
    3.1%  (threshold: 2%)

  Endpoint mismatches
    GET /auth/me                             ✓ Newman  ✗ Playwright
    POST /auth/refresh                       ✗ Newman  ✓ Playwright

  ✗ Parity check failed — fix failing specs then re-run validate.

Supported Foundation Models

All 7 providers are wired and selectable via the setup wizard. API keys are stored in .env — never in the config JSON.

| Provider | Setup | Model examples | |---|---|---| | Anthropic Claude | ANTHROPIC_API_KEY | claude-opus-4-6, claude-sonnet-4-6 — optional baseUrl for proxy | | OpenAI | OPENAI_API_KEY | gpt-4o, o1, o3-mini | | OpenAI-compatible endpoint | OPENAI_API_KEY + baseUrl | Any hub (Azure AI Foundry, etc.) | | Azure OpenAI | AZURE_OPENAI_API_KEY + endpoint + deployment | Any GPT-4 deployment | | AWS Bedrock | IAM credentials + AWS_REGION | Claude, Llama 3, Titan | | Google Vertex AI | GCP service account + project | Gemini 1.5 Pro/Flash | | Mistral AI | MISTRAL_API_KEY | mistral-large, codestral | | Ollama (local) | No key — http://localhost:11434/v1 | Llama 3.1, Phi-3, Gemma 2, CodeLlama |

Idempotent Run Model

The tool inspects the output folder before every run and behaves accordingly:

| State | What happens | |---|---| | empty | Scaffolds full monorepo (config, fixtures, utils, CI) then migrates the module | | initialised | Reads existing structure, adds only the new module, merges base.fixture.ts | | has-module | Dry-run by default — prompts to skip or overwrite. --force skips the prompt |

This means you can safely run it on every CI push — it will never clobber a module already migrated.

Migration Pipeline

Postman JSON files (./postman-collections/<module>/)
  │
  ├─ [1/6] Scaffold         pnpm-workspace · playwright.config · base.api · utils · CI · eslint.config.mjs
  ├─ [2/6] Parse            Postman JSON + *.environment.json → CollectionIR
  │                          (method, path, headers, body, auth strategy,
  │                          pre-request scripts, pm.test() assertions,
  │                          setNextRequest chains, env var usage)
  │                         Reads Postman environment files — auto-sets API_URL in .env
  │                          from baseUrl variable if present (no manual editing needed)
  ├─ [3/6] Type generation  response body shapes → models/*.ts TypeScript interfaces
  ├─ [4/6] Fixture + API    auth strategy → auth.fixture.ts
  │   client generation      CollectionIR requests → api/{module}.api.ts typed methods
  │                          (path params: {{variable}} and :param both resolved)
  ├─ [5/6] Test generation  Foundation Model → tests/{module}/*.spec.ts
  │                          Positive requests → happy path @smoke + validation @validation + auth @security
  │                          Negative requests (4xx) → detected by name/assertions, tested at expected error status
  │                          Postman status codes extracted and enforced — LLM cannot guess/change them
  │                          Postman example body values injected into prompt (not just types)
  │                          Sibling endpoint schemas available for correct setup/teardown data
  └─ [6/6] Quality checks   tsc --noEmit → structured violations per file + line
                             Auto-fix pass 1: near-miss assertion names → aliases in utils/assertions.ts
                             Auto-fix pass 2: LLM repair — TS2339 method mismatches in specs
                               (reads real api/{module}.api.ts, rewrites broken calls via same adapter)
                             Auto-fix pass 3: tsc re-run to verify repairs
                             ESLint pass (warns, never blocks migration)

Running Generated Tests

cd playwright-api-tests

# 1. Install dependencies
pnpm install
npx playwright install --with-deps chromium

# 2. Configure environment
# .env is created automatically. If your Postman environment file contains a
# baseUrl variable, API_URL is populated automatically — no manual editing needed.
# Falls back to localhost:8080 if no Postman baseUrl is found.
# API_USER / API_PASS are only needed if your API has an auth layer

# 3. Run
pnpm test:smoke        # happy path only — fast feedback
pnpm test              # full suite
pnpm test:contract     # schema/contract tests
pnpm test:security     # auth/401 tests
pnpm test:staging      # against staging URL
pnpm typecheck         # tsc --noEmit — catch spec issues

Configuration File

.postman-playwright.json is saved in your working directory. Sensitive values use ${ENV_VAR} references — the actual keys live in .env only.

{
  "provider": "openai",
  "inputDir": "./postman-collections",
  "outputDir": "./playwright-api-tests",
  "version": "0.1.0",
  "credentials": {
    "type": "openai",
    "apiKey": "${OPENAI_API_KEY}",
    "model": "gpt-4o",
    "baseUrl": "https://your-hub.example.com/v1"
  }
}

Run npx @magentic-platform/postman-playwright setup at any time to switch provider or update credentials.

Monorepo Architecture (Tool Source)

This tool is itself a pnpm monorepo of composable TypeScript packages:

apps/
  cli/                          ← @magentic-platform/postman-playwright — Commander.js CLI entry point

packages/
  agents/
    collection-parser/          ← Postman JSON → CollectionIR (fully implemented ✓)
    type-generator/             ← response shapes → models/*.ts (✓)
    fixture-generator/          ← auth detection + fixture generation (✓)
    test-generator/             ← Foundation Model → .spec.ts (✓)
    quality-agent/              ← tsc + ESLint on output, auto-fix assertion aliases (✓)
    validator/                  ← Newman vs Playwright parity diff (✓)

  skills/                       (each has SKILL.md per agentskills.io spec + TypeScript implementation)
    auth-skill/                 ← JWT/OAuth2/ApiKey/Basic strategy detection + code gen (✓)
    assertion-skill/            ← pm.test() → expect() conversion rules (✓ defined, ⚠ not called in pipeline)
    chain-resolver-skill/       ← setNextRequest chains → sequential steps (✓)
    fixture-skill/              ← module fixture + base fixture composition (✓)
    type-inference-skill/       ← JSON shape → TypeScript interface (✓ basic, ⚠ no LLM inference yet)

  mcp-servers/
    postman-mcp/                ← MCP tools: read_collection, list_requests, get_request_context, summarize_collection (✓ defined, pipeline uses collection-parser directly)
    playwright-mcp/             ← MCP tools: validate_typescript, run_eslint, run_spec (✓ wired — used by quality-agent)
    aws-mcp/                    ← MCP tools: S3, DynamoDB (✓ defined, ⚠ not wired — for future artifact storage)
    git-mcp/                    ← MCP tools: commit, push, create_pr (✓ wired — used by --commit flag)

  foundation/
    model-adapter/              ← Unified LLM interface — all 7 providers (✓)
    shared-types/               ← CollectionIR, GeneratedFile, agent contracts (✓)
    logger/                     ← Structured JSON logging (✓)

  playwright-framework/         ← Output scaffold templates + BaseApi + scaffolder (✓)

Known Gaps & Roadmap

Currently missing from pipeline

| Gap | Impact | Notes | |---|---|---| | Assertion skill not used | LLM generates assertions from pm.test() context, but not via structured assertion-skill | Assertion skill converts pm.test() → expect() — could improve precision further | | MCP servers not connected | All 4 MCP server packages are defined but not invoked by any agent | Architectural future step | | Data factory generation | data/factories/ directory created but no factory files generated | Needs a factory-generator agent | | npx playwright install not auto-run | Playwright browser install still manual | Quality checks auto-run pnpm install, but user still needs npx playwright install | | Type inference is heuristic-only | type-inference-skill uses structural analysis only — no LLM call for complex/nested types | LLM-assisted inference planned | | chained request specs | setNextRequest chains are resolved to steps but not woven into spec output | chain-resolver-skill works, test-generator prompt doesn't use the output yet |

Recently resolved

| Item | Resolution | |---|---| | Postman environment auto-detection | Collection parser now reads *.environment.json files and populates .env with baseUrl automatically | | Negative test misclassification | Test-generator detects negative tests by name patterns + Postman assertions — no longer forces happy-path @smoke on 4xx requests | | Postman status codes ignored | Expected status codes are extracted from pm.test() assertions and enforced in the LLM prompt — no more guessing 200 vs 204 | | Request body values stripped | summariseBody() now includes Postman example values alongside types — LLM gets name: string (example: "Widget") instead of just name: string | | Root-path URL 404 | BaseApi.rel() handles root path (/ or '') by using absolute API_URL, avoiding trailing-slash mismatches with Fastify/Express | | --commit flag | Wired via git-mcp — warns gracefully if git is not available | | status command | Implemented — shows per-module migration state, spec count, last run results |

Planned improvements

[ ] Wire assertion-skill: pass structured converted pm.test() rules into test-generator (currently using raw pm.test() lines)
[ ] Generate data/factories/ — entity factories for per-test isolation
[ ] Auto-run npx playwright install after successful migration
[ ] LLM-assisted type inference for complex nested response shapes
[ ] Weave chain-resolver output into test-generator for sequential setNextRequest specs

Versioning & Publishing

This monorepo uses Changesets for consistent semver across all packages.

Daily workflow — when you make changes

# 1. Make your code changes, then describe them in a changeset
pnpm changeset
# → prompts: which packages changed? patch / minor / major? what changed?
# → writes a .changeset/<name>.md file — commit this alongside your code

# 2. Commit both together
git add .
git commit -m "feat: my change"

Releasing

# Bump all affected package versions + auto-update CHANGELOGs
pnpm version

# Build + publish all changed packages to npm
pnpm release

Automated via CI

Pushing to main triggers .github/workflows/release.yml (powered by changesets/action):

If there are pending changesets → opens a "Version Packages" PR with all bumps + changelogs
When that PR is merged → publishes to npm automatically

Prerequisites:

npm account in the @magentic-platform org
NPMJS_TOKEN secret set in GitHub repo settings (granular token with read & write for the package)

What gets published (dist/ only — all workspace packages are bundled in):

dist/index.js   ← ~95 KB single-file bundle, shebang included
package.json

Semver guide

| Change type | Command | Example | |---|---|---| | Bug fix | patch | 0.1.1 → 0.1.2 | | New feature (backwards-compatible) | minor | 0.1.2 → 0.2.0 | | Breaking change | major | 0.2.0 → 1.0.0 |

Contributing

See CONTRIBUTING.md and CODE_OF_CONDUCT.md.

Issues and PRs welcome — especially for the items in the roadmap above.