postman-api-load-agent

An AI-assisted API performance and load-testing agent. It takes a Postman Collection (v2.1) and Environment, generates a JMeter 5.6.x test plan, executes it in non-GUI mode, and produces Markdown/JSON performance reports plus JMeter's own HTML dashboard — orchestrated from Node.js/TypeScript with a Windows/PowerShell-first workflow.

Status

Fully implemented and tested: Postman discovery/analysis, Postman → JMeter (.jmx) conversion (including auth, body modes, correlation/extraction of values between requests), JMX validation, JMeter execution in non-GUI mode, results (.jtl) analysis, and Markdown/JSON report generation. AWS distributed execution (src/aws/) is an intentional stub reserved for a future task — see Known limitations.

Quick start

./run.ps1 setup
./run.ps1 validate
./run.ps1 run smoke --dry-run

or, cross-platform:

npm install
cp .env.example .env
npm run validate
npm run run -- smoke --dry-run

That runs the bundled sample collection. To run your own instead, point straight at your export and pick your own load — no profile, no copying files into this project, no other .env setup beyond JMETER_HOME:

npm run run -- --collection "path/to/My API.postman_collection.json" --users 50 --duration 300 --dry-run

Give this to your end users (one command, their own files)

If someone just wants to hand a Postman collection + environment at a load and get a report back — no git clone, no cd, no other setup on their end beyond what's below — this is the whole flow to send them.

One-time setup per machine:

npm install -g postman-api-load-agent

Requires a JRE (Java 8+) and Apache JMeter 5.6.x unzipped locally, with JMETER_HOME set as a real environment variable pointing at the folder containing bin/jmeter.bat/bin/jmeter.sh.

Then, every time, one command — swap in their own collection, environment, users, and duration:

postman-api-load-agent run --collection "My API.postman_collection.json" --environment "My Environment.postman_environment.json" --users 50 --duration 300

• File paths can be relative to wherever they run the command from, or absolute — no need to copy anything into this project first.
• It'll ask them to type yes before firing real traffic; add --dry-run to a first run to sanity-check everything without load, or --yes to skip the prompt.
• Output (jmeter/, reports/, generated docs) lands next to their --collection file, so different users/runs never collide.
• If their npm registry access is restricted, see If you can't use the public npm registry for the local-checkout + npm link alternative — the run command afterward is identical.

Installing as a global command (no codebase to manage day-to-day)

The steps above are for actively working inside this repo. For anyone who just wants to point at their own collection and run a load test — never clone or touch this project's source at all — install the published CLI once per machine:

npm install -g postman-api-load-agent

That's it — one command, no git clone, no local checkout to keep around.

If you can't use the public npm registry

If your machine can't reach the public npm registry (offline/restricted network), fall back to installing from a local checkout instead:

git clone https://github.com/hard-shah-simform/API_Load_Postman_Jmeter.git
cd API_Load_Postman_Jmeter
npm install
npm link

(npm install -g <git-url> directly, with no local clone, looks tempting but is not reliable on Windows npm - it can leave a broken symlink into an ephemeral temp/cache folder instead of a real install. The four commands above are the verified-working one-time setup for this path; you never have to run them again, and never have to touch the cloned folder again either - npm link makes the postman-api-load-agent command available globally from any directory, backed by that clone.)

Either way, it's a one-time setup per machine (it also needs JMeter 5.6.x + Java installed locally, same as any other workflow here — see below). From then on, from any directory, with no other files required beyond their own Postman exports:

postman-api-load-agent run --collection "My API.postman_collection.json" --environment "Staging.postman_environment.json" --users 50 --duration 300

• JMETER_HOME must be set as a real environment variable (system/user env var, or a shell profile export) — there's no project .env file to put it in anymore. postman-api-load-agent validate checks this and everything else below.
• --environment is optional; omit it if the collection doesn't need one.
• If the collection uses variables like {{BASE_URL}} / {{TOKEN}} that it doesn't define itself, just export the environment from Postman and put the .postman_environment.json file next to the collection - it's picked up automatically, no extra flag needed. See Automatic environment detection.
• Output (jmeter/, reports/, docs/*.md) is written next to the --collection file itself when you point --collection at a path anywhere on disk — so you never have to cd into a particular folder first, and different collections/runs never collide. (If you instead reference a collection by name under this project's own postman/collections/, output goes to the current working directory instead, matching the project structure above.) Either way, nothing ever writes into the global npm install location itself.
• Everything under Running a test below — profiles, --dry-run, --yes, production safety — works identically; just replace npm run run -- with postman-api-load-agent run.

Prerequisite: JMeter + Java

Whether run from a checkout or installed globally, this agent only ever orchestrates a real local JMeter installation — it never bundles or downloads JMeter/Java itself. Each machine that runs a test needs:

1. A JRE (Java 8+).
2. Apache JMeter 5.6.x, unzipped anywhere.
3. JMETER_HOME pointing at that folder (the one containing bin/jmeter.bat/bin/jmeter.sh) — via .env in a checkout, or a real environment variable for a global install.

npm scripts

| Script | Description | | --- | --- | | npm run build | Compiles TypeScript to dist/. | | npm run test | Runs the Vitest test suite. | | npm run validate | Validates local environment setup (directories, .env, JMETER_HOME, Postman inputs). | | npm run run -- <profile\|nothing> [flags] | Runs the agent orchestrator (see Running a test below). |

Project structure

postman-api-load-agent/
│
├── postman/                 # Postman Collection / Environment JSON inputs
│   ├── collections/
│   └── environments/
│
├── jmeter/                  # JMeter templates, generated .jmx plans, results
│   ├── templates/
│   ├── generated/           # Generated .jmx test plans (gitignored contents)
│   └── results/             # Per-run <timestamp>/{results.jtl, jmeter.log, html-report/}
│
├── config/                  # Versioned, non-secret app configuration
│   ├── default.json         # Paths, JMeter defaults, reporting formats
│   └── load-profiles.json   # smoke/baseline/load/stress/spike/soak definitions
├── scripts/                 # PowerShell helper scripts (setup, JMeter check)
├── reports/                 # Generated Markdown/JSON reports (gitignored)
├── docs/                    # Documentation
│
├── src/
│   ├── agent/                # CLI entry point, orchestrator, presentation/safety helpers
│   ├── postman/               # Postman parsing, analysis, loading
│   ├── jmeter/                  # JMX generation, validation, correlation, execution
│   ├── config/                   # Config/env loading, variable resolution
│   ├── reporting/                 # JMeter .jtl results analysis + report generation
│   ├── aws/                        # Distributed execution on AWS (stub, disabled)
│   └── utils/                       # Logger, filesystem helpers, secret masking
│
├── tests/                   # Vitest tests (253 tests across 20 files)
├── .env.example
├── .gitignore
├── package.json
├── tsconfig.json
├── README.md
└── run.ps1                  # PowerShell entry point wrapping npm scripts

See docs/directory-structure.md for a full explanation of every directory, docs/architecture.md for the pipeline design, and docs/getting-started.md for setup details.

Configuration

• Non-secret defaults (JMeter thread-group defaults, relative paths, report formats) live in config/default.json.
• Load profiles (smoke/baseline/load/stress/spike/soak — users, ramp-up, duration) live in config/load-profiles.json.
• Machine-specific and secret values (JMETER_HOME, AWS credentials, LOG_LEVEL, DEFAULT_*) are read from environment variables via .env (copy from .env.example; never commit .env).

Adding a Postman collection

Any user can run this against their own collection without touching this project's files, two ways:

• Point straight at your export — pass its full/relative file path to --collection, e.g. --collection "C:\Users\me\Downloads\My API.postman_collection.json" or --collection ../my-api/My API.postman_collection.json. Nothing needs to be copied anywhere first.
• Or, to keep it versioned in this project: export your collection from Postman as Collection v2.1 JSON, place it in postman/collections/, then run npm run validate to confirm the agent can see it, and reference it by name/file name with --collection "<name>".

Adding an environment

Environments work the same way, and are entirely optional — omit --environment if your collection doesn't need one.

• Point straight at your export — pass its file path to --environment, exactly like --collection above.
• Or keep it versioned: export the matching Postman environment as JSON, place it in postman/environments/, and reference it by name/file name with --environment "<name>", or set DEFAULT_ENVIRONMENT in .env.

Either way:

• Never commit a real secret value. If an exported environment has a non-empty "type": "secret" variable, ConfigManager.findCommittedSecretWarnings flags it — clear the value in the checked-in file and supply the real value via .env/the shell environment at run time instead.
• An environment name containing "prod" (case-insensitive, e.g. Production, prod-us-east) is treated as production and is refused unless you pass --allow-production — and even then you must type PRODUCTION to confirm. See Production safety.

Automatic environment detection

When no environment is specified (no --environment, no DEFAULT_ENVIRONMENT) and the collection references variables it doesn't define, the agent looks for an environment that defines all of them, in this order:

1. Local export - a *.postman_environment.json file next to the --collection file (or in postman/environments/ for a collection referenced by name):

   My folder/
     My API.postman_collection.json
     Staging.postman_environment.json   <- used automatically

2. Your Postman account - when POSTMAN_API_KEY is set (a real environment variable or .env), environments are fetched with the Postman API. Optional: POSTMAN_WORKSPACE_ID limits the search to one workspace, and POSTMAN_ENVIRONMENT picks an environment by name.

   export POSTMAN_API_KEY=PMAK-xxxx
   postman-api-load-agent run --collection "My API.postman_collection.json" --users 50 --duration 300

   The Postman API only returns each variable's shared Initial value. A Current value that was never shared/synced (common for tokens) is not available - set an Initial value in Postman, or use a local export.

Exactly one matching environment is used; if several match, the run stops and asks you to choose (--environment locally, POSTMAN_ENVIRONMENT for the cloud) rather than guessing. Nothing changes for collections that already resolve, or when you name an environment yourself. Production-looking environment names are still refused without --allow-production.

As a last resort, values can also be passed with --var KEY=VALUE (repeatable; overrides collection/environment values of the same name).

Running a test

npm run run -- [profile] [--users N] [--ramp-up N] [--duration N] \
  [--collection NAME_OR_PATH] [--environment NAME_OR_PATH] [--var KEY=VALUE ...] [--dry-run] [--yes] [--allow-production]

profile is entirely optional. Pass one of smoke, baseline, load, stress, spike, soak (defined in config/load-profiles.json) to start from its preset users/ramp-up/duration, or skip it and just say what you want directly:

npm run run -- --collection "My API.postman_collection.json" --users 50 --duration 300 --ramp-up 30

Anything you don't set falls back to DEFAULT_USERS/DEFAULT_RAMP_UP_SECONDS/ DEFAULT_DURATION_SECONDS from .env (or their built-in defaults if .env doesn't set them either) — --users/--ramp-up/--duration work whether or not a named profile is given, and override that profile's values for this run only when one is.

Smoke

npm run run -- smoke --collection "My API" --environment "Staging"

1 user, 1s ramp-up, 30s duration. No confirmation prompt (low-impact).

Baseline

npm run run -- baseline --collection "My API" --environment "Staging"

5 users, 30s ramp-up, 120s duration. No confirmation prompt.

Load

npm run run -- load --collection "My API" --environment "Staging"

50 users, 120s ramp-up, 600s duration. Requires typing yes to confirm (or pass --yes).

Stress

npm run run -- stress --collection "My API" --environment "Staging"

200 users, 180s ramp-up, 900s duration. Requires confirmation.

Spike

npm run run -- spike --collection "My API" --environment "Staging"

150 users, 10s ramp-up, 120s duration. Requires confirmation.

Soak

npm run run -- soak --collection "My API" --environment "Staging"

20 users, 60s ramp-up, 3600s duration. Requires confirmation.

Production safety

• An environment whose name looks like production is refused outright unless --allow-production is passed.
• Even with --allow-production, the run always stops and requires typing PRODUCTION (all caps) to proceed — --yes never bypasses this, regardless of profile.
• smoke/baseline skip the ordinary "type yes" confirmation (low impact); every other profile (load/stress/spike/soak, or the default profile) requires it unless --yes is passed.
• JMeter is only ever invoked in non-GUI mode (-n) — there is no code path that launches its GUI.
• A run only reports success when JMeter exited 0, did not time out, AND the results file actually exists on disk.

Dry run

--dry-run runs discovery, variable resolution, correlation detection, JMX generation, and JMX validation exactly as a real run would (so failures — e.g. a missing JMETER_HOME or an invalid collection — surface immediately), then prints the full execution plan and stops before asking for confirmation. Nothing is spawned, and no results/performance-report files are written (the .jmx and the docs/*-report.md conversion reports are, since those describe the conversion itself, not a test run).

npm run run -- smoke --dry-run --collection "Example API Collection" --environment "Example Environment"

# API SMOKE TEST (DRY RUN)

Collection: Example API Collection
Environment: Example Environment
API count: 1
Profile: Smoke

Users: 1
Ramp-up: 1 sec
Duration: 30 sec

JMX file: .../jmeter/generated/example-api-collection.jmx
Expected result location: .../jmeter/results/<timestamp>/ (results.jtl, jmeter.log, html-report/)
Dry run: no JMeter process was started, and no results/report files were written.

Natural-language commands

This agent is designed to be driven by an AI coding assistant (such as Claude Code) as well as directly from the CLI. Describe what you want in plain English and have the assistant translate it into the deterministic CLI form above — the assistant should always show you the resulting command (and run it with --dry-run first for anything above smoke/baseline) rather than inventing new flags:

| You say | Assistant runs | | --- | --- | | "Smoke test the Checkout API against staging" | npm run run -- smoke --collection "Checkout API" --environment "Staging" | | "Run a load test, 100 users over 5 minutes" | npm run run -- load --users 100 --ramp-up 300 | | "Stress test for 15 minutes, 300 users, don't ask me to confirm" | npm run run -- stress --users 300 --duration 900 --yes | | "Just show me what a spike test would do, don't run it" | npm run run -- spike --dry-run | | "Soak test prod for an hour" | Refused unless you explicitly add --allow-production, and you'll still be asked to type PRODUCTION. |

Because every profile, flag, and safety gate is fixed and validated (see src/agent/presentation.ts, src/jmeter/LoadProfileManager.ts), natural language only ever selects among these known, tested options — it never introduces new, unvalidated behavior.

Viewing reports

After a real run completes, three outputs are written:

• reports/performance-summary.md — human-readable overall + per-API metrics table (requests, error %, average/P90/P95/P99, throughput) plus neutral "performance observations" for samplers crossing a threshold.
• reports/performance-summary.json — the same data, machine-readable.
• jmeter/results/<timestamp>/html-report/index.html — JMeter's own interactive HTML dashboard (open in a browser).

The terminal also prints a concise summary (total requests, error %, average, P95, P99, throughput, and the paths above) when a run finishes.

For a per-request, pass/fail view (rather than the aggregate error % in the summary above), open the generated .jmx in the JMeter GUI and enable its View Results Tree listener — see JMX file structure.

JMX file structure

Every generated .jmx (jmeter/generated/<collection>.jmx, and its timestamped archive copy under jmeter/jmx/) follows the same layout, top to bottom, matching the order nodes appear in the JMeter GUI's tree view when you open the file.

Whole-file scope (apply to every request in the plan):

• (the collection name) — TestPlan. Root node. Functional mode off, "tear down on shutdown" on.
• User Defined Variables — Arguments. Every collection/environment variable found in the Postman export (e.g. baseUrl, correlation variables like createdPostId), sorted alphabetically. Values are left blank on purpose: real values are resolved at run time from .env/environment JSON, never baked into the checked-in .jmx.
• HTTP Request Defaults — ConfigTestElement. Present but intentionally empty. This agent always writes each sampler's full absolute URL into its own "Path" field rather than splitting it into domain/port/protocol, so there's nothing non-invented to put here.
• HTTP Header Manager — HeaderManager. Empty at this level. Headers are only ever added on the specific request that declares them (see below), never guessed at globally.
• HTTP Cookie Manager — CookieManager. Standard cookie jar, standard policy, so JMeter carries cookies between requests the way a browser/Postman would.
• View Results Tree — ResultCollector (guiclass="ViewResultsFullVisualizer"). Disabled by default. Open the .jmx in the JMeter GUI, right-click this node, choose Enable, then hit the green Start button to watch each request go green (pass) or red (fail) live, with full request/response detail per sampler. It's off by default because this listener holds every sample in memory for the run's duration: fine for a one-off GUI look, risky left on for an unattended load/stress/soak run. It has no effect on this agent's own automated runs, which always execute JMeter in non-GUI mode (-n), where the listener is simply inert either way.

Per-run scope:

• (profile name, e.g. "smoke profile") — ThreadGroup, one per load profile run. Users/ramp-up/duration for the selected profile (config/load-profiles.json, or --users/--ramp-up/--duration overrides). Uses the scheduler (fixed duration), not a fixed loop count.
  • (folder name, if the Postman collection used folders) — GenericController, one per Postman folder, nested to match the collection's own folder structure. A request with no folder sits directly under the Thread Group instead.

Per-request scope (repeated once for every convertible Postman request; see Known limitations for what's skipped instead of guessed at):

• (request name, e.g. "GET All Posts") — HTTPSamplerProxy. The actual HTTP call: method, path (with {{var}} converted to ${var}), query params/body arguments.
  • HTTP Header Manager — HeaderManager, only added when the request declares its own headers (or needs an implied Content-Type for its body/auth); not present on requests with none.
  • HTTP Authorization Manager — AuthManager, only for Postman basic/digest auth on that request. Bearer/API-key auth is converted to a header or query param instead (no separate node); other auth types are left unauthenticated and reported in docs/postman-analysis.md.
  • Response Code Assertion — ResponseAssertion, only added when the request's Postman test script asserts a specific status code (e.g. pm.response.to.have.status(200)); checks Assertion.response_code against that value.
  • Extract <variable> — JSONPostProcessor, only added when docs/correlation-report.md detected this request's test script setting a variable from its own JSON response (e.g. a login response's token); extracts it so a later sampler can reference ${variable}.

Every element above is followed by its own hashTree (JMeter's file format pairs each element with a hashTree of its children, even when that tree is empty) — you won't see those as separate nodes in the GUI; it's just how JMeter nests the tree in XML.

Enabling AWS monitoring / distributed execution

Not implemented yet. src/aws/index.ts exports launchDistributedRun(), which always throws "AWS distributed execution is not yet implemented." and is never called by the orchestrator — running a local test never touches AWS. AWS_REGION/AWS_PROFILE/AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY in .env.example are reserved for this future work; if you do set real AWS credentials in .env today, they are still registered with SecretMasker (so they'd never be logged in cleartext) but otherwise unused. There is no monitoring/CloudWatch integration in this codebase.

Known limitations

• AWS distributed execution is not implemented — every run executes JMeter locally, single-machine.
• Auth inheritance is not resolved — only a request's own request.auth block is converted; a request that relies on its parent folder's or the collection's inherited auth is generated without authentication (see docs/postman-analysis.md).
• Correlation (auto-chaining {{token}}-style values between requests) only covers a plain pm.*.set("x", pm.response.json().field)-shaped test script. A computed value, a value read from a cookie, or a dynamically built variable name is reported in the correlation report (docs/correlation-report.md) rather than guessed at — you'd add that extractor manually in the generated .jmx.
• Unsupported request pieces are skipped, not dropped silently: an unsupported HTTP method, auth type, or body mode is recorded in docs/jmeter-conversion-report.md (per-collection warnings/skips) and the generated sampler is emitted without that piece — never invented.
• A form-data file field cannot be converted (Postman's export has no file content) — add it manually in JMeter.
• The JTL analyzer expects JMeter's default CSV result format, not the XML result format — this is what JMeter writes by default and is what JMeterRunner configures.
• Windows-first tooling (run.ps1, scripts/*.ps1); the underlying Node.js/TypeScript code itself is cross-platform.