YAMLTest

Declarative YAML-based test runner for HTTP endpoints, shell commands, and Kubernetes resources.

Define tests in YAML, run them from the CLI or import them programmatically. No test framework boilerplate required.

Installation

From npm (once published):

npm install -g yamltest

From a local clone (before publishing):

# Install globally from the repo directory
npm install -g /path/to/YAMLTest

# Or run directly without installing
node /path/to/YAMLTest/src/cli.js -f your-tests.yaml

# Or link it for development (makes YAMLTest available system-wide, auto-updates)
cd /path/to/YAMLTest && npm link

As a dev dependency in another project:

npm install --save-dev /path/to/YAMLTest

Quick start

YAMLTest -f - <<EOF
- name: httpbin returns 200
  http:
    url: "https://httpbin.org"
    method: GET
    path: "/get"
  source:
    type: local
  expect:
    statusCode: 200
    bodyContains: "httpbin.org"
EOF

Output:

  ✓ httpbin returns 200 312ms

  1 passed | 1 total

CLI

USAGE
  YAMLTest -f <file.yaml>
  YAMLTest -f -              # read from stdin (heredoc)

OPTIONS
  -f, --file <path|->   YAML file to run, or - for stdin
  --retries <n>         Force the retry count for every test, overriding the
                        per-test `retries` and the default. Use --retries 0 to
                        run once when debugging a failure.
  --check               Validate YAML structure only; do not run tests
  -h, --help            Show this help

ENVIRONMENT
  DEBUG_MODE=true       Enable verbose debug logging
  YAMLTEST_ECHO=true    Echo every local command's output live as it runs
  NO_COLOR=1            Disable ANSI colour output

When a test fails and you want to debug it, run the same file with --retries 0 so it executes exactly once instead of retrying up to the default 120 times:

yamltest -f test.yaml --retries 0

Exit codes: 0 = all passed, 1 = one or more failed.

Input validation

All test definitions are validated against a strict schema before any test executes. Invalid input produces clear error messages pointing to the exact location of the problem:

Validation failed:

  Test #1 ("login") /http/method: must be one of: GET, POST, PUT, DELETE, PATCH
  Test #3: must define exactly one of: http, command, wait, httpBodyComparison
  Test #5 ("check pods") /source: missing required property "selector"

Validation checks include:

• Exactly one test type per definition (http, command, wait, or httpBodyComparison)
• Required fields per test type (e.g., command.command, wait.target)
• Value types and allowed enums (e.g., source.type must be local or pod)
• Conditional requirements (e.g., source.type: pod requires selector; setVars requires expect for HTTP/command tests)
• Unknown property detection on sub-objects (catches typos like htpp instead of http)

Programmatic API

const { runTests, executeTest, validateTestDefinitions } = require('yamltest');

// Run one or more tests from a YAML string (array or single object)
const result = await runTests(yamlString);
console.log(result.passed, result.failed, result.skipped, result.total);
// result.results → [{name, passed, error, durationMs, attempts}]

// Run a single test (low-level)
await executeTest(yamlString); // returns true or throws

// Validate test definitions without executing them
const yaml = require('js-yaml');
const definitions = Array.isArray(yaml.load(yamlString))
  ? yaml.load(yamlString)
  : [yaml.load(yamlString)];
validateTestDefinitions(definitions); // throws on validation errors

Test format

Tests are defined as flat YAML objects (or an array of them). All fields except the test type key (http, command, wait, httpBodyComparison) are optional.

- name: my-test          # optional display name
  retries: 3             # retry up to N times on failure (default: 0)
  http: ...              # ← test type
  source:
    type: local          # local | pod
  expect:
    statusCode: 200

Test types

HTTP test

Test any HTTP endpoint locally or from within a Kubernetes pod.

- name: health check
  http:
    url: "https://api.example.com"   # required (or auto-discovered for Service selectors)
    method: GET                       # GET (default) | POST | PUT | DELETE | PATCH
    path: "/health"                   # default: /
    headers:
      Authorization: "Bearer ${API_TOKEN}"
      Content-Type: application/json
    params:
      key: value                      # query parameters
    body: '{"foo":"bar"}'             # request body (string or object)
    skipSslVerification: true         # disable TLS verification
    maxRedirects: 0                   # redirects to follow (default: 0)
    cert: /path/to/cert.pem           # mTLS client certificate
    key:  /path/to/key.pem
    ca:   /path/to/ca.pem
  source:
    type: local
  expect:
    statusCode: 200                   # or [200, 201, 202]
    body: "exact body match"
    bodyContains: "substring"         # or array, or {value, negate, matchword}
    bodyRegex: "pattern.*"            # or {value, negate, caseInsensitive}
    bodyJsonPath:
      - path: "$.user.id"
        comparator: equals
        value: 42
    headers:
      - name: content-type
        comparator: contains
        value: application/json

Asserting a connection failure

Use expect.connectionError to assert that the request never completes — the connection is refused, DNS fails, or the TLS handshake is rejected. This replaces the old workaround of a command test (curl ... || true with equals: "000").

- name: mTLS is enforced (handshake must fail without a client cert)
  http:
    url: "https://secure.example.com"
    path: "/"
  source:
    type: local
  expect:
    connectionError: true

connectionError: true passes on any transport-level failure (DNS, TCP, or TLS) and is what most tests should use. To assert on a specific failure, use the object form — any of code, contains, or matches (all provided constraints must hold):

  expect:
    connectionError:
      code: "ECONNREFUSED"        # exact error code (see below)
      contains: "ECONNREFUSED"    # substring of the error message
      matches: "ECONN(REFUSED|RESET)" # regex against the error message

The code is matched exactly against Node's error code. Common ones: ECONNREFUSED (port closed), ENOTFOUND (DNS failure), EPROTO (TLS spoken to a plaintext port), DEPTH_ZERO_SELF_SIGNED_CERT (untrusted cert). The exact string depends on your Node/OpenSSL version, so the reliable way to find it is to run the test once and read the reported error rather than guessing.

Notes:

• connectionError cannot be combined with response-based expectations (statusCode, body, bodyContains, bodyRegex, bodyJsonPath, headers) — a failed connection produces no response to assert on.
• If the request unexpectedly succeeds, the test fails.
• Configuration errors (such as an unreadable cert/key/ca file) are not connection errors and still fail the test loudly.
• connectionError cannot be combined with setVars — a failed connection has no response to extract variables from.
• Reliably supported only for source.type: local. With pod-based sources, only usePortForward can detect a connection failure, and the reported error code reflects the local port-forward tunnel's failure (typically ECONNRESET), not the remote one. The default debug-pod and pod-exec modes wrap every transport error in a generic message, so connectionError will always fail the test there — even when the connection really did fail.

Environment variable substitution

Any $VAR or ${VAR} in the url, path, headers, body, or params fields is resolved from the environment:

http:
  url: "${API_BASE_URL}"
  path: /v0/agents/my-agent${RUN_SUFFIX}
  headers:
    Authorization: "Bearer ${API_TOKEN}"
  body: '{"id": "${REQUEST_ID}"}'
  params:
    apiKey: "${API_KEY}"

Pod-based HTTP test

Execute the HTTP request from inside a Kubernetes pod (useful for internal service testing):

source:
  type: pod
  selector:
    kind: Pod
    metadata:
      namespace: production
      labels:
        app: test-client
    context: my-cluster          # optional kubectl context
  container: my-container        # optional
  usePortForward: true           # use kubectl port-forward instead of debug pod
  usePodExec: true               # use kubectl exec + curl

Auto-discovery for Kubernetes Services

Omit http.url when source.selector.kind is Service and the IP/port are discovered automatically from the LoadBalancer status:

- name: auto-discover service
  http:
    method: GET
    path: /health
    scheme: https                # optional, defaults to http
    port: 443                    # optional port name/number/index
  source:
    type: local
    selector:
      kind: Service
      metadata:
        namespace: production
        name: my-service
  expect:
    statusCode: 200

Command test

Run any shell command and validate its output.

- name: check kubectl version
  command:
    command: "kubectl version -short"
    parseJson: false              # parse stdout as JSON (default: false)
    echo: true                    # echo the command's output live (default: false)
    env:
      MY_VAR: value               # extra environment variables
    workingDir: /tmp              # working directory
  source:
    type: local                   # or pod (uses kubectl exec)
  expect:
    exitCode: 0
    stdout:
      contains: "Client Version"  # or: equals, matches/regex, negate
    stderr:
      contains: ""

Echoing output for long-running commands

By default a command's stdout/stderr is captured silently and only shown at the end (and only for failing tests). For a long-running command — a browser flow, a multi-minute integration script — that means a blank screen until it finishes. Set echo: true to tee the command's output to your terminal as it is produced, while still capturing it for the assertions:

- name: long oauth flow
  command:
    command: "./run-oauth-flow.sh"
    echo: true
  source:
    type: local
  expect:
    exitCode: 0

Set it globally instead of per-test with the YAMLTEST_ECHO=true environment variable. Echoing applies to source.type: local commands. Because the live output interleaves with the ✓/✗ summary, it is off by default.

Multiple stdout expectations (all must pass):

expect:
  exitCode: 0
  stdout:
    - contains: "Running"
    - matches: "\\d+ pods"

JSON output validation:

- name: cluster info JSON
  command:
    command: "kubectl cluster-info --output=json"
    parseJson: true
  source:
    type: local
  expect:
    exitCode: 0
    jsonPath:
      - path: "$.Kubernetes"
        comparator: exists

Wait test

Poll a Kubernetes resource until a condition is met (or timeout).

- name: wait for deployment
  wait:
    target:
      kind: Deployment
      apiVersion: apps/v1        # optional
      metadata:
        namespace: default
        name: my-app
      context: my-cluster        # optional
    jsonPath: "$.status.readyReplicas"
    jsonPathExpectation:
      comparator: greaterThan
      value: 0
    polling:
      timeoutSeconds: 120        # default: 60
      intervalSeconds: 5         # default: 2
      maxRetries: 24             # optional upper bound
  setVars:                       # optional: capture the extracted value
    READY_REPLICAS:
      value: true

To use the captured value in subsequent tests, see the setVars section below.

Selector by labels:

wait:
  target:
    kind: Pod
    apiVersion: v1               # optional
    metadata:
      namespace: production
      labels:
        app: web-server
        version: v1.2.0
  jsonPath: "$.status.phase"
  jsonPathExpectation:
    comparator: equals
    value: Running

Asserting several paths

jsonPath also accepts an array of self-contained assertions — the same {path, comparator, value, negate} shape that command and http tests use — so one wait can check several fields of the same resource instead of falling back to a command shim or repeating the whole block:

- name: validators are delivered via the config ConfigMap
  wait:
    target:
      kind: ConfigMap
      metadata:
        name: agentgateway-config
        namespace: agentgateway-system
    jsonPath:
      - path: "$.data['token-exchange-validators.yaml']"
        comparator: contains
        value: "subjectValidators:"
      - path: "$.data['token-exchange-validators.yaml']"
        comparator: contains
        value: "actorValidators:"
      - path: '$.spec.template.spec.containers[0].env[?(@.name=="LOG_LEVEL")].value'
        comparator: equals
        value: debug
    polling:
      timeoutSeconds: 60
      intervalSeconds: 2

Semantics: one kubectl get ... -o json per poll attempt, and an attempt succeeds only when every entry passes. Any failure — including a path that is not present yet — retries until polling.timeoutSeconds. This matches how command and http treat their jsonPath arrays, and the run output still shows one block per test rather than one per entry. On timeout the error names the assertion that blocked:

Timed-out (60s) waiting for ConfigMap/agentgateway-system/agentgateway-config
  → $.a to contains "x" AND $.b to equals "y"
  (last failure: JSONPath $.b comparison failed: expected to equals "y", found z)

Two rules keep the forms unambiguous, both enforced by schema validation:

| Combination | Result | | --- | --- | | jsonPath string + jsonPathExpectation | supported (the original form) | | jsonPath array (each entry carries its own comparator) | supported | | jsonPath array + jsonPathExpectation | rejected — the per-entry comparator would silently supersede it | | jsonPath array + setVars | rejected — with several paths there is no unambiguous value to capture; use the string form |

HTTP body comparison test

Compare the response bodies of two HTTP calls and assert they are identical (useful for canary / shadow traffic validation).

- name: compare two backends
  httpBodyComparison:
    request1:
      http:
        url: "http://service-v1"
        method: GET
        path: /api/data
      source:
        type: local
    request2:
      http:
        url: "http://service-v2"
        method: GET
        path: /api/data
      source:
        type: local
    parseAsJson: true            # parse bodies as JSON before comparing
    delaySeconds: 1              # wait between requests
    removeJsonPaths:             # ignore dynamic fields
      - "$.timestamp"
      - "$.requestId"

Expectation operators

All comparators can be negated with negate: true.

| Comparator | Description | Types | |---------------|--------------------------------------|----------------| | equals | Deep equality | any | | contains | Substring / JSON-stringified search | string, object | | matches | Regular expression test | string | | exists | Value is not null/undefined | any | | greaterThan | Numeric > | number | | lessThan | Numeric < | number |

Negation example

expect:
  bodyContains:
    value: "error"
    negate: true        # assert the body does NOT contain "error"

Word-boundary match

expect:
  bodyContains:
    value: "ok"
    matchword: true     # uses \bok\b regex (whole word only)

Advanced features

Run control: retries, consecutive, timeout, maxtime

Each test definition accepts four optional run-control knobs. All have defaults tuned for waiting on eventually-consistent systems, and all defaults are overridable per run via environment variables (so you never have to edit the test to tune them):

| Field | Meaning | Default | Env override | |-------|---------|---------|--------------| | retries | Max retry attempts after the first | 120 | YAMLTEST_RETRIES | | consecutive | Successful runs required per attempt (all must pass in a row) | 1 | — | | timeout | Per-attempt cap, in ms (a hung attempt is abandoned and retried) | 10000 | YAMLTEST_TIMEOUT_MS | | maxtime | Wall-clock cap on the whole retry loop — the circuit breaker | max(180000, timeout + 60000) | YAMLTEST_MAXTIME_MS (floor) |

maxtime accepts either a number of milliseconds or a duration string ("500ms", "30s", "3m", "1h"). The retry loop stops at whichever comes first: retries exhausted or maxtime elapsed. The pause between retries defaults to 1000ms (YAMLTEST_RETRY_INTERVAL_MS).

- name: flaky service
  retries: 5            # retry up to 5 times after the first attempt
  http:
    url: "http://flaky-service"
    method: GET
    path: /api
  source:
    type: local
  expect:
    statusCode: 200

- name: routing is stable, not just lucky
  consecutive: 3        # must return the expected provider 3 times in a row
  retries: 10
  http: { url: "http://gw", method: POST, path: /failover }
  source: { type: local }
  expect:
    bodyJsonPath:
      - path: "$.model"
        comparator: contains
        value: "gemini"

- name: long browser/OAuth flow
  timeout: 360000       # one attempt may take up to 6 minutes
  maxtime: "8m"         # cap the whole retry loop at 8 minutes
  command: { command: "./run-oauth-flow.sh" }
  source: { type: local }
  expect: { exitCode: 0 }

On failure, the CLI prints the observed request/response (or command result) from the last attempt inline — no need to re-run under DEBUG_MODE to see why a test failed. Bodies and outputs are truncated so a large response can't flood the log.

Multiple tests in one file

Tests run sequentially and stop at the first failure (fail-fast).

- name: first test
  http: { url: "http://svc", method: GET, path: /ready }
  source: { type: local }
  expect: { statusCode: 200 }

- name: second test
  command: { command: "kubectl get pods -n default" }
  source: { type: local }
  expect: { exitCode: 0, stdout: { contains: "Running" } }

Environment variables in url, path, headers, body, and params

http:
  url: "$API_BASE_URL"          # $VAR or ${VAR}
  path: /v0/agents/my-agent${RUN_SUFFIX}
  headers:
    Authorization: "Bearer ${API_TOKEN}"
  body: '{"id": "${REQUEST_ID}"}'
  params:
    apiKey: "${API_KEY}"

setVars — variable passing between steps

Extract values from a test response and store them for use in subsequent steps via ${VAR_NAME} syntax. setVars requires expect to be present on the test — variables are only captured after all assertions pass.

HTTP extraction sources

- name: login
  http:
    url: "http://api.example.com"
    method: POST
    path: /login
    body: '{"user":"admin","pass":"secret"}'
  source:
    type: local
  expect:
    statusCode: 200
  setVars:
    AUTH_TOKEN:
      jsonPath: "$.token"           # extract from JSON body via JSONPath
    SESSION_ID:
      header: "x-session-id"       # extract a response header (case-insensitive)
    STATUS_CODE:
      statusCode: true             # capture the HTTP status code
    RAW_BODY:
      body: true                   # capture the full response body
    CSRF_TOKEN:
      regex:                       # extract via regex capture group from body
        pattern: 'name="csrf" value="([^"]+)"'
        group: 1                   # capture group index (default: 1)

Command extraction sources

- name: read config
  command:
    command: "cat config.json"
    parseJson: true                # required for jsonPath extraction
  source:
    type: local
  expect:
    exitCode: 0
  setVars:
    DB_HOST:
      jsonPath: "$.database.host"  # extract from parsed JSON stdout
    FULL_OUTPUT:
      stdout: true                 # capture full stdout
    ERR_OUTPUT:
      stderr: true                 # capture full stderr
    EXIT:
      exitCode: true               # capture exit code
    PID:
      regex:                       # extract via regex from stdout or stderr
        source: stdout             # "stdout" (default) or "stderr"
        pattern: "PID (\\d+)"
        group: 1

Wait extraction source

- name: wait for deployment
  wait:
    target:
      kind: Deployment
      metadata:
        namespace: default
        name: my-app
    jsonPath: "$.status.readyReplicas"
    jsonPathExpectation:
      comparator: greaterThan
      value: 0
  setVars:
    READY_REPLICAS:
      value: true                  # capture the jsonPath-extracted value

value: true requires the string form of jsonPath. The array form asserts several paths at once, so there is no single value to capture — see Asserting several paths.

Chaining example: login then access protected endpoint

- name: login
  http:
    url: "http://localhost:3000"
    method: POST
    path: /login
    body: '{"user":"admin","pass":"secret"}'
  source:
    type: local
  expect:
    statusCode: 200
  setVars:
    AUTH_TOKEN:
      jsonPath: "$.token"

- name: access protected endpoint
  http:
    url: "http://localhost:3000"
    method: GET
    path: /protected
    headers:
      Authorization: "Bearer ${AUTH_TOKEN}"
  source:
    type: local
  expect:
    statusCode: 200

Shell heredoc gotcha: if you feed YAML via a heredoc (<<EOF), the shell expands $AUTH_TOKEN in the heredoc body to its value in the parent shell (usually empty) before YAMLTest ever sees the text. Use one of these patterns to prevent that:

# 1. Escape the dollar sign
YAMLTest -f - <<EOF
    command: echo \$AUTH_TOKEN
EOF

# 2. Quote the heredoc delimiter (disables all expansion)
YAMLTest -f - <<'EOF'
    command: echo $AUTH_TOKEN
EOF

# 3. Use a file — no shell expansion at all (recommended)
YAMLTest -f tests.yaml

Debug logging

DEBUG_MODE=true YAMLTest -f tests.yaml

Prints full request/response details, comparison results, and kubectl commands.

Project structure

src/
  core.js       # Test execution engine (HTTP, command, wait, comparison)
  runner.js     # Multi-test orchestration (YAML parsing, validation, fail-fast, retry)
  validate.js   # JSON Schema validation (Ajv)
  index.js      # Public API
  cli.js        # YAMLTest binary entry point
test/
  unit/         # Pure function tests (compareValue, filterJson, parseCurl, ...)
  integration/  # Real HTTP server + real shell command tests
  e2e/          # CLI binary spawned end-to-end

Running the test suite

npm test                    # all tests
npm run test:unit           # unit tests only
npm run test:integration    # integration tests only
npm run test:e2e            # end-to-end CLI tests only
npm run test:coverage       # with coverage report

CI/CD

# .github/workflows/test.yml
name: Tests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - run: npm install
      - run: npm test

License

MIT