@autotelic/openapi-test-coverage

OpenAPI test coverage analysis based on the A-TEST '19 paper "Test Coverage Criteria for RESTful Web APIs" (Martin-Lopez, Segura, Ruiz-Cortes).

Records HTTP traffic during integration tests, computes coverage against an OpenAPI 3.x spec, and reports results across 10 criteria organised into Test Coverage Levels (TCL 0--7).

Coverage Criteria

| TCL | Criteria added | |-----|----------------| | 0 | (none) | | 1 | Path coverage | | 2 | + Operation coverage | | 3 | + Input and output content-type coverage | | 4 | + Parameter coverage, status code class coverage | | 5 | + Parameter value coverage, status code coverage | | 6 | + Response body properties coverage | | 7 | + Operation flow coverage (not yet implemented) |

Install

npm install @autotelic/openapi-test-coverage

Or from git:

npm install https://github.com/autotelic/openapi-test-coverage.git

Quick Start

1. Record HTTP calls

Use the Fastify adapter to automatically record all requests and responses:

const { registerCoverageHooks } = require('@autotelic/openapi-test-coverage/adapters/fastify')

// After creating your Fastify app and registering routes:
const recorder = registerCoverageHooks(app, { autoAssertOnInject: true })

await app.ready()

Or record manually with the core Recorder:

const { Recorder } = require('@autotelic/openapi-test-coverage')

const recorder = new Recorder()

recorder.record({
  method: 'GET',
  pathname: '/v1/items',
  queryParams: { limit: '10' },
  responseStatusCode: 200,
  responseContentType: 'application/json',
  responseBody: [{ id: '1', name: 'Item' }],
})

2. Compute coverage

const { computeCoverage } = require('@autotelic/openapi-test-coverage')

const coverage = computeCoverage(recorder.getCalls(), openapiSpec, {
  excludedPaths: ['/internal/health'],
  excludedStatusCodes: ['/legacy\tGET\t404'],
})

console.log(`TCL: ${coverage.tcl}`)
console.log(`Paths: ${coverage.path.covered}/${coverage.path.total}`)
console.log(`Operations: ${coverage.operation.covered}/${coverage.operation.total}`)

3. Report

const { printReport, writeJsonReport, writeHtmlReport } = require('@autotelic/openapi-test-coverage')

printReport(coverage, { verbose: true })

writeJsonReport(coverage, 'test/output/openapi-coverage.json')
writeHtmlReport(coverage, 'test/output/openapi-coverage.html')

Adapters

Fastify

registerCoverageHooks(app, options?) registers onSend and onResponse hooks on a Fastify instance. Returns a Recorder.

const { registerCoverageHooks } = require('@autotelic/openapi-test-coverage/adapters/fastify')

const recorder = registerCoverageHooks(app, {
  // Pass an existing recorder (optional, creates one if omitted)
  recorder: myRecorder,
  // Wrap app.inject() to auto-mark responses as asserted (default: true)
  autoAssertOnInject: true,
})

Mocha

createCoverageAfterAll(options) returns a function suitable for use in Mocha's afterAll hook. It computes coverage, prints reports, and optionally enforces a minimum TCL.

const { createCoverageAfterAll } = require('@autotelic/openapi-test-coverage/adapters/mocha')

module.exports.mochaHooks = {
  async beforeAll() {
    // ... set up app, spec, recorder ...
  },
  async afterAll() {
    const reportCoverage = createCoverageAfterAll({
      getRecorder: () => app.coverageRecorder,
      getSpec: () => app.openapiSpec,
      config: {
        excludedPaths: [],
        excludedStatusCodes: [],
        excludedResponseBodyProperties: [],
      },
      console: { verbose: true },
      jsonReportPath: process.env.OPENAPI_COVERAGE_JSON,
      htmlReportPath: process.env.OPENAPI_COVERAGE_HTML,
      minTcl: 6,
    })
    reportCoverage()

    await app.close()
  },
}

API

Core

| Export | Description | |--------|-------------| | Recorder | Class that stores recorded HTTP calls. Methods: record(opts), markLastResponseAsserted(), getCalls(), clear(). | | computeCoverage(calls, spec, config?) | Compute all coverage metrics and TCL from recorded calls against an OpenAPI spec. Returns a CoverageResult. | | computeTcl(inputs) | Compute the TCL (0--6) from individual metric totals/covered counts. | | walkSpec(spec, excludedPaths?) | Extract all coverage targets (paths, operations, parameters, etc.) from a resolved OpenAPI spec. | | matchRequestToSpec(call, specPaths, basePath) | Match a recorded request to a spec path template. | | getBasePath(spec) | Derive the base path from the spec's servers[0].url. | | collectSchemaPaths(spec, schema) | Collect all property paths from an OpenAPI schema (for response body coverage). | | collectPathsFromValue(value) | Collect property paths from an actual JSON response body. |

Reporters

| Export | Description | |--------|-------------| | printReport(coverage, opts?) | Print a coloured console report. Options: { verbose: boolean, maxMissingItems: number }. | | writeJsonReport(coverage, filepath) | Write coverage as JSON. Creates parent directories if needed. | | writeHtmlReport(coverage, filepath) | Write a standalone HTML report. |

Configuration

CoverageConfig supports three exclusion lists:

• excludedPaths -- Path keys to omit entirely (e.g. ['/admin/reports']).
• excludedStatusCodes -- Tab-separated path\tmethod\tstatusCode keys to exclude from status code and response body coverage.
• excludedResponseBodyProperties -- Tab-separated path\tmethod\tstatusCode\tcontentType\tpropertyPath keys to exclude from response body property coverage.

Environment Variables

These are conventions used by the Mocha adapter; the core library is env-agnostic.

| Variable | Description | |----------|-------------| | OPENAPI_COVERAGE_JSON | Path to write a JSON coverage report. | | OPENAPI_COVERAGE_HTML | Path to write an HTML coverage report. | | OPENAPI_COVERAGE_MIN_TCL | Minimum TCL required; test run fails if below this. |

Coverage Result Shape

interface CoverageResult {
  path:                   { total: number; covered: number; missing: string[] }
  operation:              { total: number; covered: number; missing: string[] }
  statusCode:             { total: number; covered: number; missing: string[] }
  parameter:              { total: number; covered: number; missing: string[] }
  parameterValue:         { total: number; covered: number; missing: string[] }
  inputContentType:       { total: number; covered: number; missing: string[] }
  outputContentType:      { total: number; covered: number; missing: string[] }
  statusCodeClass:        { total: number; covered: number; missing: string[] }
  responseBodyProperties: { total: number; covered: number; missing: string[] }
  responseAsserted:       { total: number; covered: number; missing: string[] }
  tcl: number
}

Development

pnpm install
pnpm build        # Build CJS + ESM + .d.ts
pnpm test         # Run vitest
pnpm typecheck    # tsc --noEmit

License

MIT