@mimik/api-coverage-helper

Measures how much of the API specification of a service is exercised by its tests.

The specification is looked up the way @mimik/local looks it up: first the <account>_<name>_<version>_swagger.json file of the repository api directory, then swagger.yml at tag <version> of the Bitbucket repository declared by the swaggerFile field of package.json. The version declared there is the one which must be found: a local file of another version is reported, never used.

Reaching Bitbucket takes credentials, read from the file @mimik/local reads, ../key.json next to the service directory unless keyFile says otherwise:

{ "bitbucket": { "username": "[email protected]", "password": "an-api-token" } }

Bitbucket Cloud no longer accepts app passwords, so password holds an API token, and username the email of the Atlassian account the token was created for. The account, name and version of the swaggerFile field name a route of Bitbucket and a local file, so they may only hold letters, digits, dots, dashes, underscores and plus signs, as in 1.0.0+build.5, and the only provider the lookup reaches out to is bitbucket: an API of any other provider has to be stored locally.

The tests are read, not run. Their sources are parsed and every call carrying an HTTP method and a path is matched against the operations of the specification. Two coverages are measured:

• the operation coverage: one operation, that is one method on one path, is one unit;
• the response coverage: within the operations the tests reach, one documented status is one unit, so GET /customers/{name} documenting 200, 401 and 404 holds three of them.

Each has its own threshold: the first tells whether enough of the API is called, the second whether the paths which are called are tested beyond their happy path.

Command line

npx api-coverage                              # report on the service in the current directory
npx api-coverage --threshold 80               # exit 2 when the operation coverage is below 80%
npx api-coverage --response-threshold 60      # exit 2 when the response coverage is below 60%
npx api-coverage --format markdown            # a table of every operation
npx api-coverage --format json                # the whole result, for another tool
npx api-coverage --offline                    # fail rather than reach out to Bitbucket
npx api-coverage --dir ../mst --test-dir test # report on another service

Every path given on the command line but --dir is relative to --dir, which defaults to the current directory: --dir ../mst --api-file api/spec.json reads ../mst/api/spec.json.

Exit codes: 0 when the report was produced, 2 when a threshold was given and not met, 1 when an argument is invalid or the coverage could not be computed at all.

Add it to a service the way the other checks are added:

"scripts": {
  "api-coverage": "api-coverage --threshold 80 --response-threshold 60",
  "pre-push": "npm run docs && npm run lint && npm run test-ci && npm run api-coverage"
}

What is counted

Calls are recognised in four shapes, which cover how the fleet writes its tests:

| Shape | Example | |---|---| | a verb named member | supertest(app).get('/customers') | | a method and a path argument | baseAuth(POST, '/customers', token) | | a helper naming the method | systemPost('/itResources', body, token) | | an options object | request({ method: 'GET', url: '/customers' }) |

String constants are resolved the way the code scopes them. The innermost declaration wins, so two tests each declaring const url are read with their own value; a name imported from another test file is read from that file, so baseAuth(POST, path) with const POST = 'post' in a helper module is understood; and a name declared nowhere in scope is read from the top level of the other test files, when they all agree on its value. A function parameter, a name declared with let or var and assigned again, or any value the parser cannot see, becomes {} and matches any single specification parameter, and the base path of the specification is stripped when a test prefixes it. A specification segment mixing literal text and a parameter, as /users/{id}:activate or /v{version}/info, is matched on its literal text, so a call to /users/abc:activate covers it rather than /users/{id}. An options object is read only when given to a function called by its name, as request(…), or to a .request(…) or .fetch(…) member: one given to .send(…), .query(…) or any other member is the body or the query of a request, never the options of another one.

A call whose every segment was interpolated covers nothing: it would otherwise cover any operation of the same shape. Such calls are reported as unresolved, a request whose whole path is a value the scanner cannot see included, as supertest(app).post(path) in a helper given its path; a verb call with such a path on any other value, as cache.get(key), is not read as a request. Calls hitting a path the specification does not document are reported as unmatched. Deprecated operations are left out of the total unless includeDeprecated is set; the calls hitting them are reported as deprecated, rather than read as calls to another operation of the same shape. A test file the parser cannot read, as one holding JSX, or an entry it cannot reach, as a dangling symbolic link or a directory it may not list, is reported as unparsed: none of its calls are counted. So is a file the scanner never parses, as one written in TypeScript, listed with its extension as the reason, rather than left out of the count as if it held no call; a declaration file, as types.d.ts, holds no test, and is not listed at all. A file which is not a module is read as a script before being given up on, and a directory is walked once, however many symbolic links lead to it.

Calls chained on nock, such as nock(url).get('/orders'), or on a variable holding its result, mock a request the service makes and are never counted. Nor are the routes a mock server declares: a verb call made on an express() app or a Router(), or given a handler after its path, as app.get('/json', (req, res) => {}). Nor are the method and the path a spy is checked or stubbed with, as expect(stub).to.have.been.calledWith('GET', '/orders'), toHaveBeenCalledWith or stub.withArgs('GET', '/orders'), or written to the console. The scanner does not know the type of any other value, so any other call with a verb name and a path, such as map.delete('/orders'), is counted as an API call.

Which responses are covered

The status a call is expected to answer with is a number or a @mimik/response-helper code, read from the first of these places which names one:

| Where | Example | |---|---| | the .expect chained on the call | baseAuth(GET, '/info').expect(ERROR_CODE.NOT_FOUND) | | a check in the .end or .then callback | .end((err, res) => { if (res.status === 400) done(); }) | | a check following the awaited call | const res = await request; expect(res.status).to.equal(200); |

Other calls may come in between, as in .set('Authorization', token).expect(RESPONSE_CODE.OK), and an expectation which is not a status, such as .expect('Content-Type', /json/), is skipped. A check is an assertion on res.status or res.statusCode: expect(res.status).to.equal(…), expect(res).to.have.status(…), their should forms, or assert.equal(res.status, …); or, without one, a comparison naming the status the test passes on. Which one that is, the branch the comparison guards tells: if (res.status === 400) done() ends the test calling the done it is given, with nothing, null or undefined, if (res.status !== 201) return does not, so both expect their status; a branch guarding a failure or a retry, as if (res.status === 503) throw …, return retry() or return done(err), names a status the test does not expect, and is skipped. A negated assertion is skipped, and so is res.body.statusCode, which is the body, not the status. The response may be awaited into const res or res; the checks are then read until res is assigned again, and a block or a callback binding res anew checks another response.

A status held by a variable is followed to the value it is declared with, as const PAYLOAD_TOO_LARGE = 413. One a helper making the call is given by its callers, as a parameter, (token, statusCode = RESPONSE_CODE.OK) => …, or as a property of one, let statusCode = RESPONSE_CODE.OK; ({ statusCode } = options);, is read at every place the helper is called, through the imports of the test files, curried helpers such as const testToken = localTestToken(app, BASE_PATH), and helpers passing it on. So is the status the callers of a helper returning the request check on it, as in slotOf(id).expect(ERROR_CODE.NOT_FOUND), and a request a helper returns whose whole path is unknown, as one it is given, even when the helper expects a status itself: with const getServers = path => supertest(app).get(path).expect(200), the request is getServers('/servers'), expecting the status checked there, or the 200 without one. That 200 is kept through one helper only: when getServers('/servers') is itself what another helper returns, as in const listServers = () => getServers('/servers'), only the status the callers of listServers check is read. The request is then reported once per place, with the status given there, and with the place the request is made as via; a place already counted as a call of its own, as adminGet('/servers'), is not counted twice.

A status which cannot be followed, being reassigned, computed, or given by a value the scanner cannot see, is never guessed: the call still covers its operation, but no response, and is reported as withoutStatus. A 4XX response is covered by any 4xx status, a default response is not a unit, and a status the operation documents no response for is reported as undocumented.

The 401 and 403 responses are assumed covered, whether a test expects them or not: they are answered by the security library, whose own tests exercise them. They still count in the total, and the report tells how many were covered that way, as assumed.

• @mimik/api-coverage-helper
  • ~FORMAT : object
  • ~formatReport(result, [format]) ⇒ string
  • ~listOperations(spec, [options]) ⇒ Array.<object>
  • ~scanTests(directory, [options]) ⇒ object
  • ~computeCoverage(operations, calls, [options]) ⇒ object
  • async
    • ~apiCoverage([options]) ⇒ Promise
    • ~locateApi([options]) ⇒ Promise

@mimik/api-coverage-helper~FORMAT : object

Output formats of a coverage report, to pass to formatReport.

Kind: inner constant of @mimik/api-coverage-helper
Properties

| Name | Type | Description | | --- | --- | --- | | JSON | string | json: the whole result, for another tool. | | MARKDOWN | string | markdown: a table of every operation. | | TEXT | string | text: a summary naming what is not covered. |

@mimik/api-coverage-helper~formatReport(result, [format]) ⇒ string

Format the result of apiCoverage.

Kind: inner method of @mimik/api-coverage-helper
Returns: string - The formatted report.
Throws:

• Error A rich error when the format is not one of the FORMAT values.

| Param | Type | Description | | --- | --- | --- | | result | object | The result of apiCoverage. | | [format] | string | One of the FORMAT values. Defaults to text. |

@mimik/api-coverage-helper~listOperations(spec, [options]) ⇒ Array.<object>

List the operations an API specification defines.

Kind: inner method of @mimik/api-coverage-helper
Returns: Array.<object> - The operations, each as { key, method, path, segments, operationId, tags, summary, deprecated, statuses, defaultResponse }, where statuses lists the status codes the operation documents a response for.
Throws:

• Error A rich error when the specification declares no path or no operation, declares the same operation twice, as /things and /things/, or refers to a path item it does not hold: a path item $ref is followed only inside the definition.

| Param | Type | Description | | --- | --- | --- | | spec | object | The API specification, either Swagger 2.0 or OpenAPI 3. | | [options] | object | Options. | | [options.includeDeprecated] | boolean | When true, deprecated operations are listed too. Defaults to false. |

@mimik/api-coverage-helper~scanTests(directory, [options]) ⇒ object

List the API calls the tests of a service make, without running them.

Kind: inner method of @mimik/api-coverage-helper
Returns: object - { calls, files, failures }, where each call is { method, path, status, filename, line }, status being the status the test expects or null, files lists every test file found, the TypeScript and JSX ones included, and failures the files and the entries which could not be read or parsed, each as { filename, reason }. A file the scanner never parses, as a TypeScript one, is listed in both. A call made by a helper given its status by its callers is listed once per place the helper is called, and carries the place the request itself is made as via, { filename, line }.
Throws:

• Error A rich error when the directory itself cannot be listed.

| Param | Type | Description | | --- | --- | --- | | directory | string | Absolute name of the directory holding the tests. | | [options] | object | Options. | | [options.relativeTo] | string | Directory the reported file names are relative to. Defaults to directory. |

@mimik/api-coverage-helper~computeCoverage(operations, calls, [options]) ⇒ object

Compute the coverage of a specification by a set of calls.

Kind: inner method of @mimik/api-coverage-helper
Returns: object - { percentage, total, covered, operations, responses, unmatched, unresolved, ambiguous, deprecated }, where responses is the response coverage of the covered operations, as { percentage, total, covered, assumed, withoutStatus, undocumented }, and each operation lists its documented responses as { status, covered, assumed }.

| Param | Type | Description | | --- | --- | --- | | operations | Array.<object> | The operations of the specification, from listOperations. | | calls | Array.<object> | The calls found in the tests, from scanTests. | | [options] | object | Options. | | [options.basePath] | string | Base path the specification serves its paths under. | | [options.includeDeprecated] | boolean | When true, the deprecated operations count in the total. Defaults to false: they are still matched, and the calls hitting them are listed as deprecated. |

@mimik/api-coverage-helper~apiCoverage([options]) ⇒ Promise

Measure the API coverage of a service.

Kind: inner method of @mimik/api-coverage-helper
Category: async
Fulfil: object The result, as { api, tests, coverage, threshold, responseThreshold }, where each threshold is { value, passed }, or null when it was not given. The response coverage is coverage.responses.
Reject: Error A rich error containing the reason of the failure, or an option which cannot be used.

| Param | Type | Description | | --- | --- | --- | | [options] | object | Options. | | [options.directory] | string | Root directory of the service. Defaults to the current directory. | | [options.apiDirectory] | string | Directory holding the locally stored API files, relative to directory. Defaults to ./api. | | [options.testDirectory] | string | Directory holding the tests, relative to directory. Defaults to ./test. | | [options.packageFile] | string | Name of the package file, relative to directory. Defaults to ./package.json. | | [options.keyFile] | string | Name of the file holding the Bitbucket credentials, relative to directory. Defaults to ../key.json. | | [options.apiFile] | string | An explicit API file to use, relative to directory, bypassing both lookups. | | [options.swaggerFile] | object | Overrides the swaggerFile declaration of the package file. | | [options.offline] | boolean | When true, never reach out to Bitbucket. Defaults to false. | | [options.bitbucketApi] | string | Base URL of the repository API, for a mock or a proxy serving the routes of Bitbucket Cloud. Defaults to Bitbucket Cloud. | | [options.bitbucketTimeout] | number | Integer milliseconds, at least 1, the request to Bitbucket, retries included, may take. Defaults to 30 seconds. A network error, a connection dropped while the definition is read, a 429 or a 5xx is tried again twice within that time. | | [options.includeDeprecated] | boolean | When true, deprecated operations are counted too. Defaults to false. | | [options.threshold] | number | Percentage, between 0 and 100, the operation coverage must reach for threshold.passed to be true. | | [options.responseThreshold] | number | Percentage, between 0 and 100, the response coverage must reach for responseThreshold.passed to be true. |

Example

import { apiCoverage, formatReport } from '@mimik/api-coverage-helper';

const result = await apiCoverage({ responseThreshold: 60, threshold: 80 });

console.log(formatReport(result));
console.log(result.coverage.percentage, result.coverage.responses.percentage);

@mimik/api-coverage-helper~locateApi([options]) ⇒ Promise

Locate the API specification a service declares, without measuring anything.

The lookup is the one apiCoverage makes: options.apiFile when given, then the locally stored file of the declared version, then swagger.yml at that version's tag in Bitbucket. A local file of another version never satisfies it.

Kind: inner method of @mimik/api-coverage-helper
Category: async
Fulfil: object { spec, source, location, version, versionsFound }, where source is local, bitbucket or supplied.
Reject: Error A rich error containing the reason of the failure.

| Param | Type | Description | | --- | --- | --- | | [options] | object | The lookup options of apiCoverage: directory, apiDirectory, packageFile, keyFile, apiFile, swaggerFile, offline, bitbucketApi and bitbucketTimeout. |