@sourceloop/load-testing

v0.1.0

Published

4 days ago

Load tests for the HTTP APIs of LoopBack facades and services: TypeScript scenarios run by Artillery, with p95 limits and a baseline

0High
0Medium
0Low

loopback-extension loopback load-testing performance artillery p95

@sourceloop/load-testing

Load tests for the HTTP API of a LoopBack facade or service. We write scenarios in TypeScript, next to our other tests. The package turns each scenario into an Artillery script, and runs it against an app that already runs. It fails the run when an endpoint is too slow, has too many errors, or got slower than its baseline. It also fails the run when the project has too many heavy APIs.

This package is for load. For a serial latency gate on one operation, with no concurrent users, use @sourceloop/benchmarking. The two work well together. The benchmark guards the code, and the load test guards the running system.

Contents:

What this is, and what it is not

This package load tests the HTTP API of one running app. The app can be a facade or a service. A scenario is a list of requests that a virtual user sends. The package sends many such users at the rate we choose, measures the p95 time of each endpoint, and judges it.

It has limits on purpose.

It does not start apps or databases. We start them before the run.
It is not a browser test. Only the HTTP engine of Artillery runs.
It runs on one machine. It does not spread the load over many machines.
It does not check where the URLs point. The scenarios create and delete data. We run them only against a local or a test environment, never against production.

How a run works

load-testing run <package-dir> does these steps, in this order.

Read the environment (LOAD_TESTS_BASE_URL, LOAD_TESTS_AUTH_URL and the login variables).
Read src/__tests__/load/config.ts from dist, and check its values.
Load every *.load.ts file from dist/__tests__/load. Check the names, the limits, the phases and the values of each scenario.
Choose the scenarios to run (.only and .skip). Stop with a ConfigError when one of them sends token() and no login is set.
Check that every endpoint of the scenarios that run is in src/openapi.json.
Log in, when LOAD_TESTS_USERNAME is set.
Read the baseline file, if it exists.
For each scenario, one after another:
1. Get a fresh access token, if the old one is more than a minute old.
2. Run before.
3. Run the token check, if the scenario sends token().
4. Write the Artillery script (and the worker hooks file), and try the worker hooks once.
5. Run the phases with Artillery, and read its report.
6. Run after, then the cleanups (the last one first), then close the SQL connections.
7. Judge the scenario against its limits and the baseline.
Judge the run. Every scenario must pass, and the count of heavy APIs must be at most maxHeavyApis.
Print the report with the reporters.
Save the baseline, if LOAD_TESTS_UPDATE_BASELINE=1. A failed run is not saved, unless LOAD_TESTS_FORCE_BASELINE=1.
Exit with 0 when the run passed, and with 1 when it did not.

Step 8.6 runs also when before or the load failed, and after a signal.

Words we use

| Word | Meaning | | --- | --- | | Scenario | One load.it. A list of requests that one virtual user sends. We name it after what a user does. | | Endpoint id | METHOD path, for example GET /orders/{id}. Results and baselines use it. | | Vuser | A virtual user. It runs the steps of a scenario once. | | Phase | One step of the load, for example {duration: 60, arrivalRate: 10}. A load is a list of phases. | | Workload | The load of a scenario as one line of text, for example 10 vusers/s for 60 s. | | p95 | The time that 95% of the requests beat, in milliseconds. The time runs to the first byte of the response. It does not include the download of the body. | | Limit | The highest value that passes. Each threshold has one. | | Baseline | The mean p95 of the last 10 recorded runs of an endpoint. | | Heavy API | An endpoint with a p95 limit above P95_TARGETS.supportingApi (1000 ms). | | Hook | A function that runs at a set point of a run, for example before. | | Value | An object that the engine fills in at run time, for example randomString(). | | Datasource | A named database connection that ctx.sql can use. | | Engine | The part that runs the load and measures it. The default is Artillery. | | Reporter | The part that gets the result of the run. The default prints to the console. |

Install

npm install --save-dev @sourceloop/load-testing

What we need to know before the install.

We need Node 22 from 22.18, or Node 24. Odd releases are not supported. The engines field of the package says >=22.18.0 <23 || 24. The floor comes from Artillery 2.0.34, which asks for Node 22.18 or later.
Artillery comes with the package as a dependency, pinned to 2.0.34. We install nothing else for it. An install of this package into an empty project adds 468 packages (npm 11, Artillery 2.0.34, measured on 2026-10-09). Artillery has the MPL-2.0 license. We use it unmodified.
Artillery lists @playwright/browser-chromium, and that package downloads a browser in its install script. We do not need the browser, because this package uses only the HTTP engine of Artillery. We skip the download with npm install --ignore-scripts, or we set PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1 before the install.
pg is an optional peer dependency. We install it only when a datasource has type: 'postgres'. See Datasources.
We compile with skipLibCheck: true. The type file of Artillery 2.0.34 (artillery/types.d.ts) has an error of its own. The LoopBack tsconfig of an ARC app already sets this option.

The Artillery engine switches the Artillery telemetry off (ARTILLERY_DISABLE_TELEMETRY), so Artillery sends no usage data. Artillery also gets only a short list of the variables of the CLI. See The environment of Artillery.

The package finds Artillery when the first scenario runs, not at import. If it is missing, the run stops with this message:

artillery is missing from @sourceloop/load-testing: reinstall the package

Quickstart

This is the shortest working setup, in the facade orders-facade. We assume that the facade has an npm run build script and a committed src/openapi.json. See Set up a package.

First, src/__tests__/load/config.ts. It holds the default load.

import type {LoadTestConfig} from '@sourceloop/load-testing';

const config: LoadTestConfig = {
  phases: [{name: 'steady load', duration: 30, arrivalRate: 10}],
};

export default config;

Second, one scenario file, src/__tests__/load/orders.controller.load.ts.

import {captureFrom, load, template, token} from '@sourceloop/load-testing';

load.describe(
  'OrderController',
  {headers: {authorization: template`Bearer ${token()}`}},
  () => {
    load.it('browse orders and customers', [
      load.get('/orders', {query: {limit: 20}}),
      load.get('/orders/{id}', {
        pathParams: {id: captureFrom('GET /orders', '$[0].id')},
      }),
      load.get('/customers', {query: {limit: 20}}),
    ]);
  },
);

Third, two scripts in the package.json of the facade.

{
  "scripts": {
    "pretest:load": "npm run build",
    "test:load": "load-testing run ."
  }
}

Fourth, the environment. The login user is a test user (see Log in). The password and the secret come from the shell or from CI secrets, never from a file in git.

export LOAD_TESTS_BASE_URL=http://localhost:3000
export [email protected]
export LOAD_TESTS_PASSWORD="$LOAD_TEST_PASSWORD"
export LOAD_TESTS_CLIENT_ID=load-test-client
export LOAD_TESTS_CLIENT_SECRET="$LOAD_TEST_CLIENT_SECRET"

Last, run it. The orders facade must already run at the base URL.

npm run test:load

The CLI prints Scenario: <name> before each scenario, and Artillery prints its own progress while the load runs. At the end the console reporter prints the result. This is the first run, so no baseline exists yet.

Results

OrderController › browse orders and customers
Workload: 10 vusers/s for 30 s
Vusers:   300 started, 300 completed, 0 failed (limit 1%)

Endpoint          Requests  Req/s  p95 ms (limit)  Base ms  Err % (limit)  Result
GET /orders       300       9.9    41.2 (500)      -        0 (1)          pass (no baseline)
GET /orders/{id}  300       9.9    36.8 (500)      -        0 (1)          pass (no baseline)
GET /customers    300       9.9    52.3 (500)      -        0 (1)          pass (no baseline)

Coverage: 3 of 41 endpoints in src/openapi.json
No src/__tests__/load/.out/baseline.json. Run once with LOAD_TESTS_UPDATE_BASELINE=1 to record it.

A vuser is one virtual user. It runs the steps of a scenario one time. Failed means that its run stopped on an error.

The run passes, and the exit code is 0. To record the baseline, run again with LOAD_TESTS_UPDATE_BASELINE=1. Later runs compare each p95 with it. See Baseline. The numbers above are an example, and ours will differ.

Set up a package in a facade and service monorepo

Our monorepos hold facades and services. A facade takes the request from the client, calls one or more services, and each service reads its own database.

Where the tests live

We put the load tests in the facade in most cases. A request to the facade passes through the whole path, from the facade to the services and to the databases. So one test covers all of it. A service can have its own load tests too, for an endpoint that clients do not reach through a facade. The set-up is the same for both, and each package has its own config.ts, scenarios and baseline.

Build first

The package does not compile our TypeScript. It loads the files that tsc made, from dist. For src/__tests__/load/<name>.load.ts it loads dist/__tests__/load/<name>.load.js. So we build before each run. The pretest:load script of the quickstart does it.

A missing built file stops the run, and the message names the file.
A built file that has no source file is never loaded.
The build step must keep dist/__tests__. Most services of this catalog run openapi-spec in build or postbuild. Their preopenapi-spec hook runs rm -rf dist/__tests__, and it runs on every call of openapi-spec. So after it, we compile again. The pretest script of such a service does it, and pretest:load can do the same, for example "pretest:load": "npm run clean && lb-tsc".

The OpenAPI file

The CLI checks every endpoint of the scenarios against src/openapi.json. The run stops when a request names an endpoint that the file does not have. The report also shows how many endpoints of the file the scenarios cover.

ARC apps have an openapi-spec script. It boots the built app from dist, and writes the spec to the path that we pass. We build the app first, then we run the script file with node. We generate the file, and we commit it.

npm run build
node ./dist/openapi-spec src/openapi.json

In an app that the ARC generator made, npm run openapi-spec -- src/openapi.json also works, because its preopenapi-spec hook builds the app first. Some scripts have several commands, as in some catalog services. Then npm gives the path to the last command only. So we call node directly.

We run this again after each change of a route, so the file stays current. The CLI does not fetch the spec from the running app. The apps serve it behind Basic auth when authenticateSwaggerUI is on in the core config.

Keep load files out of the unit tests

The unit test run of the package loads dist/__tests__, and that folder now holds the load files. A load file is not a unit test. We tell Mocha to ignore the folder in .mocharc.json.

{
  "exit": true,
  "recursive": true,
  "require": "source-map-support/register",
  "ignore": ["dist/__tests__/load/**"]
}

Ignore the generated files

The CLI writes its generated files to src/__tests__/load/.out/. These are the Artillery scripts, the worker hooks files, the reports and baseline.json. We add this line to .gitignore and to .prettierignore of the package. Prettier reads only the ignore files of the folder where it runs, so without the line npm run lint fails after a load run.

src/__tests__/load/.out

The baseline is the one file that we can keep between CI runs. See Baseline.

Folder layout and naming

The CLI finds every *.load.ts file under src/__tests__/load/, also in subfolders, in the order of the paths. We do not list them.

src/__tests__/load/
  config.ts
  orders.controller.load.ts      scenarios, request bodies, short hooks
  order-items.controller.load.ts
  fixtures/
    run.ts                       run id, name prefix
    api.ts                       small calls through ctx.api
    customer.ts                  customers
    order.ts                     orders and their statuses
    order-item.ts                items of an order
    index.ts                     exports all of the above

A load file holds the scenarios. Its hooks only call functions of fixtures/. A load file never imports another load file.
A fixture file makes and removes the rows of one entity. A function that makes data starts with given, for example givenCustomer. It registers the cleanup of what it made with ctx.defer, right after the create. A function that removes data starts with cleanup.
A fixture file can import the fixtures of an entity that it needs. There must be no import cycle. If an order needs a customer, order.ts imports customer.ts, and customer.ts never imports order.ts.
The CLI loads only *.load.ts files. A file in fixtures/ is an import.
We name a scenario after what a user does, for example "browse orders and customers". The report shows the name as the heading of the scenario. A name such as "returns 200" tells the reader nothing.

A fixture function takes the RunContext that before gets. We import the type with import type, because the package lints with @typescript-eslint/consistent-type-imports. This is fixtures/customer.ts. The fixture sends the token itself, because ctx.api does not. See The ctx of before and after.

import type {RunContext} from '@sourceloop/load-testing';

export async function givenCustomer(ctx: RunContext): Promise<string> {
  const api = ctx.api.with({
    headers: {authorization: `Bearer ${ctx.token}`},
  });
  const customer = await api.post<{id: string}>('/customers', {
    name: 'loadtest-customer',
  });
  ctx.defer(() => api.delete(`/customers/${customer.id}`));
  return customer.id;
}

The hook of a scenario calls it, for example runVars.customerId = await givenCustomer(ctx).

The public types

The package exports these types. We import them with import type, and the values (load, vars, token and so on) with a normal import.

| Use | Types | | --- | --- | | Hooks and ctx | RunContext, Vars, UserContext, BeforeEachContext, HookRequest, HookResponse, Hooks, Api, ApiOptions, ApiClientOptions, Sql, QueryResult | | Scenarios and requests | ScenarioOptions, GroupOptions, RequestOptions, LoadRequest, Method, ParallelGroup, FlowItem, StepInput, RawStep, Scenario, Phase, Thresholds, HeaderMap | | Values | Value, Var, Capture, Token, Template, Uuid, RandomNumber, RandomString | | config.ts | LoadTestConfig, TokenCheck, ArtilleryConfig, Datasource, PostgresDatasource, PostgresOptions, CustomDatasource, SqlConnector, Connection | | Engines and reporters | Engine, EngineRun, Measurement, Reporter, RunResult, ScenarioResult, EndpointResult, EndpointStats, VuserCounts, HeavyApis, HeavyEndpoint |

config.ts

src/__tests__/load/config.ts holds the settings of the package. It exports a LoadTestConfig as its default. The CLI loads its built file, like a scenario file. Only phases is required.

The type checks the keys when we build. When the CLI starts, it also checks the values, before it logs in. A problem stops the run with a ConfigError that lists each problem on its own line.

| Key | Type | Default | Use | | --- | --- | --- | --- | | phases | Phase[] | none, required | The default load. A list with at least one phase. See Set the load. | | thresholds | {default?: Thresholds} | p95 500, p95Regression 20, minDelta 10, errorRate 1 | Default limits for each endpoint. See Thresholds and the verdict. | | maxHeavyApis | whole number, 0 or more | 0 | How many heavy APIs the package can have. | | datasources | Record<string, Datasource> | none | The databases that ctx.sql can use. See Datasources. | | tokenCheck | TokenCheck | the default token check | Replaces the default check of the token life. See Log in. | | reporters | Reporter[] | [consoleReporter()] | Who gets the result. An empty list prints no result. See Report. | | engine | Engine | artillery() | Who runs the load. See Engines. | | artillery.config | ArtilleryConfig | none | Artillery config options for every scenario. See Use any Artillery feature. | | artillery.env | string[] | none | The exact names of more variables to pass to Artillery, only when the CLI has them. For example ['MY_REGION']. See The environment of Artillery. |

An example with every key. A real package sets only the keys that it needs.

import type {LoadTestConfig} from '@sourceloop/load-testing';
import {
  P95_TARGETS,
  artillery,
  consoleReporter,
} from '@sourceloop/load-testing';

const config: LoadTestConfig = {
  phases: [
    {name: 'warm up', duration: 10, arrivalRate: 1, rampTo: 10},
    {name: 'steady load', duration: 60, arrivalRate: 10},
  ],
  thresholds: {
    default: {p95: P95_TARGETS.userBlockingApi, errorRate: 1},
  },
  maxHeavyApis: 0,
  datasources: {
    orders: {
      type: 'postgres',
      url: process.env.ORDERS_DB_URL,
      database: 'orders',
    },
  },
  tokenCheck: ({claims}) => {
    if (claims.exp === undefined) {
      throw new Error('the token has no exp claim');
    }
  },
  reporters: [consoleReporter()],
  engine: artillery(),
  artillery: {config: {http: {timeout: 30}}, env: ['MY_REGION']},
};

export default config;

config.ts has no headers. We set headers on load.describe or on load.it.

The library reads no database variable. config.ts reads the environment, and we choose the name of the variable. When config.ts loads, it opens nothing.

The check of artillery.env gives these problems.

artillery.env must be an array of variable names, for example ["MY_REGION"]
artillery.env[<i>] must be a text that is not empty
artillery.env[<i>] must not have "="
artillery.env[<i>] "<name>" is not allowed: these variables are for the CLI only, for a name that starts with LOAD_TESTS_ in any letter case.

Write scenarios

load.describe and load.it

load.it(name, requests, options?) adds one scenario, like it of Mocha. load.describe(name, options?, fn) groups the scenarios of a file, like describe. A load.describe can hold other load.describe blocks. Its function runs at once, and it cannot be async. A file needs at least one load.it.

import {load, template, token} from '@sourceloop/load-testing';

load.describe(
  'OrderController',
  {headers: {authorization: template`Bearer ${token()}`}},
  () => {
    load.it('browse orders and customers', [
      load.get('/orders', {query: {limit: 20}}),
      load.get('/customers', {query: {limit: 20}}),
    ]);

    load.it('browse products', [load.get('/products', {p95: 800})]);
  },
);

The scenarios run one after another. Files run in file-name order, and a file runs in the order of its load.it calls. They share the app, its pools and the baseline, so they do not run in parallel. Each scenario has its own load, hooks and data.
The name of a scenario in the report and in baseline.json is group › name, for example OrderController › browse products. The CLI names the generated files after it, in lower case with - between the words. Two scenarios cannot make the same file name, and a name needs a letter or a digit.
The file name baseline is reserved for the baseline file. The CLI refuses a scenario whose file name would be baseline, for example baseline or Baseline!, when it loads the scenarios. The message reads The scenario name "<name>" gives the file name "baseline", which the baseline file uses. Rename the scenario. The name baseline orders is fine.
The endpoint id is METHOD /path, with the path as we wrote it, for example GET /orders/{id}.

Run only some scenarios

| Call | Effect | | --- | --- | | load.it.only(...), load.describe.only(...) | Only the scenarios with only run. | | load.it.skip(...), load.describe.skip(...) | The scenario does not run. The report lists it as Skipped. |

On a load.describe, .only and .skip apply to every scenario inside it.
skip wins over only. Only a scenario with .only and without .skip runs. A .only on a scenario that also has .skip runs nothing. When no other scenario has .only, nothing runs. The report lists every scenario as skipped. The scenarios that .only leaves out are listed as skipped too. This holds only on a local run. With CI set, such a .only is an error (see below).
A .only that stays in a file hides the other scenarios. When the variable CI is set to a value other than empty, 0 or false, a .only stops the run. The CLI ignores the letter case, so False and FALSE also mean that this is not CI. Any other value means CI. A .only stops the run in CI, also when the same scenario has .skip. The message names the scenarios, like mocha --forbid-only. We remove the .only before we push.
When CI is set and every scenario is skipped, the run stops with No scenario runs: every scenario has .skip. So a green gate cannot hide that nothing ran. A local run prints Skipped: lines and exits with 0.
When .only leaves scenarios out, the CLI prints a line such as Only 1 of 2 scenarios run (.only).

Requests

load.get, load.post, load.put, load.patch and load.delete make a request. Each takes the path and an options object. A path can hold {name} parts, which pathParams fills. A name is any text without braces or a slash, for example {id} or {order-id}.

| Option | Use | | --- | --- | | pathParams | Fills the {name} parts of the path. A string, a number or a value. | | query | The query string. A number or a boolean becomes its text. | | json | The JSON body. A value works at any depth, through plain objects and arrays. A Date or a class instance goes as it is. | | headers | Headers of this request. They win over the headers of the scenario. | | p95, errorRate, p95Regression, minDelta | Limits of this endpoint. See Thresholds and the verdict. | | beforeRequest, afterResponse | Worker hooks of this request. See Hooks. | | artillery | {raw}, the Artillery request fields. See Use any Artillery feature. |

A request object belongs to one place in a flow. To send the same request twice, call load.get twice.

Values

A value is an object that the engine fills in when a vuser sends a request. It works in the json, pathParams, headers and query of a request. The scenario holds no {{ }} engine syntax.

| Call | Gives | | --- | --- | | vars(name) | What before set in vars, or what beforeEach returned. | | captureFrom(from, jsonPath) | A part of the response of an earlier request of the scenario. | | token() | The access token of the login user. | | randomString(length = 8) | length letters and digits, new for each use. | | randomNumber(min, max) | A whole number from min to max, new for each use. | | uuid() | A UUID, the same in all requests of one vuser. | | template | Joins strings, numbers and values into one text. |

To put a value inside a string, we use the tag template.

import {captureFrom, load, randomString, template, uuid} from '@sourceloop/load-testing';

load.it('create an order for a new customer, then delete both', [
  load.post('/customers', {
    json: {name: template`loadtest-${randomString()}-customer`},
  }),
  load.post('/orders', {
    json: {
      customerId: captureFrom('POST /customers', '$.id'),
      externalId: uuid(),
    },
  }),
  load.delete('/orders/{id}', {
    pathParams: {id: captureFrom('POST /orders', '$.id')},
  }),
  load.delete('/customers/{id}', {
    pathParams: {id: captureFrom('POST /customers', '$.id')},
  }),
]);

load.it('search orders of a customer', [
  load.get('/customers', {query: {limit: 1}}),
  load.get('/orders', {
    query: {
      filter: template`{"where":{"customerId":"${captureFrom('GET /customers', '$[0].id')}"}}`,
    },
  }),
]);

template takes strings, numbers and values.
A plain template literal fails on purpose. ${randomString()} in a normal backtick string calls toString() of the value, and this throws a TypeError that says to use template. Without the throw, the text would be [object Object].
For the same reason, a value cannot go into JSON.stringify. toJSON() throws a TypeError. We build a JSON text that holds a value, such as a filter, with template (see the second scenario above).
A plain string that contains {{ in path, pathParams, headers, query or json stops the run with a ScenarioError. This keeps the syntax of one engine out of the scenario. The CLI does not check the strings under artillery. A value that a hook or a response gives at run time has its own check. See Text that the engine refuses.
The same check covers the keys of objects in json, headers, query, pathParams and the headers of a scenario. The message reads POST /orders: json key "{{ x }}" contains "{{": rename the key. For a nested key it reads json.deep[0] key "a{{" contains "{{": rename the key. A key cannot use template, so we rename it. A body with a key named constructor, or an object from Object.create(null), gets the same checks.
The check reads one string at a time. Two strings that touch can still form {{ after the engine joins them. So in a template, we never put a lone { next to a value.
captureFrom works in all four places, also inside a template. The source must be an earlier request of the same scenario, found by its endpoint id. It cannot be a request of the same load.parallel group. A capture that finds nothing stops the vuser.
vars('name') needs a name from the vars of before, or from what beforeEach returns. The CLI stops before the load when no hook sets it.
There is no list of custom values. For a value that these calls cannot make, we compute it in a hook (before, beforeEach or beforeRequest), and read it with vars(name).
The calls check their arguments when the scenario file loads. For example randomString(0) and randomNumber(5, 1) throw a TypeError. randomString(length) needs a whole number from 1.
randomNumber(min, max) needs whole numbers from 0 up, and min must not be above max. A negative or fractional value throws a TypeError with randomNumber: the minimum and the maximum must be whole numbers from 0 up. Artillery reads only letters, digits, _, , and spaces inside $randomNumber(...).
We do not write {{ $env.LOAD_TESTS_TOKEN }} by hand. The CLI gives the token to Artillery only for a scenario that uses token(). See Send the token.

Text that the engine refuses

Artillery fills a template with a text replace, and it expands a text again after the fill. So a value that holds {{ ... }} can read $env or loop for ever. A value that holds $&, $` or $' can loop or grow, because the replace reads them as patterns. A $$ becomes one $, so pa$$word in a longer text would reach the API as pa$word. We call these five texts the refused text: {{, $&, $`, $' and $$. A $1, a $0 and a $<name> pass.

The package checks the values that arrive at run time. The check reads strings, arrays, and the values and the keys of objects. It reads one value at a time. It never prints the value.

A text in the vars of before, at any depth, fails the scenario before its load with a RunError. The next scenario still runs. The check reads the vars as JSON, as the generated file holds them. So a result of toJSON counts, and a value undefined is skipped. A cycle in vars gives a TypeError from JSON.stringify. The <path> looks like vars.order.note or vars.ids[1]. after and the cleanups still run.
A beforeEach result with the refused text fails the vuser. The message starts with load-testing hook beforeEach: and then holds the message of vars below. The worker finds it, so it appears after the load starts. The worker reads the result as JSON too, so a toJSON result counts. A value that JSON cannot write is a hook error.
A captured value (captureFrom) with the refused text fails the vuser before it sends another request. The check runs first in the afterResponse list of the request, before a hook of ours.
An access token with the refused text fails the scenario with a RunError, before the CLI writes any file. The next scenario still runs. The message has no token in it.
A base URL with the refused text fails the scenario with a RunError, before the CLI writes any file. Artillery reads the target as {{ $env.LOAD_TESTS_BASE_URL }}, so such a text would change the target. The message does not show the URL.

The messages read like this. The first line is for vars, the second for a capture, the third for the token, and the fourth for the base URL.

<path> contains text that the engine reads as a template ("{{", "$&", "$`", "$'" or "$$"). Put the value in vars without it, or leave it out.
load-testing hook capture: "{{", "$&", "$`", "$'" or "$$" in the value of <endpoint id> at <JSON path>. The engine reads it as a template. The vuser stopped before it sent another request.
The access token contains text that the engine reads as a template ("{{", "$&", "$`", "$'" or "$$"). Log in again to get another token.
The base URL contains text that the engine reads as a template ("{{", "$&", "$`", "$'" or "$$"). Use a URL without it.

A refused text in vars is rare. A name or a note that comes from an API can hold it, so we keep such a value out of vars.

The checks belong to the Artillery engine. A custom engine gets none of them from the package. It must refuse such text itself, if its own engine reads templates. See Engines.

Where we see a failure of the worker.

The load-testing process exits with 1, because the scenario fails.
Artillery prints errors.<message> <count> in its own summary, with two spaces and no dots for a long message. A short message gets dots between the message and the count. Artillery cuts a line at 79 columns, so a long message starts with the cause.
The report of the package shows the line Hooks: <n> error(s), not app errors (FAIL).
The full message is the key errors.<message> under aggregate.counters in src/__tests__/load/.out/<slug>.report.json. The <slug> is the label of the scenario, group › name, in lower case. Each run of other characters becomes one -.

jq '.aggregate.counters | with_entries(select(.key | startswith("errors.")))' \
  src/__tests__/load/.out/<slug>.report.json

Parallel requests

load.parallel(requests, {limit?}) sends requests at the same time in one vuser. The flow goes on when all of them finish. In a hook, we use Promise.all instead.

import {captureFrom, load} from '@sourceloop/load-testing';

load.it('open the order dashboard', [
  load.get('/orders'),
  load.parallel(
    [load.get('/customers'), load.get('/products'), load.get('/order-statuses')],
    {limit: 2},
  ),
  load.get('/orders/{id}', {
    pathParams: {id: captureFrom('GET /orders', '$[0].id')},
  }),
]);

A group holds only requests. It cannot hold a load.step or another group.
limit is the most requests that run at the same time. It must be a whole number of 1 or more.
Each request of a group keeps its endpoint id, its limits and its row in the report.
A request after the group can capture from a request in the group. A request in the group cannot capture from another request of the same group. The run stops with a ScenarioError, because the response can be missing.
A group can sit in the loop of a load.step.

Group options and the merge order

load.describe(name, options?, define) and load.it(name, requests, options?) both take headers, phases and thresholds. load.it also takes the hooks, tokenCheck and artillery.

import {load, template, token} from '@sourceloop/load-testing';

load.describe(
  'OrderItemController',
  {
    headers: {authorization: template`Bearer ${token()}`},
    thresholds: {p95: 800},
  },
  () => {
    load.it('list order items', [load.get('/order-items', {p95: 400})], {
      headers: {'X-Trace': 'load'},
      phases: [{duration: 20, arrivalRate: 2}],
      thresholds: {errorRate: 2},
    });
  },
);

The layers, strongest first:

the request
load.it
load.describe (the inner one first, then the outer one)
config.ts
the library default

| Option | How the layers combine | | --- | --- | | headers | They merge by name. Names ignore case. The winner keeps its spelling. | | phases | The list of the strongest layer replaces the others. There is no merge. | | thresholds | They merge by field. |

In the example, the scenario gets the authorization and X-Trace headers. It gets its own phases, a p95 of 800 and an errorRate of 2. The request limit p95: 400 wins on GET /order-items.

Set the load

The load is phases, a list of phases that run one after another. A phase uses the field names of Artillery. We set it in one of two places.

phases in config.ts is the default of the package. Every scenario uses it.
phases in the options of a scenario (or of a load.describe) replaces the default. The lists do not merge, so we name all the phases that we want. We use it for a flow that the app cannot take at the default rate.

In config.ts:

import type {LoadTestConfig} from '@sourceloop/load-testing';

// One flat phase: 10 new vusers a second for 60 seconds.
const config: LoadTestConfig = {
  phases: [{duration: 60, arrivalRate: 10}],
};

export default config;

In orders.controller.load.ts:

import {load} from '@sourceloop/load-testing';

// A slow start, a ramp, a pause and a count of vusers.
load.it('browse orders slowly', [load.get('/orders')], {
  phases: [
    {duration: 30, arrivalRate: 1},
    {duration: 60, arrivalRate: 1, rampTo: 5},
    {pause: 10},
    {duration: 30, arrivalCount: 100},
  ],
});

| Phase | Meaning | | --- | --- | | {duration, arrivalRate} | arrivalRate new vusers a second | | {duration, arrivalRate, rampTo} | the rate goes from arrivalRate to rampTo | | {duration, arrivalCount} | arrivalCount vusers in all | | {pause} | no new vusers | | name on a phase | a label that Artillery shows in its output | | maxVusers on a phase | the most vusers that run at once |

duration and pause are seconds, or a text such as 1m. The CLI checks the values before the load. These rules hold.

A phase needs a duration, and one of two forms of load: arrivalCount, or arrivalRate with an optional rampTo. A pause needs a pause time.
duration, arrivalCount and pause must be whole numbers above 0. arrivalRate and rampTo can be 0.
A phase with rampTo also needs arrivalRate.
A text value for duration, pause or arrivalCount must read as more than 0. The CLI reads it as Artillery reads a time: "30" and "1m" pass, and "0", "0m", "-5" and "soon" give the messages above. A text value for arrivalRate or rampTo is not read by the CLI. It only checks that the text is not empty. Artillery reads the text, and a text that it cannot read fails later.

The messages read like this.

phases[i] needs arrivalRate or arrivalCount
phases[i] needs arrivalRate with rampTo
phases[i].duration must be a whole number above 0, or a text such as "1m", and the same for pause
phases[i].arrivalCount must be a whole number above 0
phases[i].arrivalRate must be 0 or more, and the same for rampTo

The report shows every phase in its Workload line, for example 10 vusers/s for 60 s. A ramp reads ramp 1 to 5 vusers/s over 60 s. A count reads 100 vusers over 30 s. A pause reads pause 10 s. The phases join with , then . A phase with maxVusers ends with (at most N vusers at once). An example is 10 vusers/s for 60 s (at most 50 vusers at once). The name of a phase is not part of the text.

The baseline file keeps this text. A run with another workload does not compare its p95 with the old one. A change of maxVusers counts as a changed workload. See Baseline.

Every vuser of a scenario runs its steps once. A scenario that needs a longer load than the access token lives stops before the load starts. See Log in.

Hooks

A scenario can have hooks. They create the data that it needs, and they remove it.

| Hook | Runs | Where | | --- | --- | --- | | before(ctx, vars) | once per run, before the load | the CLI | | after(ctx, vars) | once per run, always | the CLI | | beforeEach({vars, vu}) | when a vuser starts | a worker of the engine | | afterEach({vars}) | when the flow of a vuser ends without an error | a worker of the engine | | beforeRequest(req, {vars}) | before each send of that request | a worker of the engine | | afterResponse(req, res, {vars}) | after each response of that request | a worker of the engine |

before and after run in the CLI. They use full TypeScript. They can import and call anything. before puts values in vars. after gets the same vars, also when before stopped half way. To run calls at the same time in a hook, we use Promise.all.

The hooks of before call the app with ctx.api. That client sends no token, so a call to a protected endpoint makes a client with the token first, as the example does. This needs a login. See Log in.

import {load, template, token, vars} from '@sourceloop/load-testing';

load.it(
  'read the deliveries of an order',
  [
    load.get('/orders/{id}/deliveries', {
      pathParams: {id: vars('orderId')},
      headers: {authorization: template`Bearer ${token()}`},
    }),
  ],
  {
    async before(ctx, runVars) {
      const api = ctx.api.with({
        headers: {authorization: `Bearer ${ctx.token}`},
      });
      const customer = await api.post<{id: string}>('/customers', {
        name: 'loadtest-customer',
      });
      ctx.defer(() => api.delete(`/customers/${customer.id}`));
      const order = await api.post<{id: string}>('/orders', {
        customerId: customer.id,
      });
      ctx.defer(() => api.delete(`/orders/${order.id}`));
      runVars.orderId = order.id;
    },
  },
);

The ctx of before and after

| Member | Use | | --- | --- | | ctx.token | The access token of the login user. Undefined without a login. | | ctx.claims | The claims of that token. {} without a login, or for a token that is not a JWT. | | ctx.api | The HTTP client of the app. It sends no token by itself. | | ctx.sql(db, text, params?) | One SQL statement on a datasource of config.ts. | | ctx.defer(cleanup) | Registers a cleanup for something that the hook made. |

ctx.api adds the base URL to each path. It sends no Authorization header and no other default header. For a call that needs the token, we make a client that has it with ctx.api.with.

A path must start with /. Otherwise the call throws a RunError that reads <METHOD> <path>: the path must start with "/", and it sends nothing. A path such as @host/x would change the host and send the token there. A path such as //host/x starts with /, so it goes to the base host as a path.

import {load, template, token, vars} from '@sourceloop/load-testing';

load.it(
  'read the first order',
  [
    load.get('/orders/{id}', {
      pathParams: {id: vars('orderId')},
      headers: {authorization: template`Bearer ${token()}`},
    }),
  ],
  {
    async before(ctx, runVars) {
      if (ctx.token === undefined) {
        throw new Error('Set the LOAD_TESTS_ login variables.');
      }
      const api = ctx.api.with({
        headers: {authorization: `Bearer ${ctx.token}`},
      });
      const orders = await api.get<{id: string}[]>('/orders', {
        query: {limit: 5, archived: false},
        timeoutMs: 10_000,
      });
      runVars.orderId = orders[0]?.id;
      const invoice = await api.raw('GET', `/orders/${orders[0]?.id}/invoice`);
      runVars.invoiceSize = (await invoice.arrayBuffer()).byteLength;
    },
  },
);

| Call | Gives | | --- | --- | | get(path, options?), delete(path, options?) | The body, parsed as JSON. | | post, put, patch (path, body?, options?) | The body, parsed as JSON. | | raw(method, path, body?, options?) | The Response, for a file, a stream or a header. | | with({headers}) | A new client that adds these headers to each call. |

The options of a call are headers, query and timeoutMs.
A header with the value undefined removes a default header of the client.
A query value goes through String(), never through JSON. The client skips a value of undefined. It adds query to a query that the path already has.
timeoutMs is 60 s by default. The requests of the login have the same limit.
The client sends the body of a request in this order. undefined sends no body and no content type. A FormData, URLSearchParams, Blob, ArrayBuffer, typed array or string goes as it is, and the client sets no content type. Anything else goes as JSON with content-type: application/json, unless we set a content type.
A response with no body gives undefined. We ask for <void> in the type.
A 2xx body that is not JSON throws a RunError. We use raw() for it.
Any status that is not 2xx throws an HttpError, also from raw().
A network failure throws a RunError with the text <METHOD> <path> failed: <cause>. A call that takes longer than its timeout throws a RunError with <METHOD> <path> timed out after <N> s. This holds also when the body stalls while the client reads it.
ctx.api follows redirects. The login does not. See The flow.
The package exports ApiClient. new ApiClient(baseUrl, options) makes a client for our own code. options can hold token, headers, timeoutMs and signal. token adds the bearer header. The type of signal is () => AbortSignal | undefined. The client adds baseUrl to the path as it is, so new ApiClient('http://h/') makes the URL http://h//orders. The CLI removes a slash at the end of its base URL, and this class does not.

Cleanups with ctx.defer

ctx.defer(cleanup) registers a cleanup for the data that a hook made. We call it right after the create, as the example above does.

At the end of the run, the CLI calls the cleanups, the last one first. So a row goes before the rows that it depends on.
This happens after after. It also happens when before stopped half way, when the load failed, and when a signal came.
A cleanup can use ctx.api. It works after a signal too.
A cleanup that throws does not stop the others, and the run fails. When the scenario ran to the end, the report shows each failure as a FAIL: cleanup failed: line. When before or the load failed, the CLI prints <scenario>: cleanup failed: <message> on stderr instead.
A cleanup must accept that its row is gone already.
Vusers run in workers, so they cannot register a cleanup. We give the rows of the vusers a name tag, and remove them in after.

Why not only after? after runs once, and it does not know how far before got. Say that before creates a customer and an order, and the order create fails. Then after must check each id to see what exists. A new create needs one more check in after, and a missed check leaves rows behind. With ctx.defer, the function that creates the data also registers its cleanup, right after the create works. So the list holds only what exists. A cleanup can even go before the create. That is safe when the cleanup runs with nothing to remove, because it finds the rows by the name of the run. It also covers a create that committed, but whose reply we lost.

We still use after for rows that no hook created, such as the rows that the vusers made. It runs before the deferred cleanups, so these rows go before the rows that they point to. One rule needs care. We register first the cleanup that must run last.

Worker hooks

The workers do not load our scenario file. The engine copies the source of beforeEach, afterEach, beforeRequest and afterResponse into a file for the workers (.out/<slug>.processor.js). So these rules hold.

A worker hook can use only its arguments and JavaScript globals, such as URL, Buffer, crypto, process and setTimeout. It cannot use an import or a variable of its module.
A worker hook must be synchronous. A worker does not wait for a promise. An async hook would run its work after the vuser went on, and nobody would see a rejection. The CLI fails the scenario before its load when one of the four hooks returns a promise. The next scenario still runs.
beforeEach must give the same result for the same arguments, and it must have no side effects. It must return an object, the vars of the vuser.
Before the load, the CLI runs each hook once, with made-up arguments, from the generated file. A hook that uses a name from outside fails at once, with a ReferenceError in the message. For beforeRequest, afterResponse and afterEach, only a ReferenceError fails the scenario, because the made-up arguments can cause other errors. A side effect of a hook runs once there. This test runs in the CLI process, so it sees all environment variables. A worker sees only the list that the next rule names.
The file writes vars (what before sets) in plain text. We never put a secret or the token in vars. token() is safe. The CLI passes the token to Artillery in the environment and does not write it to disk. It does so only for a scenario that uses token().
A worker hook runs inside Artillery. So process.env there holds only the allowed names, plus LOAD_TESTS_BASE_URL, and LOAD_TESTS_TOKEN when the scenario sends token(). See The environment of Artillery.
A beforeEach result with the refused text fails the vuser. See Text that the engine refuses.
afterEach does not run for a vuser whose flow stopped with an error.
An error in a worker hook stops that vuser. The report counts these errors apart from app errors, in the Hooks: line, and the scenario fails.

import {load} from '@sourceloop/load-testing';

load.it(
  'tagged requests',
  [
    load.get('/orders', {
      beforeRequest: (req, {vars}) => {
        req.headers['x-run-id'] = String(vars.runId);
      },
      afterResponse: (_req, res) => {
        if (res.statusCode >= 500) console.error('server error');
      },
    }),
  ],
  {
    before: async (_ctx, vars) => {
      vars.runId = Date.now();
    },
    beforeEach: ({vu}) => ({tag: `vu-${vu}`}),
  },
);

vu counts the vusers that this worker started, from 0. Every worker counts from 0, so it is not unique in the run.

Ctrl-C and signals

On SIGINT (Ctrl-C), SIGTERM or SIGHUP the CLI does these steps.

Stop the ctx.api calls of before.
Stop Artillery, and wait until it has exited. It gets SIGKILL after 10 seconds. The wait makes sure that no late request of the load creates a row after the cleanup removes it.
Run after and the cleanups.
Run the reporters, with stopped: true. The console reporter prints no coverage or heavy API lines then.
Exit with 128 plus the signal number: 130 for SIGINT (Ctrl-C), 143 for SIGTERM and 129 for SIGHUP.

A second signal ends the process at once, with the exit code of the first. A second signal that comes less than 500 ms after the first is ignored, because npm can forward one Ctrl-C twice. A stopped run saves no baseline.

Ctrl-C also works during the login (step 6 of How a run works) and the token refresh (step 8.1). The CLI prints Cancelled. and exits with the code of step 5 above (130, 143 or 129) at once. It does not wait for a request that is still open.

The CLI ignores write errors on the terminal, but only EPIPE, EIO and ERR_STREAM_DESTROYED. After SIGHUP the terminal can be gone, and a failed write cannot stop the cleanups. Any other write error, such as a full disk, is thrown.

When the CLI process exits for any reason, it kills every running Artillery process with SIGKILL. Examples are a second signal, process.exit and an uncaught error. A SIGKILL of the CLI itself cannot be caught, so the Artillery processes can survive that case.

Test data

The scenarios make and delete data in the app under test. A good scenario leaves nothing behind, and it never depends on rows that someone made by hand.

Create and remove data

There are two ways to create data.

Through the API, with ctx.api in before. This goes through the same code as a real client, so it makes audit rows, events and cache entries.
With SQL, with ctx.sql in before. This goes around the app. It makes no audit rows or events, and it does not clear a cache that the app keeps. We use it for data that no endpoint can create, or to find rows by name in after.

We always register a cleanup with ctx.defer right after a create. See Hooks. A vuser can also create and delete its own data as steps of the flow. The first request creates a row, captureFrom takes its id, and the last request deletes it. Then no two vusers share a row.

This scenario needs data of its own for each run. The before hook makes the customer and the products, and ctx.defer removes them.

import {
  captureFrom,
  load,
  randomNumber,
  randomString,
  template,
  token,
  vars,
} from '@sourceloop/load-testing';

load.it(
  'create an order with 3 items, set its discount, then read its deliveries',
  [
    load.post('/orders', {
      json: {
        customerId: vars('customerId'),
        reference: template`loadtest-${randomString()}-order`,
      },
    }),
    load.post('/order-items/bulk', {
      json: [1, 2, 3].map(() => ({
        orderId: captureFrom('POST /orders', '$.id'),
        productId: vars('productId'),
        quantity: randomNumber(1, 5),
      })),
    }),
    load.patch('/orders/{id}', {
      pathParams: {id: captureFrom('POST /orders', '$.id')},
      json: {discount: 10},
    }),
    load.get('/orders/{id}/deliveries', {
      pathParams: {id: captureFrom('POST /orders', '$.id')},
    }),
    load.delete('/orders/{id}', {
      pathParams: {id: captureFrom('POST /orders', '$.id')},
    }),
  ],
  {
    headers: {authorization: template`Bearer ${token()}`},
    async before(ctx, runVars) {
      const api = ctx.api.with({
        headers: {authorization: `Bearer ${ctx.token}`},
      });
      const customer = await api.post<{id: string}>('/customers', {
        name: 'loadtest-customer',
      });
      ctx.defer(() => api.delete(`/customers/${customer.id}`));
      const product = await api.post<{id: string}>('/products', {
        name: 'loadtest-product',
      });
      ctx.defer(() => api.delete(`/products/${product.id}`));
      runVars.customerId = customer.id;
      runVars.productId = product.id;
    },
  },
);

Give each vuser a unique value

We put a random part in a string with template. The engine makes a new value for each vuser, in every worker. We share nothing.

import {
  load,
  randomNumber,
  randomString,
  template,
  uuid,
} from '@sourceloop/load-testing';

load.post('/products', {
  json: {
    name: template`loadtest-${randomString()}-product`,
    stock: randomNumber(1, 9),
    externalId: uuid(),
  },
});

With 1,000 vusers and the default length, the chance that two vusers get the same randomString is 2 in a billion. A part of a UUID is worse. Eight hex characters give 1 chance in 8,600. uuid() is unique, but it has 36 characters. A column with a size limit can need a shorter value. We use randomString there.
randomNumber can give the same number to two vusers. We do not use it where each vuser needs a unique value.
uuid() is the same in all requests of one vuser, so it also links them.
Artillery has no unique index of a vuser. It does not split a CSV payload between the workers. In local mode, each worker gets the whole payload and starts at the first row. So we do not use a CSV to give each vuser its own row.
vu in beforeEach is not unique in the run either.
The Artillery engine renders these values as {{ $randomString(8) }}, {{ $randomNumber(1, 9) }} and {{ $uuid }}. If Artillery changes them, we change the engine only.

Datasources

A datasource is a named database connection. We declare datasources in config.ts. The first argument of ctx.sql is one of the names. A facade can reach several services, and each service has its own database. So we declare one datasource for each database that a scenario touches, for example orders, inventory, audit, suppliers, billing and users.

A datasource has one of three forms.

{type: 'postgres', url, ...} is Postgres. The package has the adapter, and the project installs the driver.
{type: 'custom', connector} is any other database, with a connector that we write.
A bare connector is the same as custom, shorter.

In config.ts:

import type {LoadTestConfig, SqlConnector} from '@sourceloop/load-testing';

// The server URL, with no database path.
const server = process.env.ORDERS_DB_URL;

declare function openDatabase(url: string): Promise<{
  run(text: string, params?: unknown[]): Promise<unknown[]>;
  close(): Promise<void>;
  onClose(listener: (err: Error) => void): void;
}>;

// A connector for any other database.
function warehouse(url: string): SqlConnector {
  return {
    async connect(onError) {
      const db = await openDatabase(url);
      db.onClose(onError);
      return {
        query: async (text, params) => ({rows: await db.run(text, params)}),
        end: () => db.close(),
      };
    },
  };
}

const config: LoadTestConfig = {
  phases: [{duration: 60, arrivalRate: 10}],
  datasources: {
    orders: {type: 'postgres', url: server, database: 'orders'},
    audit: {type: 'postgres', url: server, database: 'audit'},
    billing: {type: 'custom', connector: warehouse('wh://billing')},
    inventory: warehouse('wh://inventory'), // a bare connector
  },
};

export default config;

In orders.controller.load.ts:

import {load} from '@sourceloop/load-testing';

load.it('read orders', [load.get('/orders')], {
  async before(ctx, runVars) {
    const rows = await ctx.sql<{id: string}>(
      'orders',
      'SELECT id FROM orders WHERE reference LIKE $1',
      ['loadtest-%'],
    );
    runVars.orderId = rows[0]?.id;
  },
});

The package reads no database variable. Our config.ts reads the environment, and we choose the name of the variable.
type: 'postgres' needs the pg package in our project, with npm install --save-dev pg. The package loads pg from our project folder, never from its own, at the first ctx.sql call of the datasource. When pg is missing, the call fails with a ConfigError.
The calls to one datasource share one connection and run one after another, also inside Promise.all. Calls to different datasources run at the same time.
A Postgres datasource takes url, or host, port, user and password. It also takes database, ssl, connectionTimeoutMs, statementTimeoutMs and applicationName.
We read the URL from an environment variable, never from a literal in config.ts. We keep the certificate checks of ssl on. We turn them off only for a local database.
With url, database replaces the path of the URL. So one server URL serves many datasources. With url, the options host, port, user and password are ignored, with no message.
connectionTimeoutMs is 10,000 ms by default, so a database that does not answer cannot hold the run for ever.
statementTimeoutMs is a whole number of milliseconds from 0. Another value in a datasource is a ConfigError with the message datasources.<name>.statementTimeoutMs must be a whole number of milliseconds from 0 (0 turns the limit off). connectionTimeoutMs follows the same rule.
- By default the client waits at most 60 s for a statement (query_timeout). The server gets no statement_timeout, so it keeps running a statement after the client gave up. The CLI then drops that connection. The next ctx.sql call opens a new one.
- A value above 0 sets both limits. It sends statement_timeout to the server as a startup parameter. A proxy such as PgBouncer can refuse that parameter.
- Behind such a proxy we leave the option unset (client limit only), or we set 0 (no limit at all).
With neither url nor host, the first call fails with a ConfigError that says what to set. A run whose scenarios use no SQL needs no URL.
A name that config.ts does not declare gives a ConfigError that lists the declared names.
The package opens one connection for each datasource, at the first call, and keeps it. It drops a connection that breaks, and the next call opens a new one. The connections close after the cleanups of each scenario.
params fill $1, $2 and so on in the text. A call with several statements and no params gives the rows of the last one.
A connector has one method, connect(onError). It opens one connection and returns {query, end}. It calls onError when an idle connection breaks, so that the package can open a new one. It opens nothing when we create the connector.

Log in

The login is optional. The CLI logs in only when LOAD_TESTS_USERNAME is set. Then it also needs the password, the client id and the client secret. LOAD_TESTS_BASE_URL is always required. The Environment table lists all six variables.

The flow

The login follows the two-step login of the ARC authentication service (@sourceloop/authentication-service).

POST /auth/login with username, password, client_id and client_secret. The answer holds a code.
POST /auth/token with the code and the clientId. The answer holds an accessToken and a refreshToken. Both must be non-empty texts. If not, the CLI throws a LoginError: <call> gave no accessToken or <call> gave no refreshToken.
Before each scenario, the CLI gets a new token with POST /auth/token-refresh. It sends the refreshToken, and the old access token as a bearer token. It skips the refresh when the access token is less than a minute old. The auth service revokes the access token that a refresh replaces. The signed payload has no random part (jti), so two refreshes in the same second give the same token string. The second refresh then revokes the token that it just gave. We refresh at most once a minute to avoid this. The age comes from the iat claim. A token that has no iat, such as an opaque token, counts from the time that the CLI got it.

All three calls go to LOAD_TESTS_AUTH_URL, or to the base URL when that variable is not set. A call that answers with a status other than 2xx throws a LoginError with the status and the first 200 characters of the body.

The login and the refresh do not follow redirects. A 307 or a 308 would send the password and the client secret again to the new host. A status from 300 to 399 stops with a LoginError. It names the status and the origin of the location header, when the header parses. It never shows the whole location. The message reads POST <url> gave <status>, a redirect to <origin>. The login does not follow redirects, so the password goes only to the URL that we set. Set LOAD_TESTS_AUTH_URL to the final URL. We set LOAD_TESTS_AUTH_URL to the final URL.

We use https for an auth URL that is not on this machine. The package does not enforce it, because it accepts http for any host.

A network failure names the URL and the cause. These are the messages.

POST <url> failed: <cause>, for example with the cause connect ECONNREFUSED 127.0.0.1:4010. The cause is never empty. When fetch gives several errors, the text joins their messages with ; .
POST <url> timed out after 60 s. This holds also when the body stalls.
POST <url> gave <status> with a body that is not JSON, for a 2xx answer whose body is not JSON.
POST <url> gave <status> with a body that is not a JSON object, for a 2xx answer whose body is JSON but not an object (null, a list, a number or a text).

Ctrl-C stops a login or a refresh at once. See Ctrl-C and signals.

Send the token

The package does not add an Authorization header. A request asks for the token with token(). We set the header once, on the load.describe or on the load.it.

import {load, template, token} from '@sourceloop/load-testing';

load.describe(
  'Orders',
  {headers: {authorization: template`Bearer ${token()}`}},
  () => {
    load.it('browse orders', [load.get('/orders')]);
  },
);

token() works in headers, query, pathParams and json.

Without a login, ctx.token is undefined and ctx.claims is {}. When a scenario that runs sends token(), the whole run stops with a ConfigError before any scenario or hook runs. The message names these scenarios.
With a login, the engine sends one token for the whole load of a scenario. after and the cleanups use the same token as ctx.token.
The CLI gives the token to Artillery only for a scenario that uses token(). Another scenario gets no token variable. ctx.token and ctx.claims in before and after do not change.
ctx.api does not send the token. See Hooks.

The token check

The token check runs for a scenario that sends token(), when there is a login. It runs after before and before the load.

The default check stops the scenario when the load time plus 120 seconds is at least the time that the token has left. The 120 seconds are for the last requests of Artillery, after and the cleanups. The check reads the exp claim of the token. A token without exp passes.

The check also stops the scenario with a RunError when it cannot read the time of a phase. We write a time as a number of seconds, or as a text such as 20m.

The token life comes from the login client of the auth service. A short life limits the length of a load.

tokenCheck replaces the default check. So a custom check must test the expiry itself, as the example does. We set it in config.ts for the package, or in the options of a load.it. The innermost wins. It gets {token, claims, phases}. It throws an error, or rejects when it is async, to stop the scenario before its load. The run waits for an async che

Published

Vulnerabilities

Links

Maintainers

Keywords

Readme

@sourceloop/load-testing

What this is, and what it is not

How a run works

Words we use

Install

Quickstart

Set up a package in a facade and service monorepo

Where the tests live

Build first

The OpenAPI file

Keep load files out of the unit tests

Ignore the generated files

Folder layout and naming

The public types

config.ts

Write scenarios

load.describe and load.it

Run only some scenarios

Requests

Values

Text that the engine refuses

Parallel requests

Group options and the merge order

Set the load

Hooks

The ctx of before and after

Cleanups with ctx.defer

Worker hooks

Ctrl-C and signals

Test data

Create and remove data

Give each vuser a unique value

Datasources

Log in

The flow

Send the token

The token check