@lokalise/load-testing-utils

Building blocks for a local load-test stack: bring up a service with its datastores and fakes, run k6 against it, report what the run cost, and tear it all down again.

Each service keeps its own short runner for the steps only it knows (which migrations, which seed, which fakes, which DSNs). This package holds the rest, which is the same for every service and otherwise gets copied from one runner to the next.

• Install
• A runner in outline
• Processes
• Profiling on Windows
• Containers
• k6
• Environment
• Arguments
• Resource report
• Database probe

Install

pnpm add -D @lokalise/load-testing-utils

The database probe (@lokalise/load-testing-utils/db-probe) takes postgres.js connections, so a runner that uses it also needs postgres. Everything else has no dependencies.

A runner in outline

import { rmSync } from 'node:fs'
import { join } from 'node:path'
import {
  appendReportSection,
  bindAddressEnv,
  composeDown,
  composeUp,
  formatResourcesSection,
  k6TargetHost,
  measureResources,
  parseRunnerArgs,
  ProcessSupervisor,
  readRunTotalsFile,
  refuseIfPortTaken,
  refuseProfilingOnWindows,
  resolveK6Mode,
  runK6,
  scrapeMetrics,
  scrapeResources,
  waitForHealth,
} from '@lokalise/load-testing-utils'

const args = parseRunnerArgs(process.argv.slice(2), {
  commands: ['run', 'up', 'k6', 'down'],
  flags: { keep: false, docker: true, profiling: false },
})
const supervisor = new ProcessSupervisor({ logDir: join(HERE, '.logs') })
const compose = { file: join(HERE, 'docker-compose.perf.yml'), project: 'my-service-perf' }
const k6Mode = resolveK6Mode(args.k6Mode)

if (args.command === 'down') {
  supervisor.stopAll(supervisor.recordedProcesses())
  composeDown(supervisor, compose, { profiles: ['profiling'] })
  supervisor.clearState()
} else {
  if (args.flags.profiling) refuseProfilingOnWindows('service', { hint: 'pass --no-service' })
  await refuseIfPortTaken('service', 3000, { hint: 'run with --no-service to measure it' })
  composeUp(supervisor, compose)
  // migrations and seeding: service-specific
  supervisor.start('service', 'npx', ['tsx', 'src/server.ts'], {
    cwd: SERVICE_DIR,
    env: { ...perfEnv, ...bindAddressEnv(k6Mode, ['APP_BIND_ADDRESS']) },
  })
  await waitForHealth('service', 'http://localhost:3000/health')
  supervisor.writeState()

  const metricsUrl = 'http://localhost:9080/metrics'
  const summaryPath = join(K6_DIR, 'k6-summary.json')
  rmSync(summaryPath, { force: true })
  const { result: exitCode, delta } = await measureResources(
    () => scrapeResources({ metricsUrl, probeUrl: 'http://localhost:3323/db-stats?statements=all' }),
    () =>
      runK6(supervisor, {
        mode: k6Mode,
        cwd: K6_DIR,
        script: 'journeys.js',
        args: args.passthrough,
        env: { BASE_URL: `http://${k6TargetHost(k6Mode)}:3000` },
        docker: { hostDir: PERF_DIR },
      }),
    { sampleMetrics: () => scrapeMetrics(metricsUrl) },
  )
  appendReportSection(
    join(K6_DIR, 'k6-report.md'),
    formatResourcesSection(delta, readRunTotalsFile(summaryPath)),
  )
  process.exitCode = exitCode

  if (!args.flags.keep) {
    supervisor.stopAll()
    composeDown(supervisor, compose, { profiles: ['profiling'] })
    supervisor.clearState()
  }
}

Processes

ProcessSupervisor starts and stops the processes a stack is made of.

| Method | What it does | |---|---| | start(name, command, args, { cwd, env }) | Starts a long-running process. Every output line goes to the terminal prefixed [name], and all of it to <logDir>/<name>.log | | run(name, command, args, { cwd, env }) | Runs to completion with inherited stdio, and throws on a non-zero exit | | runToExit(command, args, { cwd, env }) | Runs with inherited stdio and resolves with the exit code. Asynchronous, so the started processes' output pipes keep draining meanwhile | | stopAll(processes?) | Stops the started processes and everything under them, last started first | | writeState(extra?) / readState() / clearState() | Records the running pids and their start times in stateFile (default <logDir>/stack-state.json) | | recordedProcesses() | The pids a --keep run recorded, for a down from another terminal to pass to stopAll. A pid whose process has a different start time now (after a crash or a reboot, say) is skipped and logged | | detach() | Stops relaying and lets this process exit while the started processes keep running. Needs detachable: true |

env is merged over process.env.

On Windows, npx, pnpm, npm and yarn are .cmd shims that Node refuses to spawn without a shell. The supervisor sends those through cmd.exe as a single line (override the list with shimCommands), so their arguments must not contain spaces. Stopping a process there kills its whole tree with taskkill /T /F, because a shim leaves the real process one level down.

Leaving a stack up

A runner that keeps its stack up after k6 (run --keep) has to return to the shell while the processes it started keep running. With piped output it cannot: the pipes hold its event loop open, and a child whose pipe loses its reader fails its next write. Create the supervisor with detachable: true and call detach() once the state is written:

const supervisor = new ProcessSupervisor({ logDir, detachable: true })
// ... start the stack, run k6
supervisor.writeState()
if (keep) supervisor.detach()

A detachable process writes straight to its log file, and the terminal relay reads the file as it grows, so lines reach the terminal up to 100 ms late and the ChildProcess it returns has no stdout or stderr. Its stdin is empty, so a tool that exits when stdin closes (esbuild --watch, say) exits at once. On Windows it is also started detached, because Windows would otherwise kill it with the runner. A Ctrl+C there no longer reaches it either, so a runner that may be interrupted before detach() should call stopAll() from its own SIGINT handler. recordedProcesses() finds it from another terminal as before.

Profiling on Windows

@pyroscope/nodejs cannot start on Windows, so a process the runner starts there with the profiler on runs unprofiled and leaves Pyroscope empty. refuseProfilingOnWindows(name, { platform?, hint? }) throws on win32 with a message naming the ways out: run the runner from WSL2, or start the process in WSL2 or its container. Call it before bringing anything up, and only when profiling is on and the runner starts the process itself; hint names the flag that skips starting it. See the @lokalise/pyroscope-profiling notes on a Windows dev box.

Containers

const compose = { file: 'docker-compose.perf.yml', project: 'my-service-perf', cwd: HERE }
composeUp(supervisor, compose, { profiles: ['profiling'] })   // up -d --wait
composeDown(supervisor, compose, { profiles: ['profiling'], removeVolumes: false })

Pass every profile the file defines to composeDown: a down without a profile leaves that profile's containers running.

k6

resolveK6Mode('auto') picks a local k6 when one answers on PATH and the grafana/k6 image otherwise. runK6 runs either:

• locally, in cwd, with env as the process environment
• in Docker, with docker.hostDir mounted at docker.containerDir (default /k6), the working directory and script translated to their container paths, env passed as -e flags, and host.docker.internal mapped to the host

A containerised k6 reaches the stack through host.docker.internal (k6TargetHost(mode)), which on Linux cannot see a socket bound to 127.0.0.1. bindAddressEnv(mode, ['APP_BIND_ADDRESS', ...]) returns 0.0.0.0 for those variables in Docker mode and 127.0.0.1 otherwise, leaving out any the shell already exported. The extra network hop makes a Docker run's latencies read a little worse, so compare Docker runs with Docker runs.

buildK6Command returns the command without running it.

Environment

| Function | What it does | |---|---| | parseEnvFile(contents) / readEnvFile(path) | KEY=value lines, # comments, optional double quotes. No interpolation, export or multi-line values | | retargetPorts(values, portVariables, env?) | Rewrites localhost:<port> in values and bare *_PORT values, for every default port whose variable ({ '5451': 'PERF_POSTGRES_PORT' }) is set in env | | omitExported(values, env?) | Drops the keys env already has, so an exported variable keeps winning over the file, as with node --env-file |

Anything service-specific, such as swapping a database name in a DSN, is a map over the result.

Arguments

parseRunnerArgs(argv, { commands, flags, defaultCommand?, help? }) reads <command> [flags] [k6 args]:

• flags are booleans with defaults, keyed in camelCase: purgeProfiles answers to --purge-profiles and --no-purge-profiles
• --k6=local|docker|auto sets k6Mode
• a bare -- (pnpm's separator) is dropped
• everything else goes to passthrough, in order, for k6 run

splitValueArgs(args, ['items']) takes --items=50 out of the passthrough as ['--items', '50'], for a seeder that reads that form. k6 exits on a flag it does not know, so those must not reach it.

Resource report

A k6 summary sees one end of a request. scrapeResources reads the other: the service's Prometheus endpoint (prom-client default metrics) and the database probe. measureResources(scrape, body, options?) scrapes before and after body and returns the difference, and formatResourcesSection(delta) renders it as markdown for appendReportSection(reportPath, section).

It reports CPU and GC seconds as differences, resident memory and heap at the end, and statements and rows per database engine. A counter that went backwards means the process restarted mid-run, and is left out rather than reported as a negative number. A source that did not answer becomes a warning line instead of a zero.

Statements. Each engine's table ranks what the run itself cost each statement: the difference between the two scrapes, per statement, most expensive first (top, default 10). That needs the probe read with ?statements=all at both ends. Without it an engine gets a warning instead of a table, since the cumulative ranking puts migrations and earlier runs on top.

Event loop lag. prom-client resets its event-loop histogram on every scrape, so each reading of nodejs_eventloop_lag_p99_seconds covers the time since the previous one. Pass sampleMetrics and measureResources scrapes the metrics every sampleIntervalMs (default 5000) while body runs. The report prints the worst and the median interval p99, the highest max, and the number of intervals. The closing scrape is the last interval. Without a sampler it is the only one, and the p99 covers the whole run, which averages a burst away. Anything else that scrapes the endpoint during the run splits an interval.

Given the run's totals, the section also reports CPU as a share of one core over the k6 run, and CPU milliseconds per request. A Node service on one event loop saturates near 100%, and CPU per request is the figure that compares across runs with different load. formatResourcesSection(delta, run) takes the totals as { seconds, requests }, or as { reason } to print a warning line saying why those rows are missing. readRunTotals(data) reads either from a k6 summary object, and readRunTotalsFile(path) from the JSON a handleSummary wrote. A script that makes no HTTP requests (gRPC or WebSocket only) still gets the share of a core. Delete the file before k6 starts, as the runner above does: a k6 that fails before its summary leaves the previous run's file behind. The k6 script writes the file:

export function handleSummary(data) {
  return { 'k6-summary.json': JSON.stringify(data) }
}

CPU seconds cover everything runK6 does, while the share of a core divides by the test run alone. The first Docker run also pulls the k6 image with the service idle, so rerun it for a clean figure. CPU per request divides by every HTTP request k6 made, so a script that also calls other hosts reads low.

Database probe

k6 has no database client without a custom binary, so the probe is a small HTTP server the runner reads before and after a run.

import postgres from 'postgres'
import {
  createDbProbeServer,
  PROBE_APPLICATION_NAME,
  readCockroachStats,
  readPostgresStats,
} from '@lokalise/load-testing-utils/db-probe'

const pg = postgres(process.env.POSTGRES_URL, { max: 2 })
const crdb = postgres(process.env.COCKROACH_URL, {
  max: 2,
  connection: { allow_unsafe_internals: 'true', application_name: PROBE_APPLICATION_NAME },
})

createDbProbeServer({
  engines: {
    Postgres: (top) => readPostgresStats(pg, top),
    CockroachDB: (top) => readCockroachStats(crdb, { top }),
  },
}).listen(3323, '127.0.0.1')

GET /db-stats?top=10 answers every engine's counters plus its ten most expensive statements, ranked by total database time rather than calls: a per-row write and the batched statement replacing it differ a hundredfold in calls and little in time. The figures are cumulative. GET /db-stats?statements=all adds every statement as allStatements (up to maxStatements, default 5000), which is what a report diffs. A list that stopped at maxStatements carries allStatementsTruncated, and a report then leaves out the statements missing from the opening list rather than count their whole history as the run's. An engine that throws is reported as unavailable, with the error as its reason, and does not cost the report the others. GET /health answers 200.

• Postgres reads pg_stat_statements, which has to be in shared_preload_libraries and created in the database. Without it the probe counts committed transactions from pg_stat_database and says so in reason. Written rows always come from pg_stat_database. The probe puts PROBE_QUERY_MARKER after the first keyword of each of its queries and leaves them out of the pg_stat_statements figures. After, because newer versions of the view store a statement from its first token, so a leading comment never reaches it. Entries an earlier probe recorded without the marker stop growing, since the probe's queries no longer match them, but stay in the cumulative figures until pg_stat_statements_reset(); a run's difference is unaffected. The fallback can't filter the probe: it counts one transaction per scrape from the probe itself. Statements another role ran come back as <insufficient privilege> unless the probe's role has pg_read_all_stats. They still count in the totals, and the statements table leaves them out with a reason saying how many.
• CockroachDB reads crdb_internal.statement_statistics, which needs no extension. It combines the node's in-memory statistics with the ones it has flushed to system.statement_statistics. node_statement_statistics holds only the in-memory half, which the node clears on every flush (sql.stats.flush.interval, 10 minutes by default), so a run spanning a flush would report a wrong difference. A statement shows up in the statistics about half a second after it runs, so a scrape straight after a run can miss the last of it. From v26.1 cockroach refuses crdb_internal reads unless the session sets allow_unsafe_internals, as above. Cockroach's own jobs and the probe's own session (matched by application_name) are left out. It keeps no written-rows counter here, so the report omits that row for it.