210-mutate

Mutation-testing toolkit for CPSC 210, in two parts: an authoring API (withMutants / @Mutate) that solution code imports to declare mutants, and a runner — the 210-mutate CLI (also used by the grader) — that runs each mutant against the reference tests and reports which ones survive.

Declaring Mutants

Every source file that defines mutants must import the authoring API from @ubccpsc/210-mutate.

Each mutant has three parts:

• An id — a globally unique name (e.g., Mutant1) the runner uses to activate and track it.
• An implementation — the altered version of the function; this is the actual mutation.
• Metadata describing it:
  • description: a short explanation of the mutant's changed behaviour.
  • difficulty: one of easy, medium, or hard — a grader can map these to point values.
  • hint: a suggestion for a test case that would kill the mutant.

[!IMPORTANT] Mutant ids must be globally unique.

There are two ways to define mutants:

1. Using the withMutants wrapper function (required for top-level functions). withMutants(target, original, mutants) takes the target (a label for the function being mutated — conventionally its name), the original correct implementation, and the mutants keyed by id. Assign the result to the export students import and test:

   import { withMutants } from "@ubccpsc/210-mutate";
   
   // define the original impl
   function sumFn(a: number, b: number): number {
       return a + b;
   }
   
   // define a mutant for sumFn
   function sub(a: number, b: number): number {
       return a - b;
   }
   
   // the exported const must have the same name as the function students will test.
   export const sum = withMutants("sum", sumFn, {
       Mutant1: {
           implementation: sub,
           metadata: {
               description: "Replace addition with subtraction.",
               difficulty: "easy",
               hint: "Check arithmetic operator correctness."
           }
       },
       Mutant2: {
           // define the mutant impl inline
           implementation: (a: number, b: number) => 0,
           metadata: {
               description: "Return a constant zero instead of the computed sum.",
               difficulty: "easy",
               hint: "Add assertions for non-zero addition results.",
           }
       }
   })

2. Using a method decorator (only works inside classes). Unlike withMutants, @Mutate(id, implementation, metadata) infers the target from the Class.method it decorates and takes each mutant positionally; stack the decorator to attach several mutants to one method:

   import { Mutate } from "@ubccpsc/210-mutate";
   
   class Calculator {
       offset = 2;
   
       @Mutate("Mutant1", sub, {
           description: "Replace addition with subtraction.",
           difficulty: "easy",
           hint: "Check arithmetic operator correctness."
       })
       @Mutate("Mutant2", sumWithOffset, {
           description: "Adds a fixed offset to the sum.",
           difficulty: "hard",
           hint: "Assert an exact result, e.g. sum(2, 3) === 5.",
       })
       sum(a: number, b: number): number {
           return a + b;
       }
   
       // ...
   }
   
   // define a mutant for sum
   function sub(a: number, b: number) {
       return a - b;
   }
   
   // define another mutant for sum that uses a field of on the class
   function sumWithOffset(this: Calculator, a: number, b: number): number {
       return a + b + this.offset;
   }

Running the analysis (CLI)

This package provides a CLI for running mutation analysis locally for PrairieLearn activities. Ensure this package is installed as (dev)dependency of the activity. Run the provided 210-mutate command from the activity root against the tests/ dir.

Note: Because the activity's vitest config is rooted at the activity (not tests/), you must provide --test-glob pointing at the tests:

# full analysis: discover mutants, check the baseline, then run every mutant
pnpm 210-mutate run --test-glob "test/**/*.ts" tests/

# just list the mutants that are defined (discovery only — no --test-glob needed)
pnpm 210-mutate list tests/

run is the default verb, so 210-mutate tests/ is the same as 210-mutate run tests/; the path defaults to the current directory.

A typical run summary:

Baseline: valid
Mutants:  4/5 killed

Survived:
  Mutant3  [hard] Swaps operands (equivalent).
           hint: Add a test with a negative amount.

A survivor means the reference tests don't distinguish that mutant from the original — either a gap worth a new test, or a genuinely equivalent (unkillable) mutant. Use list while authoring to sanity-check that every mutant registered with the id you expect.

Options (for run)

| flag | meaning | | --- | --- | | --json | emit the full report as JSON (for scripts/CI) instead of the human summary | | --baseline-runs <n> | how many times to run the baseline to detect flakiness (default 2) | | --source-glob <glob> | where to find solution modules (default ./src/**/*.{ts,js,mts,cts,mjs,cjs}) | | --test-glob <glob> | test files to run, relative to the path, overriding the project's vitest config. Omit it to defer to that config (or vitest's defaults). | | --fail-on-survivors | exit non-zero if any mutant survives (a CI gate) |

Exit codes

• 0 — ran against a healthy baseline (survivors are reported but don't fail the run unless --fail-on-survivors)
• 1 — a mutant survived and --fail-on-survivors was set
• 2 — analysis couldn't run: the baseline was invalid, or no tests / no mutants were found

How mutants register

You never add mutants to a central list — defining a mutant is registering it. At a high level:

• The runner (the CLI or the grader) and the test suite run in separate processes: Vitest executes tests in worker processes, and your solution code runs there, not in the runner. So the two sides coordinate over a side-channel rather than shared memory.
• When a solution module loads, each withMutants() / @Mutate call records its mutant ids and metadata on that channel for the runner to read back. To make sure every mutant is found — even in a module the student's tests never import — the runner first does a discovery pass that loads all solution modules (that's what --source-glob selects).
• To evaluate a mutant, the runner runs the suite once with that mutant marked active. The wrapper installed by withMutants() / @Mutate checks which mutant is active and swaps in its implementation for that run; every other call runs the original code. Tests fail ⇒ the mutant is killed; tests pass ⇒ it survived.

The cross-process plumbing (how "active" and "discovered" actually cross the boundary) lives in src/transport.ts, documented in one place.

Development

Layout

• authoring.ts — client API (withMutants/Mutate), imported by solution code
• session.ts — the runner: runMutation + the MutationSession over Vitest
• report.ts — pure result types + interpreters (classifyMutant/assessBaseline)
• transport.ts — the documented cross-process bridge (start here for the plumbing)
• cli/ — the 210-mutate command

Build & test

npm run build          # tsc -> dist/
npm test               # unit tests (report.ts logic)
npm run test:parity    # regression harness (see below)

vitest is a peer dependency, installed as a devDependency here so the package builds and tests locally.

Regression parity

test/parity.mjs runs run --json against fixtures in test/fixtures/ and diffs a normalized report against a committed expected.json. It's a standalone script, not a vitest test — runMutation starts its own Vitest, and Vitest-inside-Vitest fights over process globals.

• npm run test:parity — check output still matches the goldens (CI gate).
• npm run test:parity:update — regenerate goldens after an intentional behavior change; review the diff before committing.

Fixtures cover every outcome plus a stateful canary that only passes if each run starts from fresh module state.

Diagnostics

Set MUTATE_DEBUG=1 to have each run print Vitest diagnostics to stderr:

MUTATE_DEBUG=1 210-mutate run tests/          # or MUTATE_DEBUG=1 npm run test:parity

[210-mutate:debug] run diagnostics
[210-mutate:debug]   cwd: /path/to/project
[210-mutate:debug]   include: config-default
[210-mutate:debug]   modules found: 1
[210-mutate:debug]   - /path/to/project/test/sum.test.ts [passed] tests=2

This can help debug issues with mutant and/or test discovery.

Publishing the package

npm version patch
git push origin main
git push origin --tags

A v* tag triggers .github/workflows/publish.yml. It uses npm install, not npm ci: the macOS-generated lock omits Linux-only optional native deps (TypeScript 7's platform binary, Vitest's WASM resolver) that npm ci rejects.