Litmus — Hardware Test Framework

The act of running a test is the act of producing irrefutable evidence

Most hardware test frameworks treat reaching the end of a test method without throwing an error as a passing result. Litmus takes a different approach: a test passes only when it explicitly asserts the passing condition. The absence of failure is not accepted as positive evidence that the hardware is good.

In many frameworks, failure modes that were never modeled during development remain hidden until they surface in production. Litmus makes these gaps visible at design time. When a test method throws anything other than a TestFailure (or subclass), it is reported as a bug in the test itself. The floor technician sees exactly what went wrong, knows when to escalate to a software engineer, and clear accountability is established on both sides of the operational boundary.

Quick Start

npm run build
npm run dev       # build + run src/index.mts
npm run inspect   # open the artifact inspector on the latest run
npm run suite     # interactive suite picker → run mode

Minimal example — a hardware device with two test cases:

import {
    Test,
    TestParams,
    TestUtils,
    TestFailure,
    TestFailure
} from "#lib/Test/Test.mjs";

// Declare the context your test needs. This is the shape of the object
// your configure() method will return, and what every test method in the
// class reads from `this.ctx`. The framework captures it automatically as
// structured evidence (runData) on the RunResult — zero extra work.
type MemoryCtx = { totalKb: number };

// Every piece of hardware you test is a Test subclass. The <MemoryCtx>
// generic narrows `this.ctx` throughout the class — no casts, no `any`.
class SystemMemory extends Test<MemoryCtx> {

    // The setup phase. Resolve addresses, open shells, read calibration
    // values, poll readiness gates — whatever the test needs to reach a
    // known-good starting state. Return it as a plain object; the framework
    // merges it into `utils.ctx` for every downstream test method and
    // descendant test.
    //
    // A throw from configure is an *infrastructure* failure, not a hardware
    // failure — the Test Engineer's recovery protocol is "check the harness,"
    // not "check the hardware." Always rethrow as TestFailure so
    // the Inspector categorizes it correctly.
    protected async configure( utils: TestUtils ): Promise<MemoryCtx> {
        try {
            const { stdout } = await utils.exec( "cat /proc/meminfo" );
            const match = stdout.match( /MemTotal:\s+(\d+)/ );
            return { totalKb: match ? parseInt( match[1] ) : 0 };
        } catch( e ) {
            throw new TestFailure( e );
        }
    }

    // A pure table of contents — no test logic lives here. Compose a tree
    // using three factories:
    //   this.case( name, fn )     — inline leaf test
    //   this.sequence( name, [] ) — serial, bails on first failure
    //   this.group( name, [] )    — concurrent, collects every failure
    //
    // Method references pass directly; the factory auto-binds `this`, so
    // private methods can read `this.ctx` without arrow wrappers. This is
    // also the right place to branch on hardware variant or params —
    // buildSuite() runs after configure(), so `this.ctx` is already populated.
    protected buildSuite(): Test {
        return this.group( "presence-checks", [
            this.case( "meminfo-readable", this._readable ),
            this.case( "total-nonzero",    this._totalNonzero ),
        ]);
    }

    // Test methods carry the verdict logic. Three deliberately strict rules:
    //   1. Return "PASS" explicitly. There is no pass-by-default — the
    //      Promise<"PASS"> return type makes an implicit return a compile error.
    //   2. Throw TestFailure to report a *hardware* failure. The
    //      (message, expected, received) triplet renders directly in the
    //      Inspector — write it for a tired human reading at 2 AM.
    //   3. Any other throw (plain Error, etc.) is reported as a bug in
    //      *your* test, not a failure of the hardware.
    private async _readable( utils: TestUtils ): Promise<"PASS"> {
        const { stdout } = await utils.exec( "cat /proc/meminfo" );
        if ( stdout.includes( "MemTotal" ) ) return "PASS";
        throw new TestFailure( "/proc/meminfo missing MemTotal", "MemTotal present", stdout.slice( 0, 80 ) );
    }

    private async _totalNonzero( _utils: TestUtils ): Promise<"PASS"> {
        // ctx from configure() is available here, fully typed.
        const { totalKb } = this.ctx;
        if ( totalKb > 0 ) return "PASS";
        throw new TestFailure( "MemTotal is zero or unreadable", "> 0 kB", String( totalKb ) );
    }
}

// Module-scope export. The Inspector discovers named exports and lets the
// Test Engineer run the whole device — or any single leaf inside it —
// with ancestor configure/cleanup replayed automatically.
export const systemMemory = new SystemMemory({ name: "system-memory" });

Documentation

| I want to… | Read | |---|---| | Understand framework philosophy, audiences, and design decisions | docs/overview.md | | Understand lifecycle contracts, failure taxonomy, and guarantees | docs/contracts.md | | Learn idiomatic patterns: configure, buildSuite, survey, audit | docs/idioms.md | | Read the Inspector from a floor technician's perspective | docs/inspector.md |

Classes at a Glance

| Class | Purpose | |---|---| | Test<TCtx> | Base for every test node — hardware devices extend this | | TestSequence | Serial composition — bails on first failure | | TestGroup | Concurrent composition — collects all failures | | TestFailure | True hardware failure — throw from a test method | | TestFailure | Infrastructure failure — throw from configure() | | ExternalIntegrationFailure | Harness-level abort — raised automatically when a beforeEach / afterEach hook throws; transitions the enclosing node and every ancestor to Aborted | | SshInterface | SSH remote shell — acquire via utils.useShell() or utils.getShell() | | ArpResolver | MAC → IP resolution — acquire via utils.getArpResolver() | | Logger | Scoped per-node logger — available as utils.logger |

Composition Factories

All three are instance methods on Test.

| Factory | Execution | discreteLogFile default | |---|---|---| | this.case( name, fn ) | Inline leaf — calls fn with utils | false — rolls up into parent | | this.sequence( name, tests ) | Serial — bails on first failure | true — own log file | | this.group( name, tests ) | Concurrent — collects all failures | true — own log file |

When to Use What

| I need to… | Use | |---|---| | Model a physical hardware component (blade, NIC, drive, PSU) | Extend Test<TCtx> | | Write an inline check inside a device's buildSuite() | this.case() | | Run steps that depend on each other in order | this.sequence() | | Run independent checks simultaneously | this.group() | | Bridge a run to an external system (shop-floor DB, lab coordinator, audit pipeline) | RunParams.beforeEach / afterEach on the root run() call — see docs/contracts.md |

The Three Terminal States

Every run of a Test resolves to exactly one of three peer states:

| State | Meaning | Recovery protocol | |---|---|---| | Pass | A test method returned "PASS" explicitly. | — | | Fail | A test method threw TestFailure. The DUT produced a verdict and it was negative. | Check the hardware. | | Aborted | An integration hook (beforeEach / afterEach) threw. The harness around the test broke before a verdict could be reached. | Check the harness — not the unit under test. |

Aborted is a peer to Pass / Fail, not a flavor of failure. The framework deliberately refuses to collapse them.

Examples

Runnable source examples live in src/examples/. Build the project (npm run build) then point src/index.mts at the example you want to run, or wire it directly via npm run dev.

| File | What it demonstrates | |---|---| | hardware-device-lifecycle.mts | configure → ctx → buildSuite → private methods | | test-composition.mts | this.sequence vs this.group — when to use each | | networked-device.mts | ArpResolver MAC→IP + SSH via utils.useShell() | | networked-device-via.mts | Multi-hop topology with via routing | | remote-device-ssh.mts | SSH exec, reconnect, and shell reuse patterns | | ssh-tunneling.mts | Jump-host composition through SshInterface | | run-data-evidence.mts | useSurvey, useAudit, and runData evidence capture | | waitfor-patterns.mts | utils.waitFor() — polling gates in configure and main |

npm Scripts

| Script | What it does | |---|---| | npm run build | ESLint fix + tsc | | npm run clean | Remove dist/ and src/artifacts/ | | npm run dev | Build + execute dist/index.mjs | | npm run inspect | Open artifact inspector (auto-discovers latest run) | | npm run suite | Interactive suite picker → run mode | | npm run lint / npm run lint:fix | ESLint check / auto-fix | | npm run test | Vitest run | | npm run test:watch | Vitest watch mode |

Environment Variables

Access all environment variables through utils.env in test code — never process.env directly. The full snapshot is stored in every RunResult for post-run auditing.

| Variable | Purpose | |---|---| | PHYSICAL_LOCATION | Stamped on every RunResult — identifies the test fixture or station |