Hermes Echo

Discover what your CLI actually does. Catch behavioral changes your tests missed.

What it does

Hermes Echo finds your CLI’s console scripts from setup.py / pyproject.toml, including nested subcommands, and runs them like a user would. You accept the observed exit code, stdout, and stderr as Echo Contracts and commit them. On every pull request it re-runs those probes and comments when behavior drifts — it caught a version string change on hermes-relay PR #10 that tests missed.

Quick start (GitHub Actions)

Add this to .github/workflows/hermes-echo.yml:

name: Hermes Echo
on:
  pull_request:

permissions:
  contents: read
  pull-requests: write

jobs:
  echo:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      # Zero-config (auto-detects your console script):
      - uses: g1n0mag1k/[email protected]
      # Explicit:
      # - uses: g1n0mag1k/[email protected]
      #   with:
      #     command: myapp

pull-requests: write is required to post the drift comment on your PR.

fetch-depth: 0 is required because Hermes Echo executes probes against both the PR branch and its base revision.

Accept contracts locally, commit .hermes/contracts/, and PR comments will show contract drift.

Real comment from hermes-relay PR #10:

🔬 Hermes Echo · ECHO DETECTED A CHANGE

7 probes executed · 6 unchanged · 1 possible changes · 0 skipped

Echo Contracts

4 of 5 accepted contracts verified

| Probe | Contract | Status | |-------|----------|--------| | export-receipts | exit 2, stderr present | ✓ HOLDS | | help | exit 0, stderr empty | ✓ HOLDS | | keystore-init | exit 2, stderr present | ✓ HOLDS | | keystore-public-key | exit 2, stderr present | ✓ HOLDS | | version | exit 0, stderr empty | ✗ DRIFTED | | | Stdout drifted: 1 line(s) changed | |

Stdout diff (version):

-Hermes Relay v1.0.0
+Hermes Relay v1.0.1

Handling nondeterministic output

Hermes Echo normalizes timestamps, UUIDs, and absolute paths before comparing observations. Commands that produce different output on every run (daemons, servers, interactive prompts) are automatically skipped.

Does behavioral drift fail CI?

No. Hermes Echo always exits 0. It reports drift as a PR comment but does not block merging. You decide what to do with the information.

Local CLI usage

npm install -g @hermes-tools/hermes-echo
hermes-echo doctor --command myapp
hermes-echo accept
hermes-echo accept validate

How contracts work

• Accepted probes are stored as .hermes/contracts/*.yml (names like env-show, fmt-check)
• Each contract locks exit code, full stdout, and full stderr
• On PRs, Hermes Echo verifies those contracts and posts a drift table plus a git-style stdout/stderr diff

Compliance

Accepted Echo Contracts are version-controlled records of observed CLI behavior — what exited, what printed, and when you accepted it. Teams with formal software validation requirements may find version-controlled behavioral contracts useful as supporting evidence.

Built by Hermes Relay