DocuSOR — Reducing Token Cost by 10x via Context Verification for AI Agents

DocuSOR turns your context (Agent.md, ReadMe.md, Runbooks etc) into a test suite. It executes commands from fenced shell blocks in your docs, optionally brings up Docker Compose, evaluates assertions (httpOk, httpStatus, httpJsonPath, portOpen, certValid, fileExists, fileContains, commandSucceeds, versionAtLeast, logContains, spawn, and more) and waits (waitFor), and emits Markdown/JSON reports. Use it locally or in CI to continuously provide up-to-date context for your agents.

Open-core model:
This repo is the open-source DocuSOR CLI (Apache-2.0). The managed platform — run history, dashboards, and team features — lives at docusor.ai.

Contents

• Features
• Installation & Running
• Authoring Your README (Runnable Docs)
• Assertions & Waits — Full Reference
• Outputs, Exit Codes, and Reports
• GitHub Actions (CI) — Drop-in Workflow
• Configuration & Defaults
• Troubleshooting
• Development
• FAQ
• Security
• Contributing
• Code of Conduct
• Changelog
• Roadmap
• Trademark & Logo Notice
• License
• Appendix: Helpful Snippets

Features

• Runs shell code fences directly from README.md. Supported fence languages: bash, sh/shell, zsh, fish, pwsh/powershell, and python/py — each block runs with its matching interpreter.
• Built-in assertions:
  • HTTP: httpOk <url>, httpStatus <METHOD> <url> <code> [jsonBody], httpHeader <url> <header> "<value>", httpBodyContains <METHOD> <url> "<text>", httpJsonPath <url> <path> <value>
  • Network: portOpen [host] <port>, portClosed [host] <port>, dnsResolves <host> [expectedIp], certValid <host:port> [minDaysLeft]
  • Files: fileExists [container:<svc>] <path>, fileAbsent [container:<svc>] <path>, dirExists [container:<svc>] <path>, fileContains [container:<svc>] <path> "<text>", fileMatches [container:<svc>] <path> "<regex>", jsonValid [container:<svc>] <path>
  • Commands: commandSucceeds [container:<svc>] "<cmd>", commandOutputContains [container:<svc>] "<cmd>" "<text>", commandExitCode [container:<svc>] "<cmd>" <code>
  • Versions: versionAtLeast [container:<svc>] "<cmd>" <minVersion>, versionEquals [container:<svc>] "<cmd>" <version>
  • Containers: containerRunning <svc>, containerHealthy <svc>, processRunning [container:<svc>] "<pattern>"
  • Environment: envSet <KEY> [expectedValue]
  • Logs: logContains container:<svc> "text"
  • Spawn: spawn <cmd i.e. npm run start>
  • Negation: prefix any assertion with not to invert it (e.g. not portOpen 5432)
• Built-in waits:
  • httpOk <url> [timeoutSec]
  • httpStatus <METHOD> <url> <code> [timeoutSec]
  • portOpen [host] <port> [timeoutSec]
  • fileExists [container:<svc>] <path> [timeoutSec]
  • commandSucceeds [container:<svc>] "<cmd>" [timeoutSec]
  • logContains container:<svc> "text" [timeoutSec]
• Auto-detects Docker Compose → up -d before run, down -v after.
• Generates:
  • DOCUSOR_REPORT.md — human-readable report
  • docusor-report.json — machine-readable report
• Stops on first failure (MVP behavior).
• Works locally and in CI.

Installation & Running

Use the DocuSOR CLI globally or as a dev-dependency. Requires Node ≥ 18. Published on npm as docusor.

Global install:

npm install -g docusor
# use the docusor install command to get the docusor.md file so your agent can make your context docusor compatible
docusor install
docusor run README.md

Local (dev-dependency):

npm install --save-dev docusor
# use the docusor install command to get the docusor.md file so your agent can make your context docusor compatible
npx docusor install
npx docusor run README.md

You can point DocuSOR at any markdown file:

docusor run docs/GettingStarted.md

CLI commands

| Command | Description | |---|---| | docusor run [DOC] | Run a markdown doc as a test suite (default: README.md). | | docusor list [DOC] | List the steps a doc would run, without executing them (alias for run --dry-run). | | docusor init | Scaffold a starter docusor-check.md you can edit. | | docusor install | Copy/refresh docusor.md (the agent reference) into the current directory. | | docusor demo | Copy README_DEMO.md — a self-contained doc exercising every assertion — so you can docusor run README_DEMO.md to see it in action. | | docusor update | Update to the latest published version (npm install -g docusor@latest). | | docusor -v / docusor version | Print the installed version. | | docusor help | Show usage; add --help to any command for details. |

run options

| Option | Description | |---|---| | --dry-run, --list | Print the steps without executing them. | | --continue-on-fail, --keep-going | Run every step instead of stopping at the first failure. | | --fail-fast | Stop at the first failure (default). | | --continuous | Carry shell state (cd, exported env vars) forward across blocks so the whole doc runs as one session (bash/sh/zsh). | | --no-compose | Skip Docker Compose auto-detection/orchestration. | | --timeout <sec> | Override the per-step timeout (default 300). | | --env-file <path> | Load environment variables from a .env file before running steps. | | --json <path> | Path for the JSON report (default docusor-report.json). | | --report <path> | Path for the Markdown report (default DOCUSOR_REPORT.md). | | --format junit, --junit-out <path> | Also emit a JUnit XML report (default docusor-junit.xml). | | -q, --quiet | Suppress progress logs; print only the final result. | | --verbose | Print stdout for passing steps too. | | --no-color | Disable ANSI colors (also honored via NO_COLOR). |

Authoring Your README (Runnable Docs)

Place setup steps in a fenced bash block. Use # assert: and # waitFor: as inline directives; non-directive lines execute as shell commands.

# Start your app (Compose will also be auto-started if compose.yml is present)
docker compose up -d
# If not using docker compose and using localhost, use spawn directive to spin up running instance
# spawn: npm run start

# Wait until service is reachable and healthy
# waitFor: portOpen 127.0.0.1 8080 30
# waitFor: httpOk http://localhost:8080/health 30

# Verify API responds
# assert: httpStatus GET http://localhost:8080/health 200

# Verify logs contain startup line (compose service "web")
# assert: logContains container:web "Started server on port 8080"

# Verify a file was created
# assert: fileExists ./config/generated.yml

# Require a command to succeed (host and container examples)
# assert: commandSucceeds "echo ok"
# assert: commandSucceeds container:web "curl -sSf http://localhost:8080/health"

Rules & notes:

• container:<svc> uses the Docker Compose service name.
• waitFor has a default timeout of 60s if omitted.
• httpStatus optional JSON body is supported for POST/PUT (see reference below).
• Non-directive lines (# that are not assert: / waitFor:) are treated as normal shell comments.

Choosing a shell

The fenced code-block language selects the interpreter for that block:

| Fence language | Interpreter | |-----------------------|----------------| | bash | bash -c | | sh / shell | sh -c | | zsh | zsh -c | | fish | fish -c | | pwsh / powershell | pwsh -Command| | python / py | python3 -c |

# Steps in this block run with zsh
echo "hello from zsh"

Consecutive non-directive lines in a block run together as a single script, so shell state persists within that run: cd, export, variables, and multi-line constructs (loops, if, here-docs, multi-line Python) all work as written. An # assert:, # waitFor:, or # spawn: directive ends the current script segment and runs in order; the next group of shell lines starts a fresh interpreter, so state does not carry across a directive boundary. (Assertions themselves run in their own process and can't read shell variables you exported in a preceding segment — assert on observable state like files, ports, or HTTP.)

By default each fenced block also starts fresh, so cd/export do not carry from one block to the next. Run with --continuous to carry the working directory and exported environment variables forward across blocks, letting the whole document behave like a single shell session (supported for bash/sh/zsh).

Skipping a block. Add a skip flag to a block's info string to keep it in the docs but exclude it from execution — it is parsed out and never run:

```bash skip
# Documented for reference only; docusor won't run this.
rm -rf ./build
```

The flags ignore, no-run, and docusor-skip behave the same way.

If the chosen interpreter isn't installed on the host, the step fails with a clear Shell not found on PATH: <shell> message rather than a cryptic spawn error. Install the interpreter locally (e.g. zsh, fish, pwsh, python3) to run those blocks.

Assertions & Waits — Full Reference

Assertions

httpOk <url>

Passes on any 2xx HTTP status.

# assert: httpOk http://localhost:8080/health

httpStatus <METHOD> <url> <code> [jsonBody]

Exact status match; body is optional JSON (for POST, PUT, etc.).

# assert: httpStatus GET http://localhost:8080/health 200
# assert: httpStatus POST http://localhost:8080/login 200 '{"user":"a","pass":"b"}'

httpHeader <url> <header> "<value>"

Passes when the response header contains the given substring (case-insensitive header name).

# assert: httpHeader http://localhost:8080/ content-type "application/json"

httpBodyContains <METHOD> <url> "<text>"

Passes when the response body contains <text> (accepts any status).

# assert: httpBodyContains GET http://localhost:8080/ "Welcome"

httpJsonPath <url> <path> <value>

Fetches JSON and compares the value at a dot/bracket path (e.g. data.items[0].id) to <value>.

# assert: httpJsonPath http://localhost:8080/api/status status ok
# assert: httpJsonPath http://localhost:8080/api/user data.roles[0] admin

portOpen [host] <port>

Checks if <port> is open on host (defaults to 127.0.0.1).

# assert: portOpen 127.0.0.1 5432
# assert: portOpen 6379

portClosed [host] <port>

Passes when <port> is not accepting connections (defaults to 127.0.0.1).

# assert: portClosed 127.0.0.1 5432

dnsResolves <host> [expectedIp]

Passes when <host> resolves. If expectedIp is given, one of the resolved addresses must match.

# assert: dnsResolves api.internal
# assert: dnsResolves db.internal 10.0.0.5

certValid <host:port> [minDaysLeft]

Connects over TLS and checks the certificate is not expired. If minDaysLeft is given, the cert must have at least that many days remaining. Port defaults to 443.

# assert: certValid example.com:443
# assert: certValid example.com:443 14

fileExists [container:<svc>] <path>

Checks for a file either on the host or inside a container.

# assert: fileExists ./config/generated.yml
# assert: fileExists container:web /app/.cache/boot.log

fileAbsent [container:<svc>] <path>

Passes when a file (or path) does not exist, on the host or in a container.

# assert: fileAbsent ./tmp/lockfile
# assert: fileAbsent container:web /app/.env.local

dirExists [container:<svc>] <path>

Passes when <path> exists and is a directory, on the host or in a container.

# assert: dirExists ./dist
# assert: dirExists container:web /app/uploads

fileContains [container:<svc>] <path> "<text>"

Passes when the file's contents include <text>.

# assert: fileContains ./config/generated.yml "port: 8080"
# assert: fileContains container:web /app/boot.log "ready"

fileMatches [container:<svc>] <path> "<regex>"

Passes when the file's contents match the given JavaScript regular expression.

# assert: fileMatches ./version.txt "^v\\d+\\.\\d+\\.\\d+$"

jsonValid [container:<svc>] <path>

Passes when the file contains syntactically valid JSON.

# assert: jsonValid ./package.json
# assert: jsonValid container:web /app/config.json

commandOutputContains [container:<svc>] "<cmd>" "<text>"

Runs <cmd> (host uses sh -c; container uses sh -c inside the service) and passes when stdout contains <text>.

# assert: commandOutputContains "node --version" v20
# assert: commandOutputContains container:web "cat /etc/os-release" alpine

commandExitCode [container:<svc>] "<cmd>" <code>

Runs <cmd> and passes when its exit code equals <code>.

# assert: commandExitCode "test -f ./ready" 0
# assert: commandExitCode container:web "pgrep node" 0

versionAtLeast [container:<svc>] "<cmd>" <minVersion>

Runs <cmd>, extracts the first x[.y[.z]] version from its output, and passes when that version is greater than or equal to <minVersion> (semver comparison). Missing minor/patch components default to 0, so 18 means 18.0.0.

# assert: versionAtLeast "node --version" 18
# assert: versionAtLeast "python3 --version" 3.10
# assert: versionAtLeast container:web "nginx -v 2>&1" 1.25

versionEquals [container:<svc>] "<cmd>" <version>

Like versionAtLeast, but requires an exact major.minor.patch match. Combine with not to assert a version is anything but a specific one.

# assert: versionEquals "node --version" 20.11.1
# assert: not versionEquals "node --version" 18.0.0

envSet <KEY> [expectedValue]

Passes when the environment variable is set (and non-empty). If expectedValue is given, the value must match exactly. Checks the run's env, then process.env.

# assert: envSet DATABASE_URL
# assert: envSet NODE_ENV production

containerRunning <svc>

Passes when the Compose service's container is in the running state.

# assert: containerRunning web

containerHealthy <svc>

Passes when the service's Docker healthcheck reports healthy. Fails if the service defines no healthcheck.

# assert: containerHealthy web

processRunning [container:<svc>] "<pattern>"

Passes when a process matching <pattern> (via pgrep -f) is running, on the host or in a container.

# assert: processRunning "node server.js"
# assert: processRunning container:web "nginx"

logContains container:<svc> "text"

Scans ~1000 recent log lines for the given text.

# assert: logContains container:web "Started server"

commandSucceeds [container:<svc>] "<cmd>"

The command’s exit code must be 0 (success). Works on host or in a container.

# assert: commandSucceeds "echo ok"
# assert: commandSucceeds container:web "curl -sSf http://localhost:8080/health"

Negation — not <assertion>

Prefix any assertion with not to invert its result. It passes when the underlying assertion fails, and vice versa.

# assert: not portOpen 5432
# assert: not fileExists ./should-be-gone.txt
# assert: not httpOk http://localhost:8080/admin

Waits

Wait directives poll until success or timeout.

waitFor: httpOk <url> [timeoutSec]

# waitFor: httpOk http://localhost:8080/health 6

waitFor: httpStatus <METHOD> <url> <code> [timeoutSec]

Polls until the endpoint returns the exact status code. (Request body is not supported for waits.)

# waitFor: httpStatus GET http://localhost:8080/ready 200 30

waitFor: portOpen [host] <port> [timeoutSec]

# waitFor: portOpen 127.0.0.1 8080 60
# waitFor: portOpen 8080 30

waitFor: fileExists [container:<svc>] <path> [timeoutSec]

# waitFor: fileExists ./boot.done 30
# waitFor: fileExists container:web /app/ready 60

waitFor: commandSucceeds [container:<svc>] "<cmd>" [timeoutSec]

# waitFor: commandSucceeds "test -f ./ready" 30
# waitFor: commandSucceeds container:web "curl -sSf http://localhost:8080/health" 60

waitFor: logContains container:<svc> "text" [timeoutSec]

# waitFor: logContains container:web "Started server on port 8080" 60

spawn: [cmd ex. npm run start]

For local runs without Docker Compose, spawn will spin up a local instance of app

# spawn: npm run start

Outputs, Exit Codes, and Reports

DocuSOR emits two outputs by default:

1. DOCUSOR_REPORT.md — human-readable summary

   • Includes a table of steps (✅ / ❌).
   • Collapsible stdout/stderr for failures (GitHub-friendly).

2. docusor-report.json — machine-readable details

   • Per-step status, timings, stdout/stderr.
   • Suitable for further processing by CI, dashboards, or the managed platform at docusor.ai.

Exit Codes

• 0 — all checks passed
• 1 — verification failed (one or more assertions/waits failed)
• 2 — fatal error (missing README, invalid directive, internal error)

Sample Markdown Report

# Context Verified - Agent Ready: ✅

**Step 1:** `docker compose up -d` — ✅ Passed
**Step 2:** `portOpen 127.0.0.1 8080` — ✅ Passed
**Step 3:** `httpOk http://localhost:8080/health` — ✅ Passed
**Step 4:** `httpStatus GET http://localhost:8080/health 200` — ✅ Passed
**Step 5:** `logContains container:web "Started server on port 8080"` — ✅ Passed
**Step 6:** `fileExists ./config/generated.yml` — ✅ Passed
**Step 7:** `commandSucceeds "echo ok"` — ✅ Passed

GitHub Actions (CI) — Drop-in Workflow

Run your runnable docs on every push/PR. DocuSOR exits non-zero when a step fails, so the job fails whenever your context drifts.

name: DocuSOR
on: [push, pull_request]

jobs:
  verify-context:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - name: Verify context
        run: npx docusor@latest run README.md --format junit
      - name: Upload reports
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: docusor-report
          path: |
            DOCUSOR_REPORT.md
            docusor-report.json
            docusor-junit.xml

Tip: --format junit emits docusor-junit.xml so your CI's test UI can surface each step as a test case.

Configuration & Defaults

• Input file

  • Default: README.md in the current working directory.
  • Override by passing a path: docusor docs/GettingStarted.md.

• Docker Compose auto-detection

  • If docker-compose.yml or compose.yml exists in the repo root:
    • Run docker compose up -d before any steps.
    • Run docker compose down -v after all steps (or on failure).

• Failure policy

  • Stops on first failure (MVP behavior).
  • A continue-on-fail mode is planned.

• HTTP timeouts

  • Assertion requests: ~8 seconds per HTTP call.
  • Waits poll up to their specified timeout (or a default, e.g. 60s).

• Security

  • Commands run in your environment (or your containers).
  • Review docs before executing them, especially from untrusted sources.

Troubleshooting

• Docker not available

  • Ensure Docker Engine/Desktop is running.
  • On Linux, check access to /var/run/docker.sock.

• Services slow to start

  • Use waitFor: portOpen ... before waitFor: httpOk ....
  • Increase the timeout value (e.g. waitFor: httpOk ... 90).

• Wrong container:<svc> name

  • Must match the Compose service name in docker-compose.yml / compose.yml.

• Windows quirks

  • Recommended: use WSL2 or run on Linux/CI.
  • Volume mounting and paths may behave differently on native Windows.

• Logs too short for logContains

  • logContains scans ~1000 recent log lines.
  • Ensure your service logs the searched text at least once.

Development

Local dev:

npm install
npm run test-run   # runs the CLI against README.md (or a test README)

FAQ

Why can't agents validate context themselves? They can, and that's the problem. Your wasting those oh so precious tokens on validating context that should've been correct BEFORE the agent used it. When you provide valid context of your application, reduce the tax of discovery on every task. Even if memory is cached, thats wasted context window on an entire codebase when verified context is enough.

Why not just rely on tests?
Tests validate code; DocuSOR validates context + environment end-to-end (prereqs, commands, runtime behavior). They are complementary. Tests can pass while context the agent relies on is incorrect, even with access to the codebase. This is a fundamental waste of tokens.

How do I integrate with CI/CD? Yes! DocuSOR runs anywhere Node runs and exits non-zero on failure, so it drops straight into any pipeline (see the GitHub Actions workflow above), and docusor-report.json is ready for further processing. For managed run history, dashboards, and team features, see docusor.ai.

Does this support non-HTTP apps (Kafka, DBs, microservices)?
Yes, via portOpen, commandSucceeds, logContains, fileExists. More domain-specific assertions (dbQuery, kafkaRoundTrip, etc.) are planned.

Can I point at docs other than README.md?
Yes. Pass a path when invoking:

docusor your-md.md

How do I quickly try it? Run docusor demo to drop a ready-made README_DEMO.md into your directory, then docusor run README_DEMO.md. It spins up a local server and exercises every assertion type end-to-end.

How do I update DocuSOR? Run docusor update (equivalent to npm install -g docusor@latest). Check your installed version any time with docusor -v.

Does DocuSOR send my data anywhere? No. The CLI is fully local and stateless — no telemetry, no account, and no network calls beyond the ones your own assertions make (e.g. httpOk, certValid). Optional run history and dashboards live in the managed platform at docusor.ai.

Can I assert tool versions? Yes. Use versionAtLeast "<cmd>" <minVersion> for a semver >= check or versionEquals "<cmd>" <version> for an exact match — e.g. versionAtLeast "node --version" 18.

Which languages and shells are supported? Fenced blocks tagged bash, sh/shell, zsh, fish, pwsh/powershell, or python/py run with their matching interpreter. bash/sh work everywhere; install the others locally (e.g. zsh, fish, pwsh, python3) to use them.

Security

DocuSOR executes commands from your docs. Treat docs like code:

• Review commands before running them, especially from untrusted repos.
• In CI, prefer isolated runners/containers and least-privilege credentials.
• Do not mount secrets or production data unless necessary.

To report vulnerabilities, open a private security advisory on GitHub or email the maintainers.

Contributing

PRs are welcome.

Guidelines:

• Keep assertions deterministic and side-effect-free whenever possible.
• Minimize dependencies and keep the CLI lightweight.
• Add README examples to cover new features.
• Include basic tests or a sample doc that exercises your change.

Dev quickstart:

npm install
npm run test-run

Feel free to open an issue for discussion before a large change.

Code of Conduct

Be respectful. Harassment, hate speech, and discrimination are not tolerated. Maintain a welcoming environment for all contributors and users.

This project follows a “be a good human” standard; a more formal Code of Conduct may be added as the community grows.

Roadmap

Planned / exploring:

• continue-on-fail option and richer summary reports.
• Additional assertions:
  • dbQuery (basic DB connectivity / query check)
  • kafkaRoundTrip (produce/consume sanity checks)
• docusor.yaml config:
  • Timeouts, env vars, working dir, includes/excludes.
• JUnit/XML reporter for CI test summaries.
• DocuSOR Cloud:
  • Trusted badge + required GitHub status check.
  • Org-level dashboards, historical trends, SSO/SCIM, private runners, and more.

Changelog

See CHANGELOG.md for detailed version history, including all releases, new features, bug fixes, and deprecations.

Trademark & Logo Notice

“DocuSOR”, the DocuSOR word mark, the dinosaur logo, and the “Docs Verified” badge are trademarks and/or copyrighted works of Omari Gaskins.

They are not licensed under the Apache-2.0 license.

You may use the name “DocuSOR” and the “Docs Verified” badge to accurately describe this project, but you may not:

• Use the name or logo in a way that suggests affiliation with or endorsement by the DocuSOR project or its author without permission.
• Rebrand your own service as “DocuSOR” or use confusingly similar branding.

© 2025 Omari Gaskins. All rights reserved on logo and branding assets.

License

The code in this repository is licensed under the Apache License, Version 2.0.

You may obtain a copy of the License in the LICENSE file in this repository or at:

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

The name “DocuSOR” and associated logos/badges are not covered by this license. See Trademark & Logo Notice.

Appendix: Helpful Snippets

Minimal runnable README example

# Install dependencies
npm install

# Start the app
npm start &

# waitFor: portOpen 127.0.0.1 3000 60
# assert: httpOk http://localhost:3000/health

Example for separate docs file

# docs/GettingStarted.md

# Clone the repo
git clone https://github.com/your-org/your-app.git
cd your-app

# Start dependencies
docker compose up -d

# waitFor: portOpen 127.0.0.1 5432 60
# assert: portOpen 127.0.0.1 5432

Run:

docusor docs/<your-doc-name>

Happy verifying. ✅