MIDAS Harness

Start a conversation (0.4.0)

midas
midas --cwd ./my-project
midas chat --sandbox workspace-write
midas --continue

Type a message at You >, then continue asking follow-up questions in the same Codex session. midas with no arguments defaults to chat. Existing run/eval commands remain available. --continue loads the last chat's conversation in the selected folder; --resume RUN_ID selects a saved run. Startup options define current permissions/model; resuming does not silently enable prior write permissions. Default is read-only.

Use /help, /status, /reset, /exit. /permissions workspace-write enables file edits in the project, and /permissions read-only switches back. /model NAME sets the model for future turns; /model default uses the Codex configuration. /multiline collects lines until a single . sends the message. Ctrl+C cancels input/current work; EOF or /exit leaves. --timeout and other budgets apply per message, including that message's repair attempts.

/verify acceptance.json loads a trusted verifier relative to the project; /verify checks the most recent answer, and /verify off disables checks. Protected entries must be individual existing files (e.g. tests/test_app.py, not tests). /reset keeps saved history but starts fresh model context. Conversation pointers and reported cumulative tokens persist in .midas/runs.db; work records use existing MIDAS history. Answers appear as Codex message events complete, with tool progress between them; this is a line-based terminal chat, not a full-screen TUI.

0.3.0: verify, repair, measure and remember

midas run "Fix the failing tests" --sandbox workspace-write --verify acceptance.json --repairs 2 --timeout 180 --max-tools 20 --max-tokens 100000 --memory read-write
midas eval --output ./evaluation-preview --dry-run
midas eval --output ./evaluation-001 --repeats 2 --timeout 120
midas eval --output ./heldout-001 --split holdout --memory-db .midas/experience.db

Example acceptance.json (a trusted local configuration):

{
  "command": ["python", "-m", "pytest", "tests/test_app.py", "-q"],
  "protect": ["tests/test_app.py"],
  "timeout": 30
}

The command is an argv array, never a shell string. It runs locally with your permissions; only use trusted checks/code. This is independent grading, not a security sandbox for hostile code. Test files are fingerprinted before execution; changed or removed protected files fail the check. The JSON spec is held in memory, so editing it does not alter grading. Also supported: files mapping relative paths to exact UTF-8 contents and answer_equals. Supply all relevant acceptance/test configuration files in protect. File hashes detect changes, not a malicious program restoring files or forging a test process result.

MIDAS checks results and passes failure evidence into at most --repairs additional turns (default 1, maximum 5). Repeated identical failures end repair early. Three identical tool actions and outputs without an observed file change stop execution; tune --repeat-limit. --timeout covers execution and verification together. --max-tools counts observed tool completions. Tokens include reported input plus output, including cached input once; reasoning/cache counters are not added again. Codex reports usage at turn boundaries, so --max-tokens prevents further turns after observed exhaustion and can overshoot. It is not a hard billing limit. No retries are started after API failures, cancellations or timeouts.

Experience mode is off by default. read, write, and read-write operate on the local .midas/experience.db. Only independently accepted results are stored, with originating run and verifier fingerprint. Retrieval uses lexical similarity, up to three experiences, a 2,000-character serialized budget and 90-day age limit. Historical summaries are passed as untrusted reference data; acceptance is checked again. Similarity is a simple baseline, not a learned skill compiler. Summaries can contain project content; keep the database private.

Evaluation uses three small bundled tasks (two dev, one holdout), separate fresh task folders for each arm, seeded randomized order, and equal per-arm time/tool/observed-token limits. Baseline is one official Codex run plus the same independent grader. MIDAS receives verification feedback and bounded repair turns. Both use the same configured model. --suite FILE accepts a trusted JSON object with name and tasks; each task has unique id, goal, initial files, verify, and optional split (dev/holdout). --limit, --repeats, --seed, --model select the experiment. Reusing an output directory is rejected. Manifest, append-only result rows and incremental summary allow inspection even if evaluation is interrupted. Memory is off unless --memory-db is supplied; that database is copied once and read without learning during evaluation. Use only training experiences: freezing a database does not prove it is uncontaminated. Local workspaces provide reproducibility, not isolation against adversarial agents reading other folders.

Reports include success counts, Wilson intervals, paired wins/losses, median/p90 time, reported tokens and tool calls. Unknown tokens are counted explicitly. Dollar cost remains unknown. Repeated tasks are correlated and this tiny suite is not evidence of broad AGI. Compare memory-off and memory-on runs using identical suite, seed, model and budgets to measure retrieval's effect. Do not claim improvement from a single successful example.

MIDAS is a terminal-first, evidence-driven harness for running, verifying, and improving AI task-solving strategies.

Version 0.3.0 runs real tasks through your locally authenticated Codex CLI, and handles arithmetic locally. Codex answers are independently checked when --verify is supplied. Harness code evolution, RL and benchmark perfection are not features of this release.

Codex account setup

npm install -g @openai/codex@latest
npm install -g midas-harness
midas login codex
midas doctor codex
midas run "Explain this project" --cwd ./my-project
midas run "Fix failing tests" --cwd ./my-project --sandbox workspace-write --timeout 600
midas run "Continue" --resume RUN_ID --cwd ./my-project
midas inspect RUN_ID --cwd ./my-project

Already logged in? Skip login. MIDAS does not read credential files; Codex manages authentication. A saved ChatGPT or API-key login retains its billing rules. Environment API-key overrides are removed for this backend. doctor checks installation and saved login, not server/model availability.

Install a local build with npm install -g ./midas-harness-0.3.0.tgz. Check midas --version after installation.

Default provider auto handles arithmetic locally and sends other tasks to Codex. Use --provider local to prohibit model calls, or --provider codex to force Codex. --profile applies only to local execution. --model NAME overrides the configured Codex model. Default timeout is 300 seconds; default sandbox is read-only. File modifications require --sandbox workspace-write.

Resume accepts a MIDAS run ID saved in the chosen directory. JSON output is available with --json. Exit codes: 0 completed, 1 failed/incomplete/budget-exhausted, 2 invalid arguments, 124 timeout, 130 cancellation. Verification-enabled runs can perform the configured repair turns. API failures are not blindly retried. Cancellation does not roll back files already changed.

Without a verifier, Codex completion means a successful turn and process exit. With a verifier, the workflow requires acceptance checks to pass. Unknown dollar cost, verification and efficiency remain null. Token counts are reported as returned by Codex. Local .midas/ events can contain private prompts, file contents and tool outputs; common credentials are redacted but this is not exhaustive secret detection. Output limits are 32 MiB total and 1 MiB per line.

The official interface is documented at https://learn.chatgpt.com/docs/non-interactive-mode . MIDAS_CODEX_BIN can select a native Codex executable. Windows is locally tested; macOS/Linux use the same sources but are not yet platform-certified.

Run anywhere with npm

MIDAS requires Node.js 18+ and Python 3.11+. No Python package installation is needed because the npm package includes the current MIDAS Python core.

npx midas-harness demo
npx midas-harness run "12 * (8 + 2)"

For a persistent global command:

npm install --global midas-harness
midas demo

If Python is installed in a non-standard location, set MIDAS_PYTHON to its executable path.

Run from a source checkout

uv run midas run "12 * (8 + 2)"
uv run midas run "2 ** 10" --profile core
uv run midas demo
uv run midas history

You can also run the package directly after installation:

python -m midas run "(25 + 5) / 3"

MIDAS stores local run metadata and append-only traces under .midas/ in the current directory.

Design rule

The code favors clarity over cleverness: short functions, explicit state, type hints, ordinary Python data structures, and comments that explain decisions rather than syntax.