@bigsteele/ai-audit

Big Steele's AI audit. Point it at a codebase and it reports, file by file, what the software automates, what its AI does, what its agents can do, and how much of the software an AI agent could run. One score out of 100, a letter grade, and a level:

| Level | Name | Means | |---|---|---| | 0 | Manual | operated by people through screens | | 1 | Automated | programs can call it; work runs on schedules and webhooks; no model in the loop | | 2 | AI-assisted | models answer or draft inside the product; people do the doing | | 3 | Agentic | agents act through tools or an MCP server; a person owns the operation | | 4 | Autonomous | an AI agent could run the whole software: operations reachable by machine, failures visible |

100 and A+ means an AI agent could run the whole thing.

npx @bigsteele/ai-audit --repo .

Writes ai-audit-<app>.md (the report) and ai-audit-<app>.json (the same facts for a program) into the repository, named for the app from its manifest (or its folder, when the manifest carries a scaffold name), so two audits never overwrite each other. Add --out <dir> to write elsewhere, --stdout to print the report instead, --gate <score> to set the pass mark (default 25 and level 1 or above).

Exit code 0 when the gate passes, 2 when it does not. The two output files land in the repository unless --out says otherwise; add ai-audit-*.md and ai-audit-*.json to .gitignore if you do not want them committed.

What it reads, and what it never does

It reads the files in the tree, skipping node_modules, build output, dot-directories (planning trees, editor folders), a nested repository of its own, a copy of the repository inside itself, and every .env* file. It never opens an env file, never runs anything, and never writes source text or a value into the report: findings are file paths and counts. If any output string looks like a credential, nothing is written and the exit code is 1, because that means a secret is committed in the repository and that is the thing to fix first.

What the software is for decides what counts

A flat rubric punishes a platform for not being a voice product. So the repository may declare a profile, which weights the four groups, and waive named criteria with a reason. Both print in the report; a reader sees exactly what was set aside and why.

| Profile | Surfaces | AI | Agents | Autonomy | Set aside by default | |---|---|---|---|---|---| | product (default) | 25 | 25 | 25 | 25 | nothing | | platform | 35 | 15 | 20 | 30 | voice/vision, retrieval | | agent | 15 | 25 | 35 | 25 | nothing | | internal | 30 | 20 | 20 | 30 | voice/vision |

Declare it with --profile platform, or in ai-audit.config.json at the repository root:

{ "profile": "platform", "waive": { "webhooks": "nothing upstream sends this system events" } }

A waiver needs a reason of at least a sentence. The criteria that make the grade a grade cannot be waived and always count: API handlers, a model provider, server-side AI, tools, a loop, reach, observability, reliability, guardrails. A refused waiver is printed as refused, so nobody thinks it applied. The level floors never move: a level 3 still needs tools and a loop or an MCP server, whatever the profile.

The rubric

Four groups of 25 points, each point earned by a cited fact:

• Automated surfaces. API handlers, an OpenAPI or GraphQL description (a generated capability inventory or typed contract schemas earn half), inbound webhooks, schedules, queues.
• AI systems. A model provider on the server, prompts as code, structured output, retrieval over your own data, voice or vision.
• Agents. Tools a model may call, a loop that lets it act until done, an MCP server, orchestration or MCP client use, memory between runs.
• Autonomy and operability. Reach (machine surfaces against human pages), surfaces that write, observability, retries and idempotency, evals, guardrails and a human gate, documentation written for agents.

The report ends with "what raises the grade": every unearned point with what earning it takes, largest first. Documentation is a multiplier there, not a line item: undocumented handlers count at a quarter toward Reach, so the description item carries the Reach it would unlock ("+2 direct, +3.5 through Reach") and is ranked by the sum.

What it is not

It measures how ready the architecture is for agents, not whether the product works today. It is not a security review, not a health check, and it runs nothing; a broken deploy scores the same as a working one. The report says so in its footer.

No terminal? (Lovable, Bolt, v0, Replit)

docs/LOVABLE-PROMPT.md has two prompts to paste into Lovable's chat. Route A makes Lovable's agent apply this same rubric to the project it can see and write docs/ai-audit.md itself, with no GitHub and nothing installed; route B, for projects connected to GitHub, makes Lovable create a workflow that runs the real tool there. The tool's grade is the official one; route A is the same rubric read by an agent.

For the class

The entry rule is level 1 or above: an automated platform. Run the audit, attach ai-audit.md. The grade tells you where you stand between a manual tool and a business an agent could run.

Every report ends with the next step: upload it at bigsteele.com/scan and get a written read of the three changes that move the grade most, in order. No call required to get it.

Big Steele · bigsteele.com