agent-comprehension-debt

Self-hosted CLI that measures whether AI-authored code has ever been genuinely understood by a human

A Project by SILICON VALLEY GLOBAL PH INC

agent-comprehension-debt estimates comprehension debt: the growing gap between how much code in a repository was written by an AI coding agent and how much of that living code any human has actually engaged with since. Created and maintained by SILICON VALLEY GLOBAL PH INC.

Install:

npm install -g @siliconvalleyglobal/agent-comprehension-debt
# or, without installing globally:
npx @siliconvalleyglobal/agent-comprehension-debt scan

Honest positioning

This is not the first tool to analyze git history for knowledge risk. Established bus-factor tools (Hercules, truckfactor, git-of-theseus, and others) already do that well for human-to-human turnover: "if these people left, would the remaining team still know the code?"

What is different here:

1. The signal. We measure AI-authorship plus absence of subsequent human engagement, not concentration of human ownership. Green tests and "a human understands this" are different claims. Bus factor does not answer the second one when the last author is an agent.
2. The shape. This is a free, open, self-hosted CLI. Analysis runs on your machine against your git history. Optional review data is read from your already-authenticated GitHub or GitLab API. Nothing is sent to a product SaaS, a telemetry endpoint, or us.

It is also not test coverage. Coverage asks whether code is executed by tests. This asks whether a human has engaged with AI-authored lines after they landed.

The classifier is only as good as the attribution your tools actually write. Many agents add Co-authored-by trailers; many teams strip them. Absence of a trailer is not proof of human authorship — that case is documented as unknown-leaning-human, with an optional low-confidence heuristic. See docs/RESEARCH.md.

The problem

AI agents now write a large share of new code in many codebases. That code passes tests and ships. Comprehension debt accumulates silently until it surfaces at the worst possible time: an incident in code nobody on the team can actually explain.

How it works

1. Git history ingestion — git blame --line-porcelain and git log via the system git binary (mailmap-aware). Unchanged blobs are skipped on later runs so this can run regularly on large repos.
2. Identity resolution — .mailmap first (all git formats), then a conservative Hercules-style cluster (email, then uncommon name, GitHub noreply login). Shared/generic mailboxes do not merge different names. Optional GitAuthority-style local-part heuristic is off by default.
3. Authorship classification — pluggable AuthorshipSignals: commit trailers actually used by current tools (Claude Code, Copilot, Cursor, Codex, Gemini, Aider, Devin, …), author/committer bot identities, optional suite provenance package, then a documented heuristic fallback.
4. Human engagement — later human edits to the file, optional PR approvals/comments (GitHub/GitLab, read-only), and mark-reviewed self-reports. Recency uses exponential decay (configurable half-life), not a hard cutoff.
5. Risk score — per-file comprehension coverage 0–100, weighted by size/complexity and optional path criticality. Highest-risk files: large, AI-authored, low engagement, high criticality.

CLI

Run from inside a git repository (or pass -C).

agent-comprehension-debt scan              # ingest / incrementally update
agent-comprehension-debt report            # ranked highest-risk files
agent-comprehension-debt file src/auth.ts  # per-file breakdown
agent-comprehension-debt mark-reviewed src/auth.ts
agent-comprehension-debt trend             # coverage over stored snapshots

Useful flags:

agent-comprehension-debt scan --force          # ignore blame cache
agent-comprehension-debt scan --json
agent-comprehension-debt report --format csv -o debt.csv
agent-comprehension-debt report --format json -o debt.json
agent-comprehension-debt -C /path/to/repo scan

Local data lives in .agent-comprehension-debt/ (cache, snapshots, self-reports). Add that directory to .gitignore. It is JSON/CSV you own — Grafana, a spreadsheet, or a homepage can read it. There is no proprietary export format.

Scoring formula

All coefficients are in .agent-comprehension-debt.json. There are no hidden magic numbers in the ranker.

For each file:

aiShare            = aiLineCount / lineCount

depthNeed          = clamp(
                       1 + complexitySurcharge(lineCount, method),
                       minDepthNeed,
                       maxDepthNeed
                     )

# log-lines (default): surcharge = weight * max(0, log2(lineCount / baselineLines))
# raw-lines:           surcharge = weight * (lineCount / baselineLines)
# token-density:       log-lines surcharge scaled by non-whitespace token density

adjustedEngagement = min(1, meanEngagement / depthNeed)

coverage           = 100 * ( (1 - aiShare) + aiShare * adjustedEngagement )

rankScore          = (100 - coverage) * log2(1 + lineCount) * criticality

meanEngagement is the average, over living AI-attributed lines, of the strongest recency-weighted event that applies to that line:

recencyWeight(ageDays) = 0.5 ^ (ageDays / halfLifeDays)

eventWeight            = baseWeight[kind] * recencyWeight

Default baseWeights: self-report 1.0, PR approval 0.85, PR review comment 0.4, later human edit to the same file 0.35 (file-level, not line-level — blame already reassigns lines a human subsequently edited).

Human-authored living lines count as fully covered under this metric. A 20-line AI file with a fresh mark-reviewed scores much higher than a 400-line AI module with the same check-box, because depthNeed rises with size.

Criticality does not change coverage. It only boosts rankScore so src/auth/ can surface above a large unengaged generated fixture file.

Configuration

.agent-comprehension-debt.json at the repo root (all keys optional):

{
  "complexity": {
    "method": "log-lines",
    "baselineLines": 50,
    "weight": 0.15,
    "minDepthNeed": 1,
    "maxDepthNeed": 3
  },
  "recency": {
    "halfLifeDays": 180
  },
  "engagement": {
    "selfReport": 1.0,
    "prApproval": 0.85,
    "prReviewComment": 0.4,
    "fileFollowupEdit": 0.35
  },
  "identity": {
    "heuristicClustering": false
  },
  "reviewApi": {
    "enabled": true
  },
  "authorship": {
    "patterns": "all",
    "enableHeuristicFallback": true,
    "extraEmailRegexes": [],
    "extraNameRegexes": []
  },
  "criticality": {
    "src/auth/**": "high",
    "src/payments/**": "high",
    "docs/**": "low"
  },
  "defaultCriticality": 1,
  "excludeGlobs": ["*.min.js", "pnpm-lock.yaml", "package-lock.json"],
  "maxFileBytes": 1000000,
  "blameConcurrency": 4
}

authorship.patterns may be "all" or a list of ids: claude-code, copilot, cursor, codex, gemini, aider, devin, windsurf, cline, warp, amazon-q, continue, tabnine, opencode, jules.

criticality values: high (2.0), medium (1.5), low (0.5), or a number.

Privacy

This process:

• Reads the local git directory and .agent-comprehension-debt/ on disk.
• Optionally, only if reviewApi.enabled is true and a token is already in your environment (GITHUB_TOKEN / GH_TOKEN or GITLAB_TOKEN / GL_TOKEN), makes read-only HTTPS calls to api.github.com (or the GitHub Enterprise / GitLab host of origin). Those calls fetch PR/MR reviews for commits that introduced AI lines.

It does not open a network connection to SILICON VALLEY GLOBAL, an analytics vendor, or any other third party. No anonymous usage ping. Tokens are sent only to the forge they belong to, as Authorization / PRIVATE-TOKEN headers, and are not logged.

If you do not want forge calls, set "reviewApi": { "enabled": false } or leave forge tokens unset.

Library API

import {
  scanRepository,
  loadConfig,
  findRepoRoot,
  createAuthorshipRegistry,
} from "@siliconvalleyglobal/agent-comprehension-debt";

const repoRoot = findRepoRoot(process.cwd());
const result = await scanRepository({
  repoRoot,
  config: loadConfig(repoRoot),
});
console.log(result.summary.weightedCoverage);

AuthorshipSignal and EngagementSignal are the extension points. See CONTRIBUTING.md.

If @siliconvalleyglobal/agent-provenance is installed, it is loaded with optional import() and registered before trailer matching.

Requirements

• Node.js 18+
• Git on PATH (we shell out; we do not bundle libgit2)

Roadmap

• [ ] First-class consumer for a dedicated provenance/attribution suite package
• [ ] Line-range git log -L follow-up (stronger than file-level human edits)
• [ ] Native Grafana dashboard JSON, still fed by the same CSV/JSON files

License

MIT © SILICON VALLEY GLOBAL PH INC