honto-contract

Did it actually work?

A contract checker for LLM and agent output. You declare what a valid answer must look like; honto rejects everything else.

Zero dependencies. Synchronous. Works on any string.

The failure dataset behind this checker, and the production watchdog templates we run on top of it, are available separately — see below.

The problem

Monitoring tells you the call succeeded. It does not tell you the answer was wrong.

These all return HTTP 200, have plausible length, and raise no error:

"I'm sorry, but I can't help with that."        → recorded as a completed deliverable
"This is just a sample, not a real plan."       → passed a 200-character minimum
{"score": 85, "action": "watch"}                → returned for a request that asked for prose
"productivityですのためにをreferencingularや the" → fluent-looking, meaningless
"Fields: `agent_id`, `automation_id`..."        → invented; no such fields exist

We ran an autonomous agent organisation for {{days}} recorded days. {{records}} failures. {{contractViolations}} of them ({{contractPct}}%) were of this kind — the run reported success and the output was wrong. Utilisation showed 97–100%. Deliverables shipped: zero.

What that percentage is a percentage of. We recorded {{measuredScopeTotal}} failures. {{measuredScopeTyped}} of them ({{measuredScopePct}}%) carry a type, and those types are the ones this checker defines — see index.js. So the figure above describes what our checker was watching for, not the distribution of everything that can go wrong with an agent. A failure nobody was watching for is in neither the numerator nor the denominator. We are telling you this because you would find it yourself the moment you opened the file.

honto is the check that closes that gap.

Install

npm install honto-contract

Use

const { check } = require('honto-contract');

const problems = check(output, {
  minChars: 200,
  lang: 'en',
  must: [/^Verdict:/m],
  mustNot: [/^DATE:/m],        // signature of a different job in your pipeline
  groundedIn: sourceMaterial,  // reject specifics that are not in the source
});

if (problems.length) throw new Error(problems.join(' / '));

Or let it throw:

const { assertContract } = require('honto-contract');
assertContract(output, { minChars: 200, lang: 'en' });

What it checks

| Option | Rejects | |---|---| | minChars | Output shorter than the minimum | | must | Missing required markers (strings or RegExp) | | mustNot | Present forbidden markers — use this to catch answers to a different question | | shape: 'prose' \| 'json' | Prose requests answered with scoring JSON, and vice versa | | lang: 'ja' \| 'en' \| 'any' | Wrong language, Chinese leaking into Japanese, Japanese/English mixed into nonsense | | groundedIn | Identifiers and licence names asserted but absent from the source material | | (always) | Refusals and hedges recorded as deliverables | | (always) | Claims of customers, downloads, or track record the caller does not have |

Why mustNot matters

Prompt caches and slot reuse make a model answer the previous question. The response is fluent and well-formed — it just belongs to a different job.

Declare the signature of your other job types and they get caught:

mustNot: [/^DATE:/m, /^CASE_NAME:/m, /"score"\s*:/]

We lost most of a day to this before adding it.

The dataset

The failures that produced these rules are published as a dataset: Autonomous AI Agent Failure Logs — real production failures, one JSON object per line, with the stage that failed and the contract that was violated. It grows as the system runs.

Licence

MIT

What actually breaks

These are not hypothetical failure modes. They are what a running agent organisation produced, counted from {{records}} recorded failures over {{days}} recorded days (the dataset, CC BY 4.0 — a training-ready cleaned/ split is included, updated as the organisation keeps running).

Every one of them returned HTTP 200. Every one has plausible length and grammar. None of them raise an error, and none are visible to a tracing tool — a trace shows that the call happened, not whether the answer was usable.

| Count | Type | Example message | |---:|---|---| {{typeTable}}

What the types mean:

• missing_required — a required marker was absent: a verdict line, a heading, a field name. The text reads fine; the thing that made it machine-usable is gone.
• too_short — shorter than the contract requires. Usually not a truncation: the model spent its whole budget on reasoning and returned an empty body, with done_reason: length and a successful HTTP 200.
• other — recorded, not yet classified. Kept rather than dropped, so the tail stays visible.
• forbidden_present — a phrase the contract forbids appeared anyway.
• broken_language_mix — Japanese and English fused into something neither language. Passes a length check and a kana check. Reads as damage only to a person.
• not_japanese — Japanese requested, not a single kana returned. A check that looks for "mostly Japanese" misses it; one that looks for kana does not.
• chinese_in_japanese — characters that exist in Chinese but not Japanese. Fluent to a glance, wrong to a reader.
• answer_to_other_question — a different job's answer came back (prompt cache reuse across slots).
• unearned_claim — customers, adoption numbers, or superlatives the company does not have. A model asked to write a product page will invent traction.
• refusal — a refusal recorded as a deliverable.
• reasoning_leaked — the chain of thought arrived as the answer. Long, coherent, unusable.

One more is in the checker but has not appeared in our own logs yet: hedge ("this is only a sample, not a real plan").

The list grows. Each new failure we hit becomes a check, and the check stays.

壊れた出力を送ってください / Send us a broken output

私たちは自社の運用で見た壊れ方しか知りません。あなたが見た壊れ方は、 まだこの検出器に入っていない可能性が高いです。見逃した例も、誤って落とした例も、 そのまま設計に使います。伏せ字で構いません。

We only know the failures we saw in our own runs. If this checker missed something — or rejected something it should have passed — that case is almost certainly not in it yet. Redaction is fine; the shape is what matters.

• 壊れた出力を貼る / Paste a broken output
• 要望 / Request

まだ顧客はいません。要望をそのまま設計に使います。 No customers yet. Requests go straight into the design.

有料の段（日本の法人向け）

上のデータセットは無料です。有料の段は、無料版に無いものだけで構成しています。

{{tiersTableJa}}

お支払いは銀行振込です（適格請求書を発行します）。 当社から振込先を変更する連絡は一切しません。 特定商取引法に基づく表記

データパック単体を買う

購読ではなく、失敗の型データセットだけを1回で買う口です。

| 商品 | 価格 | |---|---| | {{productFailureLogs}} | {{priceFailureLogs}} |

購入希望のメールを開く（件名と本文の型が入ります）

折り返し、金額と振込先をお送りします。お支払いは銀行振込です。 当社から振込先を変更する連絡は一切しません。 特定商取引法に基づく表記

なお、同じ記録は Hugging Face で CC BY 4.0 のまま置いてあります。 請求書で購入する必要がなければ、そちらをそのままお使いください。

申込

[email protected] へ、次の形でメールをお送りください。

購読の申込

会社名: （法人名）
担当者: （お名前）
プラン: 梅／竹／松 のいずれか
請求先メール: （請求書の送付先）
GitHub: @（招待するアカウント）

請求書とリポジトリへの招待をお返しします。打ち合わせも説明会もありません。

0.2.0

Three changes, all of them found by running this checker on our own documents.

It no longer punishes precise writing. A document that describes a failure contains that failure's own words — The model hedged: "this is just a sample", or a style guide rejecting "trusted by hundreds of teams" when you have none. 0.1.0 flagged both. Quoted text and denials (we have no customers) now pass.

asserts(text, pattern) answers that question directly: is this pattern actually claimed here, or merely mentioned? Exported because the same question comes up outside this library, and a second copy of the test would drift.

codeOf(problem) turns a problem sentence into a stable name — too_short, chinese_in_japanese, unearned_claim. Sentences make bad keys: they change when the wording improves, and they differ per language. We found this when 89% of a published failure dataset landed in one undifferentiated bucket while the real taxonomy sat inside free text.

const { check, asserts, codeOf } = require('honto-contract');

check(text, { minChars: 200, lang: 'ja' });        // sentences, for people
check(text, { minChars: 200, messages: 'ja' });    // …in Japanese
codeOf('too short (29 chars, minimum 300)');       // 'too_short' — for counting
asserts(doc, /trusted by/);                        // false if quoted or denied

Where this came from

We run an organization of {{agents}} agents on local models, unattended. For {{days}} days we recorded every failure — not that a run failed, but what was wrong with the answer. {{records}} records.

The distribution was not what we expected. Refusals and off-target answers, the failures most evals are built around, were {{dramaPct}}%. {{shapePct}}% of everything was shape: the model answered, and skipped the line the next stage reads.

What actually breaks when you run LLM agents unattended for {{days}} days — the numbers, the types, and what we changed.

Paid tiers

The checker above is MIT and always free. The dataset is CC BY 4.0 and always free. The paid tiers only add what the free release does not have. We have no customers yet — you would be the first.

{{tiersTableEn}}

What you would be paying for is the ongoing updates and the prepared form — not the code, and not the records themselves, which stay free.

Also available: {{productHarnessPro}} — {{priceHarnessPro}}. The production watchdog & gate templates from the same harness (zero-dependency Node.js, tests included). Buy — {{priceHarnessPro}}

And a video course: {{productCourse}} — {{priceCourse}}. 8 lessons, 11 focused minutes, distilled from the same failure ledger. Buy — {{priceCourse}}

No support desk, no SLA: this is one person and a fleet of agents on one machine. Ask anything at [email protected].

The records themselves: GXCafe/ai-agent-failure-logs (CC-BY-4.0). This library is the checker we extracted from that work.

The data behind this library

Every check in this library exists because a real production incident demanded it. The incidents themselves — every contract violation our gate caught across months of 24/7 operation, labeled into a 12-pattern taxonomy with the countermeasure that held — are packaged as a dataset:

{{productContractPatterns}} — {{priceContractPatterns}} {{whatContractPatterns}}

Before you consider it: you do not need it to use this library. A free snapshot of the raw failure ledger is on Hugging Face: https://huggingface.co/datasets/GXCafe/ai-agent-failure-logs. Start there.

Buy — {{priceContractPatterns}}

Single-organization commercial licence. Prices are exclusive of tax; the tax that applies to your location is added at checkout. Japanese commercial-transaction disclosure.