FDA Warning Letter Classifier

Turn a messy FDA Warning Letter into clean, structured data — the company, the facility, the drug, the violations, how serious it is — with a model that is physically incapable of making things up.

  A 10-page FDA Warning Letter (plain text)
                  │
                  ▼
        ┌───────────────────┐
        │   this classifier  │
        └───────────────────┘
                  │
                  ▼
  {
    "company": "Bausch & Lomb Inc.",
    "facility": { "location": "Tampa, FL", "fei": "1000113778" },
    "warning_letter_date": "2026-09-04",
    "drug": { "name": null, "redaction_note": "product name redacted (b)(4)" },
    "is_sterile_product": true,
    "violation_categories": ["CGMP", "sterility", "aseptic_processing", ...],
    "contamination": { "present": true, "organisms": ["Serratia marcescens", ...] },
    "recall_concern": true
  }

This was built as a learning project: how to use TypeSafe's Jev model — a new kind of AI that makes decisions instead of writing text — to solve a real problem in the pharmaceutical / regulatory world, where being wrong has consequences and "the AI hallucinated it" is not an acceptable answer.

The problem, and why it's interesting

Every week the FDA publishes Warning Letters — public notices telling a drug maker what they did wrong. They're long, inconsistent, and full of legal boilerplate. People who track drug safety read thousands of them by hand.

The obvious idea: "just ask ChatGPT to pull out the drug name and the violations." It works… until it doesn't. Here's the trap that makes pharma different:

FDA often redacts the product name as (b)(4) — a legal blackout for trade secrets. A text-writing AI, asked "what's the drug?", will happily invent a plausible-sounding answer. In a regulatory database, a confidently wrong drug name is worse than no answer at all.

So the whole project is built around one rule:

The AI is never allowed to write an answer. It can only choose one.

Meet Jev: a model that decides, not writes

Jev (by TypeSafe) is a "System One" model — think of it as programmable common sense. You don't prompt it for a paragraph. You hand it some context and ask typed questions, and it hands back typed answers with probabilities:

• "Which of these is the drug?" → it picks one option (or unknown)
• "Is this a sterile product?" → 0.99 (yes)
• "How severe is this?" → 2.8 out of 3

Because it can only pick from options you supply, it cannot hallucinate a drug name. If the letter redacts the product and nothing supports a choice, it picks unknown — and it does, reliably. That single property is why this approach fits pharma.

This is what "classifier" means here: not a black box that guesses, but a set of small, auditable decisions — each with a probability you can inspect.

How it works (in plain terms)

Three steps, and the AI only touches the middle one:

1. Read — plain code scans the letter and pulls out every candidate: the company, dates, organisms, and every product name actually written in the text. Code does the reading; nothing is invented.
2. Decide — Jev looks at the candidates and makes bounded calls: which one is the drug, is it sterile, what violations apply, how serious. It only selects and judges — it never writes a new name.
3. Assemble — code copies the chosen answers into a clean object, and flags anything the model was unsure about for a human to double-check.

Every product name in the output is a word that appears verbatim in the letter. If it's not in the letter, it doesn't make it into the answer.

Every answer comes with a probability

This is the part that makes Jev different: it never just says "drug: yes." It says "drug: yes, 80%" — and the tool keeps the number, right next to the yes/no. You get to see how sure the model was about every single call.

Here's real output from an Abbott medical-device letter (the FreeStyle Libre glucose monitor), trimmed to the interesting parts:

Which named things are actually a subject of the violations? — a probability each:

| Candidate found in the letter | Subject? | Jev's probability | |---|---|---| | FreeStyle Libre 3 | ✅ yes | 99% | | iCGM | ✅ yes | 98% | | FreeStyle Libre | ✅ yes | 98% | | FreeStyle Libre 2 | ❔ unsure | 67% → flagged for review | | "Part 820" (a citation, not a product) | ❌ no | 31% |

Each violation judgment is a probability too:

| Question to Jev | Probability | Result | |---|---|---| | Is there a CGMP violation? | 98% | ✅ fires | | Is it a sterile product? | 3% | ❌ no | | Any contamination? | 5% | ❌ no | | Any recall concern? | 50% | ❔ too close to call → flagged |

And picking the specific drug carries its own confidence:

"drug": {
  "name": "FreeStyle Libre Flash and Continuous Glucose Monitoring",
  "confidence": 0.45          // several product names tied — Jev wasn't certain which
}

The rule the tool applies is simple and comes straight from the TypeSafe playbook: a confident answer (98%) is accepted automatically; a wobbly one (anywhere from 30–70%) is routed to a human instead of guessed. Every uncertain call lands in a review list with its exact number and the reason:

"needs_review": true,
"review": [
  { "field": "drug", "value": "FreeStyle Libre Flash…", "certainty": 0.43,
    "reason": "drug selection confidence below 0.65" },
  { "field": "has_recall_concern", "certainty": 0.50,
    "reason": "probability in the uncertain band [0.30, 0.70]" }
]

Nothing is a bare yes/no. Every yes/no has the number it came from sitting right beside it — and a full _jev block dumps the raw probability distribution over every candidate, so any decision can be audited after the fact.

How well does it work?

Tested on a corpus of 230 real FDA Warning Letters spanning every type — drug manufacturing, compounding pharmacies, medical devices, biologics, dietary supplements, veterinary. Crucially, the numbers below are on a held-out set of 48 letters the system was never tuned against — the honest test of whether it generalizes.

| What it extracts | Accuracy (unseen letters) | |---|---| | Company name | 100% | | Letter date | 100% | | Facility ID (FEI) | 100% | | Document type | 100% | | "Organisms linked to complaints?" | 100% | | Facility location | 92% | | Sterile product? | 90% | | Recall concern raised? | 94% | | Violation categories | 0.93 (F1) | | Products found in the letter | 0.98 (F1) | | The specific drug | 85% | | Every subject product | 0.81 (F1) |

The whole 230-letter corpus is classified in about a minute, and it runs entirely on TypeSafe — no other AI service in the loop.

And the headline result — the reason the project exists:

On letters where the product name is redacted (b)(4), the model returns unknown, every time, instead of inventing a drug. It even declines when handed a list of plausible decoy products. That's the discipline a text-writing model can't give you.

It also knows when a letter isn't about a drug

FDA sends Warning Letters about food, produce, and sanitation too. The tool labels every letter with what it regulates — drug, biologic, device, compounding, food_or_supplement, veterinary — and when a letter is really about, say, a syrup maker's sanitation, it reports regulated_product: "food" and leaves the drug field empty instead of pretending a food is a drug. Knowing the limits of your own scope is part of being trustworthy.

Try it

You'll need Node.js 20+ and a free TypeSafe API key from console.typesafe.ai.

The quickest way — the command-line tool

Point it at a URL or a file and it prints a readable summary:

export TYPESAFE_AI_API_KEY=your-key-here

# classify straight from an FDA warning-letter URL
npx classify-warning-letter https://www.fda.gov/.../warning-letters/<slug>

# or a letter saved locally
npx classify-warning-letter ./path/to/letter.txt

You get a digest like this:

FDA Warning Letter — Bausch & Lomb Inc (Tampa, FL · FEI 1000113778)
2026-09-04 · CDER · regulated as: drug
Drug:        (none) — product name redacted (b)(4) in the letter
Violations:  CGMP, sterility, aseptic_processing, environmental_monitoring
Sterile: yes   Contamination: yes (Serratia marcescens…)   Recall: yes
⚠ Needs human review:
  - has_recall_concern  (50%)  probability in the uncertain band [0.30, 0.70]

Add --json for the full structured object, --openfda to enrich the drug from openFDA, or --extract-only for a deterministic pass that needs no API key:

npx classify-warning-letter <url|path> --json
npx classify-warning-letter <url|path> --extract-only

Use it as a Claude skill

There's a bundled /classify-warning-letter skill: give Claude a URL or path and it runs the classifier and explains the result. Copy .claude/skills/classify-warning-letter/ into your own ~/.claude/skills/ to use it anywhere.

From a local checkout

npm install
cp .env.example .env         # paste your key into TYPESAFE_AI_API_KEY
npm run demo                 # classify the bundled Bausch & Lomb letter
npm run start -- <url|path>  # classify anything

Use it in your own code

import { classifyWarningLetter } from "./src/index.js";

const result = await classifyWarningLetter(letterText);

console.log(result.drug.name);            // the product, or null if redacted
console.log(result.violation_categories); // ["CGMP", "sterility", ...]
console.log(result.needs_review);         // true if a human should double-check
console.log(result.review);               // exactly which answers were uncertain

If the product name is blacked out, result.drug.name is null and result.drug.cross_reference_leads offers possible matches for a human to verify — never asserted as fact.

What's inside

| File | What it does | |------|------| | src/cli.ts | The classify-warning-letter command (URL or path → digest/JSON) | | src/resolve.ts | Turns a URL or a file path into letter text + metadata | | src/fetch.ts | Fetches an FDA page and parses it to clean text (shared with the corpus tool) | | src/extract.ts | Reads the letter: company, facility, dates, organisms | | src/enumerate.ts | Finds every candidate product name written in the letter | | src/redaction.ts | Works out what the (b)(4) blackouts are hiding | | src/citations.ts | Maps legal citations (21 CFR …) to violation types | | src/classify.ts | The Jev calls — the only place the model is used | | src/assemble.ts | Builds the final answer and flags uncertain fields | | src/review.ts | Decides which answers a human should double-check | | eval/ | 230 labelled letters + a scoring harness (npm run eval) |

Honest limitations

• It's a research/learning project, not a certified regulatory tool. Treat its output as a fast first pass, not the final word.
• The specific drug is the hardest field (~85%): device and biologics products have unusual names, and some letters genuinely name several products. That's why every answer carries a probability and an is uncertain flag — the system is built to say when it's not sure rather than guess.
• Extraction is tuned for FDA's letter formats; a very different document may need new patterns.

What I took away from building this

• "Selection, not generation" is a real design pattern. For anything where a wrong answer is costly, having the model choose from evidence instead of writing prose changes what you can trust.
• Probabilities are a feature, not decoration. Because every answer comes with a confidence, the system can route the shaky ones to a human — which is how you make AI you can actually rely on.
• Keep the code in charge. The model supplies judgment; ordinary code does the counting, copying, and rules. That division is what keeps the output honest.

Built while learning TypeSafe / Jev. 🤖