Handwriting Number Recognition

Recognize handwritten numbers drawn on an HTML canvas. Built to analyze answers to math questions, it recognizes the digits 0 to 9, individually or as a group like 1234.

const analyzerResult = drawingAnalyzer.analyze(userDrawing);
console.log(analyzerResult.characterString); // "1234"

About this package

This package was built by the same developer who created @edumedia/handwriting-character-recognition at EduMedia, working together with Claude Opus 5.

It is a ground-up rewrite of the recognition engine, using the original package as both the reference implementation and the benchmark to beat. The original supplied the problem definition, the CharacterDrawing API, the hand-collected dataset in public/data.json, and the accuracy bar to clear. Every change here was validated against it.

The result is a successful case of applying AI to boost the outcome of an existing project: the API stayed compatible, the accuracy on grouped numbers went from 54.4% to 95.7%, and recognition got roughly 40x faster.

The gains came from measuring real failures rather than guessing. Several features in this package exist because a specific misread was traced to its geometric cause and the fix was calibrated on the dataset before being written: the 3-vs-7 curve counter, the figure-eight detector for 8, and the deslanting pass all started as a single wrong answer from a real user.

Operators (<, >, =) are not supported here. The original package handles those; this one is numbers only.

Install

npm i @edumedia/handwriting-number-recognition

Usage

Import the DrawingAnalyzer, give it a CharacterDrawing, and do what you want with the result.

import {
  DrawingAnalyzer,
  type CharacterDrawing,
} from "@edumedia/handwriting-number-recognition";

// Create your DrawingAnalyzer instance
const drawingAnalyzer = new DrawingAnalyzer();

// Initialize a drawing
const userDrawing: CharacterDrawing = {
  id: "user", // only used for logging, you can write anything here
  strokes: [],
  strokeWidth: 10, // needed to tell whether two lines are touching
  // Dimensions of the canvas on which the strokes are drawn
  width: 280,
  height: 280,
};

// Let your user fill `userDrawing.strokes` with a great UI

// Analyze the drawing
const analyzerResult = drawingAnalyzer.analyze(userDrawing);

// Do something with the result
console.log(analyzerResult.characterString);

No data file to host. The template library is compiled into the bundle at build time, so there is nothing to download, no /data.json to serve, and no network request at startup. The package has zero runtime dependencies and works in any JavaScript environment, browser or server.

Collecting strokes

A stroke is one continuous pen trace: everything between pointerdown and pointerup. Push the raw [e.clientX, e.clientY] of each event; the library is translation invariant and does its own normalization, so do not offset or scale the points yourself.

let stroke: CharacterStroke = [];

canvas.addEventListener("pointerdown", (e) => {
  stroke = [[e.clientX, e.clientY]];
  userDrawing.strokes.push(stroke);
});

canvas.addEventListener("pointermove", (e) => {
  if (e.buttons !== 0) stroke.push([e.clientX, e.clientY]);
});

If your canvas is stretched by CSS, its backing store is a different size from its on-screen box, and clientX/clientY are in on-screen pixels. Keep every value in that same unit, or strokeWidth will be wrong relative to the points and segmentation will suffer:

const rect = canvas.getBoundingClientRect();
const scale = rect.width / canvas.width;
drawing.width = rect.width;
drawing.height = rect.height;
drawing.strokeWidth = lineWidth * scale; // lineWidth is in canvas pixels

This bit us in the demo: strokes were logged in client pixels while strokeWidth was in canvas pixels, off by a factor of 1.5.

Reading a single digit

Pass 1 as the second argument to force the whole drawing to be read as one digit. This is useful when you already know the answer is a single digit, because it removes any chance of a 4 being split into 41.

drawingAnalyzer.analyze(userDrawing, 1);

Result

type AnalyzerResult = {
  resultType: "character" | "debug";
  /** The whole recognized number, e.g. "42". "" if not confident enough. */
  characterString: string;
  /** Per digit details, left to right. */
  digits: DigitCandidate[];
  /** Mean confidence of the recognized digits, 0 -> 1. */
  confidence: number;
};

Each entry in digits carries its own character, confidence, boundingBox, the strokes it was built from, and the topMatches of the nearest-neighbour search.

When the analyzer is not confident enough, it gives no answer rather than a probable mistake: if any digit's confidence is below minConfidence (default 0.35), the whole characterString is noResultCharacter (default ""), so the user can simply draw again. The best guesses are still available in digits[i].character / digits[i].confidence. The same "" is returned when nothing was drawn (then digits is empty).

The defaults are exported as NO_RESULT_CHARACTER and MIN_CONFIDENCE.

const result = drawingAnalyzer.analyze(userDrawing);
if (result.characterString === "") askToDrawAgain();

Options

const drawingAnalyzer = new DrawingAnalyzer({
  showLogs: false,
  maxDigits: undefined,
  noResultCharacter: "",
  minConfidence: 0.35,
  extraTemplates: undefined,
  multiDigitOneStroke: false,
});

| Option | Default | Description | | --- | --- | --- | | showLogs | false | Print debug information in the console and fill result.debug. | | maxDigits | auto | Upper bound on how many digits to look for. | | noResultCharacter | "" | characterString returned when the drawing is not read confidently, or is empty. | | minConfidence | 0.35 | Every digit must reach this confidence (0 -> 1) for the reading to be returned. 0 always returns the best guess. | | extraTemplates | none | Extra samples added to the built-in library, keyed by character. Lets you fine tune recognition at runtime without a rebuild. | | multiDigitOneStroke | false | Allow one stroke to be cut into several digits, for numbers written without lifting the pen. |

multiDigitOneStroke

Off by default, deliberately. Deciding where to cut a continuous stroke is guesswork, and in testing it split single digits far more often than it recovered a joined pair: it reads about half of one-stroke numbers correctly but drags overall accuracy down. Turn it on only if your users really do write numbers without lifting the pen.

extraTemplates

If recognition struggles with a particular style of handwriting, the quickest fix is to add examples of it. This works at runtime, with no rebuild:

const drawingAnalyzer = new DrawingAnalyzer({
  extraTemplates: {
    "1": [oneWithFlagAndBase, anotherOne],
    "7": [sevenWithCrossbar],
  },
});

Each value is an array of CharacterDrawing, the same shape you pass to analyze. Prefer this over a rebuild when the extra samples are specific to one deployment or one group of users.

Migrating from @edumedia/handwriting-character-recognition

The API is compatible. Change the import, and delete the data hosting.

- import { DrawingAnalyzer } from "@edumedia/handwriting-character-recognition"
+ import { DrawingAnalyzer } from "@edumedia/handwriting-number-recognition"

new DrawingAnalyzer(), analyze(drawing) and result.characterString all behave the same, so most apps need no other change.

What to check:

• Remove /data.json from your public folder and drop the dataPath option. It is no longer read. Passing it is harmless but does nothing.
• onlyNumbers is gone. This package is numbers only. If you need <, > or =, keep using the original package.
• Do not feed it operators. They are not rejected, they are misread as digits: a > comes back as "7", a = as "77". There is no confidence cliff to detect this on, so only call analyze when you expect a number.
• Unsure readings return "". The original returned "-" for an unrecognized digit (e.g. "4-"); this one blanks the whole number when any digit is below minConfidence. Pass noResultCharacter: "-" if your app tests for "-".
• analyze(drawing, 1) now works. It threw in the original.
• Your input is no longer mutated. The original modified the strokes you passed in; this one treats them as read-only, so you can reuse or re-analyze a drawing safely.
• result.digits replaces result.guessResults. The per-digit data is richer and always present, not only in debug mode.

Accuracy

Measured on the 1340 single-digit samples in public/data.json, plus 395 synthetic multi-digit numbers assembled from them (npm run eval).

| | This package | Original | | --- | --- | --- | | Single digit, leave-one-out | 98.43% | n/a¹ | | Single digit, in-library | 99.85% | 99.10% | | Grouped numbers | 95.95% | 53.67% | | Time per recognition | ~2.4 ms | ~91 ms |

¹ The original loads the whole dataset at runtime, so no sample can be held out from it. Only the in-library row is a like-for-like comparison, and it flatters the original: every sample it is tested on is already in its library.

Leave-one-out is the honest number for this package. Each sample is recognized with itself removed from the library, so it measures performance on handwriting the engine has never seen.

A reading blanked for low confidence ("") counts as a miss in these numbers, so they include the cost of refusing to guess (6 of the 1340 leave-one-out samples, 1 of the 395 grouped numbers).

Breakdown of the grouped-number benchmark:

| Case | This package | Original | | --- | --- | --- | | 2 digits, spaced | 95.52% | 61.19% | | 2 digits, touching | 97.01% | 56.72% | | 2 digits, overlapping | 95.52% | 59.70% | | 3 digits | 95.52% | 47.76% | | 4 digits | 92.54% | 35.82% | | 8 drawn as two circles | 100.00% | 61.67% |

Most of the gap is segmentation. The original decides where to split a drawing from its geometry and then classifies each piece. This package searches groupings and scores them by how well they actually recognize, so the classifier gets a vote in where the cuts go. That is why touching and overlapping digits hold up, and why an 8 drawn as two separate circles is not read as 00.

Building with your own data

The library is generated at build time, so adding data means regenerating it. Use this when you want your samples baked into the package for everyone; use extraTemplates when they are specific to one deployment.

1. Get the data in place

public/data.json maps a character to an array of drawings:

{
  "0": [
    {
      "id": "0_1703a17d-002c-4f68-b40c-94ff662bb2fb",
      "strokes": [[[53, 36], [54, 38]]],
      "strokeWidth": 10,
      "width": 215.5,
      "height": 215.5
    }
  ],
  "1": []
}

Each drawing is exactly the CharacterDrawing you pass to analyze, so you can record real user input straight into this format. Keys other than 0-9 are ignored, which is why the original dataset's <, > and = can stay in the file.

There are two ways to gather drawings, and tooling to fold either into the dataset.

Bulk collection. npm run dev, then open /train/. The page names a random digit, the tester draws it and presses Next, over and over. The label is the digit that was asked for, so it does not depend on the recognizer being right. Download samples writes a training-*.json; drop it in new-data/. Worth passing around on phones and tablets: the dataset's weak spots are styles it has never seen, not sample count.

Mistake reports. The analyzer demo has a recorder: when the answer is wrong, type what you meant and record it. Exports go in feedback/. Far fewer samples, but each is a drawing that actually failed, so it is worth more per sample.

npm run visualize              # draw the pending samples in the terminal
npm run visualize -- --compare # next to the closest sample already stored
npm run test-data              # what adding them would change, nothing written
npm run add-data               # dry run: report what would be added
npm run add-data -- --write    # commit them, then archive the file

npm run add-data -- --from-feedback --write   # the feedback/ folder instead

Both commands read new-data/ by default and take --from-feedback to read feedback/. They accept single digits only, strip stray taps, and skip anything already stored, so re-running is safe.

test-data adds the samples in memory, re-runs every benchmark, names the samples that break a reading that used to be right or look mislabelled, and prints the --exclude=file#index,... to hand to add-data --write.

Always eyeball npm run visualize before writing: a mislabelled drawing becomes ground truth, and a wrong template does more damage than a missing one.

2. Regenerate the library

npm run train

This canonicalizes every sample and packs it into src/library.generated.ts as a base64 blob. Do not edit that file by hand.

3. Check nothing regressed

npm run eval
npm run feedback   # the cases the new samples were meant to fix

Watch the leave-one-out figure, not the in-library one: new samples always recognize themselves, so in-library accuracy will look good even if the addition made things worse in general.

Expect leave-one-out to fall when you add feedback data. Every recorded sample is one the recognizer got wrong, so adding them makes the pool harder and drags the average down even though the engine improved. Adding the 11 samples from one real session moved it from 98.38% to 97.87%, yet the 741 samples already in the dataset still scored exactly 98.38% and the reported failures went from 19/40 to 29/40. Judge a data addition by npm run feedback and by whether previously passing samples still pass, not by the headline average.

Useful flags:

node scripts/evaluate.mjs --only=mine     # skip the reference comparison
node scripts/evaluate.mjs --multi=400     # how many multi-digit cases
node scripts/evaluate.mjs --seed=777      # different synthetic sample
node scripts/evaluate.mjs --verbose       # list every failure

4. Build

npm run build

See Releasing to publish it.

Notes on adding data

• Balance matters more than volume. The dataset's weakest digits are the ones with the fewest style variants, not the fewest samples. A 1 written with both a flag and a base makes up under 2% of samples and is still the most common failure.
• Re-run npm run train after touching anything in the recognition pipeline. Templates are stored canonicalized, so a change to src/canonical.ts invalidates the generated library.
• Keep strokeWidth honest. It is how the engine decides whether two lines touch. A wrong value affects segmentation.

Development

npm run dev        # demo, side by side with the reference package
npm run tsc        # type check
npm run visualize  # draw pending collected samples in the terminal
npm run add-data   # fold collected drawings into public/data.json
npm run train      # regenerate the template library from public/data.json
npm run eval       # accuracy benchmarks
npm run feedback   # replay recorded misreads
npm run build      # build the package

The demo at npm run dev runs both engines on the same drawing, which is the fastest way to tell a genuine bug from a hard sample. It also records misreads to a JSON file that npm run feedback replays, so a fixed case stays fixed.

Releasing

Releases are manual, with changesets:

npx changeset                 # describe the change, pick patch / minor / major
npm run release:version       # bump package.json, update CHANGELOG.md
git add -A && git commit -m "release x.y.z"
npm login                     # once, with an account that can publish to @edumedia
npm run local-release         # tsc + build + check-exports, publish, tag vx.y.z
git push --follow-tags

Commit the version bump before local-release: changeset publish tags the current commit, so the tag must point at the commit that has the new version. For the very first publish (1.0.0, already in package.json and CHANGELOG.md) skip the first two steps: changeset publish publishes any version not yet on npm.

check-exports runs attw on the packed tarball: it checks that ESM and CommonJS consumers both get the code and the types (dist/index.d.ts and dist/index.d.cts). Do not publish if it reports a problem.

License

MIT