tdcv2

v0.2.2

Published

13 days ago

Deterministic test-data generator: coherent records, exact proportions, and any text format you describe. Same seed, same bytes. Also usable as a faker — one call, one value — from the same data packs.

TDC — The Data Constructor

TDC and tdcv2 — two names, two things. TDC is the language: what a .tdc file contains and what the TDC001-style codes in error messages refer to. tdcv2 is the package and the command that run it. The v2 is there for a dull reason — tdc was already taken on npm and PyPI by unrelated packages — and it is not the version of the language: the package will still be called tdcv2 at 1.0.

Declarative test-data generator: deterministic, exact proportions, any text format you can describe.

You write a short config saying what the data is and how it should look. TDC produces it — as CSV, JSON, SQL, YAML, Markdown, or a format nobody has named yet, because the layout is something you spell out rather than something you pick off a list.

Three things separate it from a faker.

It repeats. The same config and the same seed produce byte-identical output — today, next year, on your laptop and on CI. A failing test that depends on generated data can be re-run and will fail the same way.

The proportions are exact, not approximate. Ask for 30% cancelled orders over a thousand rows and you get three hundred, not "about three hundred". The shares are apportioned over the whole run and then scattered across it, so the count is right and the arrangement still looks like data.

Records hold together. A child sequence can be declared against a slice of its parent — a distribution that applies only to the rows where the customer is a company, say — so the record is coherent rather than a row of independently plausible columns.

One more, for the curious: you can hand it an SVG or PNG curve and TDC will use the shape as the probability distribution.

Two ways to use it

They are different tools that happen to share one set of data, and most people need both at different moments.

Reach for a value. Import the library, call an address, get a string — the job a faker does. Nothing is tied to anything else.

import { tdc } from 'tdcv2';

tdc.person.lastName(); // Jones
tdc.common.finance.iban(); // DE62299399441396459682
tdc.country.usa.docs.ssn(); // 699209702

Describe a dataset. Write a config saying what the records are and how they should look, then generate as many as you want — as CSV, JSON, SQL, or a format you spell out yourself. This is where the exact proportions, the parent-child distributions and the coherent records live.

<tdc>
  <env count="1000" seed="demo" local="en">
    <sequence name="Status"><gen type="text" value="paid,refunded" percent="97,3"/></sequence>
    <sequence name="Refund" parent="Status.refunded"><gen type="number" value="5..500"/></sequence>
  </env>
  <block><line><data>${{Status}},${{Refund}}</data></line></block>
</tdc>

Exactly thirty of those thousand rows are refunds, and only those rows carry an amount. No sequence of loose calls gives you that.

The rest of this page walks both: one value first, then a first config.

Documentation

📖 nickliapin.github.io/tdcv2 — every implemented feature, with search and working examples, in English, Русский and Español.

Source, issues and the other four implementations: github.com/NickLiapin/tdcv2.

One config, five implementations

This package is the TypeScript reference implementation, and the one that also carries the CLI and the language server. The same config runs in Python, Java, C# and Rust and produces the same bytes — not "the same kind of data", the same bytes. A shared fixture suite holds all five to it on every change: 130 configs rendered byte for byte, plus the diagnostics each one reports, the command-line contract, and the Parquet output down to the file header. A config written against one implementation is a config that runs on any of them.

Use it when the data has to be reproducible across a team, a language boundary, or a year.

Install

npm install -D tdcv2

The tdcv2 command comes with it, and so does a starter set of data — the common, en and USA packs — which is enough for names, addresses, companies and identifiers out of the box.

Just one value, like a faker

The values come from the same data packs a config draws on, so the name in your unit test and the name in your million-row fixture come from one list.

import { tdc } from 'tdcv2';

tdc.person.lastName(); // Jones
tdc.person.male.firstName(); // Robert
tdc.company.industry(); // Pharmaceuticals

tdc.common.id.uuid(); // 3ff6ff76-6ea7-4fad-8b99-3075a14cc7e9
tdc.common.finance.iban(); // DE62299399441396459682
tdc.country.usa.docs.ssn(); // 699209702 — with its real check digits

tdc.lang.ru.person.lastName(); // after `npx tdcv2 pack add ru`

tdc.person.lastName.many(5); // [ 'Bush', 'Armstrong', 'Andrews', … ]
tdc.gen.number('18..80'); // 66

A dot in the code is a dot in the address: person.male.firstName here is person.male.firstName in a config and in the reference — one vocabulary, not two. A bare address reads against the active locale; common., country.<code>. and lang.<code>. name a pack outright. Pin a seed when the value should be part of the test rather than a variable in it:

const t = tdc.seed('demo');
t.person.lastName(); // Jones, today and next year

Every call is independent — nothing here ties one value to another. The moment two values have to agree, you want a config, which is the rest of this page. Full reference.

A first config

<tdc>
  <env count="3" seed="demo" local="en">
    <sequence name="Id"><gen type="increment" value="1"/></sequence>
    <sequence name="Name"><gen type="template" value="person.lastName"/></sequence>
  </env>
  <block><line><data>${{Id}},${{Name}}</data></line></block>
</tdc>

npx tdcv2 demo.tdc

1,Williams
2,Johnson
3,Smith

The same three names, every time, in every implementation — that is what the seed buys.

Data packs

A pack is the data: the name lists, cities, streets and locale rules that type="template" draws from. The starter set ships with the package; everything else is downloaded when you ask for it.

npx tdcv2 init                 # write a tdcv2.config.json, once per project
npx tdcv2 pack list            # what the registry has
npx tdcv2 pack add ru france   # download and wire up

Everything lands in one folder, at its own address path — tdcv2-packs/ru/…, tdcv2-packs/countries/france/… — and that folder is registered in your config once, however many packs you go on to install.

One registry, one tdcv2.config.json, one store, shared by all five implementations: a pack installed from here is a pack the others find. The full story is in the data-packs guide.

Using it from code

Library API

import { TDC } from 'tdcv2';

const config = `
<tdc version="0.1">
  <env count="3" seed="demo">
    <sequence name="Gender">
      <gen type="text" value="Male,Female" percent="50,50"/>
    </sequence>
    <sequence name="Code">
      <gen type="number" value="0000..9999"/>
    </sequence>
  </env>
  <block>
    <line><data>\${{_count}},\${{Gender}},\${{Code}}</data></line>
  </block>
</tdc>`;

const dataset = new TDC({ configString: config });

dataset.toString(); // "1,Female,1116\n2,Male,2111\n3,Male,1968\n"
dataset.writeFile('./out.csv');
dataset.toStream().pipe(fs.createWriteStream('./out.csv'));

// The rows as objects, ignoring the <block>/<line> layout.
dataset.toArray(); // [{ Gender: 'Female', Code: '1116' }, …]
for (const row of dataset.iterate()) {
  console.log(row['Gender']);
}
dataset.getAt(0); // { Gender: 'Female', Code: '1116' }

getAt reads a single record without building the ones before it, so asking for row nine million of a ten-million-row run costs one row's work.

preflight() returns a diagnostic when a run looks too large for the way you are about to produce it, and undefined when it does not — worth checking before a big one:

const warning = dataset.preflight({ output: 'streaming' });
if (warning) console.warn(warning.message);

CLI

After package installation:

npm install -D tdcv2
npx tdcv2 demo.tdc -o out.csv

From this repository during development:

npm --workspace typescript run build
node typescript/dist/cli/main.js fixtures/tdc_csv.xml --count 10 --seed demo

Use --data-path <dir> for configs that reference @data/... sources:

npx tdcv2 demo.tdc --data-path ./data -o out.csv

Parser API

import { parse, parseStrict, TdcParseError } from 'tdcv2';

// Non-throwing variant — inspect diagnostics
const { tree, diagnostics } = parse('<tdc version="0.01"><data>hi</data></tdc>');
if (diagnostics.length === 0) {
  // tree is a DocumentContext (ANTLR parse tree root)
}

// Strict variant — throws with all diagnostics on any error
try {
  const tree = parseStrict(source);
} catch (err) {
  if (err instanceof TdcParseError) {
    for (const d of err.diagnostics) {
      console.error(`${d.source} ${d.line}:${d.column}: ${d.message}`);
    }
  }
}

Working on the repository

Everything above is what the package gives you. This part is for a checkout of the repository.

git clone https://github.com/NickLiapin/tdcv2.git
cd tdcv2/typescript
npm ci
npm test

The parser is generated from the grammar the five implementations share, so npm test and npm run build regenerate it first — there is no separate step to remember. npm run check is the gate CI uses: lint, types, the whole suite, the shared fixtures, the documented examples, and a coverage floor.

| Script | What it does | | ----------------------- | --------------------------------------------------- | | npm run build | Regenerate the parser, compile to dist/ | | npm test | Regenerate, then run the suite once | | npm run test:watch | The suite, in watch mode | | npm run test:coverage | The suite with a coverage report | | npm run typecheck | Types only, no emit | | npm run lint | ESLint over source and tests (lint:fix to repair) | | npm run format | Prettier (format:check to check only) | | npm run check | All of the above, plus the fixtures — what CI runs | | npm run clean | Remove dist, coverage and the generated parser |

node ../scripts/five-ways.mjs runs all five implementations against the shared fixtures at once, which is the check that matters when the engine changes.

Built on: Node.js 20+, TypeScript 5.6 in strict mode, ANTLR4 (antlr-ng to generate, antlr4ng at runtime) over grammar/, Vitest, ESLint 9 and Prettier. The PRNG, the Parquet writer, the Snappy encoder and the date arithmetic are written here rather than pulled in, because no dependency's choice of rounding or compression is allowed to change the bytes.