Schemistry: Type-Safe Schema Composition

For anyone consuming this package or working on it. Sections 1 and 2 are what bites a stranger on their first day; sections 3 onward are the design.

1. Importing, and Adding Your Own Keywords

A subpath specifier names a FILE under dist, without dist/ and without an extension

package.json maps "./*" to ./dist/*.js, so the package supplies the dist/ prefix and the .js suffix for you. Write either yourself and it is applied twice.

import { Projection } from '@idealic/schemistry/Data/Projection'; // ✅

The prefix and the extension are independent mistakes, and each one alone is enough to break the import:

| what you write | what it resolves to | | --- | --- | | @idealic/schemistry/dist/Data/Projection.js | dist/dist/Data/Projection.js.js ❌ | | @idealic/schemistry/dist/Data/Projection | dist/dist/Data/Projection.js ❌ | | @idealic/schemistry/Data/Projection.js | dist/Data/Projection.js.js ❌ | | @idealic/schemistry/Data/Projection | dist/Data/Projection.js ✅ |

Dropping both is necessary and not sufficient — the specifier must also name a file rather than a directory. "./*" is a pattern, and a pattern does no directory or index lookup. So a correctly-spelled directory fails too, and it fails without doubling anything:

| what you write | what it resolves to | | --- | --- | | @idealic/schemistry/Schema | dist/Schema.js — no such file ❌ | | @idealic/schemistry/Data | dist/Data.js — no such file ❌ | | @idealic/schemistry/Schema/index | dist/Schema/index.js ✅ | | @idealic/schemistry/Schema/Schema | dist/Schema/Schema.js ✅ |

Every failure above is the same one, because resolution SUCCEEDS in every row — the pattern always produces a path — and only the load fails:

Error [ERR_MODULE_NOT_FOUND]: Cannot find module
  '…/node_modules/@idealic/schemistry/dist/dist/Data/Projection.js.js'

Read the path in that message against dist, rather than looking for a doubled segment. A doubled path means you wrote dist/ or .js; an undoubled path that still fails means you named a directory, or a file that is not there. Both report as a missing module, which is why this reads like a broken install and is not one.

Most of the library is re-exported from the package root, so reach for a subpath only when you want a module the root does not carry.

Adding your own keyword: put the augmentation in a MODULE

Adding a schema keyword means teaching the compiler and teaching the runtime. Both go in one file, and that file must be a module — it needs a top-level import or export of its own:

import { Schema } from '@idealic/schemistry';

declare module '@idealic/schemistry/Schema/Schema' {
  namespace Schema {
    interface ExtrasRegistry {
      db?: { table: string };
    }
  }
}

Schema.registerExtra('db', {
  shape: { type: 'object' },
  requirement: 'an object naming the table it constrains',
  export: false, // does not follow a SOURCE field into a projection's derived schema
});

That import is load-bearing twice over, which is why the augmentation and the registration belong in the same file. It binds the Schema value that registerExtra is called on, and it makes the file a module — and in an ambient file, where neither is true, the augmentation does not work.

Put the same declare module block in an ambient .d.ts — no top-level import, no export, which is otherwise the natural home for one — and it fails silently. The namespace stays intact and your keyword is simply absent, so the error is the one you would get from writing no augmentation at all:

error TS2353: Object literal may only specify known properties,
  and 'db' does not exist in type 'Readonly<{ … }>'

Point that same ambient block at the package root — declare module '@idealic/schemistry' — and it fails loudly instead, taking the whole namespace with it:

error TS2694: Namespace '"@idealic/schemistry".Schema'
  has no exported member 'Composite'.

Neither specifier rescues an ambient file, and there is no version of this where the specifier is the fix. If the augmentation must live in a standalone .d.ts, add a bare export {}; to it — that alone makes both specifiers work. In a file that already imports something, both specifiers work too, and @idealic/schemistry/Schema/Schema is the one to reach for, being the module that actually declares ExtrasRegistry.

Getting the specifier itself wrong is the one mistake here that announces itself:

error TS2664: Invalid module name in augmentation,
  module '@idealic/schemistry/dist/Schema/Schema.js' cannot be found.

⚠️ TS2664 also fires on a correctly-spelled specifier in an ambient file under node16/nodenext. Before rewriting the specifier, check whether the file is a module — that is the more common cause of the two.

Import values from the package root, not from the defining module. Augmenting @idealic/schemistry/Schema/Schema costs nothing, but importing from it drags a couple of dozen TS2694 errors out of the library's own declarations under skipLibCheck: false. import { Schema } from '@idealic/schemistry' is clean, and the two go together in the snippet above.

Do not stop at the type. interface ExtrasRegistry teaches the compiler; Schema.registerExtra teaches the runtime. With only the first, the keyword typechecks and the walker still declines it — getSchemaVerdict answers undecidable, with the keyword named:

`db` is not a keyword this library knows, so nothing was compared against it
 — register it with `Schema.registerExtra` or remove it

Call registerExtra at module top level, before anything can validate. A registration that runs inside a lazy initializer or behind a dynamic import() can lose the race against the first validation, and an unregistered keyword is a refusal rather than a silent pass. Registration is per module instance, so a tree that resolves this package twice gets two registries and a keyword registered in one is invisible in the other.

2. Projections: a Nullable Node Carries No Shape

In a Projection target, adding "null" to a node's type changes what that node does — not just what it permits.

• { type: 'object', jsonPath: 'credential_ref' } — no properties of its own — lifts, and inherits properties and required from the source node it points at.
• { type: ['object', 'null'], jsonPath: 'credential_ref' } — no properties of its own — pulls the value whole and derives no structure at all.

Both targets below are compiled against one source node, so the target's type is the only thing that differs:

// source
{ "type": ["object", "null"],
  "properties": { "id": { "type": "string" }, "kind": { "type": "string" } },
  "required": ["id"] }

// derived from target { "type": "object", "jsonPath": "credential_ref" }
{ "type": "object",
  "properties": { "id": { "type": "string" }, "kind": { "type": "string" } },
  "required": ["id"] }

// derived from target { "type": ["object", "null"], "jsonPath": "credential_ref" }
{ "type": ["object", "null"] }

So making a node nullable silently swaps it from inherit the source's shape to carry no shape at all. Nothing warns, and both compile.

It is correct rather than accidental. A pull is the only route that returns a null as a null; a node walked as a container would answer {} or [] where a null belongs, which trades a missing shape for a fabricated value. But it is not guessable from the outside, and the cost lands on whoever reads the derived schema later — a redactor, a form renderer, a downstream stage. And it is worse than finding a node that claims nothing: a reader that searches the derived schema for a keyword — a redaction gate looking for the fields it must hide — finds no rows, which is the same answer it gives for a record that genuinely carries none. The node's silence is indistinguishable from a clean result.

You cannot recover the shape by writing it out. A nullable node carrying properties is refused, and the refusal does not depend on how you spell it — with a pointer, without one, or with required:

// ❌ refused, every spelling
{ type: ['object', 'null'], jsonPath: 'credential_ref', properties: { id: { type: 'string' } } }
{ type: ['object', 'null'], properties: { id: { type: 'string', jsonPath: 'credential_ref.id' } } }

// Projection refused at 'credentialRef': 'properties' declares a sub-schema and this node
// declares 'type' object | null, so the walk did not enter it as a container — the declaration
// would be dropped and the derived schema would permit what it rules out. Name ONE type for the
// walk to map. Dropping 'properties' to take the value whole also compiles, and it is the lossier
// of the two: the derived node then carries no 'properties' at all, so a reader accounting
// keywords over it finds nothing to report where the source declares something

// ⚠️ names one type and the structure comes back, and on a NULL record it applies null and its
//    own pair reads invalid — a contradiction this library's getSchemaVerdict will report.
{ type: 'object', jsonPath: 'credential_ref' }

// ⛔ names one type and the structure comes back, and on a NULL record it applies {} and its own
//    pair reads VALID. Nothing reports it. Quieter than the line above and therefore worse.
{ type: 'object', properties: { id: { type: 'string', jsonPath: 'credential_ref.id' } } }

// ✅ for the validator: stay nullable and take the value whole — the only shape here whose
//    applied value always satisfies the schema derived beside it.
// ⛔ and NOT for a keyword-accounting reader: the derived node is {"type":["object","null"]} and
//    carries no properties, so a walk counting a keyword over it answers 0 where the ⚠️ lift
//    above answers 1. It does not find a node claiming nothing; it finds nothing to report.
{ type: ['object', 'null'], jsonPath: 'credential_ref' }

Each marker above is per line, deliberately. A marker covering several shapes is a claim about every one of them and is only as true as the weakest — and the two shapes under the old shared ⚠️ differ in the property that matters most, which is whether anything says a word when they go wrong.

Spelling the same idea as a hand-written anyOf of an object arm and a null arm is refused too, but for an unrelated reason — cannot determine which arm to build: the arms share no const-discriminated property — so it is not a way around this.

Naming one type buys the structure and gives up the null

Neither of the two shapes marked ⚠️ and ⛔ above is refused, at compile or at apply. A target may demand a bare object from a source declaring object | null, deliberately — ["object", "null"] is how a source models an absence it means, and refusing every target that narrows one would be the library calling that design malformed. So the two lines compile, they do the right thing for every record that carries a value, and they part company on a record whose credential_ref is null.

The lift — { type: 'object', jsonPath: 'credential_ref' } — derives {"type":"object","properties":{…},"required":["id"]} and applies null. Hand that pair to this library's own getSchemaVerdict and it answers invalid, {"constraint":"type","argument":"object","value":null}: the projection contradicts the schema it derived in the same call. This is not an object quirk — a { type: 'string' } target over a ["string","null"] source, and a { type: 'number' } target over ["number","null"], each apply null under their own bare type and earn the same verdict.

The written-out walk — { type: 'object', properties: { id: … } } — derives {"type":"object","properties":{"id":{"type":"string"}}} and applies {}. That pair is valid, which makes it the quieter of the two: an empty object standing where the record said there was nothing, indistinguishable downstream from a credential reference that genuinely carried no id. It is the fabricated value this section warns about above, arriving inside one of the section's own examples. Adding required: ['id'] trades the fabrication for a refusal at apply time — and that refusal reports the source contradicts its own declaration, which is the wrong party: the source declared object | null and honestly held a null.

So the choice is nullability or structure, and it is a choice about which reader you are protecting. The library refuses to let you claim both on one node at compile time, and it does not refuse a bare type over a nullable source — so if you name one type, keeping nulls out of apply is yours to do, by a guarantee the source actually makes or by a guard ahead of the call. If you cannot promise that, take the value whole — and know what taking it whole costs, because it is not free and it is not loud. The derived node claims nothing, and a reader who looks at the node can see that it claims nothing. A reader who searches for a keyword cannot: Schema.fieldsCarrying over {"type":["object","null"]} answers zero rows, the same answer it gives for a schema that carries none.

That function does have a third answer for I could not see inside this — hand it {type:['object','null'], properties:{…}} and it refuses rather than under-reporting. It cannot fire here, because the projection dropped the properties before the walk ever saw them. The decline exists; taking the value whole is what removes its trigger. So the rule is: take the value whole when a human or a renderer will read the derived node, and name one type — with a guard ahead of apply — when a keyword-accounting gate will walk it.

3. Philosophy & Core Concepts

The Single Source of Truth

The primary mission of Schemistry is to concentrate power onto the Schema, making it the single source of truth for the entire application. By defining data structures once in the schema, we drive everything else: runtime validation, TypeScript types, LLM integration, and application logic.

While this approach provides security benefits (acting as a "sandbox" that limits attack surfaces), the core goal is schema-driven development. This ensures that runtime data and build-time types are always synchronized without manual intervention or code generation steps.

The "From Schema" Paradigm

A central mechanism of the library is the "from schema" tool—a TypeScript macro that computes types directly from schema definitions.

• Dynamic Type Inference: We compute types dynamically from the JSON schema constant definition.
• Type Safety: These inferred types are used throughout the codebase, ensuring that the TypeScript layer perfectly reflects the schema definitions.

The Schema Registry

Schemistry includes a built-in mechanism to register schemas, creating a centralized library of definitions.

• Flat Structure: Schemas exist at the top level and reference each other via IDs ($ref), preventing deep nesting hell.
• Automatic Resolution: All internal functions automatically handle these references, simplifying schema management.

Canonical Schema & Semantic Divergence

We embrace the concept of a Canonical Schema.

• Internal Representation: Internally, a schema can be unwrapped into an expression optimized for library functions.
• Semantic Integrity: We accept that the TypeScript type describing a schema (inferred via FromSchema) and the schema's runtime structure may diverge structurally, provided they retain the same semantic meaning.
• Example: A schema might be structurally defined as an allOf intersection, but its inferred type is a merged object. This divergence is acceptable because it simplifies usage while strictly enforcing the data shape.

Performance & Caching

Deep type inference is computationally expensive. To handle this:

• Caching: Helpers return objects with types pre-computed and cached.
• Reusability: Types are cached on the schema object itself. When chaining operations, we carry the cached type forward, avoiding the need to re-infer from scratch at every step.

4. Evolution: From Composite to Strict

The "Composite" Trap

Initially, the library used a composite, wide type for all schemas, assuming any schema object could have any property.

• The Problem: This prevented type narrowing. Mutually exclusive properties (like in discriminated unions) couldn't be correctly inferred because the wide type swallowed the specific constraints.

The Shift to Strictness

We shifted to a strict approach where schema.any is a union of distinct types (object, array, string, etc.).

• The Challenge: This broke legacy helpers designed for the composite model.
• The Resolution: We are moving away from the composite type entirely. Composite types should be "unwrapped" into their specific constituent types.

5. Roadmap: Type-Safe Architecture

Moving forward, our focus is on implementing robust, type-safe helpers using a new architectural strategy.

The ToSchema<T> Strategy

We are adopting a strategy that leverages the inherent commutativity and functional properties of schemas.

The Pipeline:

The transformation pipeline relies on the isomorphism between Schemas and TypeScript Types:

1. Extract (FromSchema): We lift the schema into the TypeScript type domain. While this may shed non-structural metadata, it preserves the structural truth of the data.
2. Operate: We apply standard, composable TypeScript operations (e.g., Intersection, Union, Pick, Omit).
3. Reconstruct (ToSchema): We lower the resulting type back into a Schema definition.

Schema.Intersection<A, B>(a: A, b: B): ToSchema<FromSchema<A> & FromSchema<B>>

Why This Works: Schemas act as a bridge between runtime values and static types. Because this relationship is commutative, we can confidently move between these domains. We prioritize structural correctness over metadata preservation, allowing us to treat schemas as pure, functional building blocks that can be composed, transformed, and reconstructed with mathematical certainty.

Goals: New Helpers & Extensibility

This strategy allows us to implement complex operations that were previously difficult or impossible to type correctly:

• Intersection & Union: combining schemas with full type fidelity.
• Pick & Omit: manipulating object schemas while retaining strict typing.

Maintainability: This approach significantly simplifies the implementation of new helpers. By focusing on the TypeScript type transformation rather than maintaining perfect schema structure alignment, we can create robust tools without the immense complexity of handling every schema permutation at the type level.

Partial Schema Support

Currently, our strict typing requires schemas to be fully defined (e.g., an object must explicitly have type: 'object'). JSON Schema, however, allows for "partial" definitions (e.g., defining properties without a type), effectively inferring the type from the context.

• Current Limitation: Our FromSchema implementation limits us; it does not infer types from partial objects. A schema with just properties is not treated as a real object, which breaks type inference.
• The Workaround: We currently require fully resolved objects and types (explicit usage of the type property everywhere).
• Internal State: We use a CouldBePartial type internally to mark places where partial support might be added.
• The Blocker: We need to determine if adding support for partial inference in FromSchema would increase TypeScript type complexity and depth to an impractical level.
• Future Goal: If feasible without severe performance penalties, we aim to enable partial support across the ecosystem.