@fefeding/typeorm-storage

🌐 Other languages: 简体中文

A TypeORM-compatible JSON file storage engine (a framework-agnostic, business-decoupled public package): it persists data as JSON files on disk in the form data/<table>/<primaryKey>.json, and provides a subset of TypeORM's DataSource / Repository, QueryBuilder, and an in-memory SQL evaluator, serving as a transparent persistence fallback layer for database-less scenarios.

This package is a standalone public package that does not depend on @midwayjs/* or any application-internal modules; the application-specific decisions needed to run it (directory resolution, non-entity table registration, etc.) are injected by the integrator via configuration.

Installation

npm install @fefeding/typeorm-storage
# or
pnpm add @fefeding/typeorm-storage

Runtime peer dependencies (must be installed by the host): typeorm@^0.3.31, reflect-metadata@^0.2.2.

Configuration Injection (required before first use)

The engine does not bundle any storage root directory or non-entity table registration strategy; you must call configureJsonStorage() to inject them before the first access:

import {
  configureJsonStorage,
  getJsonDataSource,
  createJsonRepository,
  sqlTableFor,
  FsStorageBackend,
  type JsonStorageBackend,
} from '@fefeding/typeorm-storage';

// Storage root (e.g. <data>/db) and non-entity table registration (tables accessed only via raw SQL, without an @Entity definition)
configureJsonStorage({
  dbDir: '/path/to/data/db',
  nonEntityTables: {
    some_raw_table: { primaryColumn: 'id' },
  },
});

• If dbDir is not injected and a collection tries to persist, the engine throws a clear error pointing you to call configureJsonStorage({ dbDir }) first. It does not silently fall back to an implicit directory.
• nonEntityTables is optional; once injected, isNonEntityTable / getNonEntityPrimaryColumn / getNonEntityUniqueColumns / listNonEntityTables return based on the injected data, and idempotent upserts in raw SQL dedupe by the injected primary/unique columns.
• Configuration supports partial, incremental injection with later writes overriding earlier ones; tests can call resetJsonStorageForTest() for isolation.

DataSource Options Alignment (drop-in replacement)

In addition to the configureJsonStorage() + getJsonDataSource() model ("inject first, then get the facade"), JsonDataSource also supports construction directly in the form of a TypeORM DataSourceOptions, acting as a drop-in replacement for new DataSource({ type: 'mysql', ... }):

import { JsonDataSource } from '@fefeding/typeorm-storage';

// Same shape as new DataSource({ type: 'mysql', host, ... }); type is fixed to 'json'
const ds = new JsonDataSource({ type: 'json', dbDir: '/path/to/data/db' });
await ds.initialize();                       // no-op, compatible with TypeORM chaining

const repo = ds.getRepository(SomeEntity);
await repo.save({ /* ... */ });

• At construction, type must be the literal 'json' (distinct from TypeORM's 'mysql' / 'postgres' driver types); passing any other value (e.g. 'mysql') throws immediately rather than silently falling back to the default fs directory.
• At construction, dbDir / backend / nonEntityTables / reloadTtlMs / doubleQuotedIdentifiers / enforceUniqueConstraints are injected into the global backend via configureJsonStorage(), so new JsonDataSource(options) and configureJsonStorage(options) + getJsonDataSource() are equivalent and coexisting — the three access paths (no-arg construction / getJsonDataSource() singleton / parameterized construction) all point to the same backend with a consistent on-disk directory.
• Pass-through fields are ignored: TypeORM DataSourceOptions fields such as name / entities / synchronize / logging / entityPrefix are ignored (only accepted as placeholders so existing config objects can be passed), and do not change the JSON persistence behavior.

Framework Integration (MidwayJS example): zero changes to existing TypeORM code

One of the selling points of this engine: business code needs no changes at all. Calls in your store / service such as @InjectEntityModel(Entity), @InjectDataSource('default'), and repo.save / find / query all stay exactly the same; you only need one component at application startup to rebind these two Midway TypeORM decorator keys to the json-orm facade / repository shim, and inject the storage backend based on the environment.

This package is framework-agnostic and does not contain a Midway component; the component code below is provided by the integrator (application side). It reuses Midway TypeORM's decorator keys (ORM_DATA_SOURCE_KEY / ORM_MODEL_KEY), so it loads mutually exclusively with the original @midwayjs/typeorm component without interference.

1. Write an integration component (application side, not in this package)

import { Configuration, Inject, MidwayDecoratorService } from '@midwayjs/core';
import { ORM_DATA_SOURCE_KEY, ORM_MODEL_KEY } from '@midwayjs/typeorm';
import {
  configureJsonStorage,
  createJsonRepository,
  getJsonDataSource,
  sqlTableFor,
} from '@fefeding/typeorm-storage';
import { APP_NON_ENTITY_TABLES, resolveJsonDbDir, isJsonStorageMode } from './storage/json-mode';

@Configuration({ namespace: 'jsonStorage' })
export class JsonStorageConfiguration {
  @Inject() decoratorService: MidwayDecoratorService;

  async onReady(): Promise<void> {
    // Only inject in JSON fallback mode (e.g. DISABLE_DB=true or no DB_HOST); DB mode still uses @midwayjs/typeorm
    if (!isJsonStorageMode()) return;

    // Inject the application-side storage config (storage root + business non-entity tables), before any engine access
    configureJsonStorage({
      dbDir: resolveJsonDbDir(),
      nonEntityTables: APP_NON_ENTITY_TABLES,
    });

    // @InjectEntityModel(Entity) → JSON Repository shim
    this.decoratorService.registerPropertyHandler(
      ORM_MODEL_KEY,
      (_propertyName: string, meta: { modelKey: Function }) =>
        createJsonRepository(meta.modelKey, sqlTableFor),
    );
    // @InjectDataSource('default') → JSON DataSource facade
    this.decoratorService.registerPropertyHandler(
      ORM_DATA_SOURCE_KEY,
      () => getJsonDataSource(),
    );
  }
}

Key points:

• registerPropertyHandler resolves @InjectEntityModel(Entity) to the json-orm JsonRepository shim, and @InjectDataSource('default') to the JsonDataSource facade. Their method signatures match TypeORM's, so business code is unchanged.
• This component loads mutually exclusively with the @midwayjs/typeorm component (choose one in src/configuration.ts based on isJsonStorageMode()): DB mode is fully unaffected, and only JSON mode loads this component.
• Non-entity tables (accessed only via raw SQL, without an @Entity definition) are registered via nonEntityTables, so ds.query('SELECT ... FROM t_raw_table') can also persist / read.

2. Local storage (default, zero-config)

Just inject dbDir, and the engine auto-constructs FsStorageBackend, persisting to <dbDir>/<table>/<primaryKey>.json (equivalent to the pre-refactor local JSON storage):

// Method A: injection (first line of component onReady)
configureJsonStorage({ dbDir: path.join(getProjectDataPath(), 'db') });

// Method B: drop-in construction (see "DataSource Options Alignment" above)
const ds = new JsonDataSource({ type: 'json', dbDir: path.join(getProjectDataPath(), 'db') });
await ds.initialize();

3. Switch to COS / object storage / other backends

Implement the JsonStorageBackend interface (all async) and inject it as backend; business code is still unchanged:

import { configureJsonStorage, type JsonStorageBackend, type JsonStorageFileStat } from '@fefeding/typeorm-storage';

class CosStorageBackend implements JsonStorageBackend {
  // All methods implemented via COS/S3 SDK; only the contract essentials are listed below
  async readFile(relPath: string): Promise<string | null> { /* GET object, return null on 404 */ }
  async writeFileAtomic(relPath: string, data: string): Promise<void> { /* atomic write: temp key + Copy/overwrite, or versioned overwrite */ }
  async readDir(relPath: string): Promise<string[]> { /* ListObjects enumerated by prefix */ }
  async deleteFile(relPath: string): Promise<void> { /* DELETE object */ }
  async mkdir(_relPath: string): Promise<void> { /* object storage has no directory concept, usually no-op */ }
  async stat(relPath: string): Promise<JsonStorageFileStat | null> {
    // Return a stat that reflects content changes for a "table directory"; prefer providing etag (e.g. object ETag/version),
    // which the engine uses for cross-process cache invalidation. Returning null declares "does not exist" and does not trigger a reload.
    return { mtimeMs: Date.now(), size: 0, etag: await this.dirEtag(relPath) };
  }
  async withLock<T>(relPath: string, fn: () => Promise<T>): Promise<T> {
    // Implement cross-process mutual exclusion with remote atomic operations (e.g. object storage If-Match / version lock / atomic rename),
    // and support orphan lock reclamation; a single process can approximate an in-memory mutex chain (see comments in examples/05-custom-backend.ts)
    return fn();
  }
  async nextId(relPath: string, key: string): Promise<number> { /* remote atomic increment */ }
  async ensureSeqAtLeast(relPath: string, key: string, value: number): Promise<void> { /* raise the sequence floor */ }
}

configureJsonStorage({
  backend: new CosStorageBackend(cosClient, { bucket: 'my-bucket', prefix: 'ai-work/db' }),
});

• backend takes precedence over dbDir; if neither is present, the first access throws (no silent fallback).
• The interface contract and a minimal usable implementation are in examples/05-custom-backend.ts (JsonStorageBackend / JsonStorageFileStat types can both be imported from this package).
• withLock must support orphan lock reclamation and use remote atomic operations to guarantee cross-process mutual exclusion; a single-process in-memory backend may approximate this (noted in the example), but a production object-storage backend must use a real remote lock.

4. Switch storage via configuration (recommended)

Extract the storage method into an environment variable / configuration instead of hard-coding, so neither business code nor component code needs to change:

// Application config (config.default.ts or a standalone storage config), decided at startup
const backendKind = process.env.JSON_STORAGE_BACKEND || 'fs'; // 'fs' | 'cos'

if (backendKind === 'cos') {
  configureJsonStorage({
    backend: createCosBackend(/* read bucket / region / credentials from config */),
  });
} else {
  configureJsonStorage({ dbDir: resolveJsonDbDir() });
}

This way: the original TypeORM business code is unchanged, and only the startup config layer decides between local JSON and object storage; combined with isJsonStorageMode() (DISABLE_DB=true or no DB_HOST), you can switch wholesale between a "database-less deployment" and "real MySQL". JsonDataSource's type: 'json' is only used for driver type discrimination and does not participate in backend selection.

Storage Backend (injectable, async)

All engine persistence I/O goes through the injectable JsonStorageBackend (async interface) and does not directly depend on fs: readFile / writeFileAtomic (internal tmp+rename atomic write) / readDir / deleteFile / mkdir / stat (cross-process cache invalidation detection) / withLock (cross-process critical-section mutual exclusion) / nextId (backend-level atomic increment) / ensureSeqAtLeast (explicit large-primary-key sequence floor raise).

The default backend is FsStorageBackend (fs/promises implementation, equivalent to the pre-refactor local JSON persistence). Injection methods:

// Method A: inject dbDir only (auto-constructs FsStorageBackend, backward compatible)
configureJsonStorage({ dbDir: '/path/to/data/db' });

// Method B: inject a custom backend (e.g. object storage COS/S3), takes precedence over dbDir
const cosBackend: JsonStorageBackend = createCosBackend(/* ... */);
configureJsonStorage({ backend: cosBackend });

All data-access methods (find / save / update / delete / upsert / query, the SELECT/INSERT/UPDATE/DELETE evaluation of native SQL, collection read/write, sequence allocation) are fully asynchronous, with no blocking synchronous I/O on the data path. Callers using the DataSource / Repository / EntityManager public APIs need no changes (these methods already return Promise).

Multi-process safety

• Unique sequencer: nextId() allocation completes a read-modify-write inside the backend's withLock critical section, strictly incrementing and unique across processes; when an explicitly written primary key is larger than the current sequencer, ensureSeqAtLeast raises it to avoid future collisions.
• Cross-process cache invalidation: each process keeps an in-memory full-table snapshot to support SQL scan performance; before reading, it checks for changes via backend.stat(table directory) (etag preferred; when neither side has an etag, use an increase in mtimeMs), and reloads if another process has changed it. Throttling is controlled by reloadTtlMs (configureJsonStorage({ reloadTtlMs }), default 0 means validate on every access); when >0, the cache can be reused within the TTL window (trade-off: you may briefly read a pre-change snapshot written by another process).
• Write path forced validation: write / delete / nextId are not throttled by reloadTtlMs; before writing, they always do one invalidation check first, ensuring the watermark of the record after writing is consistent with the in-memory snapshot, without masking writes from other processes.
• Single-row writes still use tmp+rename file-level atomicity (temp filenames include pid and a random component; concurrent writes to the same row produce no interleaved content and do not throw ENOENT), and introduce no whole-table write lock.
• Lock ownership validation: withLock identifies the holder with a token and validates the token before release; if a lock is judged orphaned and acquired by another, the original holder will not accidentally delete the other's lock (avoiding two processes entering the critical section simultaneously).

Public API Overview

| Category | Exports | | --- | --- | | Config | configureJsonStorage / resetJsonStorageForTest / JsonStorageOptions / NonEntityTableInfo / JsonStorageBackend / JsonStorageFileStat / FsStorageBackend | | DataSource | JsonDataSource / JsonEntityManager / getJsonDataSource / resetJsonDataSourceForTest / sqlTableFor | | Repository | JsonRepository / createJsonRepository / FindOptionsLoose | | Query/SQL | JsonQueryBuilder / executeRawQuery / SqlTable / GetSqlTable / toParams | | Collection | JsonCollection / getCollection / resetCollectionsForTest / JsonRow | | Entity metadata | resolveEntityMeta / resolveEntityMetaByTable / JsonEntityMeta / JsonEntityColumn and the four non-entity-table functions | | SQL lexer/expr | tokenizeSql / TokenCursor / matchesWhere / looseEqual / compareValues / likeMatch / parseExpr / evalExpr / truthy / hasAggregate / AGG_FUNCS / Expr / EvalContext / AliasInfo / jsonPathGet / jsonUnquote / toDate / mysqlDatetime |

For full types and behavior, see the engine tests under src/__tests__, and the cheat sheet published with the package at skills/typeorm-storage/.

Supported TypeORM Capability Matrix

The table below lists the engine's coverage of TypeORM's public API. ✓ implemented = behavior matches TypeORM; ~ shim = reachable at runtime but with semantic differences (see next section); × unsupported = not yet implemented; calling it throws or is a silent no-op.

Repository (entity repository)

| Capability | Status | Notes | | --- | --- | --- | | find / findBy / findOne / findOneBy / findAndCount / findAndCountBy | ✓ | object conditions + sort/pagination/select subset | | findOneOrFail / findOneByOrFail | ✓ | throws EntityNotFoundError on miss | | count / countBy | ✓ | includes object conditions | | save / insert | ✓ | auto-increment PK, UUID PK, UPSERT, ODKU semantics | | update / delete / increment / decrement | ✓ | empty condition fails loud (forbids accidental full-table damage) | | deleteAll / updateAll | ✓ | explicit full-table ops (guarded) | | upsert | ✓ | conflictPaths defaults to PK, unique columns specifiable | | softDelete / restore / softRemove / recover | ✓ | depends on @DeleteDateColumn | | exists / exist / existsBy | ✓ | | | sum / average / minimum / maximum | ✓ | supports where filter and column name | | preload / merge / hasId / getId / create | ✓ | | | clear | ✓ | clears the whole table | | query / sql / createQueryBuilder | ✓ / ~ | raw SQL + query builder | | find's relations option | × | relations/nested entities not implemented (see known limitations) |

EntityManager (JsonEntityManager)

All read/write methods (save / find / count / increment / upsert / softDelete / query … roughly 40 in total) are aligned one-by-one with Repository, and additionally provide updateAll / deleteAll. The manager field and createEntityManager() are also available.

DataSource (JsonDataSource)

| Capability | Status | | --- | --- | | initialize / connect / destroy / close / setOptions | ✓ | | getRepository / getTreeRepository / getCustomRepository / getMetadata / hasMetadata | ✓ | | synchronize / dropDatabase / runMigrations / undoLastMigration / showMigrations | ~ (no-op, see next section) | | createQueryRunner | ~ (hasTable always true, schema methods are no-op) | | transaction / createEntityManager / query / sql | ~ / ✓ |

Type note: JsonRepository / JsonDataSource / JsonQueryBuilder are compatible shims of TypeORM's public interfaces, runtime behavior aligned, but they do not implements the corresponding interfaces (Repository<Entity> / DataSource / SelectQueryBuilder). If a consumer needs const ds: DataSource = getJsonDataSource() to pass type checking, for now you need an as unknown as DataSource assertion; fully eliminating it requires implementing a real TypeORM Driver (see known limitations).

QueryBuilder (JsonQueryBuilder)

| Capability | Status | | --- | --- | | select (incl. * / alias / DISTINCT), from / addFrom | ✓ | | where (string / object / whereInIds / Brackets nesting) | ✓ | | leftJoin / innerJoin / leftJoinAndSelect | ~ (Cartesian-product placeholder only, no foreign-key nested fill) | | groupBy / having / addGroupBy / addHaving | ✓ (aggregation + HAVING) | | orderBy (ASC/DESC, NULLS FIRST/LAST, object form, empty-arg clear) / addOrderBy | ✓ | | limit / offset (MySQL LIMIT offset, count) / skip / take | ✓ | | update().set().where().orderBy().limit() / delete().where().orderBy().limit() | ✓ | | orUpdate (ODKU) / into(entity, columns) | ✓ | | Fetch: getMany / getOne / getManyAndCount / getCount / getExists / getRawMany / getRawOne / getRawAndEntities / getQuery | ✓ | | withDeleted (soft-delete filter toggle) | ✓ | | subQuery / whereExists | × |

FindOperator (object condition operators)

Equal · Not · In / Any · IsNull · LessThan / LessThanOrEqual · MoreThan / MoreThanOrEqual · Between · Like / ILike · And · Or · ArrayContains / ArrayContainedBy / ArrayOverlap · JsonContains · Raw ✓

Array operators act on JSON array columns (PostgreSQL style); Raw receives (dbColumn, aliases?) => string expression evaluated by the engine's evaluator.

Native SQL functions (in-memory evaluator)

Scalar: COALESCE / NULLIF / IF / IFNULL / NOW / CURRENT_TIMESTAMP / UNIX_TIMESTAMP / FROM_UNIXTIME / DATE_FORMAT / TIMESTAMPDIFF / DATEDIFF / ABS / ROUND / FLOOR / CEIL / CONCAT / CONCAT_WS / LEAST / GREATEST / SIGN / MOD / POWER / SQRT / LPAD / RPAD / LOCATE / INSTR / REVERSE / REPLACE / STR_TO_DATE / YEAR / MONTH / DAY / HOUR / MINUTE / SECOND / DATE / CURDATE / CURTIME / DATE_ADD / DATE_SUB / TIMESTAMPADD ·

JSON: JSON_EXTRACT / JSON_UNQUOTE / JSON_CONTAINS / JSON_OVERLAPS / JSON_SET / JSON_OBJECT / JSON_ARRAY · Aggregation: COUNT / SUM / AVG / MIN / MAX / GROUP_CONCAT.

The full function registry is in FUNCS / AGG_FUNCS of json-sql-expr.ts.

Write operation return: UPDATE / DELETE return { affected, raw }; the raw SQL UPDATE/DELETE returns an OkPacket shape where affectedRows is the number of matched rows and changedRows is the number of actually changed rows (corresponding to MySQL info's Rows matched / Changed). To judge "did this actually change a value", you should use changedRows.

Semantic Differences from MySQL & Known Limitations

This engine is a compatible fallback layer of "single-process JSON files + in-memory SQL evaluation" against TypeORM, not a real database. The following differences should be known to integrators to avoid pitfalls in semantic assumptions.

Write semantics

• Transactions are no-op: DataSource.transaction() / EntityManager.transaction() / QueryRunner transaction methods simply pass through the callback execution, providing no cross-table/cross-statement atomicity or rollback. JSON mode cannot guarantee "fail-whole-rollback"; if your business strongly depends on transaction consistency, use a real database.
• DDL is no-op: synchronize() / dropDatabase() / runMigrations() etc. are no-ops; createQueryRunner() is only for schema probing, and hasTable always returns true. Therefore code like if (!(await hasTable(t))) await createTable(t) will not execute table creation — JSON mode needs no table creation anyway, but callers should be aware.
• Multi-process consistency: the sequencer (nextId) and write lock guarantee cross-process uniqueness and mutual exclusion via the backend's withLock, but "read-modify-write" critical sections outside transaction are still not strongly consistent; for high-concurrency writes, rely on primary-key/unique-constraint conflict detection.

Relations and entity capabilities (× not yet supported)

| Capability | Notes | | --- | --- | | Relations (@ManyToOne / @OneToMany / @OneToOne / @ManyToMany / @JoinColumn / @JoinTable) | Decorators not parsed; find({ relations }) and leftJoinAndSelect only do a Cartesian-product placeholder, not foreign-key nested-entity fill | | EntitySchema | Metadata is only read from decorators (getMetadataArgsStorage); entities defined via EntitySchema cannot be found | | Subscribers / lifecycle hooks (@BeforeInsert and the other 10 hooks, EntitySubscriberInterface) | Writes do not pass through any hooks | | @Tree / TreeRepository | Not implemented | | BaseEntity static methods | Not provided (convenience wrappers based on Repository are missing) |

SQL dialect gaps

• Subqueries / CTE: FROM (SELECT …), IN (SELECT …), EXISTS (SELECT …), WITH not yet implemented.
• Set operations: UNION / UNION ALL not implemented.
• Window functions: OVER (PARTITION BY …) not implemented (aggregation only supports GROUP BY form).
• JOIN types: RIGHT / FULL JOIN not implemented; LEFT / INNER are Cartesian-product placeholders only (see above).
• Identifiers: double-quoted identifiers ("col") in raw SQL are unsupported on some paths; it is recommended to use backticks or no quotes uniformly.
• Functions listed in the "Native SQL functions" section above are the supported set; anything beyond that throws an "unknown function" error.

Type compatibility (publishing note)

JsonRepository / JsonDataSource / JsonQueryBuilder are compatible shims of TypeORM's public interfaces, runtime behavior aligned, but they do not implements the corresponding interfaces. Therefore:

const ds: DataSource = getJsonDataSource();          // ❌ type check fails
const ds = getJsonDataSource() as unknown as DataSource; // ✓ assertion needed for now

To fully eliminate this gap (giving consumers the complete TypeORM type experience), you need to implement a real TypeORM Driver (JsonDriver + JsonQueryRunner + RdbmsSchemaBuilder no-op), so that Repository / EntityManager / QueryBuilder / relations / hooks / migrations are provided by TypeORM itself — this is the prerequisite for the engine to achieve "full TypeORM support".

Examples

The examples/ directory provides complete, runnable examples covering the most common integration scenarios. In this repo you can run them without building first: the root tsconfig.tsx.json enables legacy decorator metadata and maps @fefeding/typeorm-storage to src via paths; run with tsx (see examples/README.md). When consuming the published package, the import resolves to dist.

npm install
npm run example:01   # or: npx tsx --tsconfig=tsconfig.tsx.json examples/01-basic-entity.ts

| Example | Scenario | | --- | --- | | 01-basic-entity.ts | Define entity + Repository save/find/findOneBy/findAndCount/update/delete | | 02-query-builder.ts | QueryBuilder: string/object conditions, LEFT JOIN, GROUP BY aggregation, paginated count | | 03-raw-sql.ts | query: SELECT/INSERT/UPDATE/DELETE, ODKU upsert, TRUNCATE; positional/named parameters | | 04-non-entity-table.ts | Non-entity table (raw SQL only) + nonEntityTables registration | | 05-custom-backend.ts | Implement JsonStorageBackend to inject a custom (in-memory) backend | | 06-find-operators.ts | Repository where FindOperators: In/Like/Not/IsNull/MoreThan/Between | | 07-sql-functions.ts | Native SQL functions / aggregation (GROUP BY+HAVING) / And·Or combos / JSON contains query | | 08-soft-delete.ts | @DeleteDateColumn soft delete: softDelete / restore / withDeleted | | 09-collection.ts | Low-level JsonCollection / getCollection: row-level read/write by PK, auto-increment, non-entity table |

Key conventions (two namespaces, merge-write semantics, multi-process safety, etc.) are in examples/README.md.

Build & Test

pnpm build        # produces dist/esm (NodeNext) and dist/cjs, with fixup-cjs repairing .d.ts
pnpm typecheck    # type check only (ESM config)
pnpm test         # vitest runs engine tests under src/__tests__ (157 cases)

Note for integrators: this package is published as an npm package; prepublishOnly is bound to build, so what you install is the already-built dist, and integrators do not need to build this package themselves.

License

MIT