@heswell/vuu-data-engine-local
v4.3.0
Published
In-browser Vuu server (VuuModule, tables, EngineDataSource) built on @heswell/vuu-table and @heswell/vuu-viewport
Readme
@heswell/vuu-data-engine-local
A browser-hosted Vuu data layer built on the runtime-agnostic data engine packages
@heswell/vuu-table and
@heswell/vuu-viewport. The same
engine runs in the vuu-websocket (Bun) server.
The package runs alongside @vuu-ui/vuu-data-test and has the same public API. It is
meant to replace the VuuModule / TickingArrayDataSource core in that package. The
two packages don't depend on each other. To move an app across, see
MIGRATING.md.
Peer dependencies
All @vuu-ui packages are peers declared as ^4.0.0-alpha.20, so the engine uses the host
application's single copy of each (a second @vuu-ui/vuu-utils or @vuu-ui/core would mean
a second set of React contexts). They are @vuu-ui/vuu-utils and @vuu-ui/vuu-filter-parser
(runtime), @vuu-ui/core (React provider, /react entry only), and the type-only vuu-data-types,
vuu-filter-types, vuu-protocol-types and vuu-table-types.
Architecture
| Component | Role |
| -------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Table | A façade over the engine Table that keeps the existing Table API. Inserts, updates and deletes go straight to the engine. Every change notifies subscribed viewports. |
| LocalEngine | An isolated in-browser Vuu server: the registered VuuModules and a TableContainer of their tables. No React or DOM dependency. |
| TableContainer | A registry of tables, including join tables, keyed by {module, table}. Each engine has its own. |
| EngineDataSource | A DataSource implementation backed by an engine Viewport. It handles windowing, sort, filter, groupBy, selection (including select-all), freeze, base filter, link filter and permission filter. Row updates are batched per frame. |
| ModuleDataSource | Extends EngineDataSource with the module features: RPC and menu services, edit sessions, and visual links. TickingArrayDataSource is a deprecated alias. |
| VuuModule | Creates ModuleDataSource instances. Row permissions are passed to the engine as a permissionFilter predicate. Edit sessions can be customised per table with edit hooks. |
| RuntimeVisualLink | When the parent's selection changes, sets the child's engine link filter. |
| UpdateGenerator / TickingGenerator | Simulated ticking data that writes into the engine tables. |
Usage
Importing the package has no side effects: no modules, tables or timers are created
until you ask for them. The main entry needs no DOM, so it runs in plain Node or Bun with
no shims: it uses requestAnimationFrame when present and falls back to setTimeout
otherwise. The React bindings (createDataSourceProvider, createLocalVuuServer,
LocalDataSourceProvider) are in @heswell/vuu-data-engine-local/react, because
@vuu-ui/core needs a DOM (ResizeObserver) at import time.
import {
createLocalEngine,
registerDemoModules,
} from "@heswell/vuu-data-engine-local";
import { createLocalVuuServer } from "@heswell/vuu-data-engine-local/react";
// Your own modules, in an isolated engine
const engine = createLocalEngine({ modules: [new OrdersModule()] });
const dataSource = engine.createDataSource({
module: "ORDERS",
table: "parentOrders",
});
await dataSource.subscribe({ range: { from: 0, to: 100 } }, callback);
engine.dispose(); // disposes its modules and stops their timers
// A local Vuu server for a local-mode AuthenticationProvider (new engine per server)
const server = createLocalVuuServer({
connectionId: "orders",
modules: [new OrdersModule()],
});
// server.DataSourceProvider, server.engine
// The demo SIMUL, BASKET and TEST modules
const { simulModule } = registerDemoModules(createLocalEngine());Engines
| API | Description |
| --------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| createLocalEngine({ modules? }) | A new, isolated LocalEngine. Two engines, or two modules in one engine, can use the same table names. |
| engine.register(module) / unregister(name) | Add or remove a module and its tables. A module is attached to the first engine it is registered with. |
| engine.getModule(name), hasModule, modules | Look up modules. |
| engine.getTableList() | VuuTable[] ({module, table}) for every registered module. |
| engine.getTableSchema(vuuTable) | A table's schema. |
| engine.createDataSource(vuuTable, vp?, cfg?) | A ModuleDataSource for a table. |
| engine.tableContainer | The engine's TableContainer. Tables are keyed by {module, table}; a bare table name is accepted only if exactly one module has a table with that name. |
| engine.dispose() | Unregisters every module and disposes those attached to this engine (stopping ticking data and timers). |
| createDataSourceProvider(engine) (/react) | A React provider installing the legacy and core data contexts for the engine. |
| createLocalVuuServer({ connectionId, modules, engine? }) (/react) | A LocalVuuServer ({connectionId, DataSourceProvider, engine}). Uses a new engine unless one is passed. |
| registerDemoModules(engine?) | Creates the SIMUL, BASKET and TEST demo modules and registers them with engine (default: the default engine). |
| getDefaultEngine() / resetDefaultEngine() | The shared engine behind the deprecated globals below. |
Edit sessions: hooks and service overrides
The built-in edit-session RPC services (createSessionTable, editCell, addRow, deleteRow,
undoRowChange, endEditSession, …) can be overridden by rpcName, per module with
moduleServices or per table with services. Precedence: table services, then module
services, then the built-ins. Services registered for a source table also apply to its
session tables.
Most customisation needs only edit hooks, which mirror vuu-server's. Implement them per
table with editHooks (these win), or for every table as protected methods of the module.
| Hook | Default |
| ------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| canEditSource(column, ctx) | true: editCell may write directly to a source table (outside a session). |
| isEditableColumn(table, column, ctx) | Any column except the key and the system columns (vuuCreatedTimestamp, vuuUpdatedTimestamp, vuuMsg, vuuAction). The key of a row added in the session may be edited. |
| validateCellEdit(edit) | Accept. Return an error message to refuse the edit. |
| getStaleColumns(sourceTable, sourceRow, change) | defaultGetStaleColumns: for editCell, the edited columns that changed in the source since the row was copied into the session; for deleteRow, any column. |
| canSaveSession(prepared, ctx) | true. Return false or a reason to refuse the save. May be async. |
| applySessionSave(sessionTable, prepared, ctx) | defaultApplySessionSave: writes the changes to the source table atomically. May be async. |
class OrdersModule extends VuuModule<"parentOrders" | "childOrders"> {
protected editHooks = {
parentOrders: {
canSaveSession: ({ changes }) =>
changes.every(({ cellUpdates }) => Number(cellUpdates.qty) > 0) ||
"qty must be positive",
// amend orders through the order service instead of writing the table
applySessionSave: (_sessionTable, { changes }) =>
orderService.amend(changes),
},
};
// ...
}deleteRow and deleteSelectedRows behave as on vuu-server: they are only valid in an edit
session, and mark the session row vuuAction = "deleteRow" whatever the mode ("hard",
vuu-ui's default, or "soft"). The row stays in the session until it is saved, when
applySessionSave receives it as a deleteRow change, or undoRowChange restores it. A
row added in the session is simply dropped. Outside a session deleteRow returns an error;
deleteSourceRow (protected, not registered) keeps the old direct delete for subclasses
such as SIMUL's DELETE_ROW.
endEditSession({ save: true }):
- Rows already handled by
beforeSessionSave(with avuuMsg) are skipped. - Changes are prepared. A row added with an existing key fails with
"duplicate key"; a source row that is missing or stale fails with exactly"stale update"(vuu-ui'sVuuDataSourcematches this string and throwsStaleUpdateError). Settingparams.forceskips the stale check. The affected session rows get avuuMsg(stale:<key>:<columns>,<action>:<key>:source row missing,addRow:<key>:key already exists). canSaveSession, thenapplySessionSave. A refusal or failure (false, anERROR_RESULT, or a thrown error) leaves the session open, so the user can retry or cancel. Exported messages:STALE_UPDATE,DUPLICATE_KEY,SAVE_NOT_PERMITTED,SAVE_FAILED.afterSessionSave, then the session rows are reset (deleted rows removed,vuuActionandvuuMsgcleared, snapshots refreshed).
Join tables
createJoinTable(joinTable, table1, table2, joinColumn, joinType = "inner") (and
tableContainer.createJoinTable) creates a join that is kept up to date as either table
changes. Pass "leftOuter" to keep every table1 row. For a row with no match, the
table2 columns are null (not 0), as on vuu-server, and vuu-ui renders them empty.
Sort places nulls first when ascending; relational filters (<, >=, …) never match
null; and aggregations skip nulls.
By default the join table has every column of both tables, in alphabetical order. To control the columns, pass an options object as the fifth argument instead of a join type:
// an ordered list of column names, possibly a subset (must include table1's key)
createJoinTable(joinTable, instruments, prices, "ric", {
joinType: "leftOuter",
columns: ["ric", "exchange", "bid", "ask"],
});
// or a declared TableSchema for the join table, e.g. the schema vuu-server returns
createJoinTable(instrumentPricesSchema.table, instruments, prices, "ric", {
joinType: "leftOuter",
schema: instrumentPricesSchema,
});columns and schema are mutually exclusive, and joinType still defaults to "inner".
Column types come from the source tables; a column in both tables takes its value from
table1. Unknown or duplicate columns, a missing key column, a declared serverDataType
that differs from the source column, or a schema whose table or key does not match
(joinTable, table1's key) throw an error naming the join table and the problem.
With schema, the join's Table.schema is exactly the declared schema (no keys added), so
it matches getTableSchema / TABLE_META_RESP and can be published as a module's
schemas. If the declared schema has no rangeLimits, the engine default
(maxRangeEnd: 1_000_000, maxRangeWidth: 1_000) still applies, exposed as
table.rangeLimits rather than added to the schema. Joins created without schema
(default or columns) generate a schema that includes those default rangeLimits.
createJoinTable returns a JoinTableHandle (Table<JoinTable>), whose engineTable is
typed as the @heswell/vuu-table JoinTable. Call table.destroy() to detach a join from
its source tables, after which source changes no longer propagate to it. destroy() is
available on every Table and is safe to call more than once. You rarely need to call it
yourself:
module.dispose()(called byengine.dispose()for modules the engine owns) destroys the join tables the module publishes.tableContainer.clear()(also called byengine.dispose()) destroys tables created by that container'screateTable/createJoinTable.engine.unregister()andtableContainer.removeTable()do not destroy tables, as they may still be in use.
Ranges
subscribe({ range }), dataSource.range = … and setRange accept a vuu-utils Range or
a plain {from, to}.
Deprecated globals and the /legacy entry
tableContainer, LocalDataSourceProvider and ensureVuuModule use the default engine
(getDefaultEngine()). @heswell/vuu-data-engine-local/legacy re-exports the main and
/react entries and,
on import, registers the demo modules with the default engine, exporting the instances
simulModule, basketModule, testModule, moduleAdminModule, userAdminModule and
notificationsModule as 2.x did. It is the only entry point with side effects
("sideEffects": ["./src/legacy.ts"]).
Migrating from 3.x
These are breaking changes; the next release should be 4.0.0.
| 3.x | 4.x |
| --------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| createDataSourceProvider, createLocalVuuServer, LocalDataSourceProvider exported by the main entry. | Import them from @heswell/vuu-data-engine-local/react (or /legacy). The main entry no longer needs a DOM. |
| deleteRow with mode: "hard" removed the session row, so the save never saw the delete. | The row is marked vuuAction = "deleteRow" in either mode and applied on save (or restored by undoRowChange). |
| deleteRow outside an edit session deleted the source row. | Returns an error, as vuu-server does. Use an edit session, or expose the protected deleteSourceRow as a service. |
Migrating from 2.x
These are breaking changes; the next release should be 3.0.0.
| 2.x | 3.x |
| --------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Importing the package created and registered SIMUL, BASKET, TEST, MODULE_DISCOVERY, USER_ADMIN and NOTIFICATIONS, and exported the instances. | Nothing is created on import. Call registerDemoModules() or create the modules yourself, or import /legacy. |
| new SomeModule() registered itself in a global module container. | Register modules with an engine: createLocalEngine({ modules }), engine.register(), or createLocalVuuServer. |
| createLocalVuuServer registered its modules globally, so LocalDataSourceProvider saw them. | Each server has its own engine (server.engine). Pass engine: getDefaultEngine() to share the default engine. |
| tableContainer was global and keyed by bare table name. | Each engine has its own, keyed by {module, table}. The global tableContainer is a deprecated alias of the default engine's; a bare name throws if two modules have a table with that name. |
| ModuleContainer / moduleContainer. | Removed: use LocalEngine. |
| Module and table services were added alongside the built-in edit services. | They override built-in services with the same rpcName. |
| editCell could change the key and system columns, and edit missing or deleted rows. | Refused by default (see isEditableColumn). Editing a missing row returns an error. |
| endEditSession saved without checking for concurrent source changes or duplicate keys. | Fails with "stale update" or "duplicate key" (unless force). After a successful save the session rows are reset. |
| Demo prices and notifications ticked from import. | They start when a data source is created, and stop on engine.dispose(). |
| Reference data (instrumentsTable, pricesTable, instrumentsData, …) was created on import. | Lazy getters and factories (getInstrumentsData(), createPricesTable(), …). |
Differences from @vuu-ui/vuu-data-test
PermissionFilteredTablehas been removed. Use thepermissionFilterpredicate on the data source instead.- Visual links use the engine link filter, so the client no longer filters rows by selection.
- Grouped rows are only available from
getRowAtIndexwhen they fall within the client range. Flat rows are available at any index. - Re-inserting a row with an existing key updates that row.
- Table keys must not be empty.
Running the vuu showcase against the engine
In the finos/vuu vuu-ui workspace:
npm run showcase:engine # equivalent to: VUU_DATA_ENGINE=local npm run showcaseThis aliases @vuu-ui/vuu-data-test to @heswell/vuu-data-engine-local in the showcase
rsbuild config. The examples themselves don't change. The examples use the eagerly created demo
modules, so from 3.0 the alias should point to @heswell/vuu-data-engine-local/legacy.
The showcase uses the published package. To publish it from vuu-websocket:
npm run pub -- --package=@heswell/vuu-data-engine-local --tag alphaPerformance
bench/datasource.bench.ts compares the legacy TickingArrayDataSource
(@vuu-ui/vuu-data-test) with the engine ModuleDataSource on the same data and scenarios.
@vuu-ui/vuu-data-test is a dev dependency, used only for this comparison.
npm run bench:data-engine-local # 100k rows
BENCH_ROWS=10000 npm run bench:data-engine-local # smaller run
npm run bench -w @heswell/vuu-data-engine-local -- --outputJson=/tmp/local-bench.jsonThe repository's Performance workflow records this comparison at 100k rows
and publishes historical charts alongside the WebSocket and outbound suites.
See CI and historical results.
requestAnimationFrame is made synchronous during the bench, so each timing includes delivering
rows to the client. The bench runs under vitest (Node) using vitest.config.ts in this
package; the tests run under bun test. test/LegacyParity.test.ts checks that both implementations give the same
results for the sort, filter and groupBy scenarios.
Mean time per operation, 100k rows, Node 24, Apple Silicon:
| Scenario | legacy | engine | speed-up | | --------------------------------- | ------- | ------- | -------- | | create + subscribe | 24.1 ms | 8.7 ms | 2.8x | | sort numeric column | 62.1 ms | 14.1 ms | 4.4x | | sort two columns | 64.6 ms | 17.1 ms | 3.8x | | filter | 5.7 ms | 1.3 ms | 4.3x | | groupBy ccy, set/clear | 1.6 ms | 1.2 ms | 1.3x | | groupBy ccy + exchange, set/clear | 2.2 ms | 2.0 ms | 1.1x | | scroll, 100 range changes | 0.82 ms | 0.45 ms | 1.8x | | 1000 ticks, random rows | 1543 ms | 0.35 ms | ~4400x | | 1000 ticks, rows in viewport | 0.63 ms | 0.10 ms | 6.6x | | 1000 ticks, sorted column | 3012 ms | 1.9 ms | ~1600x | | 1000 ticks, filtered column | 1903 ms | 0.55 ms | ~3400x | | 1000 ticks, grouped | 1205 ms | 0.39 ms | ~3100x | | 500 deletes + 500 inserts | 1628 ms | 0.60 ms | ~2700x | | 500 deletes + 500 inserts, sorted | 2564 ms | 1.57 ms | ~1600x | | switch aggregation sets, grouped | 5.9 ms | 5.4 ms | 1.1x | | groupBy ccy + exchange, aggregated, set/clear | 3.2 ms | 3.6 ms | 0.9x | | expand/collapse one group | 1840 ms | 0.044 ms | ~42000x | | 1000 ticks, grouped, aggregated, expanded | 1704 ms | 0.70 ms | ~2500x |
Notes:
- The legacy update path finds each row with a linear search, so per-update cost grows with table size. The engine uses keyed lookup and sends each changed row once.
- On updates, legacy does not re-sort or re-filter rows. The engine does, so the tick comparisons understate the engine's advantage.
- Scrolling: the engine uses an array-backed row key allocator (
RowKeys) and prunes its row cache incrementally, so range changes cost in proportion to the rows entering or leaving the range. - Aggregations: switching sets and grouping with aggregations cost about the same in both, but
with the four-column set legacy fills in only one aggregate and leaves values from the previous
set behind, so it does less work. The legacy
bench can't clear aggregations, because
aggregations = []throws on a grouped legacy data source. - Expand/collapse: legacy rebuilds the expanded rows from every open node on each change. Expanding all groups was dropped from the comparison because legacy takes several seconds per run even at 10k rows. The 1M-row bench below covers it for the engine.
1M rows, engine only
bench/scale.bench-run.ts measures the engine alone at 1M rows, with heap use. A legacy
comparison is impractical at this size. It runs under vitest in Node with --expose-gc,
using vitest.scale.config.ts. Each timing is the median of 5 runs after a warmup. Heap is
V8 heapUsed after a full GC.
npm run bench:data-engine-local:1m
SCALE_ROWS=200000 npm run bench:scale -w @heswell/vuu-data-engine-local
SCALE_OUT=/tmp/scale.json npm run bench:scale -w @heswell/vuu-data-engine-localThe Performance workflow runs it at 1M rows and charts the results as local-scale.*.
Results at 1M rows, Node 24, Apple Silicon:
| Measurement | Result | | ---------------------------------------------------------- | -------- | | heap: source row arrays (context) | 182 MB | | heap: table (rows + key index) | 222 MB | | heap: sorted data source | 0.2 MB | | heap: grouped data source (2 levels, 4 aggregations, expanded) | 16 MB | | build table | 262 ms | | create + subscribe | 4.4 ms | | sort numeric column | 223 ms | | sort two columns | 452 ms | | filter | 15 ms | | groupBy ccy | 27 ms | | groupBy ccy + exchange, with aggregations | 64 ms | | switch aggregation sets | 37 ms | | expand/collapse one group | 1.4 ms | | expand all 56 groups, then collapse all | 329 ms | | scroll, 100 range changes | 0.7 ms | | 1000 ticks, sorted column | 8.4 ms | | 1000 ticks, grouped, aggregated and expanded | 1.4 ms | | 500 deletes + 500 inserts, sorted | 5.8 ms |
A 1M-row table fits comfortably in a browser tab, and data sources over it are cheap. Ticks, churn, scrolling and expand/collapse stay in the low milliseconds. Sorting runs on the main thread and takes a few hundred milliseconds, long enough to be noticeable when it repeats.
Tests
bun test packages/vuu-data-engine-localThe tests previously ran under vitest with happy-dom. Under bun test, test/browser-globals.ts
provides requestAnimationFrame and window, and test/happy-dom.ts registers a DOM for the
React test only.
test/EngineDataSource.test.ts holds the conformance tests for the engine-backed data source.
test/GroupedDataSource.test.ts covers aggregations (every type, multi-level, live updates,
sorting by an aggregate) and group expand/collapse (by key and index, nested groups, state
across sort, updates and groupBy changes, subscriber notifications).
