@formancy/data-core
v0.3.0
Published
The Formancy Data core: the database-neutral adapter port and metadata contract. No driver, no HTTP, no Node.
Downloads
512
Readme
@formancy/data-core
The database-neutral core of Formancy Data: the port every database adapter implements, and what the adapters have to agree on so that the rest of the product never needs to know which database it is talking to.
No driver, no HTTP, no Node. The package compiles against ES2023 with no
@types/node, so a Node import here is a compile error rather than a review
finding — the same rule @formancy/core holds upstream, for the same reason: a
core that can only run on a server is a core that cannot be checked in a
browser.
What is here today
- The adapter port,
DatabaseAdapter, with two implementations from the first day. - The metadata contract:
MetadataSnapshotand what it holds, made only throughcreateSnapshot, which sorts, checks and fingerprints. A snapshot carries gaps — what the connection could not see, about the scope, a schema or an object — so a permission-filtered catalog is never mistaken for a complete one. It names the account it was taken as, whose user is in the fingerprint, and says what that account may SELECT, INSERT and UPDATE, column by column, and whether row-level security applies to it, per table —none,applies, orunknownwith the gap that says why (0027). Since 0049 each table and view also says whether the account may DELETE from it (access) and every foreign key that refers to it, from a table in any schema (referencedBy), or areferenced-bygap says the list may be short. Both are absent from a snapshot taken before, which keeps its fingerprint;createSnapshotrefuses one without the other, and a key between two tables of the snapshot said otherwise at its two ends. - The server a snapshot was read from:
server, its product, major and, for SQL Server, the connected database's compatibility level, as each adapter'spingreports them too (ServerIdentityextendsServerFacts) -- outside the fingerprint, so an upgrade is no drift.serverStanding(kind, server)says whether this release was tested on it, againstTESTED_SERVERS, the majors of the test matrix and SQL Server's compatibility floor, with a sentence a person reads; anything unknown, a snapshot withoutserverincluded, is not tested.serverNamenames a server, a SQL Server release by its year fromSQLSERVER_RELEASES. Nothing refuses a server for being untested (0068).UNTESTED_PRODUCTSnames, with their names, the products a snapshot recognises that no test runs on -- Azure SQL Database and Azure SQL Managed Instance -- derived fromTESTED_SERVERS, and the root README's What it is tested on row from it (0073). - Form generation:
generateForm(snapshot, request)returns a spec 3 formancy document, itsFormBindings, and a note on every choice it made.GENERATED_SPEC_VERSIONis that"3", which the data server also holds every stored version to (0042). A lookup is a select over its option source, withwidget: "typeahead"when its display includes a text or whole-number column, which is what a search matches, and a plain select of its first page otherwise; its note says which, and for a plain select what to show to make it searchable. A display column with no text form -- binary, rowversion, unsupported -- is refused, asbuildLookupConfigrefuses it;isDisplayTypeis that rule (0066). - Codecs:
codecFor(column)checks and canonicalises one API value for a column — decimals as strings, never rounded; integers past 2^53 as strings; formancy's date and time shapes on real days (0008); text counted in the unit its column counts, and refused with NUL or an unpaired surrogate; a 32-bit float ascanonicalFloat32names it, the shortest decimal for its float, and refused where a real would store infinity or a zero nobody wrote (0026). Both adapters read an instant to the second and a time to the minute, shorter than either engine stores them, soplanUpdateremoves an unedited echo of one (0040, below). The snapshot names what the engines disagree on about a column — a text's length unit, a binary's padding, an identity ALWAYS or BY DEFAULT, a check enforced or not — andcreateSnapshotrefuses one stored before it did.
import { generateForm } from '@formancy/data-core'
const { form, bindings, notes } = generateForm(snapshot, {
connection: 'erp',
root: { schema: 'sales', name: 'order' },
formId: 'sales-order',
title: 'Order',
lookups: [{ foreignKey: 'fk_order_customer', display: ['name'] }],
})Pass the columns the form's policy pins as pinned: they become read-only
fields filled from trusted context, never required fields the policy refuses.
A key the database numbers or fills is shown in the Record section and never
written: an identity (0026), and a column with a default and no generation
that is all of the key the request does not pin -- a uuid from
gen_random_uuid() or newsequentialid(), alone or beside a pinned tenant --
whose note says the database fills it. Where the pins leave more of the key,
a defaulted column of it is still written
(0056).
The form offers what the snapshot's account may do and nothing else: no field
over a column it may not read, a field written per operation — writes:
{ create, update }, bindings version 2 — so a column it may INSERT and not
UPDATE is written on create and read-only on update, and no operation the
database would refuse on every request. Row-level security that applies is an
access note. Bindings of version 1 are refused everywhere with "republish"
(0027).
Exact decimals and integers past 2^53 become text fields with an exact pattern,
never JavaScript numbers. Update is offered only with a proven concurrency
token. See 0009.
Delete is offered with one too, where the account may DELETE and the version
proves the record for good: a rowversion, or a version column on a table
whose key the database numbers -- an identity, or a key of one column that
the engine's never-repeating default fills, gen_random_uuid() on
PostgreSQL or newsequentialid() on SQL Server, compared whole as the
snapshot spells it, and that no foreign key of the table includes
(numbersKey). Any other default does not count -- a constant hands out
one value twice, and uuid_generate_v4() or newid() are not named --
and neither does such a default beside another key column, which a form
without the pin writes, or on a key a foreign key includes, which a lookup
through it writes; drift review stops delete when such a key's default
changes or a foreign key comes to include it
(0050,
0057).
Drift review
diffSnapshots(base, current, bindings, policy) compares the snapshot a form
was generated from with the database as it is now, through that form's
bindings and its policy's lookup filters, which the bindings do not record.
Each change says what it means for that form, blocking, review or info,
and which fields it touches; the report says what the form may still write.
import { diffSnapshots } from '@formancy/data-core'
const { changes, blocking, writable } = diffSnapshots(published, rescanned, bindings, policy)Something the connection can no longer see is an access problem, never a
deletion: the snapshot's gaps say which. A privilege the account lost is a
privilege-narrowed change that stops exactly the writes resting on it, or
blocks the form when a read rests on it — a bound or identity column, or the
confirmed concurrency token, which every read and every create names; row
security that changed is for review; a snapshot taken as another user is
account-changed, blocking while row security is in play. A lookup reads its
target's key, display and filter columns by the types it was published with,
so any change to one of those types, wider or narrower, blocks the lookup
(0028). An apparent rename is a dropped
column, a new one and a hint, never inferred. Tables the form neither binds nor
looks up are left out, so an empty report means nothing this form rests on
changed. See 0010.
One change is not about the catalog: when the current snapshot was read from
a server this release was not tested on, as serverStanding decides, every
report against it says so, as a server-untested change for review that
stops nothing -- an unchanged catalog included, since the server is outside
the fingerprint and an upgrade in place keeps it. A report whose only change
is that one means nothing the form rests on changed, on a server this
release was not tested on. A snapshot that records no server says nothing
here
(0068).
The report also says readable -- no change breaks what the form shows --
and runtime, what a request decides against this database (0041): the
verdict of the root's own definition alone, its kind, columns, keys and
foreign keys, which is all a request compares. runtime is never stricter
than review, and looser by what only review compares: a lookup's table,
privileges, row security and the account. ROOT_DEFINITION is the families
that compare that definition; diffSnapshots runs them first, and
diffRootDefinition(base, described, bindings, policy) runs only them,
over a root as a request described it, so a rule added to one reaches both.
describedOf(snapshot, ref) is a root as a snapshot would describe it, and
inSnapshotOrder puts a description in a snapshot's order.
Presentation over a generated base
A person's edits to a generated form are kept as a patch beside the base the generator wrote, and carried to the next base when the form is regenerated (0030). The patch holds exactly the studio's edits — a field's label, a section's label, a field's place within its section's grid, full width, and the generated section a field is placed in (0075) — and nothing else.
import { presentationOf, rebasePresentation } from '@formancy/data-core'
const derived = presentationOf(base, edited, bindings) // refuses every other difference, by JSON path
const { presentation, form, conflicts } = rebasePresentation(
{ base, presentation: derived.presentation, bindings },
{ base: next.form, bindings: next.bindings },
)A field is anchored by what it stands for — its column, or its lookup's
foreign key — so a label follows the column when a colliding key renumbers.
A section is anchored by the label the generator wrote and which of the
sections with that label it is. A field moved to another section is recorded
on its own entry as section, that section's anchor; it lands after the
section's own fields, and a section's order is of the fields it holds once
the moves are made, one decision placement.ts makes for applying,
deriving and rebasing alike. A section keeps at least one field. What cannot
be carried as it was is a conflict: a field gone, a field under another key,
a field now placed in another section, a move whose section is gone, cannot
be told apart or would be left with no field (move-dropped), a section
gone, a label the generator also changed (the person's is kept). applyPresentation is what the server checks a stored
form against; reassignedKeys names keys that now stand for another column,
whose grants somebody has to confirm — and the server refuses a publish that
grants on one without confirming it
(0039).
grantsOnKey(policy, key) is what grants on one means there: a read or
write role on its field, a lookup's filter alone being no grant, since every
lookup field has one and nobody without a role may search it.
describeReassigned(key) is the one sentence the server's refusal and the
studio's lists say a reassigned key in.
What it does not do: a renamed column reads as one gone and one new, and its
overrides are reported dropped rather than guessed across; a root whose own
label is Record has two sections labelled Record, and when one of them
comes or goes the rebase cannot tell which is left, so their overrides are
reported dropped (section-gone) rather than put on the wrong one; help
text, translations, required, a numeric span, a section added and a move
that empties a section are refused, not carried. A version-1 reader before
0075 refuses a presentation holding a move, naming /fields/N/section.
Lookups
The database-neutral half of a foreign-key lookup: what a select stores, what
an adapter is asked, and the decisions both adapters would otherwise make twice.
@formancy/data-postgres and @formancy/data-sqlserver implement the port.
- A key token is the string the select stores.
encodeKeyToken(['7', '1001'])isk1:7,1001; anything outsideA–Z a–z 0–9 - . _is escaped as~and four hex digits, so a token is printable ASCII and one key has one spelling.decodeKeyTokenrefuses every other spelling. A key whose token would exceed the 200 characters formancy stores is refused, never truncated. buildLookupConfig(bindings, field, { snapshot })derives what an adapter may know: the target, the key columns in order with their types, the display and search columns with theirs, the sort columns, and the page size. Every sort column says where NULLs go, last by default, because the engines disagree. It refuses bindings from another snapshot, bindings that are not what the root's foreign key references, a float, binary, boolean, time or timestamp key, and a search over a column the label does not show.validateLookupQueryis the boundary for a search request: exactly a search, an offset and a limit, trimmed and bounded.LookupAdapteris the port:search,resolveandrejects, each with the actor's trusted row filters, read withrowFilterTerms. "Every row" is{ kind: 'unrestricted' }, written on purpose; an empty list is refused.- Row filters come from
scopeRowFilters(object, filter, where), for a lookup's target and a record request's root alike. Each term carries its column's type and compares the column's canonical value with the trusted value exactly: case, accents and trailing spaces count, and the column's collation is not consulted. A column that is absent, not readable by the snapshot's account, or not text, integer, decimal, uuid or date isinvalid-policy—rowFilterColumnProblemnames why, and the server's publish check and the studio ask the same function. A trusted value not spelled as its column holds it ('042','AB 'forchar(n), an upper-case uuid), or text with an unpaired surrogate — which one driver sends as U+FFFD and the other as itself — isinvalid-context.rowFilterTermsrefuses an untyped or misspelled term at run time too. - Labels are spelled once, by
displayText, from the canonical value the adapter's record reader returns: a boolean astrue, a real as0.1, a time to the minute and a timestamp to the second, cut and never rounded, whatever the session's settings. Labels are not localised.rejectsis the shape formancy's server-sidemembersport asks for.lookupKeys,lookupPage(config, …),resolvedRows(config, …)andrejectedTokensbuild its answers: a token is asked about only when each value is spelled as its key column holds it, and is a member only when a row found under those filters re-encodes to it exactly.
A token is a reference, not a permission: it says which row was meant, and the server decides again, every time, whether this actor may name it. See 0012, and 0028 for filters and labels.
Access policy
A FormPolicy says, per published form, which roles may read, create and
update, which may read and write each field, and which root columns must equal
an attribute of the trusted PolicyContext — tenant isolation — with the same
for each lookup's target table. The host builds the context after verifying the
caller's identity; it is never built from request input.
import { checkSubmittedFields, forcedValues, rowFilter } from '@formancy/data-core'
const context = { actor: { id: 'u-17', roles: ['clerk'] }, attributes: { tenant: '42' } }
checkSubmittedFields(policy, context, bindings, 'create', ['tenant_id', 'name'])
// { ok: false, code: 'over-posting', message: 'tenant_id is pinned by a row filter: …' }
rowFilter(policy, context, 'read') // { ok: true, filter: [{ column: 'tenant_id', value: '42' }] }
forcedValues(policy, context) // the same columns, written from the context on createDeny by default: a field the policy does not name is readable and writable by
nobody, and a lookup it does not name offers nothing. A missing attribute is a
refusal, never an empty filter, because an empty filter is every tenant's rows.
Every function answers { ok: true, … } or a refusal with a stable code, and
none of them reads the form document: a hidden or disabled field is
presentation, not authorisation. Each one that returns fields, a filter or
values first authorises the operation it serves, so an actor whose roles hold
field grants and no operation grant gets nothing from any of them.
validatePolicy(policy, bindings) refuses a policy that does not fit its
form, and every function given the bindings refuses it too — the lookup
filter among them, so an options search cannot run under a policy the save
would refuse. The request planner below runs them on every read, create and
update it plans. See
0011.
A table with no tenant column of its own -- an order's lines -- is scoped
through a parent instead: through: ['order'] names lookup fields whose
filter on their target also scopes the form's rows
(0043).
const linePolicy = { …, rowFilters: [], lookups: { order: [{ column: 'tenant_id', attribute: 'tenant' }] }, through: ['order'] }
throughFilters(linePolicy, context, bindings, 'read') // { ok: true, through: [{ field: 'order', filter: [{ column: 'tenant_id', value: '42' }] }] }
throughProblems(snapshot, bindings, linePolicy) // [] -- or why its lookup cannot scope: it does not configure, or its key is textvalidatePolicy refuses a through that is not a lookup of the form, is
named twice, or whose list offers every row; throughProblems refuses one
whose key is text on either side, which no snapshot records a collation to
compare by, on both engines. throughFilters authorises the operation and
refuses a missing attribute, and asks for no grant on the field: a through
scopes rows and offers no options. authorizeOperation refuses every
operation, create included, whose through the context cannot resolve, so a
form is never offered for what every request would then be refused. And a
through names a key: grantsOnKey counts it, so a regeneration that
re-points that key is asked about (0039) whether or not anybody holds a role
on its field. Optional: a policy without it means what it meant, and a server
older than it refuses a policy that uses it.
Record requests
The one place a browser's request becomes an adapter's: the policy, the
codecs, the tokens and the bindings meet here, so both adapters receive the
same typed request for the same answers. Pure — no I/O, no clock — and
@formancy/data-server calls it on every runtime request.
import { planCreate, planUpdate, rejectedSelection, toFormAnswers } from '@formancy/data-core'
const described = await records.describe(bindings.root) // the table as the catalog has it now (0041)
if (!described.ok) return described
const planned = planCreate(snapshot, bindings, policy, context, body, described.described)
if (!planned.ok) return planned // a refusal with a stable code, or { code: 'invalid-values', fieldErrors }
for (const check of planned.memberships) {
if ((await lookups.rejects(check.config, check.tokens, check.filters)).length > 0) {
return { ok: false, code: 'invalid-values', fieldErrors: [rejectedSelection(check.field)] }
}
}
const saved = await records.insert(planned.request)
return saved.ok ? toFormAnswers(bindings, planned.fields, saved) : saved // { record, version, answers }- A record token addresses a record:
recordTokenencodes its identity values with the lookup's key-token encoding, anddecodeRecordKeyholds each value to its key column's spelling. planRead,planCreate,planUpdatecheck the bindings against the snapshot they were generated from (driftotherwise), the policy — over-posting first — and then every answer through its column's codec, reporting every bad field at once. The tenant is written from the context; an omitted field lets a default apply on create and is unchanged on update; the key and the pinned columns are never set; update needs a confirmed concurrency token. A version column, which every update increments, is never the key, a field's column or a generated one, and never pinned by a row filter.- The write planners take the table as the request described it
(0041).
planCreate(…, answers, described)andplanUpdate(…, answers, asRead, described)refusedrifta write the description stops, bydriftRefusal-- drift review's root families over it, after the policy and before the token, the version and the codecs -- and putdescribed.definitionon the request, so the adapter writes only while the table still has it. A read is decided after its statement, over the description the read returned;runtimeOperationsis what opening a form may offer. The refusals' sentences (READ_REFUSED,WRITE_REFUSED,NOTHING_LEFT) name no column; a refusal'sdriftlists the blocking changes for a log. - An update takes the record as read. An instant or a time read cut to
the shape and sent back unedited would, set, replace the stored fraction
or seconds.
planUpdate(…, answers, asRead, described)takes whattoFormAnswersmade of the recordplanReadand the adapter read for this actor just before the update, and does not set an instant or a time the actor may write and read that equals it at the version named; at another version nothing is removed, and the version guard answers. Without that read, or with another record's, an update carrying one is refusedrecord-not-read. A field the actor may not read is never compared, and needs no read (0040). - Throughs are on every read and update (0043).
planReadandplanUpdateput a typedThroughon the request for each lookup the policy names -- the root's foreign-key columns, the target and its key, and the lookup's filter scoped as every filter is -- and an adapter reads them withthroughTerms, as it reads filters withrowFilterTerms: a request withoutthrough([]for none) is thrown before anything is sent. A through field is required on create, omitted or null, and may not be cleared on update, whatever its column allows: a row whose foreign key is NULL references no parent, and nobody could reach it again. - A page of records (0044).
listable(snapshot, bindings, policy, context)decides whether this person is offered a list at all: the policy first, then a key a token can carry and the snapshot holds NOT NULL, rows something scopes where a lookup's options are filtered (unscopedLookups), and every key column in a field the person may read or pinned by their filter.listColumns(bindings, fields)is what a list may be sorted and searched by, whichGET /v1/forms/:idsends aslist.validateListQuerychecks a body from a browser;planListplans aListRequest-- the read's columns, filters and throughs, the orderlistOrdermakes total with the key, the search and the parent -- andrecordPagebuilds{ rows, hasMore }from thelimit + 1rows an adapter fetched, each row whattoFormAnswersmakes of a read of it. An adapter reads a request throughlistTerms, which throws on anythingplanListwould not have produced. A page is decided over the description its own statement carried bydriftRefusal(…, 'list'): a read's refusal, andLIST_REFUSEDwhen the table no longer holds the key unique and NOT NULL. - A file of records (0079).
validateExportQuerychecks an export's body: a list'ssearch,sortandparent, read by the same checks, and nothing else -- anoffsetor alimitismalformed, since the server reads every page. The pages areplanList's.exportColumns(form, bindings, readable)is the file's columns: the readable fields in the form'smodel.fieldsorder, headed by a string label or else the key, a lookup as two, its label and then its token under<header> (key).exportHeaderwrites the header row after a byte-order mark;exportRecord(columns, record, labels)writes one record from a page's answers and the labels the page's resolves gave, refusingunencodablefor a lookup whose key fits no token and, throughcsv.ts, for text UTF-8 cannot carry or a NUL, naming the column and never the value.exportText(type, value)is a cell's text:displayText's, except a time and a timestamp, written as read.csv.tsquotes by RFC 4180 and puts one apostrophe before a cell that starts as a formula, or with an apostrophe, unless it is a number.EXPORT_MAX_BYTES, 16 MiB, is the budget server and client share, andexportFileName(formId)the name a file is saved under,<form id>.csv. - A delete (0050).
planDelete(snapshot, bindings, policy, context, recordToken, expectedVersion, described)asks the policy first -- its filters and throughs scope a delete as an update's do -- then whether the form offers delete (offers(), where an absent flag is not offered) with a confirmed token, as its own snapshot says it can (deleteUnoffered), what a delete also does (deleteProblems), the description (driftRefusal(…, 'delete'),DELETE_REFUSED), the token and the version. What a delete also does is walked from the snapshot (0051): every key onto the root, and onto every table a cascade reaches, that cascades, sets NULL or sets a default must be named in the policy'scascadeswith its action, the account must be able to DELETE from every table reached, and a table outside the scope, areferenced-bygap, a SET NULL another key follows ON UPDATE and acascadesentry that names a key the delete does not act through are refused.deleteReachgives the request'sreach-- the tables removed, the root first, the tables cleared with their columns, and each accepted key with the table it refers to --deleteEffectswhat a page says of it, anddeleteReviewthe same walk key by key, for an administrator to accept each (0052): every key that acts with whethercascadesnames it with its action, every key that only refuses, each acceptance that names nothing the delete acts through, and indeleteProblems' own words what no acceptance can fix. An adapter reads the request throughdeleteTerms, which throws on a key that is not the identity, a missing version, filter or through, and a reach that does not begin with the root or whose tables and keys do not account for each other;reachesPastsays whether it reaches past the root. - A person's menu (0045).
menuForm(bundle, context)reads one published form alone for this person -- a bundle it cannot read throws there, for the caller to leave out -- andformMenu(forms)names, in codepoint order, every one whose policy lets this person read, withgrantedOperations-- what the policy grants of what the form offers as published, the part ofGET /v1/forms/:id's answer that asks no database -- and the children under each, comparing only whatmenuFormread.childFields(parent, child)is the structure: a lookup of the child, on the same connection, onto the parent's identity column for column in order, each foreign-key column of the referenced key's normalized type in each side's own snapshot, so the parent's record token is a value the child's field takes. The child's list narrowed by it holds the rows whose foreign key references that record by the database's own equality, which on a text key under a collation that ignores case or trailing spaces is not byte equality: a row's field can spell the key otherwise than the parent's token does. The menu names a child only when this person may list it (listable) and read that field. It asks no database: what the table allows now is decided when a form is opened. - Membership is the database's. A lookup token whose key carries the tenant
is refused when the tenant is not the context's; one that does not — a
surrogate id — cannot be judged without a query. Every selection comes back
as a
MembershipCheckfor the caller to run againstrejectsbefore it writes, and every non-member, forged or foreign, gets the samerejectedSelection. - The record port describes the table (0041):
describe(table)reads the root's kind, columns, keys and foreign keys in one statement,readanswers the same description beside the record, or besidenullwhen no row is inside the filters,listanswers it beside a page, and everyInsertRequestandUpdateRequestcarries thedefinitionthe decision was made over. An insert or an update runs only while the table's definition is that one; otherwise it isschema-changed, and the record was not written. - A database's refusal is a
RecordFailurewith a stable code.refusedis a refusal this port has no code for — a trigger's own error, a write declined without one, an error the adapter does not recognise — and is expected to be refused again.unavailablemeans only what passes: the database unreachable, or a deadlock or lock timeout the engine documents as such. Nothing was written in either case. toFormAnswersis the inverse, for the fields the actor may read: a lookup as its token, the three-state radio as'true'/'false'/null, and a whole number as a number, because the released engine reports canonical text in a number field as below its minimum.
See 0018.
Licence
Source-available, not open source. See LICENSE.md: free to
read, fork, evaluate, develop and test; a paid licence for production use.
