@monospace/extension-kit

Types and helpers for writing Monospace data connectors. These extensions let Monospace read from and write to a service it has no built-in connector for, such as a REST API.

| Import | Contents | | ------------------------------------------------------ | --------------------------------------------------------------------- | | @monospace/extension-kit/data-connector | defineDataConnector, handler and request types, error classes. | | @monospace/extension-kit/data-connector/protocol | Types for the requests the engine sends and the schema it expects. | | @monospace/extension-kit/data-connector/experimental | Helpers for declaring a connector's schema. May change in any release. |

Quick start

1. Create a project

The monospace CLI scaffolds a connector project:

npx @monospace/cli extension create ./acme-store --id acme/store --name "Acme Store"

Leave out --id and --name to be asked for them. Use pnpm dlx, yarn dlx or bunx in place of npx with other package managers.

The project has the kit as a dependency and the CLI, rolldown and typescript as dev dependencies. It contains:

• extension.config.json, which describes the extension and its connector entry, acme/data-connector.
• src/data-connectors/acme/index.ts, the connector.
• src/data-connectors/acme/schema.ts, the collections it serves.

Install its dependencies and build it with the commands extension create prints, such as npm install and npm run build.

2. Write the connector

Declare the collections the connector serves in schema.ts:

import {
	defineCollection,
	defineSchema,
	field,
	fields,
	index,
	paging,
} from '@monospace/extension-kit/data-connector/experimental';

const products = defineCollection({
	name: 'products',
	fields: {
		id: field.string({ length: 64 }),
		name: field.string(),
		price: field.decimal({ precision: 10, scale: 2 }),
	},
	indexes: { products_pkey: index.primary(['id']) },
	operations: {
		readMany: { limit: paging.limit(), offset: paging.offset(), output: fields.all() },
	},
});

export const schema = defineSchema({ collections: [products] });

Serve them in index.ts. This connector reads products from a REST API whose URL is part of the data source's configuration:

import type { MonospaceObject } from '@monospace/extension-kit/data-connector';
import {
	AccessDenied,
	ConnectionFailed,
	defineDataConnector,
	InvalidConfiguration,
	QueryRejected,
	RateLimited,
} from '@monospace/extension-kit/data-connector';
import { schema } from './schema';

function apiUrl(config: Record<string, unknown>): URL {
	if (typeof config.apiUrl !== 'string' || !URL.canParse(config.apiUrl)) {
		throw new InvalidConfiguration('apiUrl must be an absolute URL.');
	}

	return new URL(config.apiUrl);
}

async function get(url: URL): Promise<Response> {
	const response = await fetch(url).catch((error: unknown) => {
		throw new ConnectionFailed('The Acme API could not be reached.', { cause: error });
	});

	if (response.status === 401 || response.status === 403) throw new AccessDenied('The Acme API rejected the credentials.');
	if (response.status === 429) throw new RateLimited('The Acme API is rate limiting requests.');
	if (!response.ok) throw new ConnectionFailed(`The Acme API answered ${response.status}.`);

	return response;
}

export default defineDataConnector({
	preflight({ config }) {
		return {
			permissions: [{ permission: `net:${apiUrl(config).host}`, reason: 'Reads products from the Acme API' }],
		};
	},

	setup({ config }) {
		const base = apiUrl(config);

		return {
			introspect() {
				return schema.toSchemaDocument();
			},

			async query({ query }) {
				if (query.op !== 'readMany') {
					throw new QueryRejected(`${query.op} is not supported.`);
				}

				const url = new URL(query.collection, base);
				if (query.limit !== undefined) url.searchParams.set('limit', String(query.limit));
				if (query.offset !== undefined) url.searchParams.set('offset', String(query.offset));

				const response = await get(url);
				// Validate the response shape in real code.
				const records: MonospaceObject[] = await response.json();

				return { records };
			},

			async testConnection() {
				await get(base);
				return { ok: true };
			},
		};
	},
});

The connector handles every argument the schema declares. The engine returns the results of query without filtering, sorting or paging them. This schema declares only limit and offset on readMany.

3. Build and install

npm run build -- --install-to /path/to/engine/extensions

The build script runs monospace extension build. It bundles each entry into one JavaScript module and writes it with a manifest.json to dist. --install-to also copies the result into an engine's extension directory, which is ./extensions beside the engine unless MONOSPACE_EXTENSIONS__INSTALL_LOCATION sets another. Builds do not check types; run npm run typecheck for that.

The engine loads extensions at startup. Started with MONOSPACE_EXTENSIONS__AUTO_RELOAD=true, it also reloads an extension when a build replaces it, so npm run build -- --watch --install-to <dir> rebuilds and reloads on every change. Once installed, the connector is available when you create a data source in Studio.

See the CLI reference for all build options.

Adding the kit to an existing project

npm install @monospace/extension-kit
npm install --save-dev @monospace/cli rolldown

The build reads extension.config.json from the project directory. See Extension configuration for its format.

Writing a connector

A connector's entry module default-exports the result of defineDataConnector, which takes two functions:

• setup({ config }) runs when the engine starts the connector for a data source, and again after a reload or restart. It returns the handlers. Keep clients and other per-data-source state in its closure.
• preflight({ config }) is optional. It returns the network permissions the configuration needs; see Network permissions.

setup returns these handlers:

• introspect() returns a schema document describing the connector's collections, fields, indexes and relations, and the operations each collection supports. See Declaring a schema.
• query(request) serves one operation on one collection. See Handling queries.
• testConnection() is optional. It returns { ok: true } or throws. Leave it out when a check costs something, for example against a metered API; the engine then reports the check as unsupported.

The engine may call handlers concurrently.

Configuration

config is the data source's stored configuration, passed through unchanged and unvalidated. Check every value the connector depends on and throw InvalidConfiguration when one is missing or wrong. defineDataConnector<Config>() accepts a type for config, which defaults to Record<string, unknown> and is not checked at runtime.

Runtime

Connectors run in a V8 isolate inside the engine. Web APIs such as fetch and URL are available; node: modules are not. The CLI bundles each entry and its dependencies into a single module.

Testing

defineDataConnector returns a plain object, so a test can call setup and the handlers directly:

import connector from './index';

const handlers = await connector.setup({ config: { apiUrl: 'https://api.acme.example' } });
const result = await handlers.query({
	query: { op: 'readMany', collection: 'products', select: ['id', 'name'], limit: 10 },
});

Handling queries

query receives { query }, where query.op names the operation. Narrow on it to reach the operation's arguments.

| op | Arguments | Answer | | ------------ | ---------------------------------------------------------------------- | ----------------------- | | readMany | select, filter, sort, limit, offset, count | { records } | | readOne | key, select, filter | { record } | | create | data (one object per record), select | { records } | | updateMany | data, filter, select | { records } | | updateOne | key, data, select, filter | { record } | | deleteMany | filter, select | { records } | | deleteOne | key, select, filter | { record } |

Every request also carries collection, and namespace when the collection is in one.

• select lists the collection's own fields the answer should carry. A write has to answer with records only when the connector declares the matching query capability.
• key holds a value for the primary key field. A keyed operation (readOne, updateOne, deleteOne) addresses at most that row, and only if it also matches filter. When nothing matches, answer { record: null } and change nothing; a miss is not an error.
• A readMany with count: true asks for the number of records matching filter, ignoring limit and offset. Answer { totalCount }. It carries limit: 0 and no select.
• sort is a list of { field, direction, nulls? }, applied in order. Without nulls, use the service's default placement.

filter is a tree of nodes, each with exactly one of these shapes:

• { and: [...] }, { or: [...] } and { not: filter } combine filters.
• { cmp, left, right } compares two operands with one of equals, greaterThan, greaterThanOrEquals, lessThan, lessThanOrEquals, contains, icontains (case-insensitive), startsWith or endsWith. Each operand is { field } or { value }.
• { in: operand, values } matches when the operand is one of values, a list of values or a { field } reference to a list field.

The engine converts callers' { field: value } shortcuts to equals. It converts _null: true to equals with null, and _null: false to not around that. Narrow a node by checking a key against undefined. This handler serves readMany and readOne from rows held in memory:

import type {
	FilterOperandSerialized,
	FilterSerialized,
	MonospaceObject,
	QueryHandler,
	Value,
} from '@monospace/extension-kit/data-connector';
import { QueryRejected } from '@monospace/extension-kit/data-connector';

declare const rows: MonospaceObject[];

function operand(record: MonospaceObject, side: FilterOperandSerialized): Value {
	return side.field !== undefined ? record[side.field] ?? null : side.value;
}

function matches(record: MonospaceObject, filter: FilterSerialized): boolean {
	if (filter.and !== undefined) return filter.and.every((child) => matches(record, child));
	if (filter.or !== undefined) return filter.or.some((child) => matches(record, child));
	if (filter.not !== undefined) return !matches(record, filter.not);

	if (filter.in !== undefined) {
		if (!Array.isArray(filter.values)) throw new QueryRejected('in against a field is not supported.');
		return filter.values.includes(operand(record, filter.in));
	}

	if (filter.cmp === 'equals') return operand(record, filter.left) === operand(record, filter.right);

	throw new QueryRejected(`${filter.cmp} is not supported.`);
}

function where(filter: FilterSerialized | undefined): MonospaceObject[] {
	return filter === undefined ? rows : rows.filter((row) => matches(row, filter));
}

export const query: QueryHandler = ({ query }) => {
	switch (query.op) {
		case 'readMany': {
			const matching = where(query.filter);
			if (query.count) return { totalCount: matching.length };

			const start = query.offset ?? 0;
			const end = query.limit === undefined ? undefined : start + query.limit;
			return { records: matching.slice(start, end) };
		}

		case 'readOne':
			return { record: where(query.filter).find((row) => row.id === query.key.id) ?? null };

		default:
			throw new QueryRejected(`${query.op} is not supported.`);
	}
};

QueryHandler<Op> types a function for specific operations, such as QueryHandler<'readOne'>, and requires their answer shape.

Declaring a schema

introspect returns a schema document, typed as SchemaDocument in @monospace/extension-kit/data-connector/protocol. The helpers in @monospace/extension-kit/data-connector/experimental build that document from typed definitions and report most mistakes in your editor. They are experimental and may change in any release.

import {
	defaultSource,
	defaultValue,
	defineCollection,
	defineNamespace,
	defineRelation,
	defineSchema,
	field,
	fields,
	filter,
	index,
	paging,
	sort,
} from '@monospace/extension-kit/data-connector/experimental';

const sales = defineNamespace({ name: 'sales' });

const customers = defineCollection({
	name: 'customers',
	namespace: sales,
	fields: {
		id: field.uuid({ defaultValue: defaultValue.connectorManaged() }),
		email: field.string({ length: 255 }),
		createdAt: field.dateTime({
			defaultValue: defaultValue.engineManaged({ onCreate: defaultSource.currentTimestamp() }),
		}),
	},
	indexes: { customers_pkey: index.primary(['id']) },
	operations: {
		readMany: {
			filter: { fields: { id: filter.field('equals', 'in') }, logicalOperators: filter.logical('and') },
			sort: { createdAt: sort.field('ascending', 'descending') },
			limit: paging.limit(),
			offset: paging.offset(),
			output: fields.all(),
		},
		create: { data: ['email', 'createdAt'], output: fields.all() },
	},
});

const orders = defineCollection({
	name: 'orders',
	namespace: sales,
	fields: { id: field.uuid(), customerId: field.uuid({ nullable: true }) },
	indexes: { orders_pkey: index.primary(['id']) },
	operations: {
		readMany: {
			filter: { fields: { customerId: filter.field('in') }, logicalOperators: filter.logical('and') },
			output: fields.all(),
		},
	},
});

const customerOrders = defineRelation({
	name: 'customer_orders',
	from: orders,
	to: customers,
	on: [['customerId', 'id']],
	names: { forward: 'customer', reverse: 'orders' },
});

export const schema = defineSchema({
	namespaces: [sales],
	collections: [customers, orders],
	relations: [customerOrders],
});

schema.toSchemaDocument() returns the document for introspect. toSchemaDocument({ writes: false }) leaves out every operation that writes, which makes every collection read-only.

Fields

field has a constructor per type: boolean, bytes, date, dateTime, dateTimeWithTimezone, decimal, float32, float64, geometry, int8, int16, int32, int64, json, string, time, uint8, uint16, uint32, uint64, unsupported and uuid. The engine does not accept geometry values from connectors yet.

Each constructor takes the options nullable and list (both false by default), defaultValue, and apiName, a name callers use in place of the field's name. defineCollection accepts apiName as well. field.string({ length }) sets a maximum length; strings are unlimited by default. field.decimal({ precision, scale }) constrains a decimal; pass both or neither.

A field is read-only unless a data list in operations names it.

Defaults

Defaults describe values; the kit never computes them.

• defaultValue.connectorManaged() means the connector fills the field when a caller leaves it out. Pass { isAutoincrement: true } for an auto-incrementing id.
• defaultValue.engineManaged({ onCreate, onUpdate }) means the engine fills the field when a caller leaves it out. The field must be in the operation's data list, as createdAt is above. The value comes from defaultSource.currentTimestamp(), defaultSource.currentUser(), defaultSource.randomUuid('v4' | 'v7') or defaultSource.static(value).

A collection that declares create must list every non-nullable field without a default in a data list.

Indexes and relations

index.primary, index.unique and index.normal take a list of field names; the key in indexes names the index. The engine uses a collection's primary index as its stable id to address and read back rows. If there is no primary index, it uses a unique index.

defineRelation pairs fields of from with fields of to in on. Paired fields must be scalars of the same type. names.forward is how to appears on from, and names.reverse how from appears on to. onDelete and onUpdate take noAction (the default), restrict, cascade, setNull or setDefault.

Register every namespace, collection and relation in defineSchema.

Checks

Your editor checks field names, index fields, relation pairs and registrations as you type. At runtime, defineCollection, defineRelation and defineSchema throw SchemaDefinitionError, whose issues list each problem with its path. The engine checks the rules that depend on field types, indexes, relations and capabilities when a data source is created or reintrospected, and answers with a 422 that names what to declare.

Supported operations

Each collection declares in operations which of readMany, readOne, create, updateMany, updateOne, deleteMany and deleteOne it serves, and what each accepts. Only declared operations and arguments are available. operations: {} makes a collection read-only.

import {
	defineCollection,
	field,
	fields,
	fieldsOf,
	filter,
	index,
	paging,
} from '@monospace/extension-kit/data-connector/experimental';

const productFields = {
	id: field.uuid(),
	name: field.string(),
	price: field.decimal({ precision: 10, scale: 2 }),
};
const editable = fieldsOf(productFields).pick('name', 'price');

export const products = defineCollection({
	name: 'products',
	fields: productFields,
	indexes: { products_pkey: index.primary(['id']) },
	operations: {
		readMany: {
			filter: {
				fields: { id: filter.field('equals', 'in'), price: filter.field('lessThan', 'greaterThan') },
				logicalOperators: filter.logical('and', 'or'),
			},
			limit: paging.limit(),
			output: fields.all(),
			meta: { totalCount: true },
		},
		readOne: { output: fields.all() },
		updateMany: {
			filter: { fields: { id: filter.field('in') }, logicalOperators: filter.logical('and') },
			data: editable,
			output: fields.all(),
		},
		updateOne: { data: editable, output: fields.all() },
	},
});

| Argument | Operations | Declares | | --------------------------- | --------------------------- | --------------------------------------------------------------------------------------------------- | | filter | all but create | fields maps each filterable field to filter.field(...operators); logicalOperators uses filter.logical('and', 'or', 'not'). | | sort | readMany | Maps each sortable field to sort.field(...directions). | | limit, offset | readMany | paging.limit() and paging.offset(). The connector cannot set a maximum. | | meta: { totalCount: true } | readMany | Callers may ask for a count. | | data | create, updateMany, updateOne | The fields callers may set. | | output | all | The fields the operation returns. fields.all() lists every field. |

fieldsOf(fields).pick(...) builds a field list outside defineCollection that several operations can share.

The engine accepts these filter operators per field type:

| Operator | Field types | | ---------------------------------------------------------------------- | ------------------------------------------------------------- | | equals | every type except unsupported | | in | strings, numbers, decimals, dates, times, uuid | | greaterThan, greaterThanOrEquals, lessThan, lessThanOrEquals | strings, numbers, decimals, dates, times | | contains, icontains, startsWith, endsWith | strings |

A caller's sort without a direction arrives as ascending, so callers may omit the direction only on a field that declares ascending.

readOne, updateOne and deleteOne require a primary key of a single field whose type is not unsupported. A row-level permission rule on a collection is merged into the filter of the operation it restricts, so an operation that declares no filter cannot have one.

Calls the engine makes

The engine also calls the connector through readMany and by the collection's stable id to read back what a write changed and re-read the rows a write filters. Creating or reintrospecting a data source fails with a 422 that names what to declare when its declaration breaks one of these rules:

• The filters of updateMany, updateOne, deleteMany and deleteOne use only operators and logical operators that readMany.filter declares for the same fields.
• updateOne and deleteOne need equals on every stable-id field, and and, in readMany.filter.
• readMany, updateMany and updateOne list every stable-id field in output.
• An update, and a create or delete without the matching query capability, needs in on a single stable-id field, or equals on each of several with or, plus and, in readMany.filter. readMany.output lists the stable id and every field the write outputs.
• An update whose data sets no field is not offered.

The engine resolves a relation by filtering the other side by the relation's fields, with the same in or equals-with-or shape plus and, and by reading those fields from output. A relation that a declaration cannot serve this way is not offered, and the engine logs a warning that names what to declare.

Until the engine checks every call against the declaration, a connector may also receive the following without declaring them. Serve them where possible and throw QueryRejected otherwise.

• The relation filter shape above on relation or stable-id fields where no declaration can offer it. This applies to a single boolean, bytes, json or geometry field, or any unsupported field. It arrives in relation reads, nested writes and read-backs.
• not around a caller's update permission rules, joined to the update's filter with and, in the readMany that checks those rules before an update.
• A limit on a readMany that declares none. A caller's readMany carries the engine's default limit (100 unless configured otherwise) when the caller sets none. The engine's own read-backs may carry no limit.
• A key on every keyed operation, whatever its filter declares.

Extension configuration

extension.config.json describes the extension and its entries. extension create writes it with a $schema that editors use for completion.

{
	"$schema": "./node_modules/@monospace/cli/schemas/extension.config.schema.json",
	"entries": [
		{
			"engine": {
				"capabilities": { "query": ["insertReturning", "updateReturning", "deleteReturning"] },
				"entrypoint": "./src/data-connectors/acme/index.ts",
				"permissions": [{ "permission": "net:api.acme.example", "reason": "Reads the product catalogue" }]
			},
			"id": "acme/data-connector",
			"name": "Acme Store",
			"type": "dataConnector",
			"version": "0.1.0"
		}
	],
	"id": "acme/store",
	"name": "Acme Store",
	"version": "0.1.0"
}

The top-level id identifies the installed extension; each entry's id identifies a connector that data sources use. Ids take the form namespace/name in lowercase letters, digits and single hyphens. Versions are exact SemVer versions, and the extension's and each entry's are independent. description, icon, iconForeground and iconBackground are optional at both levels. dataConnector is the only entry type.

Query capabilities

engine.capabilities.query declares which writes return records for every collection the connector serves:

• insertReturning means create returns the created records with the fields requested by select.
• updateReturning means updateMany and updateOne return the updated records with the requested fields.
• deleteReturning means deleteMany and deleteOne return the deleted records with the requested fields.

Leaving out capabilities or query declares none. Without a capability, the engine gets the written records through extra calls, and the collection's declaration must support them:

• The collection needs a stable id to declare create, updateMany or deleteMany without the matching capability.
• Without insertReturning, create.output lists the stable id. The engine reads each created row back by it.
• Without updateReturning, no data list may name a stable-id field.
• Without updateReturning or deleteReturning, the engine first reads the stable ids of the rows a write matches, then writes exactly those rows through updateMany or deleteMany, filtered by stable id. This applies to updateOne and deleteOne as well. Declare that operation with the stable-id filter shape from Calls the engine makes and, for updates, a data list that covers the data of every update operation.

These extra calls are separate requests. Another writer can change a row between them, and on an eventually consistent service the read-back can return stale data. If the service returns written records, declare the capability.

The engine reads capabilities from the manifest when it builds a workspace. Reloading or installing a build whose capabilities differ deactivates the connector's existing data sources. Calls fail with a 503 whose message says a restart applies the change. Restarting or otherwise rebuilding the workspace, such as by adding a data source, makes them available with the new capabilities.

Network permissions

A connector can reach only the hosts its data source has been granted. Each permission is net: followed by a host name, a *. wildcard host name, an IPv4 address or subnet, or a bracketed IPv6 address, each with an optional port, such as net:api.acme.example:443. URLs and paths are rejected.

Hosts the connector always needs go in engine.permissions on its entry, each with a reason shown to whoever approves the grant. Hosts that depend on the configuration, such as a user-supplied API URL, come from preflight, which returns them in the same form. preflight must answer synchronously and should not make requests or start clients. It may throw InvalidConfiguration.

The engine runs a connector only when the data source's grant matches the entry's permissions together with what preflight returns. When they change, the grant has to be approved again.

Errors

Throw these classes so the engine can report a failure accurately. The message reaches API callers, so keep secrets and personal data out of it.

| Class | Throw when | | ------------------------------- | ------------------------------------------------------------------------------------------ | | InvalidConfiguration | A configured value is missing or wrong, found without asking the service. | | AccessDenied | The service rejects the credentials, or they lack a scope, access or plan. | | ConnectionFailed | The service cannot be reached or reports a temporary failure. Retrying may work. | | RateLimited | The service is throttling the connector until a limit or quota resets. | | CollectionNotFound | A collection the schema declared no longer exists in the service. | | QueryRejected | A valid request cannot be served as asked. Write the message for the caller. | | UniqueConstraintViolation | A write would duplicate a value that must be unique. | | NullConstraintViolation | A write leaves out a value the service requires. | | ForeignKeyConstraintViolation | A write references a missing record, or deletes one that others still reference. |

Each constructor takes a message and an optional { cause }. The cause is kept for debugging and does not reach API callers.

A fetch that the data source's permissions refused is reported as a permission failure, even when wrapped in ConnectionFailed. Any other thrown value is reported as an unclassified failure unless it carries a monospace: { code } property, as these classes do. ErrorCode lists the codes.

Values

Records, filter values and keys are JSON. Types without their own JSON representation use strings. Decimals use strings to preserve precision. Bytes use base64. UUIDs, dates and times use their text forms.

Single int64 and uint64 values arrive as decimal strings, because JavaScript numbers lose precision beyond 2^53. Answer with either a string or a number. Elements of int64 and uint64 lists currently arrive as numbers. Narrower integers are plain numbers.

Documentation

• CLI reference, including monospace extension create and monospace extension build
• Connectors

Feedback

Open an issue to report a bug or suggest a change.

License

MIT