@collective-intelligence/extensions-sdk-backend

Framework-neutral executable contracts for backend extensions hosted by Collective Intelligence.

Use this package for executable component base classes, host capability keys, and the runtime context supplied by a host. Common declarations, schemas, fields, and manifests remain in @collective-intelligence/extensions-sdk.

Event-listener execution contracts (EventListenerRuntime, its factory, and its host-supplied context) and the callback-bearing KnowledgeConnectorOAuthProvider also live here. Their portable declarations, request/result envelopes, and OAuth token data remain in the common SDK.

DatasetParser implements file parsing through analyze(input) and parseRecords(input), returning Awaitable<DatasetParserResult<DatasetParserAnalysis>> and Awaitable<DatasetParserResult<DatasetParserRecords>>. Bind a no-argument subclass with the base SDK's declareDatasetParser. Package runtime calls use JSON bytes: readonly number[]; the runtime validates each byte (0–255) and a positive safe integer maxRows, then reconstructs Uint8Array. Expected errors are plain JSON failure envelopes; unexpected exceptions remain rejections.

ManagedDatasetProvider exposes readiness, datasetProvision, datasetIngest, datasetQuery, datasetInspect, and datasetDelete. ExternalDatasetProvider exposes readiness, listSources, discoverDefinition, datasetQuery, and datasetInspect. Each method takes its portable dataset input plus PackageRuntimeContext and returns an Awaitable<DatasetProviderResult<T>>. Bind them with declareManagedDatasetProvider and declareExternalDatasetProvider. The host supplies connection identity and optional provider endpoint settings in context.config; an implementation validates those settings and uses context.resolveConnection with an allowed connection-template set to obtain authorized credentials. Original dataset inputs remain unchanged. Neither credentials nor runtime configuration enter the declaration. Runtime dispatch admits only the methods owned by each kind; parsers cannot execute connector, knowledge-connector, documents-connector, tool, or workflow hooks.

PackageDocumentsConnector implements file inventory and content fetch through pull(input, context) and fetchContent(input, context). Bind a no-argument subclass with the base SDK's declareDocumentsConnector. pull receives { cursor?, selectedRootRemoteIds? } and returns JSON { items, nextCursor? }. fetchContent receives { contentRevision, remoteFileId } and returns { bytes: Uint8Array, mimeType?, extension?, filename?, source? }, describing the actual stored representation after any provider export. Package runtime calls use the default [value, context, progress] invocation. Before JSON encoding, fetchContent converts bytes into bytesBase64 with standard base64 and preserves the optional string fields and source; non-Uint8Array bytes and non-string format fields are rejected. Raw Uint8Array never crosses the JSON invocation boundary.

source is the base SDK's DocumentsSource ({ webUrl: string }) and refers to the live original individual file, separately from exported/stored Version bytes. A present source survives serialization; source: null explicitly clears an observed destination; omitted source stays omitted and means no observation. Neither a collection-root URL nor an ID-derived URL substitutes for a missing per-file URL. The host owns URL usability and authorization policy.

A connector may implement fetchSource({ remoteFileId }, context) for metadata-only refresh of existing imports and unchanged revisions. Hosted callers invoke fetch_source; no content revision or byte fetch is required. The result is a DocumentsSource or explicit null. Missing optional hooks remain unsupported operations, distinct from a successful null result. Provider failures reject the invocation; they must not masquerade as missing metadata. Read-only browse operations listDrives({ cursor? }) and listFolders({ parentRemoteId, cursor? }) return { items, nextCursor? } pages of drives and direct child folders; every returned remoteId is a selectable pull / proveRootAncestry root. Knowledge-connector operations are unchanged. proveRootAncestry passes provider-reported selected-root displayName and webUrl through the hosted JSON invocation when present.

import {
  PackageTool,
  type PackageRuntimeContext
} from "@collective-intelligence/extensions-sdk-backend"

export class EchoTool extends PackageTool<{ text: string }, string> {
  execute(
    input: { text: string },
    _context: PackageRuntimeContext
  ): PromiseLike<string> {
    return Promise.resolve(input.text)
  }
}

A tool whose declaration sets healthCheck: { description, capabilities? } implements the optional healthCheck(context) method (health_check operation). It receives the same runtime context as execute (settings, connection resolution, authority) but no input, and bypasses the tool input and output schemas. It returns a PackageToolHealthReport: { status: "healthy" | "degraded" | "failing", summary?, remediation?, capabilities?: [{ name, status, summary?, remediation?, details? }] }. Probes must not mutate provider data or return workspace content; the host stamps the check time (checkedAt), redacts credential values, and persists the report. Details keep only camelCase identifier, code, count, and flag metadata: id, *Id, ids, *Ids, count, *Count, statusCode, action, category, code, errorCode, permission, provider, providerFamily, resourceType, scopeKind, authorityMode, configured, permissionGranted, and reachable; other keys are dropped.

The API exposes ordinary values and PromiseLike results. Effect services and private host implementations are intentionally outside this package.

For local validation, test/consumer is a private workspace with explicit SDK dependencies. Run bun run test:pack to verify the published boundary: it copies the consumer without local node_modules and installs the packed SDK artifacts in a temporary directory, so workspace links cannot satisfy its imports.

The fixture runs lint with its own generated configuration and declared tooling dependencies. This keeps the SDK rules active for consumer code and resolves the generated lint plugin from the fixture's scope.

Worker-local package runtime

Export one runtime over the same ordinary definitions used to declare a package:

import { createBackendPackageRuntime } from "@collective-intelligence/extensions-sdk-backend"
import { packageDefinitions } from "./definitions"

export const runtime = createBackendPackageRuntime(packageDefinitions)

The host calls runtime.invoke({ packageFqn, componentFqn, operation, input, context, signal, onProgress }). Package identity is independent of the artifact that acquired it: an aggregate built-in artifact may contain several packages. Duplicate package or component identities fail during runtime creation. The operation is checked against the component kind before constructing its class. Authors keep their existing methods; they do not wrap each method as a hosted contribution. Existing hosted contribution declarations remain separate exports.

Tool input decoders and output encoders execute beside the class. Connection credentials, workflow input and output values, and workflow field values use retained native sidecars. Workflow output encoding preserves the result envelope and directives. Field parts receives context.config; canonicalize and validateLocal receive the value and that configuration. Other methods receive input and the invocation context; connection discovery also receives the progress callback. Listener routing and field transformation methods are addressed by snake_case operation values. A void result becomes JSON null.

BackendPackageSchemaError exposes stage, location, and issues. Package exceptions retain their original identity and structured properties in the local adapter; a worker transport must preserve those fields. Results and progress must be JSON-compatible after native encoding. No constructor, codec, capability, credential, or cancellation signal is serialized into the manifest.

Every call supplies an AbortSignal, also available as context.signal. Cancellation is checked before decoding, before and after execution, and progress is suppressed after cancellation. The host owns finite deadlines, cancellation grace, forced worker termination, and shutdown. context.deadlineAt, when supplied, is the host's absolute deadline in epoch milliseconds. The adapter does not retry an invocation or authorize a caller; the host injects the authorized context and capabilities for that invocation.

Tenant state

One Worker serves every Organization that invokes the same package release, and several invocations may run in it at once. The per-call context is the only tenant carrier: read the Organization, capabilities, store and configuration from the context of the current call. Do not keep tenant data, contexts, capabilities or credentials in module-level variables or caches that outlive the call. The host authorizes each capability call from its own record of that invocation, so a context kept from an earlier call is rejected once that call ends. Workers are a shared execution environment, not a security sandbox.

Workflow nodes

Use the base SDK's declareWorkflowNode with a PackageWorkflowNode subclass. execute(input, context) returns { outputs } with one value for each output port that the node ends on, or a directive, as an ordinary value or a PromiseLike result.

A node reports a failure that its author can act on by throwing or rejecting with new PackageWorkflowNodeError({ code, message }). code is a stable lower snake_case string; message holds no secret. The host ends the node on its error output port with { code, message, details: {}, retryable: false }. When no edge leaves that port, the step fails and the run fails with it, so a failure is never lost. The host does not repeat the node after this failure, also when the author set retries on the node; it repeats a rejection of another kind. A timed-out attempt of a node that declares execution.repeatable: false is not repeated either, and neither is a rejection of another kind, because its call can have reached its target. A node must not return outputs.error itself: with no edge on the port, the step would complete. Any other rejection shows the author the generic code attempt_failed only. name, code and message are own JSON fields of the error, so a host that serializes the rejection still reads them.

Email providers

Use the base SDK's declareEmailProvider with a PackageEmailProvider subclass. send(input, context) receives one raw email and returns { messageId } as an ordinary value or a PromiseLike result. The provider renders nothing. input.scope is tenant or organization: the owner of the selection that sends. input.credentials holds the resolved credential field values of the connection template that the declaration names. input.message carries from (address and optional displayName), to, optional cc, bcc, and replyTo, subject, and optional html and text. The invocation operation is send.

A provider reports a failed send by throwing or rejecting with new PackageEmailSendError({ category, message, statusCode? }). category is invalid_message, rejected, transport, or configuration; statusCode is the HTTP status of the provider response, when there is one. Both are own JSON fields of the error, so a host that serializes the rejection still reads them.

Organization SSO

Use the base SDK's declareOrganizationSso with a PackageOrganizationSso subclass. authorizationUrl(request, context) and exchangeCode(request, context) use ordinary values or PromiseLike results. Requests carry a readonly credential record; exchange returns { subject, email, name? }. HTTP clients and other services enter through capabilities. The declaration includes providerId, connectionTemplateFqn, required and secret credential field names, and optional localized setup guides. This metadata participates in the canonical package manifest; the constructor remains worker-local.

For separate form, configuration, and port validation, the same invoke method accepts operation: "decode_schema" | "encode_schema" with a required schemaLocation from that component's retained sidecars. The adapter checks the sidecar against its portable projection and returns JSON. This preserves host boundaries that require decoded JSON; it deliberately rejects a nonportable native result. Native tool and workflow values execute through the ordinary method invocation so they stay in the worker. A location cannot select another component's sidecars.

Listener acquisition hooks are optional: parseSelector, normalizeInternal, normalizeWebhook, normalizeWebhookRequest, and poll. Missing hooks fail as unsupported operations; the runtime does not invent empty acquisition results. Existing canonicalSelector, routeKeys, and broadRouteKeys defaults remain. poll receives the watch request and authorized PackageRuntimeContext, including its cancellation signal. Internal normalization receives the JSON event envelope.

Webhook invocation uses explicit JSON envelopes:

• normalizeWebhook: input: { bodyBytes: number[] }.
• normalizeWebhookRequest: input: { bodyBytes: number[], headers: Record<string, string>, receivedAt?: string, url?: string }.

The adapter accepts only integer bytes from 0 through 255, reconstructs a Uint8Array, and preserves header names and values. Headers must be strings. url is the full URL that the vendor called, when the host knows it; a listener that verifies a signature over the URL reads it. receivedAt is the ISO 8601 time at which the host accepted the request; a listener checks a time limit of the vendor against it, because the host can verify later. The public BackendWebhookBodyInput and BackendWebhookRequestInput types name these envelopes. The corresponding class hooks receive native bytes or the native { body, headers, receivedAt?, url? } request, plus the invocation context.

Entity adapter extraProps uses BackendExtraPropsInput: input: { remote: JsonObject, schemaMeta: JsonObject | null, originalRemote?: JsonObject }. The adapter calls extraProps(remote, schemaMeta) with exactly two data arguments when originalRemote is absent. When present, it passes extraProps(remote, schemaMeta, originalRemote) so extras can use original provider metadata (such as file timestamps) even after mapped fields are pruned. It never substitutes the host context for schema metadata. Product hosts own constructing these envelopes and authorizing their context; package-internal helper methods remain outside the callable operation inventory.

PackageKnowledgeEntityAdapter<Remote, Local> and PackageKnowledgeRelationAdapter<Remote, Local> require remoteId(remote: Remote): Awaitable<string>. Providers return a stable, nonempty source identity, including any composite-key calculation. The remoteId worker operation receives the remote JSON value through the existing invocation path; local mapping and relation endpoints remain separate operations. Hosts must validate the returned identity before graph writes. There is no default id field reader and no identity derived from the local projection.

Hosts may supply PackageRuntimeContext.capabilityBindings, a ReadonlyMap<string, object> of already-authorized services bound once for the invocation. capability(key) must validate and return the same bound object. Bindings may contain synchronous data and nested asynchronous methods; the host owns transport and lifetime. This inventory grants no authority and never enters portable manifests.

Hosts may also supply optional PackageRuntimeContext.store with get, put, delete, and prefix list. Values are JSON objects only; nested arrays inside an object are allowed. Every put requires a TTL from 1 minute to 30 days. Keys are at most 512 bytes and values at most 64 KiB. Prefix list returns at most 100 keys and does not auto-walk. Isolation by organization and package FQN is host-stamped; SDK keys are caller keys. Last-write-wins; there is no compare-and-swap. Packages call requirePackageRuntimeStore(context) so omitted store fails those operations. assertPackageRuntimeStorePut is the shared cap check for authors and the host adapter in apps/backend (PackageRuntimeStoreService). The store never enters portable manifests.

Hosts may supply optional PackageRuntimeContext.contentUrl beside captureContent and readContent. contentUrl({ storageFileId }) gives { url, expiresAt }: a short-lived URL of a stored file, for a vendor whose API takes a URL in place of the bytes. The optional lifetimeSeconds asks for a longer life, for a link that a person opens; the host caps it at its maximum and expiresAt tells the real end. The host signs the URL, and anyone who holds it can read the file with no sign-in until expiresAt: ask for no more time than the recipient needs, and send the URL only to that recipient.

Tool invocations also receive read-only PackageRuntimeContext.authority (PackageRuntimeAuthority: mode admin_wide or fixed_user, with optional impersonatedUserId and impersonatedSubject) and, when a binding such as a skill tool binding selected the tool, bindingSettings, separate from the instance config. Packages may refuse work their authority does not allow; the host remains the enforcement point for visibility and approval.

A tool connection reference may name a discovered catalog collection with catalogCollection. Tool invocations then receive read-only connectionCatalogIds[slot]: the id of every resource in that collection of the bound connection's last discovery snapshot. A slot is omitted when it declares no collection, or the connection has no usable snapshot or lacks that collection; treat an omitted slot as unknown (unrestricted) and an empty list as nothing reachable.

The testAndDiscover connection operation accepts the ordinary credential input, discover: boolean, and an optional runtime-only onTested: () => Awaitable<void>. It decodes credentials once and uses one instance for test and discovery. When discovery is enabled, it awaits the tested observer before discovery; observer failure or cancellation prevents discovery. Disabled discovery skips the observer and returns { catalog: null }; enabled discovery returns { catalog }. The caller's invocation signal bounds observer acknowledgement and progress.

An entity adapter without the optional extraProps hook returns JSON null for that operation, allowing the host to apply its schema-field fallback. Implemented hooks retain their results and errors; a non-callable hook remains an error.

Entity adapters may implement normalizeImportedData(local, remote) to normalize mapped import data while retaining access to the original provider record. Declare normalizesImportedData: true to opt in: hosts dispatch this operation only when that manifest flag is true, preserving older runtime behavior when absent or false. Invoke it with operation normalizeImportedData and BackendNormalizeImportedDataInput: input: { local: JsonValue, remote: JsonValue }. The runtime passes these as two data arguments, without substituting runtime context or the cancellation signal. An absent hook returns local unchanged; non-callable hooks and implementation failures propagate to the caller. Without a declarative field map, normalization belongs in toLocal(remote).

Entity adapters may also implement normalizeFieldMapValue(value) and declare normalizesFieldMapValues: true. The host invokes it on each value read from an inbound field-map path before a rule casts or transforms that value. An absent hook returns the value unchanged; the method remains an executable sidecar.

Declare entitySchemaFqn: "*" for a connector-wide fallback adapter. Hosts select one exact schema adapter first, then one wildcard adapter; ambiguous matches are errors. The wildcard uses the existing declaration shape.

Enum value spelling

Repository-owned enum values use lower snake_case. This includes component kinds, dataset statuses, schema consumer profiles, and hosted contribution identity. Notification enum values retain their existing spelling. External protocol values, including HTTP methods and JSON Schema keywords and formats, retain the protocol spelling.

Invocation operation values such as fetch_content, dataset_query, and test_and_discover use snake_case; executable method names such as fetchContent, datasetQuery, and testAndDiscover retain their TypeScript spelling. The runtime translates operation values before dispatch.