@beetlio/connect

A TypeScript SDK and local CLI for authoring host-neutral data integrations.

Documentation · Examples · Contributing

[!WARNING] This project is experimental and currently at 0.0.0. APIs and local storage formats may change.

An integration declares its credentials, runtime validation, records, pagination, checkpoints, and sync behavior. The host owns secrets, HTTP authentication, retries, output, and state. Sync code never receives declared credentials directly.

Features

• Strict TypeScript authoring with Zod validation at runtime boundaries
• Bearer, basic, API-key, custom, and OAuth 2.0 authentication
• Authorization Code with PKCE and Salesforce device-code connection flows
• Connection verification and automatic OAuth refresh after a 401
• Cursor and offset pagination helpers
• Durable checkpoints for incremental syncs and atomic snapshot replacement
• Managed retries with Retry-After support
• NDJSON output from the local CLI

Install

Requires Node.js 24 or newer.

npm install @beetlio/connect

This installs both the SDK and the local beetl-connect executable.

Define an integration

Create a beetl.integration.ts file:

import { auth, defineIntegration, z } from "@beetlio/connect";

const Item = z.object({
  id: z.string(),
  name: z.string(),
});

export default defineIntegration({
  key: "example",
  displayName: "Example",
  connection: {
    baseUrl: "https://api.example.com",
    credentials: z.object({ token: z.string() }),
    auth: auth.bearer(),
    async verify(ctx) {
      const response = await ctx.fetch("/me");
      if (!response.ok) throw new Error("Connection verification failed");
    },
  },
  syncs: (defineSync) => [
    defineSync({
      key: "items",
      displayName: "Items",
      records: Item,
      primaryKey: ["id"],
      async run(ctx) {
        const response = await ctx.fetch("/items");
        if (!response.ok) throw new Error("Item request failed");
        await ctx.emit({ records: z.array(Item).parse(await response.json()) });
      },
    }),
  ],
});

Paths passed to ctx.fetch() must be relative to the configured origin and begin with /. The host validates configuration and injects authentication into each request.

CLI

| Command | Purpose | | --- | --- | | check | Validate an integration and list its syncs | | connect | Complete OAuth, verify the connection, and save credentials | | verify | Test stored or environment-provided credentials | | sync <key> | Run one sync and write NDJSON records |

Credential fields map from camel case to upper-snake-case environment variables. For example, apiKey maps to API_KEY and token maps to TOKEN.

set -gx TOKEN "<token>"
npx beetl-connect check --integration beetl.integration.ts
npx beetl-connect verify --integration beetl.integration.ts
npx beetl-connect sync items --integration beetl.integration.ts
set -e TOKEN

Incremental state is stored under .beetl/state. OAuth connections are stored under .beetl/connections with owner-only permissions. Connection files are not encrypted, so treat the local machine and working directory as trusted.

Sync model

Append syncs are the default: each run writes a new NDJSON file and can resume from its latest checkpoint. A sync can instead declare mode: "snapshot"; the local host replaces the previous output only after the new snapshot succeeds.

ctx.paginate() supports cursor and offset APIs. Integrations can also issue requests directly for custom pagination and checkpoint strategies. Retries apply to safe HTTP methods by default and can be configured per connection.

Examples

| Integration | Demonstrates | | --- | --- | | Basic dummy API | Definition, validation, verification, direct fetch, and snapshot output | | All features dummy API | OAuth, configuration, retries, headers, pagination, checkpoints, and logging | | Wikidata | A real unauthenticated public API with bounded cursor pagination |

The dummy integrations use the reserved api.example.com domain and local fixture servers. Wikidata structured data is available under CC0; follow Wikimedia's API usage guidelines when adapting the public example.

Run the Wikidata example from this repository without authentication:

npm run cli -- sync entities \
  --integration examples/wikidata/beetl.integration.ts \
  --connection-config '{"userAgent":"my-wikidata-sync/1.0 ([email protected])"}' \
  --sync-config '{"search":"open source","language":"en","maxResults":25}'

Replace the example email with your contact information. The command writes a timestamped NDJSON snapshot to the current directory.

Development

npm install
npm run check
npm test

Current scope

The local host supports one OAuth connection per integration. Hosted execution, multi-connection profiles, encrypted credential storage, deployment, and compatibility manifests are not implemented.

Contributing

See CONTRIBUTING.md.

License

Licensed under the Apache License 2.0.