Syncables

This repository contains a sync engine that can be used to download (and in the future also update) a collection of objects from an API. It can be configured declaratively by extending the OpenAPI spec.

Find the path that can be used to fetch a (paged) collection of items. This will typically look something like this:

paths:
 /widgets:
    get:
      parameters:
        - description: Token specifying which result page to return. Optional.
          in: query
          name: pageToken
          schema:
            type: string
      responses:
        "200":
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/CalendarList"
components:
  schemas:
    CalendarList:
      properties:
        results:
          items:
            properties:
              backgroundColor:
                type: string
          type: array
        nextPageToken:
          description: Token used to access the next page of this result. Omitted if no further results are available.
          type: string
      type: object

Under components.paginationSchemes.default you can add a pagination scheme as explained in DESIGN.md.

Under paths['/widgets']['get']['responses']['200']['content']['application/json'], add an array of object syncables, in each of which you can specify:

• type: collection or item. Defaults to collection.
• name: a string descriptor of the collection, e.g. "widgets"
• query: an object containing query parameters to add in addition to the pagination-related ones
• defaultPageSize: tell Syncable how many items per page (max) to expect by default
• forcePageSize: if possible, let Syncable tell the API which page size to use
• idField to indicate which property of response items is used as the unique identifier (currently only used for confirmationBased pagination).
• params for templated syncables an object whose keys are template identifiers like calendarId and whose values are fields from other syncables, like calendars.id. Use with care, because it will try to exhaustively fetch collections for all combinations of parameters.

Usage

Create the OAD

You start with an OAD, for instance one from APIs-guru. Probably the API you want to use is not specifying syncables yet, so you'll need to add those yourself using an overlay, something like the ones you see here.

Install syncables

Depending on your preferred package manager, you can run something like this to install syncables from npm:

pnpm install syncables

Write your code

Now you have the AOD with the definition of the syncable, and the type for the items you will sync, you can write code like this:

import { readFileSync } from 'fs';
import { components } from './types/google-calendar.js';
import { Syncer } from './syncer.js';

type Entry = components['schemas']['CalendarListEntry'];
const specFilename = './openapi/generated/google-calendar.yaml';
const specStr = readFileSync(specFilename).toString();

const syncer = new Syncer({
  specStr,
  authHeaders: {
    Authorization: `Bearer ${process.env.GOOGLE_BEARER_TOKEN}`,
  },
  dbConn:
    'postgresql://syncables:syncables@localhost:5432/syncables?sslmode=disable',
});

const allTables = await syncer.fullFetch();
// Data coming out of Syncer adheres to types from .d.ts files, autocomplete works for this in VS Code, e.g.:
const calendarEntries: Entry[] = allTables.calendars as Entry[];
console.log(calendarEntries[0].backgroundColor);

Authorization

We ignore the security entry in the OAD root (design decision: https://github.com/tubsproject/syncables/issues/48). Instead, in src/dev-example.ts you see how the securityScheme to use is explicitly picked. If your API is called ACME Widgets, then its environment variable names will start with 'ACME_WIDGETS_'. Substitute that string in the following examples of supported security schemes:

• for type='oauth2', flow='clientCredentials' you will need to specify tokenUrl (and optionally audience, scope and vendorApiKey), and add ACME_WIDGETS_CLIENT_ID and ACME_WIDGETS_CLIENT_SECRET at runtime.
• for type='oauth2', flow='authorizationCode' you will need to specify authorizationUrl, tokenUrl and scopes, and add ACME_WIDGETS_CLIENT_ID and ACME_WIDGETS_CLIENT_SECRET at runtime.
• for type='cognito' you will need to specify cognitoUrl, and add ACME_WIDGETS_CLIENT_ID, ACME_WIDGETS_USERNAME and ACME_WIDGETS_PASSWORD at runtime.
• for type='api-key' and 'string-token' you will need to add ACME_WIDGETS_API_KEY at runtime.
• for type='api-key-password' you will need to add ACME_WIDGETS_API_KEY and ACME_WIDGETS_PASSWORD at runtime.
• for type='basic' you will need to add ACME_WIDGETS_USER and ACME_WIDGETS_PASSWORD at runtime.
• for type='acube' you will need to add ACME_WIDGETS_EMAIL and ACME_WIDGETS_PASSWORD at runtime.

Dev Example

• In the Google Cloud Dashboard create an OAuth client ID with http://localhost:8000 as an authorized JavaScript origin and http://localhost:8000/callback as an authorized redirect URI.
• Enable the calendar API
• Set the GOOGLE_CLIENT_ID and GOOGLE_CLIENT_SECRET environment variables. Check: echo $GOOGLE_CLIENT_ID and $GOOGLE_CLIENT_SECRET
• pnpm build
• Similarly you can set MONEYBIRD_CLIENT_ID and MONEYBIRD_CLIENT_SECRET.
• Now you can run the example:

docker compose up -d
pnpm start
pnpm start events,contacts
docker exec -it db psql postgresql://syncables:syncables@localhost:5432/syncables -c "\d+"

It will check for existing bearer tokens in the .tokens folder and initiate OAuth flows as needed. API responses will be cached in the .fetch-cache folder for easier development.

Development

git clone https://github.com/tubsproject/syncables
cd syncables
pnpm install
pnpm generate
pnpm build
pnpm test
docker exec -it db psql postgresql://syncables:syncables@localhost:5432/db_unit_tests -c "\d+"
pnpm prettier
pnpm login
pnpm publish

Acknowledgements

The __tests__/integration/mock-server' folder is a copy of [@scalar/mock-server`](https://github.com/scalar/scalar/blob/main/packages/mock-server/src) - thanks Scalar!