@dataworks-technology/data

Official SDK for the Dataworks Data Engine — authenticate, ingest live athlete metrics, subscribe to real-time data streams, and report errors.

Features

• Self-contained bundle — Apollo Client and AppSync subscription link are bundled in; consumers install nothing extra
• Dual format — ESM and CommonJS bundles included
• TypeScript-first — complete type declarations shipped with the package
• Built-in validation — metrics are validated before ingestion
• Real-time subscriptions — WebSocket-based live data streaming via AppSync Events API
• Typed GraphQL API client — execute any query, mutation, or subscription against the Data Engine schema
• Schema type exports — full codegen types available via @dataworks-technology/data/schema
• Cognito authentication — secure login with automatic token refresh

Prerequisites

You need a Dataworks developer account. Contact your Dataworks administrator to receive:

| Credential | Description | |---|---| | cognitoEndpoint | Cognito User Pool endpoint URL | | clientId | Cognito app client ID | | ingestUrl | API Gateway endpoint for metric ingestion | | errorUrl | API Gateway endpoint for error reporting | | realtimeUrl | AppSync Events API endpoint for subscriptions | | graphqlUrl (optional) | AppSync GraphQL HTTP endpoint | | Username + password | Your developer login credentials | | apiKey (optional) | AppSync Events API key for subscription-only access |

Installation

npm install @dataworks-technology/data

yarn add @dataworks-technology/data

pnpm add @dataworks-technology/data

bun add @dataworks-technology/data

Quick Start

import { DataClient } from "@dataworks-technology/data";

// Recommended: pass a named environment and the SDK resolves all endpoint URLs.
const dataworks = new DataClient({
  environment: "uat", // or "live"
  clientId: "your-client-id",
  apiKey: "your-appsync-events-api-key", // optional, subscription-only auth
});

// Authenticate
await dataworks.login("username", "password");

// Ingest metrics
await dataworks.ingest(
  [
    {
      athletePublicId: "ath-1",
      metric: "heartrate_calculated",
      value: 172,
      timestamp: Math.floor(Date.now() / 1000),
    },
  ],
  "event-id",
  "dataset-datasource-id",
);

API

new DataClient(config)

Create a new client instance. Provide either a named environment (the SDK resolves the endpoint URLs for you) or the explicit URLs. Explicit URLs always take precedence over the environment defaults.

// Environment-based (recommended)
const dataworks = new DataClient({
  environment: "uat", // "uat" | "live"
  clientId: "abc123",
});

// Explicit URLs (still supported; overrides environment defaults)
const dataworks = new DataClient({
  cognitoEndpoint: "https://cognito-idp.eu-west-1.amazonaws.com/",
  clientId: "abc123",
  ingestUrl: "https://ingest.dataworks.live",
  errorUrl: "https://errors.dataworks.live",
  realtimeUrl: "https://realtime.dataworks.live",
  graphqlUrl: "https://api-id.appsync-api.eu-west-1.amazonaws.com/graphql",
});

API Key Authentication (Subscription-Only)

For read-only access to real-time subscriptions without Cognito credentials:

import { DataClient, SubscriptionEvent } from "@dataworks-technology/data";

const dataworks = new DataClient({
  cognitoEndpoint: "https://cognito-idp.eu-west-1.amazonaws.com/",
  clientId: "unused",
  ingestUrl: "https://unused.dataworks.live",
  errorUrl: "https://unused.dataworks.live",
  realtimeUrl: "https://realtime.dataworks.live",
  apiKey: "da2-xxxxxxxxxx", // AppSync Events API key
});

// No login required — subscribe directly
const sub = dataworks.subscribe("dataworks/*", (event: SubscriptionEvent) => {
  console.log(event.athleteId, event.metrics);
});

Note: API key authentication only supports subscriptions. Ingest and error reporting require Cognito credentials. If both apiKey and Cognito credentials are provided, Cognito takes precedence.

dataworks.login(username, password)

Authenticate with Cognito. Must be called before any other operation (unless using API key for subscriptions).

const result = await dataworks.login("username", "password");
// result: { accessToken, idToken, refreshToken, tenant }

dataworks.ingest(metrics, eventId, datasetDatasourceId)

Send metric data points to the Data Engine. Invalid metrics are dropped with warnings.

await dataworks.ingest(
  [
    { athletePublicId: "ath-1", metric: "heartrate_calculated", value: 172, timestamp: 1700000000 },
    { athletePublicId: "ath-1", metric: "speed_calculated", value: 4.2, timestamp: 1700000000 },
  ],
  "event-123",
  "ds-456",
);

dataworks.ingestWithResult(metrics, eventId, datasetDatasourceId, options?)

Ingest with structured validation diagnostics.

• Default mode (best-effort): invalid metrics are dropped and valid metrics continue.
• Strict mode (strict): throws MetricValidationError if any metric is invalid.

const result = await dataworks.ingestWithResult(metrics, "event-123", "ds-456", {
  validationMode: "best-effort",
});

console.log(result.acceptedCount, result.droppedCount, result.requestSent);

dataworks.subscribe(channel, onEvent, onError?)

Subscribe to real-time data events via WebSocket.

Tip: start with dataworks/1/1/* to inspect all metrics for a dataset-datasource/event pair, then narrow to a specific metric channel such as dataworks/1/1/heartrate.

import { SubscriptionEvent, SubscriptionError } from "@dataworks-technology/data";

const subscription = dataworks.subscribe(
  "dataworks/1/1/heartrate",
  (event: SubscriptionEvent) => {
    console.log("Athlete:", event.athleteId);
    console.log("Metrics:", event.metrics);
  },
  (error: SubscriptionError) => {
    console.error("Error:", error.message, "Code:", error.code);
  },
);

// Later: close the subscription
subscription.close();

Event Type: SubscriptionEvent

The standard Data Engine event shape:

interface SubscriptionEvent {
  name: string;              // e.g. "heartrateingested"
  timestamp: number;         // Unix timestamp (milliseconds)
  timestampSec: number;      // Unix timestamp (seconds)
  eventId: string;
  datasetDatasourceId: string;
  athleteId: string;
  metrics: SubscriptionMetric[];
}

interface SubscriptionMetric {
  metric: string;            // e.g. "current", "heartrate"
  value: number | string;
}

Custom Event Types

If your channel publishes a different format, use the generic type parameter:

interface MyCustomEvent {
  athleteId: string;
  customField: number;
}

dataworks.subscribe<MyCustomEvent>("custom/*", (event) => {
  console.log(event.customField); // TypeScript knows this exists
});

Error Type: SubscriptionError

interface SubscriptionError {
  message: string;
  code: "CONNECTION_ERROR" | "UNAUTHORIZED" | "SUBSCRIPTION_ERROR" 
      | "PARSE_ERROR" | "RECONNECT_FAILED" | "UNKNOWN";
  cause?: Error;
}

| Code | Description | |------|-------------| | CONNECTION_ERROR | WebSocket connection failed (network, TLS) | | UNAUTHORIZED | Auth token expired or invalid | | SUBSCRIPTION_ERROR | AppSync subscription error (invalid channel, server error) | | PARSE_ERROR | Malformed message or event payload | | RECONNECT_FAILED | Token refresh failed after auth expiry | | UNKNOWN | Unexpected error |

dataworks.query(query, variables?, options?)

Execute a typed GraphQL query against the Data AppSync schema.

const result = await dataworks.query<{
  listEvents: { items: Array<{ id: number; name: string }> };
}>(
  `query ListEvents($tenant: String!) {
    listEvents(tenant: $tenant) {
      items { id name }
    }
  }`,
  { tenant: "demo-tenant" },
);

console.log(result.listEvents.items);

dataworks.mutate(mutation, variables?, options?)

Execute a typed GraphQL mutation.

const result = await dataworks.mutate<{
  startEvent: { id: number; name: string };
}>(
  `mutation StartEvent($id: Int!, $tenant: String!) {
    startEvent(id: $id, tenant: $tenant) { id name }
  }`,
  { id: 42, tenant: "demo-tenant" },
);

dataworks.subscribeGraphQL(subscription, onEvent, onError?, variables?)

Subscribe to schema-level GraphQL subscriptions. Backed by Apollo Client + aws-appsync-subscription-link internally — AppSync Cognito auth and WebSocket protocol are handled automatically.

The Apollo client is lazily created on the first subscribeGraphQL() call and reused for all subsequent subscriptions on the same DataClient instance.

import type { Dataset, Maybe } from "@dataworks-technology/data/schema";

const sub = dataworks.subscribeGraphQL<{ onUpdateDataset: Maybe<Dataset> }>(
  `subscription OnUpdateDataset {
    onUpdateDataset {
      id publicId clientId name
      eventId eventName eventLocation eventType eventStartsAt
      createdAt updatedAt
    }
  }`,
  (payload) => {
    console.log(payload.onUpdateDataset);
  },
  (error) => {
    console.error(error.message);
  },
);

sub.close();

Schema Types

All Data Engine GraphQL types are exported from the schema subpath. Use these instead of hand-rolling your own:

import type {
  Dataset,
  DatasetConnection,
  Maybe,
  CreateDatasetInput,
  UpdateDatasetInput,
  DatasetConditionInput,
  Athlete,
  Participant,
  Datasource,
  // ... all entity, connection, filter, and input types
} from "@dataworks-technology/data/schema";

dataworks.reportError(error)

Report an error to the Data Engine for monitoring and alerting.

await dataworks.reportError({
  datasetDatasourcePublicId: "ds-456",
  athletePublicId: "ath-123",
  clientId: 1,
  errorTitle: "Sensor disconnected",
  errorDescription: "BLE heart rate sensor lost connection at 00:42:15",
});

dataworks.isAuthenticated

Check if the client has valid credentials.

if (dataworks.isAuthenticated) {
  await dataworks.ingest(metrics, eventId, dsId);
}

dataworks.tenant

Get the tenant from the authenticated session (or null).

console.log(`Logged in as tenant: ${dataworks.tenant}`);

Validation Utilities

Standalone validation functions are exported for pre-checking metrics before ingestion:

DataClient.validateMetric(metric)

Returns null if valid, or a string describing the validation failure.

import { validateMetric } from "@dataworks-technology/data";

const error = validateMetric({
  metric: "heartrate",
  athleteId: "1",
  value: 172,
  timestamp: 1700000000,
});

if (error) {
  console.warn("Invalid metric:", error);
}

DataClient.filterValidMetrics(metrics, warnFn)

Filters an array, keeping only valid metrics. Calls warnFn for each invalid item.

import { filterValidMetrics } from "@dataworks-technology/data";

const valid = filterValidMetrics(rawMetrics, (msg) => console.warn(msg));

Validation Rules

| Field | Rule | |---|---| | metric | Non-empty string | | athleteId | Non-empty string | | value | Number (finite, not NaN) or non-empty string | | timestamp | Positive integer (Unix seconds) |

Types

All types are exported for use in your application:

import type {
  DataClientConfig,
  LoginResult,
  Metric,
  MetricItem,
  RawMetricItem,
  MetricsPayload,
  ErrorReport,
  SubscriptionHandler,
  ErrorHandler,
  Subscription,
  ValidationMode,
  DroppedMetric,
  IngestOptions,
  IngestResult,
} from "@dataworks-technology/data";

Key Interfaces

interface Metric {
  athleteId: string;
  metric: string;
  value: number | string;
  timestamp: number; // Unix seconds
}

interface ErrorReport {
  datasetDatasourceId: string;
  athleteId: string;
  clientId: number;
  errorTitle: string;
  errorDescription: string;
}

interface LoginResult {
  accessToken: string;
  idToken: string;
  refreshToken: string;
  tenant: string;
}

Error Handling

All async methods throw on failure. Wrap calls in try/catch:

try {
  await dataworks.login("user", "pass");
} catch (err) {
  // Authentication failed (invalid credentials, network error, etc.)
}

try {
  await dataworks.ingest(metrics, eventId, dsId);
} catch (err) {
  // Ingestion failed (401 expired token, 5xx server error, etc.)
}

Calling ingest(), reportError(), or subscribe() before login() throws immediately.

Exception: subscribe() can run without login() when apiKey is configured. This mode is subscription-only — ingest() and reportError() still require login().

When both Cognito credentials (login(username, password) + clientId) and apiKey are available, the SDK always uses Cognito auth for subscriptions.

Requirements

• Node.js ≥ 18 (uses native fetch)
• ESM or CommonJS — both module formats are included
• TypeScript ≥ 5.0 (optional — works with plain JavaScript too)
• Browser — compatible with any environment that has fetch and WebSocket

Derived Metrics — Round-Trip Example

A core use case: subscribe to live metrics, apply your own calculations externally, and push derived metrics back into the Data Engine. This creates a feedback loop where the platform stores both raw and enriched data.

Concept

Data Engine → subscribe("dataworks/{datasetDatasourceId}/{eventId}/heartrate") → Your App → calculate rolling avg → ingest("heartrate_rolling_avg_calculated") → Data Engine

Any metric you ingest is treated the same as raw sensor data — it flows through the enrichment pipeline (avg/min/max/zones), gets stored, and is available via subscriptions and the GraphQL API.

TypeScript Example

import { DataClient } from "@dataworks-technology/data";

const dataworks = new DataClient({
  cognitoEndpoint: "https://cognito-idp.eu-west-1.amazonaws.com/",
  clientId: "your-client-id",
  ingestUrl: "https://your-ingest-endpoint.dataworks.live",
  errorUrl: "https://your-error-endpoint.dataworks.live",
  realtimeUrl: "https://your-realtime-endpoint.dataworks.live",
});

await dataworks.login("developer", "password");

// 1. Subscribe to raw heart rate data
const buffer: number[] = [];
const datasetDatasourceId = "1";
const eventId = "1";
const heartrateChannel = `dataworks/${datasetDatasourceId}/${eventId}/heartrate`;

dataworks.subscribe(heartrateChannel, (event) => {
  // No metric guard needed here: the channel already filters to heartrate.

  // 2. Calculate a rolling 3-point average
  buffer.push(event.value);
  if (buffer.length > 3) buffer.shift();
  const avg = buffer.reduce((a, b) => a + b, 0) / buffer.length;

  // 3. Push the derived metric back into the engine
  dataworks.ingest(
    [
      {
        athleteId: event.athleteId,
        metric: "heartrate_rolling_avg_calculated",
        value: Math.round(avg * 10) / 10,
        timestamp: event.timestamp,
      },
    ],
    event.eventId,
    event.datasetDatasourceId,
  );
});

Python Example (Cross-Platform)

The same flow works from any language. A full working Python demo is included at test/demo/round-trip.py — it proves every SDK capability end-to-end:

| Phase | What it does | SDK equivalent | |-------|-------------|----------------| | 1. Authenticate | Cognito USER_PASSWORD_AUTH | dataworks.login() | | 2. Subscribe | WebSocket → AppSync Events API | dataworks.subscribe() | | 3. Receive | Catch events on the subscriber | onEvent callback | | 4. Calculate | Write to CSV/Google Sheets, compute rolling avg + HR zones | Your business logic | | 5. Re-ingest | POST enriched metrics back to the Data Engine | dataworks.ingest() | | 6. Report Error | Send error through the error pipeline | dataworks.reportError() |

Run the demo

# Install Python dependencies
pip install websocket-client requests

# Run from the repo root (reads test.env for credentials)
python test/demo/round-trip.py

Output

PHASE 4: WRITE DATA + CALCULATE
  Enriched data:
    HR=130 → avg=130.0, zone=Zone 2 (Aerobic), delta=—
    HR=140 → avg=135.0, zone=Zone 3 (Tempo), delta=+10
    HR=150 → avg=140.0, zone=Zone 3 (Tempo), delta=+10
    HR=160 → avg=150.0, zone=Zone 4 (Threshold), delta=+10
    HR=170 → avg=160.0, zone=Zone 4 (Threshold), delta=+10

PHASE 5: RE-INGEST ENRICHED METRICS
    → heartrate_rolling_avg=130.0
    → heartrate_rolling_avg=135.0
    → heartrate_rolling_avg=140.0
    → heartrate_rolling_avg=150.0
    → heartrate_rolling_avg=160.0
  ✓ Ingested (200)

The raw heartrate values (130–170 bpm) are transformed into heartrate_rolling_avg_calculated derived metrics and pushed back into the platform. These derived metrics are then available to all consumers (Console, Visual, Moments triggers) just like any other metric.

Google Sheets (optional)

The demo can write to Google Sheets instead of CSV — formulas compute rolling averages, HR zone classification, and deltas in the spreadsheet before re-ingesting. See test/demo/README.md for setup.

Ideas for Derived Metrics

| Derived metric | Calculation | Use case | |---|---|---| | heartrate_rolling_avg_calculated | 3-point moving average | Smooth noisy sensor data | | heartrate_zone | Zone classification (1–5) | Training load analysis | | speed_efficiency | Speed / heart rate ratio | Fatigue detection | | power_to_weight | Power / athlete weight | Normalised performance | | pace_delta | Current pace − target pace | Real-time coaching alerts | | fatigue_index | Custom formula over time window | Trigger warnings via Moments engine |

Documentation

Full documentation with guides, examples, and detailed API reference:

https://data-sdk-docs.dataworks.live

Development

Internal contributors only — this section covers building, testing, and publishing the package.

This package publishes to public npm (registry.npmjs.org), not to the internal CodeArtifact registry used by @dataworks/sdk. The packages/sdk/.npmrc and publishConfig in package.json enforce this.

Build

bun run build    # tsup → dist/ (ESM + CJS + DTS)
bun run dev      # tsup --watch (rebuild on save)
bun run test     # vitest unit tests (mocked — no infra needed)

Local Development

To test SDK changes locally before publishing:

Option A — npm pack (recommended, simulates real install):

# In packages/sdk/
bun run build
npm pack                    # creates dataworks-technology-data-0.1.4.tgz

# In your consumer project (e.g. test/demo/)
npm install ../../packages/sdk/dataworks-technology-data-0.1.4.tgz

Option B — bun link (faster iteration):

# In packages/sdk/
bun run build
bun link

# In your consumer project
bun link @dataworks-technology/data

Option C — watch mode + link (live reload):

# Terminal 1: packages/sdk/
bun run dev                 # rebuilds dist/ on every save

# Terminal 2: consumer project
bun link @dataworks-technology/data
npx tsx demo.ts             # picks up latest build automatically

Tip: After local testing, remember to bun unlink or reinstall from npm before committing.

Publish

Login to npm first (one-time):

npm login --registry https://registry.npmjs.org/

Then from packages/sdk/:

bun run publish:patch   # bump patch + publish
bun run publish:minor   # bump minor + publish
bun run publish:major   # bump major + publish

E2E Tests

The e2e test suite (test/e2e-public-sdk.test.ts) validates the full external developer flow against deployed infrastructure. It runs against live AWS services and proves the public SDK works end-to-end across all three stacks (Shared, Data, SDK).

Prerequisites

Deploy all three stacks first — make deploy generates test.env with the required env vars:

| Env var | Source | Purpose | |---|---|---| | FRONTEND_CLIENT_ID | Shared stack → Cognito | App client for USER_PASSWORD_AUTH | | API_URL | Data stack → API Gateway | Metric ingestion endpoint | | ERROR_URL | Data stack → API Gateway | Error reporting endpoint | | REALTIME_URL | Data stack → AppSync Events API | WebSocket subscription endpoint | | COGNITO_BASE_DOMAIN | Shared stack → Cognito | IDP endpoint for auth | | DEVELOPER_USERNAME | Shared stack → Cognito | Test user in the developer group | | DEVELOPER_PASSWORD | Shared stack → Cognito | Test user password |

Run

bun x vitest run test/e2e-public-sdk.test.ts

What each test proves

| # | Area | Test | What it validates | How to verify manually | |---|---|---|---|---| | 1 | Login | should authenticate testdeveloper | USER_PASSWORD_AUTH works with frontend client | AWS Console → Cognito → User Pool → Users → testdeveloper exists in developer group | | 2 | Login | should extract tenant from JWT | custom:tenant claim is present in ID token | Decode the JWT at jwt.io — look for custom:tenant | | 3 | Login | should include resource server scopes | Pre-token Lambda injects dataworks.live/read and /write | AWS Console → Cognito → User Pool → Lambda triggers → Pre token generation V2 is set | | 4 | Login | should reject invalid credentials | Bad password returns error, not a token | Try dataworks.login("testdeveloper", "wrong") — should throw | | 5 | Ingest | should ingest valid metrics | JWT passes API Gateway authoriser, Lambda processes payload | AWS Console → CloudWatch → Log group for the ingest Lambda — look for the metric | | 6 | Ingest | should filter invalid metrics | Client-side validation drops bad metrics before sending | The stderr output shows Dropping invalid metric — local validation, no server call for bad data | | 7 | Ingest | should throw when not authenticated | ingest() before login() throws immediately | Client-side guard — no network call made | | 8 | Error | should reach error endpoint | Auth + API Gateway routing work (500 = Lambda ran, FK failed) | AWS Console → CloudWatch → Log group for error processor Lambda | | 9 | Error | should throw when not authenticated | reportError() before login() throws immediately | Client-side guard | | 10 | Subscribe | should throw when not authenticated | subscribe() before login() throws immediately | Client-side guard | | 11 | Subscribe | should create a subscription | Skipped in Node.js (no WebSocket global) — see below | Use the Python verification script | | 12–14 | Validation | validateMetric / filterValidMetrics | Static validation utilities work correctly | Pure functions — no infra dependency |

Verifying WebSocket subscribe (test 11)

The subscribe test is skipped in the Node.js Vitest environment because globalThis.WebSocket is not available. To prove subscribe works end-to-end, use the standalone Python script:

pip install websocket-client requests
python test/verify-subscribe.py

This script:

1. Authenticates via Cognito USER_PASSWORD_AUTH (same as the SDK)
2. Opens a WebSocket to the AppSync Events API with base64URL-encoded JWT auth subprotocols
3. Sends connection_init and receives connection_ack from AppSync
4. Sends a subscribe message and receives subscribe_success
5. Closes cleanly

Expected output:

Loading config from test.env

1. Authenticating as 'testdeveloper'...
   ✓ Login successful
   Scopes: aws.cognito.signin.user.admin https://dataworks.live/write https://dataworks.live/read
   Tenant: 2x9AOzBvDBgllLB0CXcCrPjK03C

2. Connecting to wss://event-api-chris-dev.dataworks.live/event/realtime...
   Protocol: aws-appsync-event-ws (manual)
   Received: connection_ack
   ✓ WebSocket connection established

3. Subscribing to channel '/dataworks/e2e-verify'...
   Received: subscribe_success
   ✓ Subscribed to /dataworks/e2e-verify

4. WebSocket closed cleanly

============================================================
ALL CHECKS PASSED — subscribe flow works end-to-end
============================================================

License

MIT © Dataworks Technology