@dataworks-technology/data

Official SDK for the Dataworks Data Engine — authenticate, ingest live athlete metrics, subscribe to real-time data streams, and report errors.

Features

• Zero dependencies — fully self-contained, no transitive installs
• Dual format — ESM and CommonJS bundles included
• TypeScript-first — complete type declarations shipped with the package
• Built-in validation — metrics are validated before ingestion
• Real-time subscriptions — WebSocket-based live data streaming
• Cognito authentication — secure login with automatic token management

Prerequisites

You need a Dataworks developer account. Contact your Dataworks administrator to receive:

| Credential | Description | |---|---| | cognitoEndpoint | Cognito User Pool endpoint URL | | clientId | Cognito app client ID | | ingestUrl | API Gateway endpoint for metric ingestion | | errorUrl | API Gateway endpoint for error reporting | | realtimeUrl | AppSync Events API endpoint for subscriptions | | Username + password | Your developer login credentials |

Installation

npm install @dataworks-technology/data

yarn add @dataworks-technology/data

pnpm add @dataworks-technology/data

bun add @dataworks-technology/data

Quick Start

import { DataClient } from "@dataworks-technology/data";

const dataworks = new DataClient({
  cognitoEndpoint: "https://cognito-idp.eu-west-1.amazonaws.com/",
  clientId: "your-client-id",
  ingestUrl: "https://your-ingest-endpoint.dataworks.live",
  errorUrl: "https://your-error-endpoint.dataworks.live",
  realtimeUrl: "https://your-realtime-endpoint.dataworks.live",
});

// Authenticate
await dataworks.login("username", "password");

// Ingest metrics
await dataworks.ingest(
  [
    {
      athleteId: "athlete-1",
      metric: "heartrate_calculated",
      value: 172,
      timestamp: Math.floor(Date.now() / 1000),
    },
  ],
  "event-id",
  "dataset-datasource-id",
);

API

new DataClient(config)

Create a new client instance.

const dataworks = new DataClient({
  cognitoEndpoint: "https://cognito-idp.eu-west-1.amazonaws.com/",
  clientId: "abc123",
  ingestUrl: "https://ingest.dataworks.live",
  errorUrl: "https://errors.dataworks.live",
  realtimeUrl: "https://realtime.dataworks.live",
});

dataworks.login(username, password)

Authenticate with Cognito. Must be called before any other operation.

const result = await dataworks.login("username", "password");
// result: { accessToken, idToken, refreshToken, tenant }

dataworks.ingest(metrics, eventId, datasetDatasourceId)

Send metric data points to the Data Engine. Invalid metrics are automatically filtered out.

await dataworks.ingest(
  [
    { athleteId: "1", metric: "heartrate_calculated", value: 172, timestamp: 1700000000 },
    { athleteId: "1", metric: "speed_calculated", value: 4.2, timestamp: 1700000000 },
  ],
  "event-123",
  "ds-456",
);

dataworks.subscribe(channel, onEvent, onError?)

Subscribe to real-time data events via WebSocket.

Tip: start with dataworks/1/1/* to inspect all metrics for a dataset-datasource/event pair, then narrow to a specific metric channel such as dataworks/1/1/heartrate.

const subscription = dataworks.subscribe(
  "dataworks/1/1/heartrate",
  (event) => {
    console.log("Received:", event);
  },
  (error) => {
    console.error("Subscription error:", error.message);
  },
);

// Later: close the subscription
subscription.close();

The optional onError callback receives an Error for:

• WebSocket connection errors (network failures, TLS errors)
• AppSync subscription errors (invalid channel, server-side errors)
• Reconnect failures (token refresh failed after an auth expiry)

dataworks.reportError(error)

Report an error to the Data Engine for monitoring and alerting.

await dataworks.reportError({
  datasetDatasourceId: "ds-456",
  athleteId: "athlete-123",
  clientId: 1,
  errorTitle: "Sensor disconnected",
  errorDescription: "BLE heart rate sensor lost connection at 00:42:15",
});

dataworks.isAuthenticated

Check if the client has valid credentials.

if (dataworks.isAuthenticated) {
  await dataworks.ingest(metrics, eventId, dsId);
}

dataworks.tenant

Get the tenant from the authenticated session (or null).

console.log(`Logged in as tenant: ${dataworks.tenant}`);

Validation Utilities

Standalone validation functions are exported for pre-checking metrics before ingestion:

DataClient.validateMetric(metric)

Returns null if valid, or a string describing the validation failure.

import { validateMetric } from "@dataworks-technology/data";

const error = validateMetric({
  metric: "heartrate",
  athleteId: "1",
  value: 172,
  timestamp: 1700000000,
});

if (error) {
  console.warn("Invalid metric:", error);
}

DataClient.filterValidMetrics(metrics, warnFn)

Filters an array, keeping only valid metrics. Calls warnFn for each invalid item.

import { filterValidMetrics } from "@dataworks-technology/data";

const valid = filterValidMetrics(rawMetrics, (msg) => console.warn(msg));

Validation Rules

| Field | Rule | |---|---| | metric | Non-empty string | | athleteId | Non-empty string | | value | Number (finite, not NaN) or non-empty string | | timestamp | Positive integer (Unix seconds) |

Types

All types are exported for use in your application:

import type {
  DataClientConfig,
  LoginResult,
  Metric,
  MetricItem,
  RawMetricItem,
  MetricsPayload,
  ErrorReport,
  SubscriptionHandler,
  ErrorHandler,
  Subscription,
} from "@dataworks-technology/data";

Key Interfaces

interface Metric {
  athleteId: string;
  metric: string;
  value: number | string;
  timestamp: number; // Unix seconds
}

interface ErrorReport {
  datasetDatasourceId: string;
  athleteId: string;
  clientId: number;
  errorTitle: string;
  errorDescription: string;
}

interface LoginResult {
  accessToken: string;
  idToken: string;
  refreshToken: string;
  tenant: string;
}

Error Handling

All async methods throw on failure. Wrap calls in try/catch:

try {
  await dataworks.login("user", "pass");
} catch (err) {
  // Authentication failed (invalid credentials, network error, etc.)
}

try {
  await dataworks.ingest(metrics, eventId, dsId);
} catch (err) {
  // Ingestion failed (401 expired token, 5xx server error, etc.)
}

Calling ingest(), reportError(), or subscribe() before login() throws immediately.

Requirements

• Node.js ≥ 18 (uses native fetch)
• ESM or CommonJS — both module formats are included
• TypeScript ≥ 5.0 (optional — works with plain JavaScript too)
• Browser — compatible with any environment that has fetch and WebSocket

Derived Metrics — Round-Trip Example

A core use case: subscribe to live metrics, apply your own calculations externally, and push derived metrics back into the Data Engine. This creates a feedback loop where the platform stores both raw and enriched data.

Concept

Data Engine → subscribe("dataworks/{datasetDatasourceId}/{eventId}/heartrate") → Your App → calculate rolling avg → ingest("heartrate_rolling_avg_calculated") → Data Engine

Any metric you ingest is treated the same as raw sensor data — it flows through the enrichment pipeline (avg/min/max/zones), gets stored, and is available via subscriptions and the GraphQL API.

TypeScript Example

import { DataClient } from "@dataworks-technology/data";

const dataworks = new DataClient({
  cognitoEndpoint: "https://cognito-idp.eu-west-1.amazonaws.com/",
  clientId: "your-client-id",
  ingestUrl: "https://your-ingest-endpoint.dataworks.live",
  errorUrl: "https://your-error-endpoint.dataworks.live",
  realtimeUrl: "https://your-realtime-endpoint.dataworks.live",
});

await dataworks.login("developer", "password");

// 1. Subscribe to raw heart rate data
const buffer: number[] = [];
const datasetDatasourceId = "1";
const eventId = "1";
const heartrateChannel = `dataworks/${datasetDatasourceId}/${eventId}/heartrate`;

dataworks.subscribe(heartrateChannel, (event) => {
  // No metric guard needed here: the channel already filters to heartrate.

  // 2. Calculate a rolling 3-point average
  buffer.push(event.value);
  if (buffer.length > 3) buffer.shift();
  const avg = buffer.reduce((a, b) => a + b, 0) / buffer.length;

  // 3. Push the derived metric back into the engine
  dataworks.ingest(
    [
      {
        athleteId: event.athleteId,
        metric: "heartrate_rolling_avg_calculated",
        value: Math.round(avg * 10) / 10,
        timestamp: event.timestamp,
      },
    ],
    event.eventId,
    event.datasetDatasourceId,
  );
});

Python Example (Cross-Platform)

The same flow works from any language. A full working Python demo is included at test/demo/round-trip.py — it proves every SDK capability end-to-end:

| Phase | What it does | SDK equivalent | |-------|-------------|----------------| | 1. Authenticate | Cognito USER_PASSWORD_AUTH | dataworks.login() | | 2. Subscribe | WebSocket → AppSync Events API | dataworks.subscribe() | | 3. Receive | Catch events on the subscriber | onEvent callback | | 4. Calculate | Write to CSV/Google Sheets, compute rolling avg + HR zones | Your business logic | | 5. Re-ingest | POST enriched metrics back to the Data Engine | dataworks.ingest() | | 6. Report Error | Send error through the error pipeline | dataworks.reportError() |

Run the demo

# Install Python dependencies
pip install websocket-client requests

# Run from the repo root (reads test.env for credentials)
python test/demo/round-trip.py

Output

PHASE 4: WRITE DATA + CALCULATE
  Enriched data:
    HR=130 → avg=130.0, zone=Zone 2 (Aerobic), delta=—
    HR=140 → avg=135.0, zone=Zone 3 (Tempo), delta=+10
    HR=150 → avg=140.0, zone=Zone 3 (Tempo), delta=+10
    HR=160 → avg=150.0, zone=Zone 4 (Threshold), delta=+10
    HR=170 → avg=160.0, zone=Zone 4 (Threshold), delta=+10

PHASE 5: RE-INGEST ENRICHED METRICS
    → heartrate_rolling_avg=130.0
    → heartrate_rolling_avg=135.0
    → heartrate_rolling_avg=140.0
    → heartrate_rolling_avg=150.0
    → heartrate_rolling_avg=160.0
  ✓ Ingested (200)

The raw heartrate values (130–170 bpm) are transformed into heartrate_rolling_avg_calculated derived metrics and pushed back into the platform. These derived metrics are then available to all consumers (Console, Visual, Moments triggers) just like any other metric.

Google Sheets (optional)

The demo can write to Google Sheets instead of CSV — formulas compute rolling averages, HR zone classification, and deltas in the spreadsheet before re-ingesting. See test/demo/README.md for setup.

Ideas for Derived Metrics

| Derived metric | Calculation | Use case | |---|---|---| | heartrate_rolling_avg_calculated | 3-point moving average | Smooth noisy sensor data | | heartrate_zone | Zone classification (1–5) | Training load analysis | | speed_efficiency | Speed / heart rate ratio | Fatigue detection | | power_to_weight | Power / athlete weight | Normalised performance | | pace_delta | Current pace − target pace | Real-time coaching alerts | | fatigue_index | Custom formula over time window | Trigger warnings via Moments engine |

Documentation

Full documentation with guides, examples, and detailed API reference:

https://data-sdk-docs.dataworks.live

Development

Internal contributors only — this section covers building, testing, and publishing the package.

This package publishes to public npm (registry.npmjs.org), not to the internal CodeArtifact registry used by @dataworks/sdk. The packages/sdk/.npmrc and publishConfig in package.json enforce this.

Build

bun run build    # tsup → dist/ (ESM + CJS + DTS)
bun run dev      # tsup --watch (rebuild on save)
bun run test     # vitest unit tests (mocked — no infra needed)

Local Development

To test SDK changes locally before publishing:

Option A — npm pack (recommended, simulates real install):

# In packages/sdk/
bun run build
npm pack                    # creates dataworks-technology-data-0.1.4.tgz

# In your consumer project (e.g. test/demo/)
npm install ../../packages/sdk/dataworks-technology-data-0.1.4.tgz

Option B — bun link (faster iteration):

# In packages/sdk/
bun run build
bun link

# In your consumer project
bun link @dataworks-technology/data

Option C — watch mode + link (live reload):

# Terminal 1: packages/sdk/
bun run dev                 # rebuilds dist/ on every save

# Terminal 2: consumer project
bun link @dataworks-technology/data
npx tsx demo.ts             # picks up latest build automatically

Tip: After local testing, remember to bun unlink or reinstall from npm before committing.

Publish

Login to npm first (one-time):

npm login --registry https://registry.npmjs.org/

Then from packages/sdk/:

bun run publish:patch   # bump patch + publish
bun run publish:minor   # bump minor + publish
bun run publish:major   # bump major + publish

E2E Tests

The e2e test suite (test/e2e-public-sdk.test.ts) validates the full external developer flow against deployed infrastructure. It runs against live AWS services and proves the public SDK works end-to-end across all three stacks (Shared, Data, SDK).

Prerequisites

Deploy all three stacks first — make deploy generates test.env with the required env vars:

| Env var | Source | Purpose | |---|---|---| | FRONTEND_CLIENT_ID | Shared stack → Cognito | App client for USER_PASSWORD_AUTH | | API_URL | Data stack → API Gateway | Metric ingestion endpoint | | ERROR_URL | Data stack → API Gateway | Error reporting endpoint | | REALTIME_URL | Data stack → AppSync Events API | WebSocket subscription endpoint | | COGNITO_BASE_DOMAIN | Shared stack → Cognito | IDP endpoint for auth | | DEVELOPER_USERNAME | Shared stack → Cognito | Test user in the developer group | | DEVELOPER_PASSWORD | Shared stack → Cognito | Test user password |

Run

bun x vitest run test/e2e-public-sdk.test.ts

What each test proves

| # | Area | Test | What it validates | How to verify manually | |---|---|---|---|---| | 1 | Login | should authenticate testdeveloper | USER_PASSWORD_AUTH works with frontend client | AWS Console → Cognito → User Pool → Users → testdeveloper exists in developer group | | 2 | Login | should extract tenant from JWT | custom:tenant claim is present in ID token | Decode the JWT at jwt.io — look for custom:tenant | | 3 | Login | should include resource server scopes | Pre-token Lambda injects dataworks.live/read and /write | AWS Console → Cognito → User Pool → Lambda triggers → Pre token generation V2 is set | | 4 | Login | should reject invalid credentials | Bad password returns error, not a token | Try dataworks.login("testdeveloper", "wrong") — should throw | | 5 | Ingest | should ingest valid metrics | JWT passes API Gateway authoriser, Lambda processes payload | AWS Console → CloudWatch → Log group for the ingest Lambda — look for the metric | | 6 | Ingest | should filter invalid metrics | Client-side validation drops bad metrics before sending | The stderr output shows Dropping invalid metric — local validation, no server call for bad data | | 7 | Ingest | should throw when not authenticated | ingest() before login() throws immediately | Client-side guard — no network call made | | 8 | Error | should reach error endpoint | Auth + API Gateway routing work (500 = Lambda ran, FK failed) | AWS Console → CloudWatch → Log group for error processor Lambda | | 9 | Error | should throw when not authenticated | reportError() before login() throws immediately | Client-side guard | | 10 | Subscribe | should throw when not authenticated | subscribe() before login() throws immediately | Client-side guard | | 11 | Subscribe | should create a subscription | Skipped in Node.js (no WebSocket global) — see below | Use the Python verification script | | 12–14 | Validation | validateMetric / filterValidMetrics | Static validation utilities work correctly | Pure functions — no infra dependency |

Verifying WebSocket subscribe (test 11)

The subscribe test is skipped in the Node.js Vitest environment because globalThis.WebSocket is not available. To prove subscribe works end-to-end, use the standalone Python script:

pip install websocket-client requests
python test/verify-subscribe.py

This script:

1. Authenticates via Cognito USER_PASSWORD_AUTH (same as the SDK)
2. Opens a WebSocket to the AppSync Events API with base64URL-encoded JWT auth subprotocols
3. Sends connection_init and receives connection_ack from AppSync
4. Sends a subscribe message and receives subscribe_success
5. Closes cleanly

Expected output:

Loading config from test.env

1. Authenticating as 'testdeveloper'...
   ✓ Login successful
   Scopes: aws.cognito.signin.user.admin https://dataworks.live/write https://dataworks.live/read
   Tenant: 2x9AOzBvDBgllLB0CXcCrPjK03C

2. Connecting to wss://event-api-chris-dev.dataworks.live/event/realtime...
   Protocol: aws-appsync-event-ws (manual)
   Received: connection_ack
   ✓ WebSocket connection established

3. Subscribing to channel '/dataworks/e2e-verify'...
   Received: subscribe_success
   ✓ Subscribed to /dataworks/e2e-verify

4. WebSocket closed cleanly

============================================================
ALL CHECKS PASSED — subscribe flow works end-to-end
============================================================

License

MIT © Dataworks Technology