@modaic/modaic-ts

TypeScript client for Modaic Arbiters (LLM judges).

Arbiter is a thin wrapper over the Modaic REST API and git — it never runs an LLM locally. predict() calls the API; create() / update() write the judge's config.json (signature schema) and program.json (stored prompt) and push them to Modaic Hub via git. These files are produced the same way as the Python SDK so the two interoperate.

Quickstart

import { Arbiter, Signature, Enum } from "@modaic/modaic-ts";
import { z } from "zod";

const signature = new Signature({
  instructions: "Decide whether the answer correctly addresses the question.",
  input: z.object({
    question: z.string().describe("The user's question"),
    answer: z.string().describe("The answer to judge"),
  }),
  output: z.object({
    // Arbiter outputs must be discrete — use Enum (or a Zod enum), not a plain string.
    verdict: Enum("correct", "incorrect").describe("Whether the answer is correct"),
  }),
});

// Create + push a new judge (private by default). Uses MODAIC_TOKEN.
const arbiter = await Arbiter.create({
  repo: "modaic/quality-judge",
  signature,
  model: "together_ai/openai/gpt-oss-120b",
});

// Run it (the server runs the LLM).
const result = await arbiter.predict({ question: "...", answer: "..." });
console.log(result.output, result.reasoning);

// Update later (optional new signature / metadata / extra files).
await arbiter.update({ signature, model: "together_ai/openai/gpt-oss-120b" });

// Open an existing judge at a specific revision.
const existing = new Arbiter("modaic/quality-judge", { rev: "v1" });

Configuration

| Env var | Default | Purpose | | ---------------- | ------------------------- | ------------------------------------ | | MODAIC_TOKEN | — | Access token (or pass access_token)| | MODAIC_API_URL | https://api.modaic.dev | Modaic REST API base URL | | MODAIC_GIT_URL | https://git.modaic.dev | Modaic git host | | MODAIC_CACHE | ~/.cache/modaic | Staging dir for git working trees |

Develop

bun install
bun test
bun run build

Note: serializeSignatureToConfig (signature → config.json) is provided separately and currently throws. create()/update() with a signature will not push until that lands; predict() and program.json generation work today.

Summary

Modaic API: A FastAPI application for Modaic

Table of Contents

• @modaic/modaic-ts
  • Quickstart
  • Configuration
  • Develop
  • SDK Installation
  • Requirements
  • SDK Example Usage
  • Authentication
  • Available Resources and Operations
  • Standalone functions
  • Retries
  • Error Handling
  • Server Selection
  • Custom HTTP Client
  • Debugging

SDK Installation

[!TIP] To finish publishing your SDK to npm and others you must run your first generation action.

The SDK can be installed with either npm, pnpm, bun or yarn package managers.

NPM

npm add https://github.com/modaic-ai/modaic-ts

PNPM

pnpm add https://github.com/modaic-ai/modaic-ts

Bun

bun add https://github.com/modaic-ai/modaic-ts

Yarn

yarn add https://github.com/modaic-ai/modaic-ts

[!NOTE] This package is published as an ES Module (ESM) only. For applications using CommonJS, use await import() to import and use this package.

Requirements

For supported JavaScript runtimes, please consult RUNTIMES.md.

SDK Example Usage

Example

import { ModaicClient } from "modaic";

const modaicClient = new ModaicClient({
  token: process.env["MODAIC_TOKEN"] ?? "",
});

async function run() {
  const result = await modaicClient.chat.createCompletion();

  console.log(result);
}

run();

Authentication

Per-Client Security Schemes

This SDK supports the following security scheme globally:

| Name | Type | Scheme | Environment Variable | | ------- | ---- | ----------- | -------------------- | | token | http | HTTP Bearer | MODAIC_TOKEN |

To authenticate with the API the token parameter must be set when initializing the SDK client instance. For example:

import { ModaicClient } from "modaic";

const modaicClient = new ModaicClient({
  token: process.env["MODAIC_TOKEN"] ?? "",
});

async function run() {
  const result = await modaicClient.chat.createCompletion();

  console.log(result);
}

run();

Available Resources and Operations

Arbiters

• get - Get Arbiter
• getSchema - Get Arbiter Schema Output
• createChatCompletion - Create Chat Completion
• getSupportedModels - Get Supported Models
• init - Init Arbiter
• updateMetadata - Update Arbiter Metadata

Chat

• createCompletion - Create Chat Completion

Examples

• ingest - Ingest Examples
• list - List Examples
• delete - Delete Examples
• export - Export Examples
• downloadExport - Download Export
• getDistinctHashes - Get Distinct Hashes
• hasUncalibrated - Has Uncalibrated Predictions
• getGradedCount - Get Graded Count
• patchAnnotation - Patch Example
• get - Get Example By Id

Jobs

• startConfidenceScore - Start Confidence Score Job
• getConfidenceScoreStatus - Get Confidence Score Job Status
• cancelConfidenceScore - Cancel Confidence Score Job
• startOptimization - Start Optimization Job
• cancelOptimization - Cancel Optimization Job
• getOptimizationStatus - Get Optimization Job Status
• getOptimizationLogs - Get Optimization Job Logs
• startBatchPredictions - Start Batch Predictions Job
• getBatchPredictionStatus - Get Batch Predictions Job Status
• cancelBatchPrediction - Cancel Batch Predictions Job
• streamBatchEvents - Stream Batch Predictions Events
• streamBatchResults - Stream Batch Predictions Results

Predictions

• createV1 - Create Prediction
• getConfidence - Get Prediction Confidence
• enqueueConfidence - Enqueue Prediction Confidence
• getConfidenceStatus - Get Prediction Confidence Status
• streamConfidence - Stream Prediction Confidence
• dispatch - Dispatch Prediction
• create - Create Prediction

Standalone functions

All the methods listed above are available as standalone functions. These functions are ideal for use in applications running in the browser, serverless runtimes or other environments where application bundle size is a primary concern. When using a bundler to build your application, all unused functionality will be either excluded from the final bundle or tree-shaken away.

To read more about standalone functions, check FUNCTIONS.md.

• arbitersCreateChatCompletion - Create Chat Completion
• arbitersGet - Get Arbiter
• arbitersGetSchema - Get Arbiter Schema Output
• arbitersGetSupportedModels - Get Supported Models
• arbitersInit - Init Arbiter
• arbitersUpdateMetadata - Update Arbiter Metadata
• chatCreateCompletion - Create Chat Completion
• examplesDelete - Delete Examples
• examplesDownloadExport - Download Export
• examplesExport - Export Examples
• examplesGet - Get Example By Id
• examplesGetDistinctHashes - Get Distinct Hashes
• examplesGetGradedCount - Get Graded Count
• examplesHasUncalibrated - Has Uncalibrated Predictions
• examplesIngest - Ingest Examples
• examplesList - List Examples
• examplesPatchAnnotation - Patch Example
• jobsCancelBatchPrediction - Cancel Batch Predictions Job
• jobsCancelConfidenceScore - Cancel Confidence Score Job
• jobsCancelOptimization - Cancel Optimization Job
• jobsGetBatchPredictionStatus - Get Batch Predictions Job Status
• jobsGetConfidenceScoreStatus - Get Confidence Score Job Status
• jobsGetOptimizationLogs - Get Optimization Job Logs
• jobsGetOptimizationStatus - Get Optimization Job Status
• jobsStartBatchPredictions - Start Batch Predictions Job
• jobsStartConfidenceScore - Start Confidence Score Job
• jobsStartOptimization - Start Optimization Job
• jobsStreamBatchEvents - Stream Batch Predictions Events
• jobsStreamBatchResults - Stream Batch Predictions Results
• predictionsCreate - Create Prediction
• predictionsCreateV1 - Create Prediction
• predictionsDispatch - Dispatch Prediction
• predictionsEnqueueConfidence - Enqueue Prediction Confidence
• predictionsGetConfidence - Get Prediction Confidence
• predictionsGetConfidenceStatus - Get Prediction Confidence Status
• predictionsStreamConfidence - Stream Prediction Confidence

Retries

Some of the endpoints in this SDK support retries. If you use the SDK without any configuration, it will fall back to the default retry strategy provided by the API. However, the default retry strategy can be overridden on a per-operation basis, or across the entire SDK.

To change the default retry strategy for a single API call, simply provide a retryConfig object to the call:

import { ModaicClient } from "modaic";

const modaicClient = new ModaicClient({
  token: process.env["MODAIC_TOKEN"] ?? "",
});

async function run() {
  const result = await modaicClient.chat.createCompletion({
    retries: {
      strategy: "backoff",
      backoff: {
        initialInterval: 1,
        maxInterval: 50,
        exponent: 1.1,
        maxElapsedTime: 100,
      },
      retryConnectionErrors: false,
    },
  });

  console.log(result);
}

run();

If you'd like to override the default retry strategy for all operations that support retries, you can provide a retryConfig at SDK initialization:

import { ModaicClient } from "modaic";

const modaicClient = new ModaicClient({
  retryConfig: {
    strategy: "backoff",
    backoff: {
      initialInterval: 1,
      maxInterval: 50,
      exponent: 1.1,
      maxElapsedTime: 100,
    },
    retryConnectionErrors: false,
  },
  token: process.env["MODAIC_TOKEN"] ?? "",
});

async function run() {
  const result = await modaicClient.chat.createCompletion();

  console.log(result);
}

run();

Error Handling

ModaicClientError is the base class for all HTTP error responses. It has the following properties:

| Property | Type | Description | | ------------------- | ---------- | --------------------------------------------------------------------------------------- | | error.message | string | Error message | | error.statusCode | number | HTTP response status code eg 404 | | error.headers | Headers | HTTP response headers | | error.body | string | HTTP body. Can be empty string if no body is returned. | | error.rawResponse | Response | Raw HTTP response | | error.data$ | | Optional. Some errors may contain structured data. See Error Classes. |

Example

import { ModaicClient } from "modaic";
import * as errors from "modaic/models/errors";

const modaicClient = new ModaicClient({
  token: process.env["MODAIC_TOKEN"] ?? "",
});

async function run() {
  try {
    const result = await modaicClient.chat.createCompletion();

    console.log(result);
  } catch (error) {
    // The base class for HTTP error responses
    if (error instanceof errors.ModaicClientError) {
      console.log(error.message);
      console.log(error.statusCode);
      console.log(error.body);
      console.log(error.headers);

      // Depending on the method different errors may be thrown
      if (error instanceof errors.HTTPValidationError) {
        console.log(error.data$.detail); // ValidationError[]
      }
    }
  }
}

run();

Error Classes

Primary errors:

• ModaicClientError: The base class for HTTP error responses.
  • HTTPValidationError: Validation Error. Status code 422. *

Network errors:

• ConnectionError: HTTP client was unable to make a request to a server.
• RequestTimeoutError: HTTP request timed out due to an AbortSignal signal.
• RequestAbortedError: HTTP request was aborted by the client.
• InvalidRequestError: Any input used to create a request is invalid.
• UnexpectedClientError: Unrecognised or unexpected error.

Inherit from ModaicClientError:

• ResponseValidationError: Type mismatch between the data returned from the server and the structure expected by the SDK. See error.rawValue for the raw value and error.pretty() for a nicely formatted multi-line string.

* Check the method documentation to see if the error is applicable.

Server Selection

Override Server URL Per-Client

The default server can be overridden globally by passing a URL to the serverURL: string optional parameter when initializing the SDK client instance. For example:

import { ModaicClient } from "modaic";

const modaicClient = new ModaicClient({
  serverURL: "https://api.modaic.dev",
  token: process.env["MODAIC_TOKEN"] ?? "",
});

async function run() {
  const result = await modaicClient.chat.createCompletion();

  console.log(result);
}

run();

Custom HTTP Client

The TypeScript SDK makes API calls using an HTTPClient that wraps the native Fetch API. This client is a thin wrapper around fetch and provides the ability to attach hooks around the request lifecycle that can be used to modify the request or handle errors and response.

The HTTPClient constructor takes an optional fetcher argument that can be used to integrate a third-party HTTP client or when writing tests to mock out the HTTP client and feed in fixtures.

The following example shows how to:

• route requests through a proxy server using undici's ProxyAgent
• use the "beforeRequest" hook to add a custom header and a timeout to requests
• use the "requestError" hook to log errors

import { ModaicClient } from "modaic";
import { ProxyAgent } from "undici";
import { HTTPClient } from "modaic/lib/http";

const dispatcher = new ProxyAgent("http://proxy.example.com:8080");

const httpClient = new HTTPClient({
  // 'fetcher' takes a function that has the same signature as native 'fetch'.
  fetcher: (input, init) =>
    // 'dispatcher' is specific to undici and not part of the standard Fetch API.
    fetch(input, { ...init, dispatcher } as RequestInit),
});

httpClient.addHook("beforeRequest", (request) => {
  const nextRequest = new Request(request, {
    signal: request.signal || AbortSignal.timeout(5000)
  });

  nextRequest.headers.set("x-custom-header", "custom value");

  return nextRequest;
});

httpClient.addHook("requestError", (error, request) => {
  console.group("Request Error");
  console.log("Reason:", `${error}`);
  console.log("Endpoint:", `${request.method} ${request.url}`);
  console.groupEnd();
});

const sdk = new ModaicClient({ httpClient: httpClient });

Debugging

You can setup your SDK to emit debug logs for SDK requests and responses.

You can pass a logger that matches console's interface as an SDK option.

[!WARNING] Beware that debug logging will reveal secrets, like API tokens in headers, in log messages printed to a console or files. It's recommended to use this feature only during local development and not in production.

import { ModaicClient } from "modaic";

const sdk = new ModaicClient({ debugLogger: console });

You can also enable a default debug logger by setting an environment variable MODAIC_DEBUG to true.