@bentolabs/ams

JavaScript/TypeScript SDK for BentoLabs AMS — an open-source observability platform for AI agents.

Trace, evaluate, annotate, and analyze LLM data. Bring LLM applications to production with confidence.

Quickstart

npm install @bentolabs/ams

And then in the code:

import { AMS } from '@bentolabs/ams'

AMS.initialize({ projectApiKey: '<PROJECT_API_KEY>' })

This will automatically instrument most LLM, Vector DB, and related calls with OpenTelemetry-compatible instrumentation.

Where to place AMS.initialize()

AMS.initialize() must be called:

• once in your application
• as early as possible, but after other instrumentation libraries

Instrumentation

In addition to automatic instrumentation, we provide a simple observe() wrapper for tracing individual functions.

Example

import { OpenAI } from 'openai';
import { AMS as L, observe } from '@bentolabs/ams';

L.initialize({ projectApiKey: "<AMS_PROJECT_API_KEY>" });

const client = new OpenAI({ apiKey: '<OPENAI_API_KEY>' });

const poemWriter = async (topic = "turbulence") => {
  const prompt = `write a poem about ${topic}`;
  const response = await client.chat.completions.create({
    model: "gpt-4o",
    messages: [
      { role: "system", content: "You are a helpful assistant." },
      { role: "user", content: prompt }
    ]
  });

  return response.choices[0].message.content;
}

await observe({ name: 'poemWriter' }, async () => await poemWriter('turbulence'))

Evaluations

npm install @bentolabs/ams

import { evaluate } from '@bentolabs/ams';

const writePoem = ({ topic }: { topic: string }) => {
  return `This is a good poem about ${topic}`;
};

evaluate({
  data: [
    { data: { topic: 'flowers' }, target: { poem: 'This is a good poem about flowers' } },
    { data: { topic: 'cars' }, target: { poem: 'I like cars' } },
  ],
  executor: (data) => writePoem(data),
  evaluators: {
    containsPoem: (output, target) => target.poem.includes(output) ? 1 : 0,
  },
  groupId: 'my_first_feature',
});

Run:

export AMS_PROJECT_API_KEY=<AMS_PROJECT_API_KEY>
npx ams eval my-first-eval.ts

Client

import { AMSClient } from '@bentolabs/ams';

const client = new AMSClient({ projectApiKey: '<AMS_PROJECT_API_KEY>' });