Agent Monitoring — engine (Phase A) ✅

Install & CLI

cost-per-task, latency, error rate, and alerts for agent runs. Requires Node ≥18.

npm i -g @mnikks01/agentmon    # then run `agentmon …`, or use npx without installing:
npx @mnikks01/agentmon demo                 # report on built-in sample runs
npx @mnikks01/agentmon analyze runs.json    # array of AgentRun -> metrics + alerts

The core engine for project #4, "Datadog for agents." Record agent runs as trajectories (reason → llm → tool steps) via a framework-agnostic SDK, then get cost-per-task, latency, error rate, model mix, a failure taxonomy, and alerts. Pure TypeScript, Node 24 native TS, zero-network (the SDK records what the agent reports; it makes no LLM calls itself).

Status: Phase A built + tested (2026-06-21)

• ✅ SDK / tracer — monitor.run(name).reason().llm({...}).tool({...}).end(); cost computed from a model price table.
• ✅ Cost-per-task — per-step + per-run USD from token usage (pricing.ts, overridable).
• ✅ Analytics — error rate, total/avg cost, total tokens, avg + p95 latency, model mix, per-tool call/error counts, failure taxonomy.
• ✅ Alerts — failed run (error), cost spike, slow run (configurable thresholds).
• ✅ 14/14 tests pass (scripts/test.ts): cost math, run status from tool failure, aggregation, alert firing, safe empty aggregate.

Run it

node scripts/demo.ts   # instrument 4 agent runs -> trajectory, metrics, alerts
node scripts/test.ts   # 14 assertions

SDK shape

import { Monitor } from "./src/index.ts";
const mon = new Monitor({ maxCostUsd: 0.5 });

mon.run("research-agent")
  .reason("need recent facts -> search")
  .tool({ name: "web_search", ok: true, latencyMs: 1200 })
  .llm({ model: "claude-sonnet-4-6", inputTokens: 8000, outputTokens: 600, latencyMs: 2200 })
  .end();

mon.metrics();  // { runs, errorRate, totalCostUsd, p95LatencyMs, modelMix, toolStats, failureTaxonomy }
mon.alerts();   // [{ severity, rule, message, runName }]

Structure

src/
  types.ts       # Step (llm|tool|reason), AgentRun, Alert, Metrics
  pricing.ts     # model price table -> costUsd()
  tracer.ts      # RunRecorder — the SDK surface
  store.ts       # in-memory run store (-> Postgres + OTel ingest in production)
  analytics.ts   # aggregate(): cost/latency/error/model/tool/failure metrics
  monitors.ts    # alert rules (run_failed / cost_spike / slow_run)
  index.ts       # Monitor: run() / runs() / metrics() / alerts()
scripts/
  demo.ts / test.ts

Next (per the docs)

• MCP server + web dashboard (trajectory viewer, cost/latency charts, alerts) — same pattern as the wedge projects.
• OTel ingest (framework-agnostic spans from LangGraph/CrewAI/OpenAI/Claude SDK) → Postgres.
• Eval harness (golden tasks + LLM-as-judge) and replay (V1).
• Ships standalone AND as a ContextOS (#1) module.

Production swaps

| Engine (now) | Production | |---|---| | in-memory RunStore | Postgres + OTel ingest, retention, RLS per org | | representative price table | exact/current prices via setPrice() | | threshold alerts | + anomaly detection, eval-regression gates |