Tripwire

A Convex component that reads your deployment's logs and tells you what needs you: functions that read far more than their work needs, failures that are real rather than noise, traffic that looks like probing, and whether you are on pace to run out of Database I/O.

Code computes every number. One cheap model call per window makes the judgements a number cannot: is this read waste or the job, is this error a defect or a user typing the wrong password. Findings land in a table your app can subscribe to.

987 events over 5.5 h, 11 windows of 30 min
Database I/O pace: about 2.15 GB per 30 days
32 questions asked, $0.001391

[broken · today · rule · seen 1x] wall:scorecard failed 1 of 1 call
    It returned a value its own `returns` validator rejects.
    → Bring the validator and the data back in line: widen the validator, or strip the extra fields before returning.

[waste · this week · p=0.77 · seen 2x] lookup:building read 1.3 MB in 3 calls
    429.5 KB and 422.7 documents per call, returning 86.1 KB. Every read byte counts against Database I/O.
    → Define an index on the fields it filters by and query with .withIndex(), so it reads only the documents that match.

[quota · rule · seen 8x] On pace for 2.15 GB of Database I/O in 30 days, budget 2.00 GB

That is a real run against a production app. The first finding was a page that was failing in production without anyone having noticed; it turned up the first time this code ran.

Try it before installing anything

npx convex logs --prod --jsonl --success --history 5000 > logs.jsonl   # Ctrl-C once it goes quiet
npx convex-tripwire check logs.jsonl --budget-gb 2

Without a JEV_API_KEY in the environment this runs the rules and the arithmetic only (broken contracts, runtime errors, I/O pace). With one, it asks the model too. check uses the same code as the component, so what it prints is what the component would find.

How it works

npx convex logs ──► tripwire forward ──► POST <site>/tripwire/ingest
   (your logs)      one-minute totals        │
                                             ▼
                                   rollups: one row per function per window
                                             │  window closes
                                             ▼
                          ┌──────────── the judge ────────────┐
                          │ rules       broken contracts       │  no model
                          │ arithmetic  Database I/O pace      │  no model
                          │ one call    waste? defect? abuse?  │  Jev
                          │             which fix? how urgent? │
                          └────────────────┬──────────────────┘
                                           ▼
                           findings: open · resolved · muted

Totals, not events. A watcher that stored every log line would be the heaviest reader in the deployment it watches. The forwarder folds events into one row per function per minute before they arrive; the component only adds to those rows. Totals older than a week are deleted; a per-function moving average keeps what they taught.

Code decides what is worth asking. A function that read 300 bytes is never asked about waste. A function with no errors is never asked if it is broken. A function whose return value fails its own returns validator is a defect by rule, and the model is not consulted: when that case was left to judgement on real logs it came back 0.51.

One call per window. The judge model is Jev from TypeSafe AI, which answers many typed questions in parallel over one shared state: one round trip, and the state is paid for once however many questions ride on it. Every question for every function in a window goes in a single request. On the run above that was 32 questions across 11 windows for $0.0014, about 450 ms each. You bring your own key.

It picks, it does not write. The model returns probabilities and choices, never text. A suggested fix is one entry from a catalog written by a person (add an index, skip unchanged reads, bound the read, keep a summary, split large documents, call it less), chosen per function. It is shown only when the pick is confident; a finding with no suggestion is better than one with a guess.

It grades its own judge. When you close a finding you say what it was: fixed or noise. Noise is muted for good on that function. Both are labels, and calibration reports, per probability band, how many findings you judged and how many were real. Nobody can tell you in advance whether a 0.7 from the model means 70%; your own verdicts are the only ground truth there is, so Tripwire keeps score.

Install

npm install convex-tripwire

// convex/convex.config.ts
import { defineApp } from "convex/server";
import { v } from "convex/values";
import tripwire from "convex-tripwire/convex.config";

const app = defineApp({
  env: { JEV_API_KEY: v.optional(v.string()), TRIPWIRE_SECRET: v.optional(v.string()) },
});

app.use(tripwire, {
  httpPrefix: "/tripwire/", // serves <site>/tripwire/ingest
  env: { JEV_API_KEY: app.env.JEV_API_KEY, TRIPWIRE_SECRET: app.env.TRIPWIRE_SECRET },
});

export default app;

npx convex env set TRIPWIRE_SECRET "$(openssl rand -hex 24)"
npx convex env set JEV_API_KEY <your TypeSafe key>     # optional: without it, rules and arithmetic only

Then send it logs. The component cannot read its own deployment's logs from the inside, so a small process does it, from anywhere that can run npx convex logs for your project:

TRIPWIRE_SECRET=... npx tripwire forward --prod --url https://<your-deployment>.convex.site/tripwire

It is safe to restart: it asks the component how far it got and skips what was already counted. For a scheduled job instead of a long-running process, add --once; it catches up and exits. To backfill a new install from a saved file, --file logs.jsonl.

Use

// convex/tripwire.ts
import { v } from "convex/values";
import { Tripwire } from "convex-tripwire";
import { components } from "./_generated/api";
import { mutation, query } from "./_generated/server";

export const tripwire = new Tripwire(components.tripwire);

export const setup = mutation({
  args: {},
  handler: async (ctx) => {
    await tripwire.configure(ctx, {
      windowMinutes: 30,
      ioBudgetBytes: 2 * 1024 ** 3, // the free plan's Database I/O
      about: "A marketplace for used bikes. Crons sync listings from partner feeds hourly.",
      questions: [{ key: "checkout", text: "Checkout is failing for customers." }],
      alertUrl: "https://hooks.slack.com/services/...", // Slack, Discord, or any webhook
    });
  },
});

// Reactive: a dashboard subscribed to this updates the moment a window is judged.
export const findings = query({
  args: {},
  handler: async (ctx) => await tripwire.findings(ctx),
});

// Put your own auth in front of this.
export const close = mutation({
  args: { id: v.string(), verdict: v.union(v.literal("fixed"), v.literal("noise")) },
  handler: async (ctx, { id, verdict }) => await tripwire.resolve(ctx, id, verdict),
});

export const calibration = query({
  args: {},
  handler: async (ctx) => await tripwire.calibration(ctx),
});

| | | | --- | --- | | configure(ctx, {...}) | Window size, I/O budget, a sentence about the app, your own questions, an alert webhook. | | findings(ctx, { status?, limit? }) | open by default; also resolved and muted. Each has kind, fn, title, detail, probability, urgency (0 to 3), suggestion, seen, firstSeen, lastSeen. | | resolve(ctx, id, "fixed" \| "noise") | fixed: a new finding opens if it comes back. noise: muted for good. | | calibration(ctx) | Per probability band: how many findings you judged, how many were real. | | judgements(ctx) | Every judged window: questions asked, cost, time, status. | | report(ctx, { fn, error?, readBytes?, ms? }) | Count something by hand: a failure you caught, or an app you cannot run a forwarder for. |

Finding kinds: waste, broken, regressed (worse than the function's own usual, once there are three windows of usual), abuse, quota, and custom for your own questions. The same kind on the same function is one finding while it is open; repeats raise seen. A finding not seen for a day closes itself.

Your own questions are plain sentences that can be true or false of a window's totals. They ride in the same call as everything else, so each one costs only its own few dozen tokens.

What it costs you

On your deployment: a few small writes per forwarded minute, and per window one read per active function plus the judge's writes. Tripwire ignores its own functions, so it never reports on itself.

For the model: the state is a compact table of at most 60 functions, and only functions whose facts justify a question get one. The 5.5 hours above cost $0.0014; a month of half-hour windows at that rate is well under a dollar.

Known limits

• It needs a forwarder. A component cannot subscribe to its deployment's log stream, so something outside has to run tripwire forward. Convex log streams (Pro) as a push source are not supported yet.
• The judge is only as good as your verdicts say it is. The probabilities come from a model whose calibration nobody has measured on your workload. Until you have closed enough findings for calibration to mean something, treat 0.6 to 0.7 as "worth a look", not "70% likely".
• Suggestions are coarse. Totals say how much a function read, not which line did it. add_index is the most common pick because it is the most common cause; read the evidence on the finding before acting.
• Text only. It reads log lines and numbers. It does not read your source.
• Function names and clipped log lines leave your deployment in the request to the model: up to five log lines and three error lines per function per window, 200 to 300 characters each. If your logs carry personal data, do not log it, or run without JEV_API_KEY.

The example app

example/ mounts the component and example/index.html is a dashboard for it in one file: four subscriptions, no build step. Findings appear the moment a window is judged; the Fixed and Noise buttons are the verdicts; the table on the right is the calibration.

npx convex dev                                   # a local deployment running example/
npx convex env set TRIPWIRE_SECRET dev-secret
npx convex run example:setup
npx tripwire forward --url http://127.0.0.1:3211/tripwire --secret dev-secret --file logs.jsonl
npx serve example                                # then open it; add ?url=<deployment url> for a cloud one

Development

npm install
npm test                 # unit tests, and the component in an in-memory Convex with the model mocked
npm run typecheck
npx convex dev           # runs example/ against a local deployment

src/core.ts is everything Tripwire decides, as plain functions shared by the component and the CLI. src/component/ is the Convex side: lib.ts (ingest, config, findings, verdicts), judge.ts (the scheduled judge), http.ts (the ingest route).

License

MIT