ExperimentLane

An evidence-driven experimentation engine for AI agents. Record experiments, compare them against baselines, and get a deterministic decision about what to change next — one variable at a time.

What it is

ExperimentLane tracks content and agent experiments as receipts: a baseline, a run, and the platform-exposed metrics that come back. It compares them, names a winner, picks the strongest and weakest signals, and tells the agent exactly what to change next while everything else stays constant.

Why it exists

Most agents ship a variation, see a number, and guess. That is how B-tests quietly become A/B/C/D tests and nothing is ever attributable.

ExperimentLane encodes the discipline the majority of researchers are missing: change exactly one independent variable at a time. It is a local, deterministic, no-frills engine any agent can embed, so experiments stay honest, small, and comparable.

Open engine first. The hosted power can come later — the same schema is already shaped for the Talocode API surface.

Install

npm install @talocode/experimentlane

Requires Node 18+ (local JSON store, no cloud required).

Quickstart

import { ExperimentLane, JsonStore } from "@talocode/experimentlane";

const lane = new ExperimentLane(new JsonStore({ dir: ".experimentlane" }));
await lane.init();

// 1. Define a lane for a platform's exposed metrics
const shorts = lane.createLane({ name: "youtube-shorts", platform: "youtube_short" });

// 2. Record a baseline (the control asset's platform-exposed metrics)
const base = lane.createBaseline({
  laneId: shorts.id,
  metrics: { views: 480, avg_view_percent: 75, shares: 10, subscribers: 2 },
});

// 3. Create the experiment — exactly one variable changes
const asset = lane.createAsset({ laneId: shorts.id, title: "reliabilitylane-short-2" });
const exp = lane.createExperiment({
  laneId: shorts.id,
  assetId: asset.id,
  baselineId: base.id,
  variable: "hook",
});

// 4. Record the receipt after the run
lane.addReceipt({
  experimentId: exp.id,
  metrics: { views: 624, avg_view_percent: 87, shares: 14, subscribers: 3 },
});

// 5. Compare and get one deterministic decision
const { comparison, decision } = lane.compare({ experimentId: exp.id });
console.log(comparison.winner); // "experiment"
console.log(decision.line.nextExperiment); // "change only <weakest signal>"
console.log(decision.line.everythingElse); // "hold constant"

Auth / env

The local store needs no API key. Environment variables are optional:

• EXPERIMENTLANE_DIR — override the default data directory (default .experimentlane).

API surface

Everything is compatible metrics only — absent metrics are "unavailable", never a failure.

| Method | Purpose | |--------|---------| | createLane | Define a platform and its metric priority | | createAsset | A piece of content or an agent run | | createBaseline | The control — an asset snapshot or explicit metrics | | createExperiment | An assignment changing exactly one variable | | addReceipt | The platform-exposed metrics that came back | | compare | Compute deltas, name winner, pick strongest/weakest signals | | next | Recommend the next single variable to change | | validateNext | Guardrail: reject proposals that change 2+ variables |

Platform schemas

Platforms (YouTube Shorts, TikTok, Instagram Reels, X, generic) map their exposed metrics. shares_per_view is derived from views + shares when both are present.

Guardrail codes

| Code | Meaning | |------|---------| | EXPERIMENT_OK | Exactly one independent variable changed | | EXPERIMENT_INVALID | 2+ independent variables changed — reject |

CLI

experimentlane lane create --name my-shorts --platform youtube_short
experimentlane receipt add --experiment <id> --views 624 --shares 14
experimentlane compare --experiment <id>
experimentlane next --experiment <id> --propose hook,duration   # guardrail check

Data lives in .experimentlane (override with EXPERIMENTLANE_DIR).

MCP

experimentlane-mcp exposes the engine as an MCP server: experimentlane_create, experimentlane_record_receipt, experimentlane_compare, experimentlane_get_baseline, experimentlane_next_experiment, experimentlane_history.

Related packages

Sibling installs for the Talocode ecosystem:

| Package | Install | |---------|---------| | StackLane | pip install talocode | | Tera | pip install talocode-tera | | Codra | pip install talocode-codra | | XSearchLane | npm i @talocode/xsearchlane |

Talocode ecosystem

| Product | Description | |---------|--------------| | ExperimentLane | (this package) — evidence-driven experiment engine for agents | | Tera | Capability API under Talocode Cloud | | Codra | Coding agent / skills runtime | | StackLane | Cloud backend: projects, API keys, credits, billing | | SearchLane | Search API product | | GateLane | Policy gates | | ContextLane | Context management | | ScreenLane | Screen capture pipeline | | MemoryLane | Agent memory | | Tradia | Trading agents | | DevTool | Developer tooling | | XProLane | X advanced tools | | XSearchLane | X realtime search | | Agent Browser | Browser control | | InvoiceLane | Invoicing | | GeoLane | Geolocation | | ClipLoop | Clip builder |

More: github.com/talocode · talocode.site · docs.talocode.site

License

MIT © Talocode.