Agent Tool Forge

Production LLM agent sidecar + Claude Code skill library for building, testing, and running tool-calling agents.

Two jobs, one package:

1. Sidecar runtime — deploy alongside your app. Handles the full ReAct loop, HITL gates, verifier pipeline, eval runner, and observability.
2. Skill library — Claude Code skills that generate tools, eval suites, and MCP servers via structured 12-phase dialogue.

Quick Start

As a runtime package

npm install agent-tool-forge

import { createSidecar } from 'agent-tool-forge'

const { server, ctx, close } = await createSidecar(
  { auth: { mode: 'trust' } },
  { port: 8001 }
)

// server is already listening on port 8001
// call close() on shutdown for clean teardown

With the TUI

node lib/index.js

See docs/tui-workflow.md for a start-to-finish walkthrough.

Install Claude Code Skills

# Global install (available in all projects)
cp -r node_modules/agent-tool-forge/skills/forge-tool     ~/.claude/skills/
cp -r node_modules/agent-tool-forge/skills/forge-eval     ~/.claude/skills/
cp -r node_modules/agent-tool-forge/skills/forge-mcp      ~/.claude/skills/
cp -r node_modules/agent-tool-forge/skills/forge-verifier ~/.claude/skills/

Then in any Claude Code session:

/forge-tool      # 12-phase tool creation dialogue
/forge-eval      # Generate golden + labeled eval suites
/forge-mcp       # Generate an MCP server from a ToolDefinition
/forge-verifier  # Detect tools without verifiers, generate stubs

Skills

| Skill | Purpose | |-------|---------| | /forge-tool | 12-phase structured dialogue: challenge necessity, lock the description contract, generate tool + tests + evals | | /forge-eval | Generate golden (5-10 cases) and labeled (multi-tool) eval suites with deterministic assertions | | /forge-mcp | Generate an MCP server scaffold from a ToolDefinition | | /forge-verifier | Detect tools without verifier coverage, generate verifier stubs + barrel registration |

The 12-Phase /forge-tool Dialogue

| Phase | What Happens | |-------|-------------| | 0 | Creative exploration — open-ended "what should this do?" | | 1 | Skeptic gate — challenge necessity, overlap, scope | | 2 | Description + name — lock the routing contract | | 3 | Collect fields — schema, category, consequence level, confirmation flag | | 4 | Routing — collect endpoint target, HTTP method, auth type, parameter mapping | | 5 | Dependency check — verify the tool context provides what's needed | | 6 | Confirm full spec — sign off before any code is written | | 7 | Generate all files — tool, tests, barrel registration | | 8 | Run tests — must be green before proceeding | | 9 | Generate evals — hand off to /forge-eval | | 10 | Generate verifiers — create verifier stubs for the new tool | | 11 | Done — summary of everything created |

Runtime Features

• ReAct loop — multi-turn LLM + tool execution, streamed via SSE
• HITL — four levels (autonomous → paranoid), pause/resume with 5-minute TTL
• Verifiers — post-response quality pipeline (warnings + flags, ACIRU ordering)
• Eval runner — node lib/index.js run --eval <path> executes eval JSON, checks assertions, stores results in SQLite; --record / --replay for fixture-based testing
• Observability — token tracking, cost estimation, per-tool metrics; chat audit log and eval history stored in Postgres when DATABASE_URL is set (durable across Railway/ephemeral filesystem deploys)
• Web component — <forge-chat> drop-in chat widget (vanilla JS, zero deps)

Optional Peer Dependencies

The sidecar core requires only better-sqlite3. Additional backends are loaded on demand when configured — install them only if you use them:

| Package | When needed | |---------|-------------| | redis or ioredis | conversation.store: 'redis' or rateLimit.enabled: true with Redis backend | | pg | database.type: 'postgres' — Postgres conversation store, agent registry, preferences, eval results, chat audit log, and verifier registry |

# Redis backend
npm install ioredis          # or: npm install redis

# Postgres backend
npm install pg

If a required package is missing, the sidecar prints an actionable error on startup rather than crashing at import time.

Exported Subpaths

All subpaths ship with TypeScript declarations.

import { createSidecar }      from 'agent-tool-forge'               // main entry
import { reactLoop }           from 'agent-tool-forge/react-engine'
import { createAuth }          from 'agent-tool-forge/auth'
import { makeConversationStore } from 'agent-tool-forge/conversation-store'
import { mergeDefaults }       from 'agent-tool-forge/config'
import { makeHitlEngine }      from 'agent-tool-forge/hitl-engine'
import { makePromptStore }     from 'agent-tool-forge/prompt-store'
import { makePreferenceStore } from 'agent-tool-forge/preference-store'
import { makeRateLimiter }     from 'agent-tool-forge/rate-limiter'
import { getDb }               from 'agent-tool-forge/db'
import { initSSE }             from 'agent-tool-forge/sse'
import {
  PostgresStore,
  PostgresEvalStore,
  PostgresChatAuditStore,
  PostgresVerifierStore
}                              from 'agent-tool-forge/postgres-store'
import { buildSidecarContext, createSidecarRouter } from 'agent-tool-forge/forge-service'

Documentation

| Doc | Contents | |-----|----------| | docs/tui-workflow.md | TUI walkthrough, start to finish | | docs/reference/config.md | forge.config.json field reference | | docs/reference/api.md | HTTP endpoints, SSE events, HITL flow | | docs/eval-runner-contract.md | Eval file format and assertion spec | | docs/API-DISCOVERY.md | API discovery TUI | | docs/VERIFIER-FACTORY.md | Verifier gap detection and stub generation |

Repo Structure

lib/
  sidecar.js              # createSidecar() — package entry point
  index.js                # TUI + CLI entry point
  react-engine.js         # ReAct loop, SSE streaming
  hitl-engine.js          # HITL pause/resume
  verifier-runner.js      # Post-response verifier pipeline
  eval-runner.js          # Eval execution engine
  checks/                 # Deterministic assertion checks
  fixtures/               # Record/replay fixture store
  comparison/             # Run comparison + Wilson statistics
  runner/                 # Gate evaluation + CLI
  views/                  # TUI screens
  db.js                   # SQLite persistence
skills/
  forge-tool/             # 12-phase tool creation workflow
  forge-eval/             # Golden + labeled eval generation
  forge-mcp/              # MCP server generation
  forge-verifier/         # Verifier gap detection + stub generation
templates/                # Pseudo-code reference templates
docs/
  tui-workflow.md         # Start-to-finish TUI guide
  reference/
    config.md             # forge.config.json reference
    api.md                # HTTP + SSE reference
  eval-runner-contract.md # Eval file format spec
  API-DISCOVERY.md        # API discovery workflow
  VERIFIER-FACTORY.md     # Verifier gap detection + stub generation
example/
  tools/                  # Example tool files
  verification/           # Example verifiers
docs/examples/            # Example evals (golden, labeled)
widget/
  forge-chat.js           # <forge-chat> web component

Standing on Shoulders

Tool-Forge integrates ideas and code from two excellent open-source projects:

• evalkit by wkhori — MIT License Provides the deterministic check suite (lib/checks/): content matching, tool selection verification, schema validation, and the runChecks() meta-runner. Used under MIT license with attribution in each file.

• agent-eval-kit by FlanaganSe — MIT License Provides fixture-based record/replay (lib/fixtures/), statistical comparison with Wilson confidence intervals (lib/comparison/), gate evaluation (lib/runner/gate.js), and composition operators (lib/checks/compose.js). Used under MIT license with attribution in each file.

License

MIT