firecrawl-guard

A Firecrawl-compatible gateway that gives each autonomous agent its own virtual key, endpoint scope, and hard cumulative credit budget.

Firecrawl's own controls are excellent at the request and team layer: Agent maxCredits caps one job, team rate limits protect capacity, and historical usage can attribute spend after the fact. They do not represent:

Research Agent A may consume 500 credits today while Agent B may consume 5,000.

When several workers share one Firecrawl account, a retry loop on a single agent can exhaust the team's crawl budget. Guard sits in front of Firecrawl, issues constrained virtual keys, and rejects over-budget requests before they reach Firecrawl.

This product is a compatibility gateway, not a dashboard, scraper, or billing system. Firecrawl documents that requests processed by its infrastructure can still be charged when the target site returns an error; failed Agent runs may be refunded when they report zero usage. Guard is about cumulative isolation and conservative preflight accounting.

60-second quick start

npm install -g firecrawl-guard
# or: npm install && npm run build && node dist/index.js

export FIRECRAWL_UPSTREAM_API_KEY=fc-your-real-key
export ADMIN_TOKEN=change-me
export DATABASE_PATH=./data/guard.sqlite

firecrawl-guard key create research-agent --daily-credits 50 --allow agent,search,crawl
firecrawl-guard key create production-agent --daily-credits 500 --allow agent,search,crawl
firecrawl-guard serve

Point a Firecrawl client at the gateway and authenticate with the virtual key, never the real one.

import { Firecrawl } from "firecrawl";

const firecrawl = new Firecrawl({
  apiKey: process.env.RESEARCH_AGENT_VIRTUAL_KEY,
  apiUrl: "http://127.0.0.1:8787",
});

await firecrawl.startAgent({
  prompt: "Find the founders of Firecrawl",
  maxCredits: 25,
});

from firecrawl import Firecrawl

firecrawl = Firecrawl(
    api_key=os.environ["RESEARCH_AGENT_VIRTUAL_KEY"],
    api_url="http://127.0.0.1:8787",
)

app, err := firecrawl.NewFirecrawlApp(
  virtualKey,
  "http://127.0.0.1:8787",
)

curl -s http://127.0.0.1:8787/v2/search \
  -H "Authorization: Bearer fg_..." \
  -H "Content-Type: application/json" \
  -d '{"query":"firecrawl","limit":5}'

SDKs differ in the exact option name (apiUrl, api_url, constructor arg). Where a client cannot set a custom base URL, use raw REST against the gateway.

Supported endpoints

| Method | Path | Behavior | | --- | --- | --- | | POST | /v2/agent | Caps maxCredits at remaining virtual-key budget, then forwards | | GET | /v2/agent/:jobId | Polls the job that this virtual key started and settles usage | | POST | /v2/search | Reserves worst-case search (+ optional scrape) cost, then reconciles creditsUsed | | POST | /v2/crawl | Requires an explicit finite limit, reserves a page-based maximum, then reconciles | | GET | /v2/crawl/:jobId | Polls the crawl this virtual key started and settles usage | | POST | /admin/keys | Create a virtual key (plaintext shown once) | | GET | /admin/keys | List keys (hashes only) | | DELETE | /admin/keys/:id | Revoke a key | | GET | /admin/usage | Per-key budget snapshot and ledger | | GET | /healthz | Liveness |

Unknown Firecrawl paths, including Browser and Interact, are rejected. They are not proxied silently.

What can receive a hard guarantee

Guard only forwards a request when it can compute a conservative upper bound in credits.

Agent

Hard guarantee: yes, via Firecrawl's own maxCredits.

• If the client omits maxCredits, Guard injects the remaining virtual-key budget (Firecrawl's default is 2,500).
• The upstream maxCredits is min(requested, remaining daily, remaining lifetime).
• On job completion, usage is settled from creditsUsed. Failed Agent runs that report creditsUsed: 0 return the reservation to available capacity.
• Persisted jobs are also polled in the background, so settlement does not depend on the originating client staying alive.

Search

Hard guarantee: yes, for documented search + scrape shapes.

• Search itself: 2 credits per 10 results, rounded up, per source. End-to-end ZDR (enterprise: ["zdr"]) is 10 credits per 10 results.
• Optional scrape: 1 credit per result, plus 4 for JSON mode.
• Current live Firecrawl docs say enhanced proxy has no surcharge. Guard still adds a 4-credit compatibility buffer for enhanced, auto, or omitted proxy mode because published Firecrawl pricing sources have changed across versions. Set proxy: "basic" to avoid that buffer.
• Zero-data retention on scraped results adds 1 credit per result.
• X/Twitter scraping adds 29 credits per request. Search result URLs are not known at reservation time, so Guard includes that worst case for every scraped result.
• PDF parsing is allowed only with an explicit maxPages; set parsers: [] to disable the default PDF parser.

Crawl

Hard guarantee: yes, only with an explicit finite limit and explicit parser configuration.

• Firecrawl's default crawl limit is 10,000 pages. Guard refuses crawls that omit limit.
• Guard refuses crawls that omit scrapeOptions, because the upstream default PDF parser has no request-wide PDF-page ceiling. Use scrapeOptions: { parsers: [], proxy: "basic" } for the lowest documented per-page cost.
• Reserve limit * per-page cost, including the proxy compatibility buffer (+4/page), JSON (+4/page), ZDR (+1/page), X/Twitter (+29/page), and bounded PDF extras.
• Reconcile creditsUsed when the crawl reaches a terminal status, either through a client status request or the background reconciler.

Pricing basis checked 2026-08-22 against Firecrawl's billing, scrape, search, crawl, and proxy mode documentation. Future upstream repricing requires a Guard update before the hard guarantee can cover the affected request shape.

Why unsupported cost modes fail closed

Some Firecrawl options have variable, time-based, or undocumented credit cost. Guard returns 503 unbounded_cost and does not call Firecrawl when it cannot prove a ceiling.

Rejected in v1:

• Browser / Interact (time-based billing)
• Scrape actions (browser-like)
• Default PDF parsing (1 credit / PDF page with no maxPages)
• Lockdown on Search or Crawl (Firecrawl supports it only on /v2/scrape)
• Formats whose credit cost is not documented for budgeting (audio, video, summary, changeTracking, …)
• onlyCleanContent (undocumented LLM pass)
• Agent threatProtection in normal mode (the agent can visit an unbounded URL set)
• Any undocumented request field that might affect billing

That is the product. A gateway that guesses is worse than no gateway.

CLI

firecrawl-guard serve
firecrawl-guard key create research-agent --daily-credits 500 --allow agent,search,crawl
firecrawl-guard key list
firecrawl-guard key revoke vk_...
firecrawl-guard usage
firecrawl-guard demo-seed

Environment:

| Variable | Purpose | | --- | --- | | FIRECRAWL_UPSTREAM_API_KEY | Real Firecrawl key, server-side only | | ADMIN_TOKEN | Bearer token for /admin/* | | DATABASE_PATH | SQLite file, default ./data/guard.sqlite | | PORT | Default 8787 | | HOST | Default 127.0.0.1 | | FIRECRAWL_GUARD_ALLOW_REMOTE_ADMIN | Set true to reach /admin from non-loopback addresses (needed in Docker) | | FIRECRAWL_GUARD_PUBLIC_URL | Origin used when rewriting crawl next URLs | | FIRECRAWL_GUARD_JOB_POLL_INTERVAL_MS | Base background reconciliation interval, default 5000 (allowed: 1000–300000) |

Runaway-agent demo

export FIRECRAWL_UPSTREAM_API_KEY=fc-...
export ADMIN_TOKEN=dev
export DATABASE_PATH=./data/guard.sqlite

firecrawl-guard demo-seed
# prints research-agent (50) and production-agent (500) virtual keys once

firecrawl-guard serve

In another shell:

node examples/runaway-agent.mjs \
  --research fg_... \
  --production fg_... \
  --base http://127.0.0.1:8787

Expected:

1. research-agent is limited to 50 credits.
2. A retry loop is cut off with 402 budget_exceeded.
3. production-agent continues to search successfully on the same upstream account.
4. The real Firecrawl key never exists in either agent environment.

GET /admin/usage shows isolated ledgers.

Maintainers can capture the complete proof from a disposable ledger:

npm run build
mkdir -p assets
asciinema rec --overwrite -c ./examples/record-runaway-demo.sh assets/runaway-demo.cast
agg --cols 110 --rows 30 --speed 0.5 assets/runaway-demo.cast assets/runaway-demo.gif

The runner removes FIRECRAWL_UPSTREAM_API_KEY from the agent process while keeping it available to the gateway.

Error model

| Status | When | | --- | --- | | 401 | Invalid or revoked virtual key (revoked keys may still poll jobs they already started) | | 403 | Endpoint outside the key allowlist, or unknown Firecrawl path | | 402 | Daily or lifetime budget would be exceeded | | 429 | Virtual-key concurrency exceeded, or pass-through of upstream 429 | | 502 | Upstream Firecrawl 5xx, timeout, or transport failure | | 503 | Maximum credit cost cannot be bounded |

Architecture

Single Node 22 process: Fastify + SQLite (Drizzle / better-sqlite3).

• ProxyServer — Firecrawl-compatible HTTP surface
• VirtualKeyStore — create, hash, revoke, resolve
• PolicyEngine — auth, endpoint scope, concurrency
• CostEstimator — conservative maxima from documented v2 billing
• ReservationLedger — atomic reserve / settle / release
• FirecrawlClient — forwards to a fixed upstream origin
• AsyncJobTracker — maps Agent/Crawl jobs back to the originating virtual key
• AsyncJobReconciler — resumes polling persisted reservations after clients disconnect or the gateway restarts
• Redactor — strips secrets from logs and errors

Reservations use SQLite BEGIN IMMEDIATE transactions. Fifty concurrent retries against the last remaining credits cannot oversubscribe a key.

Async Agent/Crawl job IDs live in the same SQLite ledger as their reservations. The reconciler scans reserved jobs when the server starts, polls with bounded concurrency and exponential backoff, and settles atomically only after Firecrawl reports completed, failed, or cancelled. A terminal response without creditsUsed commits the full reserved maximum. Timeouts, non-terminal responses, and upstream errors keep the reservation held; there is no TTL release.

Threat model

Assets. The upstream Firecrawl API key, virtual keys, and the usage ledger.

Trust boundary. Clients are untrusted agents. The gateway host is trusted. Firecrawl is trusted for billing fields it returns, not for client-supplied cost parameters.

Guarantees.

• The real Firecrawl credential never leaves the gateway.
• Virtual keys are random 256-bit values, shown once, stored as SHA-256 hashes.
• Authorization is never written to logs.
• Scraped response bodies are proxied, not persisted.
• Admin routes default to loopback.
• The upstream hostname is fixed (https://api.firecrawl.dev in production) so the process is not an open proxy.
• Request bodies are capped at 1 MiB.

Non-guarantees.

• Guard cannot see credits Firecrawl might later reprice retroactively.
• If Firecrawl never returns a retrievable terminal status, the reservation stays held (conservative).
• A stolen virtual key can spend up to that key's remaining budget.
• Compromising the gateway host yields the upstream key.

Docker

docker compose up --build

Mount /data for the SQLite file. Set FIRECRAWL_GUARD_ALLOW_REMOTE_ADMIN=true if you need /admin from outside the container; otherwise create keys with the CLI against the same volume.

npm

npm install -g firecrawl-guard

Requires Node.js 22+.

Tests

npm test

Optional live smoke test against a low-credit Firecrawl key:

FIRECRAWL_LIVE_KEY=fc-... npm test

Non-goals

• Payments, crypto, or x402
• Multiple upstream providers
• A web dashboard
• AI features
• Browser / Interact in v1