tomoshi
v1.1.12
Published
Tomoshibi (灯火 — lamp light) — self-hosted web search & scraping for AI agents. One binary, $5/mo.
Maintainers
Readme
Why tomoshi?
If you're paying Firecrawl/Exa per token or spawning a browser per request, you're overpaying.
| What hurts with hosted APIs | What tomoshi gives you | Outcome | | --------------------------------------- | --------------------------------------------------- | ----------------------------------- | | $0.01–0.10 per scrape + rate limits | $0 self-hosted — one binary, hobby-tier RAM | Ship RAG without a cloud bill | | 500ms Chrome spawn per request | One shared allocator + recycled tabs | ~200ms static, parallel pools | | JS SPAs return empty HTML | Smart mode — static first, fallback to Chromedp | Works on React/Vue without guessing | | Noisy HTML (nav/ads/footer) | Readability main-content + ad block | Clean markdown your LLM wants |
Part of Tomoshibi 灯火 — Go API + Svelte 5 playground + this MCP. Formerly Cinder.
Install (30 seconds)
Prerequisites: A running Tomoshibi API. Host it in one command:
git clone https://github.com/Michael-Obele/tomoshibi.git && cd tomoshibi
docker compose up -d # api 7431 + redis 7434 (searxng: --profile searxng)
# or: fly deploy (see tomoshibi README)Then add the MCP:
// .vscode/mcp.json or Claude config
{
"mcpServers": {
"tomoshi": {
"command": "npx",
"args": ["-y", "tomoshi"],
"env": {
"TOMOSHI_API_URL": "http://localhost:7431",
// optional: forwarded to the backend as a search API key on every
// request (X-Tomoshi-Env): set keys here, not in the backend's .env
"BRAVE_SEARCH_API_KEY": "BSB-...",
},
},
},
}Legacy CINDER_API_URL still works. Aliases: tomoshibi, tomoshibi-mcp (same package).
Restart your editor — tomoshi appears as 3 tools.
Run locally (development)
Prerequisites: Bun 1.2+ and a running Tomoshibi API (http://localhost:7431).
git clone https://github.com/Michael-Obele/tomoshibi.git && cd tomoshibi/packages/mcp
bun install
cp .env.example .env # TOMOSHI_API_URL=http://localhost:7431
bun run dev # HTTP + SSE server → http://localhost:7433 (reloads on change)
bun run start # same, without --watchTwo transports — pick one:
| Transport | Command | Use for |
| -------------- | ---------------------------------------- | -------------------------------------- |
| stdio | npx -y tomoshi (bin → dist/stdio.js) | local editors: VS Code, Claude, Cursor |
| HTTP + SSE | bun run dev on PORT (7433) | remote / hosted clients |
HTTP endpoints: POST /mcp (Streamable HTTP), /sse + /message (legacy SSE), /health and GET / (JSON status).
Checks and build:
bun run typecheck # tsc --noEmit
bun test # unit tests
bun run check # typecheck alias
bun run build # dist/ — what the published bin runsConfig (see .env.example): TOMOSHI_API_URL (required), TOMOSHI_API_KEY, PORT, REDIS_URL + MCP_SESSION_MANAGER=redis (sessions default to memory), RATE_LIMIT_*, OAUTH_*, LOG_LEVEL. Legacy CINDER_API_URL / CINDER_API_KEY still work.
Search key forwarding: names the backend asks for (GET /v1/env, e.g. BRAVE_SEARCH_API_KEY) are forwarded from this server's env on every request as X-Tomoshi-Env, so search API keys are configured once, here. TOMOSHI_FORWARD_ENV tunes it: "" = all requested names (default), "false" = off, or a comma list to restrict. The backend re-validates against its own allowlist and resolves the keys per request; it never stores them.
Docker
# from the repo root — starts on http://localhost:7433 (needs an API on 7431)
docker compose -f packages/mcp/docker-compose.yml up -d
curl http://localhost:7433/health
# → {"service":"tomoshi-mcp","status":"ok",...}Point it somewhere else with TOMOSHI_API_URL (the name the server actually reads):
TOMOSHI_API_URL=https://your-api.example.com \
docker compose -f packages/mcp/docker-compose.yml up -dPublished image:
docker run --rm -p 7433:7433 -e TOMOSHI_API_URL=http://host.docker.internal:7431 michaelobele/tomoshibi-mcpTools
3 tools, not 8 — resource-oriented (action enum, ≤7 philosophy).
| Tool | action | What it does |
| ------------------ | -------------------------------------------------------------------- | ------------------------------------------------------------------ |
| tomoshi_extract | scrape · scrape_multi (≤10) · links · batch · batch_status | Single/multi-page → markdown, link extraction, async batch (Redis) |
| tomoshi_discover | search · map · crawl · crawl_status | Web search (native roster → SearXNG → Brave/Serper/Tavily), sitemap, async BFS crawl |
| tomoshi_monitor | create · status · delete | Hash markdown, webhook on change |
Async actions (batch, crawl, monitor) are Redis-backed — poll *_status until done.
Example — scrape + search in one turn:
tomoshi_discoversearch"svelte 5 runes" →tomoshi_extractscrape_multion top 3 URLs → LLM-ready markdown
Scrape params for JS-rendered extraction (dynamic mode)
| Param | Type | Default | What it does |
| ----------------------- | ------------------ | ------- | ---------------------------------------------------------------------------------------------------------------------- |
| auto_extract | boolean | true | Harvest embedded page payloads (Next.js flight/__NEXT_DATA__, Nuxt, JSON-LD) into extracted_data; set false to disable. |
| actions[].type | string | - | wait_ms · wait_selector · click · scroll_down · scroll_to_bottom · wait_for_function · evaluate. |
| actions[].script | string | - | JS for evaluate (expression; return value captured) or wait_for_function (predicate returning truthy). |
| screenshot_opts.scale | "css" \| "device" | css | css = 1× device pixels (default, historical output); device = 2× device pixels (larger, crisper). |
Dynamic-mode responses also carry (omitted when empty):
evaluations—[{type, result, error, truncated}], one entry perevaluateaction (plus a{type: "wait_for_function", error}entry when a wait fails; the remaining actions still run); a JS throw is recorded inerrorand the scrape continues (results > 256 KB are clipped).extracted_data—{sources, canvases, truncated}; each source is{source, data}(__next_data,next_flight,__nuxt__,__initial_state__,json_ld), capped at 512 KB total (8 sources × 128 KB).
Caps: actions ≤ 50/request; evaluate results ≤ 256 KB; extracted_data ≤ 512 KB (8 sources × 128 KB).
Hosting
| Option | Command | Cost |
| ------------------ | ----------------------------------------------------------- | ----------------- |
| Docker Compose | docker compose -f packages/mcp/docker-compose.yml up -d | $0 (your machine) |
| From source | bun run dev (MCP) + go run ./packages/api/cmd/api (API) | $0 + Chromium |
API base: http://localhost:7431 — all endpoints under /v1 (/scrape, /search, /crawl, /batch, /monitor, /map).
Project structure
packages/mcp/
├── src/
│ ├── stdio.ts # stdio entry — what `npx tomoshi` runs (via dist/stdio.js)
│ ├── index.ts # HTTP + SSE entry (srvx) — /mcp · /sse · /health on PORT
│ ├── server.ts # McpServer wiring — registers the 3 tools
│ ├── client.ts # typed HTTP client over the Go API (shared by the tools)
│ ├── config.ts # Valibot env schema — TOMOSHI_API_URL, PORT, OAuth, limits
│ ├── adapter.ts # valibot → JSON Schema adapter for tool inputs
│ ├── auth-provider.ts # optional OAuth 2.1 (SimpleProvider)
│ └── tools/ # one file per tool, colocated bun tests
│ ├── extract.ts # tomoshi_extract — scrape · scrape_multi · links · batch · batch_status
│ ├── discover.ts # tomoshi_discover — search · map · crawl · crawl_status
│ ├── monitor.ts # tomoshi_monitor — create · status · delete
│ ├── schema.ts # shared field builders (num, url, …)
│ └── *.test.ts # bun test suites
├── dist/ # build output — published to npm, backs the bins
├── Dockerfile # deps stage → oven/bun:1-slim (non-root, wget healthcheck)
├── docker-compose.yml # mcp on 7433, needs an API on 7431
├── package.json # bin: tomoshi / tomoshibi / tomoshibi-mcp → dist/stdio.js
├── tsconfig.json # typecheck config (bun run check)
├── tsconfig.build.json # build config → dist/
└── .env.example # TOMOSHI_API_URL, PORT, sessions, rate limits, OAuthLinks
- GitHub: https://github.com/Michael-Obele/tomoshibi
- npm: https://www.npmjs.com/package/tomoshi
- API docs:
https://your-api.fly.dev/swagger/index.html(Swagger, auto-generated — debug mode only)
Name
tomoshi (灯 — ともし) — Japanese for lamp, light. Short form of Tomoshibi (灯火 — ともしび) lamp light, the glow of a paper lantern. Search as illumination: lighting up the web for AI agents.
License
MIT — see LICENSE.
