webular

Universal web research CLI — scrape, crawl, map, extract, search, summarize, and gather context, by composing free & open-source libraries into Mastra workflows/tools, orchestrated through a deterministic mise task graph, on the Bun runtime.

webular distills the unique web capabilities of seven commercial web-data products (Firecrawl, Exa, Perplexity, Tavily, Parallel, Nimble, Brave) into a single CLI that needs no paid API — every capability is implemented from open-source building blocks.

Install

bun add -g webular      # or: npm i -g webular  (requires bun + mise on PATH)

webular requires Bun (runtime). mise routes every subcommand (mise run run:<command>) and ships bundled as a pinned npm dependency — a PATH install is used when present. The first command after a fresh install provisions the pinned toolchain once (non-interactive, pinned via the shipped bun.lock); later runs skip it.

Usage

webular <command> [options]
webular --help

| Command | Capability | |---|---| | search | Keyless web search via DuckDuckGo's no-JS HTML endpoint | | scrape | Fetch one URL → clean markdown/JSON (Bun.fetch + Readability + Turndown) | | crawl | Recursive same-host BFS crawl (no robots.txt/rate-limiting — crawl responsibly) | | map | Discover all URLs for a domain (sitemap + link BFS) | | extract | Structured extraction via CSS selectors / schema | | summarize | Extractive summary (no model) | | answer | Grounded answer (search → fetch → synthesize) | | research | Multi-step research loop → cited markdown report | | parse | Local docs → markdown (PDF/DOCX/HTML/CSV/JSON/TXT) | | media | Download media; screenshot/PDF via a real browser (agent-browser) | | monitor | Track changes to a URL over time (diff + bun:sqlite) | | batch | Scrape many URLs concurrently | | doctor | Diagnose environment & toolchain | | tasks | List the mise task graph this CLI routes through | | mcp | Expose webular capabilities as an MCP server | | diagram | List the design diagrams (pre-rendered SVGs in docs/diagrams) | | audit | Report deepsec availability + scan plan (keyless; scans need LLM keys) | | act | Drive a real browser — open, snapshot, screenshot, interact (agent-browser) |

Global options: --json, -o/--output <file>, --timeout <ms> (per request attempt), -h/--help, -V/--version.

Exit codes: 0 success · 1 failure · 2 usage error · 3 partial failure (batch) · 127 toolchain missing.

webular scrape https://example.com --json
webular search "bun javascript runtime" --json
webular map https://example.com
webular research "web scraping in 2026" --depth 3 -o report.md

Architecture

you ▸ webular CLI ▸ mise (universal task graph) ▸ Mastra workflow ▸ Mastra tools ▸ FOSS libs ▸ web

The CLI never calls capability code directly: it parses argv and routes to mise run run:<command> -- <args>. mise owns a single, acyclic task graph (setup → format → lint → typecheck → build → test → validate → ci, with each command branching off setup). Each command is a Mastra workflow built from non-model Mastra tools that wrap open-source libraries.

The design is captured in seven diagrams under docs/diagrams/ (sequence, class, state, ER, user-journey, requirement, mindmap), generated with mmdc as the design basis for the implementation.

Designed per the Command Line Interface Guidelines: machine data to stdout, logs to stderr, --json everywhere, -o file sinks, forgiving parsing, helpful --help, and a doctor for diagnostics.

Development

This project is built test-first. The toolchain is pinned in mise.toml. In a fresh clone, run mise trust ./mise.toml once before anything else.

mise run ci         # the single CI path: setup→lint→typecheck→build→test→validate
mise run format     # apply formatting (mutating; deliberately OFF the check-only ci spine)
mise run test       # run the bun test suite (real services, no mocks)
mise run run:scrape -- --url https://example.com --json

Conventions enforced by the suite: ≤200 lines/file, ≤30 lines/construct, nesting depth ≤3, no orphaned exports, and no mocks/stubs — tests exercise real behavior against a local Bun.serve fixture (deterministic, offline-capable). Live-internet cases (DuckDuckGo search, a real browser) are opt-in: WEBULAR_TEST_LIVE=1 mise run test. Note: DDG blocks many datacenter IPs.

License

Apache-2.0 © Kiren Srinivasan. Built on Bun, Mastra, mise, mermaid-cli, deepsec, and agent-browser, plus a set of open-source libraries.