mlx-bun

MLX inference as a TypeScript/Bun library, with a signed executable serving OpenAI/Anthropic-compatible APIs on Apple Silicon. Embed generation in a Bun application or run the local server and browser chat app. The numerical tests compare logits bit-for-bit with mlx-lm for validated configurations.

Docs: mlx-bun.dev

Scope

The engine requires an Apple Silicon Mac running macOS 14 or later. The standalone executable includes its runtime; npm and source usage require Bun. Node.js, Linux, and Windows cannot run the native engine.

The server uses continuous batching by default. Eligible requests share the execution engine, including when only one request is active. Supported combinations and the explicit serial option are documented in server configuration.

Use the supported model roster to choose an artifact. Model architecture, weight format, and cache scheme determine compatibility. Arbitrary Hugging Face repositories and GGUF files are not automatically supported.

Install

Four ways in; all need an Apple Silicon Mac running macOS.

# Direct download — signed, notarized, no toolchain
curl -fsSL https://mlx-bun.dev/install.sh | sh

# Homebrew
brew install joshuarossi/tap/mlx-bun

# bunx — no install, needs Bun >= 1.4.0
bunx mlx-bun

# From source
git clone https://github.com/joshuarossi/mlx-bun.git && cd mlx-bun
bun install && bun run link-cli

Homebrew and direct download install the same self-contained bundle. npm ships a launcher and TypeScript source, then fetches the native runtime pack on first use. See distribution for details.

Quickstart

Start the server and open the chat UI. With no model selected, the first run downloads a starter model:

mlx-bun serve --port 8080

Use mlx-bun ls to see downloaded models. For example, after downloading a matching model, mlx-bun serve e4b selects it by name. Model selection and downloads are documented in the CLI reference.

Send a request from the terminal:

curl http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"messages": [{"role": "user", "content": "Hello!"}], "max_tokens": 128}'

import OpenAI from "openai";
const client = new OpenAI({ baseURL: "http://localhost:8080/v1", apiKey: "local" });
const res = await client.chat.completions.create({
  model: "local",
  messages: [{ role: "user", content: "Hello!" }],
});

Longer walkthroughs on the site: Installation and Quickstart.

Find your way around

• Use the app or server: start above, then consult the CLI, models, or troubleshooting.
• Build an application: the library API documents in-process Bun integration and isolated hosts for desktop apps. The HTTP API works with other runtimes.
• Evaluate or contribute: read the benchmark evidence and contribution guide. The docs index separates reference material, active design, and history.

Each reference topic has one home:

| Topic | Doc | |---|---| | Benchmark numbers (parity / performance / quality) | docs/reference/benchmarks.md | | Supported models roster | docs/reference/models.md | | Server start flags, MLX_BUN_* env, defaults | docs/reference/server-config.md | | HTTP API routes and request/response schemas | docs/reference/server-api.md | | CLI verbs | docs/reference/cli.md | | TypeScript library API | docs/reference/library-api.md | | LoRA fine-tuning (SFT / DPO / ORPO) | docs/reference/training.md | | Build, sign, notarize, publish | docs/reference/distribution.md | | Troubleshooting | docs/reference/troubleshooting.md | | Personal memory (local wiki) | docs/reference/memory.md | | Reference environment / oracle setup | docs/reference/environment.md | | Active engineering design docs | docs/design/ | | Contributing / repo rules | CONTRIBUTING.md |

The numerical contract distinguishes stock mlx-lm parity, mlx-optiq parity for supported extensions, and experimental methods. Tests compare logits against pinned oracles under matching conditions; batching and sampling can change generated trajectories. See the evidence and limits in benchmarks.md. Server policy defaults are documented separately in server configuration.

Why

mlx-bun brings MLX inference into a TypeScript application without a Python service. Bun's FFI calls mlx-c directly. Model implementations, scheduling, sampling, and reusable cache state live in the same process.

Performance depends on both native kernels and how the engine uses them. The project develops specialized kernels, avoids repeated computation, and measures complete requests as well as decode throughput. The benchmark ledger records the machines, settings, comparisons, and remaining regressions.

License

MIT. Third-party attributions: THIRD_PARTY_LICENSES.md.