llmrun
v1.0.1
Published
Run open-source LLMs on AWS EC2 from a local CLI — pick a model, spin up a right-sized GPU instance, and get a local OpenAI-compatible endpoint over AWS SSM.
Maintainers
Readme
Why llmrun?
- No local GPU required — your laptop stays cool. Models run on a right-sized AWS GPU instance you spin up on demand and let idle-stop when you're done.
- Pay for infrastructure, not tokens — no per-token pricing. You pay AWS on-demand rates (~$0.80/hr for a 7B model) only while the instance is running.
- Your data stays in your cloud — inference never leaves your AWS account. No third-party API sees your prompts, code, or documents — full isolation for sensitive or proprietary work.
Install
npm install -g llmrun
# or
pnpm add -g llmrunQuick start
llmrun init # scaffold llmrun.yaml
llmrun doctor # check prerequisites
llmrun up # pick a model → approve cost → provision → connectYour model is now available at http://localhost:8000/v1 — a full OpenAI-compatible endpoint.
Docs
Full documentation at https://tomaszczechowski.github.io/llmrun — configuration, model catalog, coding assistant integrations, and command reference.
License
Apache-2.0
