llm-upptime
v0.1.0
Published
Upptime for LLM endpoints: real streamed checks with your own key from GitHub Actions, status page + badges
Maintainers
Readme
llm-upptime
Uptime and time-to-first-token for the LLM endpoints your app depends on, measured from your own key. A GitHub template repo plus a zero-dependency Node script. On a schedule, GitHub Actions sends one tiny real streamed completion to each of your endpoints/models, stores the result in the repo, and publishes a static status page and per-model shields.io badges on GitHub Pages.
Bahasa Indonesia: lihat bagian di bawah.
Upptime checks that a URL answers. An LLM endpoint can answer 200 OK on /v1/models
and still fail every completion, stall for 20 seconds before the first token, or break the stream halfway. llm-upptime checks the
thing your users feel: does a streamed completion for this model, on this route, with this key, actually work, and how fast does
the first token arrive?
- Real checks: OpenAI-style
/chat/completionsand Anthropic-style/v1/messages, streamed, so TTFT is real. - Your key, your models: any provider or gateway, including your own. Keys stay in repository secrets.
- No server: history is JSON in your repo, the page is static, alerts are GitHub issues.
- Cheap by design: 48 checks per model per day, each capped at 5 output tokens (see Cost).
- Zero runtime dependencies, Node 18+, MIT.
Quick start (30 seconds)
Option A: template repo (no npm involved, the workflow runs the script from the repo itself)
- Click Use this template on EzkielynaGroup/llm-upptime and create your repo (public is fine: keys are secrets, history holds no keys).
- Edit
.llm-status.yml: one entry per endpoint/model (see Presets). - Settings > Secrets and variables > Actions: add each
key_secret(for exampleOPENAI_API_KEY), and list it underenv:of the "Check models" step in.github/workflows/llm-status.yml. - Settings > Pages > Source: GitHub Actions.
- Actions > LLM status > Run workflow. Your page appears at
https://<you>.github.io/<repo>/.
Option B: existing repo
npx llm-upptime init # writes .llm-status.yml and .github/workflows/llm-status.yml, never overwritesThen steps 2 to 5 above. The generated workflow runs a pinned version (npx -y [email protected]).
Try it locally (one round, prints results, writes history/):
export OPENAI_API_KEY=... # whatever your key_secret names
npx llm-upptime check
npx llm-upptime build # then open site/index.htmlExample output
From a run against the local mock server used in the tests (no network, no cost):
$ npx llm-upptime check
llm-upptime 0.1.0 · 5 model(s) · 2026-11-11 18:53 UTC
UP Chat fast HTTP 200 TTFT 48 ms total 49 ms out 5
UP Chat slow start HTTP 200 TTFT 427 ms total 427 ms out 2
UP Messages API HTTP 200 TTFT 30 ms total 30 ms out 4
DOWN Flaky upstream HTTP 500 http_500:server_error after 30 ms
ERROR Rate limited HTTP 429 http_429:rate_limit_exceeded after 2 ms
wrote history/2026-11.json
$ npx llm-upptime build
Chat fast uptime 24h 100.00% 7d 100.00% 30d 100.00% TTFT p50 30 ms
...
Flaky upstream uptime 24h 0.00% 7d 0.00% 30d 0.00% TTFT p50 no data
Rate limited uptime 24h no data 7d no data 30d no data TTFT p50 no data
wrote 13 files to site/
$ npx llm-upptime alerts --dry-run
would open: [llm-upptime] Flaky upstream is down
would open: [llm-upptime] Rate limited: checks are failing (client error)One line of history/2026-11.json:
{"t":"2026-11-11T18:52:59Z","s":"chat-fast","r":"up","c":200,"ttft":60,"ms":60,"out":5,"tps":null}A badge file (site/badges/chat-slow-start-ttft.json):
{"schemaVersion":1,"label":"Chat slow start TTFT p50","message":"409 ms","color":"brightgreen"}The page shows per model: current state, uptime 24h / 7d / 30d, p50 TTFT and p50 total time (24h), tokens/s when measurable,
a 24-hour sparkline (TTFT line plus one bar per hour), and an incident list for 30 days. It is one self-contained HTML file with
no scripts and no external assets, light and dark mode, and it never shows your base_url.
Badges

SLUG is the entry's slug (default: the name in lower case with dashes, for example OpenAI gpt-4o-mini becomes
openai-gpt-4o-mini). The uptime badge shows 7 days; the TTFT badge shows the 24-hour median.
Config
.llm-status.yml (or .llm-status.yaml, or llm-status.json if you prefer JSON). To stay dependency-free, llm-upptime reads a
strict, small subset of YAML: block mappings and lists, # comments, plain / 'single' / "double" quoted strings, numbers,
true/false, null. It refuses flow lists ([a, b]), anchors, tags, multi-line strings and values containing : without
quotes, and tells you the line number. Anything it accepts is also valid YAML with the same meaning. Unknown keys are errors, so
typos do not pass silently.
title: LLM status
alert_after: 3
timeout_ms: 30000
site_url: https://you.github.io/repo/
models:
- name: OpenAI gpt-4o-mini
protocol: openai
base_url: https://api.openai.com/v1
model: gpt-4o-mini
key_secret: OPENAI_API_KEY
max_tokens: 5| Top-level key | Default | Meaning |
|---|---|---|
| title | LLM status | Page title |
| description | none | One line under the title |
| alert_after | 3 | Open an issue after this many non-up checks in a row (3 = 90 min at the default schedule) |
| timeout_ms | 30000 | Per check, 1000 to 120000. A complete answer must arrive in time, otherwise the check is down |
| site_url | none | Linked from alert issues |
| reference | off | url + optional label; see Reference column |
| models | required | 1 to 50 entries |
| Model key | Default | Meaning |
|---|---|---|
| name | required | Display name |
| slug | from name | File names for badges, key in history |
| protocol | required | openai (POST {base_url}/chat/completions) or anthropic (POST {base_url}/v1/messages) |
| base_url | required | OpenAI style includes /v1; Anthropic style does not (the same convention as the vendor SDKs). No credentials or query strings |
| model | required | Model id sent to the endpoint |
| key_secret | none | Name of the env var / repo secret holding the key. Omit for endpoints without auth |
| max_tokens | 5 | Output cap per check, 1 to 512 (cost guard) |
| prompt | Reply with the single word OK. | User message |
| token_param | max_tokens | max_completion_tokens for OpenAI reasoning models (o-series, gpt-5 family) |
| stream_usage | true | Sends stream_options.include_usage (OpenAI style) to get output tokens; set false if a server rejects it |
| anthropic_version | 2023-06-01 | anthropic-version header |
| reference_model | none | Model id to look up in the reference feed |
Presets
Copy into models:. Model ids are examples from November 2026; use the model you actually depend on.
# Anthropic
- name: Anthropic Haiku
protocol: anthropic
base_url: https://api.anthropic.com
model: claude-haiku-5-5
key_secret: ANTHROPIC_API_KEY
max_tokens: 16 # thinking is on by default for this model; a tiny cap may end before any text (still "up")
# DeepSeek
- name: DeepSeek chat
protocol: openai
base_url: https://api.deepseek.com/v1
model: deepseek-chat
key_secret: DEEPSEEK_API_KEY
# Ezkielyna (OpenAI style; Anthropic style uses base_url https://api.ezkielyna.store)
- name: Ezkielyna DeepSeek Flash
protocol: openai
base_url: https://api.ezkielyna.store/v1
model: deepseek-v4.1-flash
key_secret: EZKIELYNA_API_KEY
reference_model: deepseek-v4.1-flash # only used if you turn on the reference column
# Groq
- name: Groq Llama 8B
protocol: openai
base_url: https://api.groq.com/openai/v1
model: llama-3.1-8b-instant
key_secret: GROQ_API_KEY
# Ollama (GitHub-hosted runners cannot reach your laptop: use a self-hosted runner with
# base_url http://localhost:11434/v1, or a reachable host behind your own auth proxy)
- name: Ollama llama3.2
protocol: openai
base_url: https://ollama.example.com/v1
model: llama3.2
# OpenAI
- name: OpenAI gpt-4o-mini
protocol: openai
base_url: https://api.openai.com/v1
model: gpt-4o-mini
key_secret: OPENAI_API_KEY
# reasoning models: token_param: max_completion_tokens
# OpenRouter
- name: OpenRouter gpt-4o-mini
protocol: openai
base_url: https://openrouter.ai/api/v1
model: openai/gpt-4o-mini
key_secret: OPENROUTER_API_KEYEvery key_secret must also appear in the workflow:
- name: Check models
run: node bin/llm-upptime.mjs check
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
GROQ_API_KEY: ${{ secrets.GROQ_API_KEY }}GitHub does not hand all secrets to a step automatically, and llm-upptime does not ask for them in bulk on purpose.
What counts as up, down or error
The same rule a production gateway status page uses: only failures on the provider's side count against uptime.
| Result | When | Counts in uptime |
|---|---|---|
| up | HTTP 2xx and a complete, well-formed answer ([DONE] or a finish_reason for OpenAI style, message_stop for Anthropic style) | yes, as up |
| down | HTTP 5xx (including 529), timeout, network error, malformed or truncated stream, or an error event inside a 2xx stream | yes, as down |
| error | HTTP 4xx (wrong key, no credit, unknown model, 429 rate limit), a redirect, or a missing secret | no, but shown and alerted |
- Uptime = up / (up + down) over the window, rounded down to 2 decimals, so 1 failure in 48 checks shows 97.91%, never 98%.
- TTFT = time from sending the request to the first streamed chunk carrying generated text (answer or reasoning text; for
Anthropic style also the start of a thinking block, because thinking text can be omitted). A server that ignores
stream: trueand returns plain JSON isupwith TTFT "n/a" and the notenot_streamed. - Total = until the stream ends. Output tokens come from the usage the provider reports.
- Tokens/s is only computed when an answer has at least 50 output tokens and 2 s of generation after the first token.
Shorter answers say little about speed, so with the default
max_tokens: 5the speed column stays "n/a". To measure it, raisemax_tokensto about 100 and use a prompt that produces a long answer, and redo the cost arithmetic below. - Reason codes are short and safe to publish:
http_503,http_429:rate_limit_exceeded,timeout,network:ECONNREFUSED,bad_stream:truncated,stream_error:overloaded_error,missing_secret. Provider error messages are not stored and are only printed (redacted) with--verbose, because Actions logs of public repos are public.
Cost guard
At the default schedule (every 30 minutes) each model gets 48 checks per day. Per model:
| | Per day | Per 30 days | |---|---|---| | Requests | 48 | 1,440 | | Output tokens (cap 5 each) | ≤ 240 | ≤ 7,200 | | Input tokens (default prompt, ≤ 30 each incl. chat template) | ≤ 1,440 | ≤ 43,200 |
Worked example at a price of $1 per million input tokens and $4 per million output tokens: 43,200 × $1 / 1M + 7,200 × $4 / 1M = $0.0432 + $0.0288 = about $0.07 per model per month. At $15 / $75 per million (a large model): $0.648 + $0.54 = about $1.19 per model per month. Check your provider's prices and any minimum charge per request. Reasoning models count reasoning toward the cap, so the cap still holds.
Built-in limits: max_tokens is capped at 512, at most 50 models per config, 4 checks in parallel, timeout_ms at most 120 s.
GitHub Actions minutes: free for public repositories. In a private repository on the Free plan, two short jobs per run
(check + Pages deploy) are about 2 billed minutes, so a 30-minute schedule uses roughly 2,900 minutes a month, more than the 2,000
included. Use a public repo, or change the cron to hourly (0 * * * *, 24 checks per model per day).
Alerts
llm-upptime alerts (a workflow step with GH_TOKEN: ${{ github.token }} and issues: write) uses the preinstalled gh CLI:
- after
alert_afterchecks in a row that are notup, it opens one issue per model, labelledllm-upptime, with a table of the failed checks. Outages and client errors (for example an expired key) get different titles; - it never opens a second issue while one is open (issues carry a hidden marker per model);
- when the model answers again it closes the issue with a comment.
GitHub notifies watchers of new issues, so this works as e-mail / mobile alerts without extra services. The decision logic is
unit-tested; the GitHub side is not. npx llm-upptime alerts --dry-run shows what would happen. The step has
continue-on-error: true, so a GitHub hiccup never stops the page from updating.
Reference column
Off by default. If a provider publishes a status JSON in the schema described in docs/status-json-schema.md, you can show its public numbers next to your own:
reference:
url: https://api.ezkielyna.store/status/models.json
label: Ezkielyna public statusThen set reference_model on the entries you want compared. The reference numbers describe the provider's whole traffic, not
your checks, and the page says so. If the feed is down or has another schema, the page still builds and the column says
"unavailable". llm-upptime also writes its own site/status.json in the same schema, so your page can be a reference for others.
Commands
| Command | Does |
|---|---|
| npx llm-upptime init | Writes .llm-status.yml and .github/workflows/llm-status.yml; existing files are left unchanged |
| npx llm-upptime check | One round of checks, appends to history/YYYY-MM.json (one file per UTC month, one line per check) |
| npx llm-upptime build | Writes site/index.html, site/badges/<slug>-uptime.json and -ttft.json, site/status.json |
| npx llm-upptime alerts | Opens / closes GitHub issues (needs gh and GH_TOKEN) |
| Option | Applies to | Meaning |
|---|---|---|
| --dir <path> | all | Project folder (default: current folder) |
| --config <file> | check, build, alerts | Config file |
| --history <path> | check, build, alerts | History folder (default history) |
| --out <path> | build | Output folder (default site) |
| --only <slug,...> | check | Check some models only |
| --dry-run | check, alerts | Do not write history / do not call gh |
| --verbose | check | Print the provider's error message (redacted) for failed checks |
Exit codes: 0 ok (failed checks are data, not errors), 2 config or usage error, 1 anything else.
FAQ
How do I monitor whether the OpenAI API (or any OpenAI-compatible API) is down for my app?
Add the exact model and base URL your app uses to .llm-status.yml, store the key as a repository secret, and let the workflow
run every 30 minutes. You get a status page, badges and a GitHub issue after 3 failed checks in a row.
How do I measure time to first token (TTFT) of an LLM API?
TTFT needs a streamed request: llm-upptime sends stream: true and measures from sending the request to the first chunk that
carries generated text. A non-streamed request only tells you the total time.
Does a 429 rate limit count as downtime?
No. 4xx answers, including 429, are recorded as error, shown on the page and can open an issue, but only 5xx, timeouts,
network errors and broken streams lower uptime. A rate limit means the endpoint works but refused your request.
How much does LLM uptime monitoring cost? At the default 48 checks per model per day with at most 5 output tokens each: about 43k input and 7k output tokens per model per month, a few cents for most small models. The arithmetic is above.
Is my API key safe in a public repo?
The key lives only in GitHub's encrypted secrets and the step's environment. llm-upptime never writes it to history, never puts it
in the page, and redacts it (and anything that looks like a key) from error output. The page never shows your base_url.
Why do scheduled checks stop after a while? GitHub disables scheduled workflows in public repositories after 60 days without repository activity. The history commits this workflow makes count as activity in our understanding (Upptime relies on the same), but if GitHub ever disables it, re-enable the workflow under Actions. GitHub also delays scheduled runs at busy times; 30 minutes is a target, not a guarantee.
How it works
checkreads the config, takes each key from the environment variable named bykey_secret, and sends one streamed request per model (4 in parallel,redirect: manual, a hard timeout). It parses the SSE stream itself, classifies the result and appends compact records tohistory/YYYY-MM.json.- The workflow commits
history/asgithub-actions[bot](with a pull-rebase retry), runsalerts, thenbuild. buildreads the last 31 days, computes uptime / percentiles / hourly buckets / incidents, fetches the optional reference feed, and writes the static site, whichactions/deploy-pagespublishes.
Source: bin/llm-upptime.mjs (CLI), src/probe.mjs (requests and stream parsing), src/stats.mjs (maths), src/site.mjs
(page, badges, status JSON), src/alerts.mjs, src/yaml.mjs, src/config.mjs, src/history.mjs.
Limits
- Synthetic checks from GitHub-hosted runners (cloud data centres; you do not choose the region): latency includes their network path, not your users'.
- One check per model per run; a 30-minute schedule cannot see short blips between runs.
- OpenAI Chat Completions and Anthropic Messages only in v0.1. The OpenAI Responses API and Gemini are planned.
- No custom request headers or extra body fields yet (for example to switch thinking off).
- Not a benchmark. Results describe your route with your key at the time of the check.
Contributing
Issues and pull requests are welcome; see CONTRIBUTING.md. npm test runs everything against a local mock
server, with no network and no cost. Security reports: SECURITY.md.
License
MIT. Built by the team behind Ezkielyna, an independent AI API gateway. llm-upptime works with any provider; Ezkielyna is one preset among others.
Bahasa Indonesia
llm-upptime mengukur uptime dan time to first token (TTFT) endpoint LLM yang dipakai aplikasi Anda, dengan API key Anda
sendiri. Dijalankan oleh GitHub Actions secara terjadwal: tiap model dikirimi satu permintaan streaming kecil yang nyata, hasilnya
disimpan di repo (history/), lalu dibuat halaman status statis dan badge shields.io di GitHub Pages. Tanpa server, tanpa
dependensi, MIT.
Mulai cepat
- Klik Use this template di repo ini (atau jalankan
npx llm-upptime initdi repo yang sudah ada; file yang sudah ada tidak ditimpa). - Isi
.llm-status.yml: satu entri per endpoint/model. Contoh preset ada di bagian Presets (Anthropic, DeepSeek, Ezkielyna, Groq, Ollama, OpenAI, OpenRouter). - Simpan setiap key sebagai repository secret (Settings > Secrets and variables > Actions) dan tulis namanya di bagian
env:langkah "Check models" pada.github/workflows/llm-status.yml. Di config hanya ditulis nama secret (key_secret), bukan key-nya. - Settings > Pages > Source: GitHub Actions, lalu Actions > LLM status > Run workflow.
Aturan hasil. up = HTTP 2xx dan stream lengkap. down = HTTP 5xx, timeout, error jaringan, atau stream rusak/terpotong; hanya
ini yang menurunkan uptime. error = HTTP 4xx (key salah, saldo habis, nama model salah, 429 rate limit) atau secret belum
diisi; tetap tampil dan bisa membuka issue, tetapi tidak dihitung dalam uptime. Uptime dibulatkan ke bawah. Tokens/s hanya dihitung
bila jawaban minimal 50 token dan 2 detik.
Biaya. Jadwal default tiap 30 menit = 48 cek per model per hari, masing-masing maksimal 5 token output: per bulan sekitar 1.440 request, ≤ 43.200 token input dan ≤ 7.200 token output per model. Dengan harga contoh $1/$4 per juta token, sekitar $0,07 per model per bulan. Untuk repo privat, perhatikan kuota menit GitHub Actions (lihat Cost guard); repo publik gratis.
Keamanan. Key hanya dibaca dari environment variable, tidak pernah dicetak, disimpan, atau ditampilkan; pesan error disamarkan.
Halaman status tidak menampilkan base_url Anda.
Catatan. GitHub menonaktifkan workflow terjadwal setelah 60 hari tanpa aktivitas repo; commit data dari workflow ini biasanya cukup menjaga repo tetap aktif. Bila tetap nonaktif, aktifkan lagi di tab Actions.
Dibuat oleh tim di balik Ezkielyna, gateway API AI independen. llm-upptime bekerja dengan penyedia mana pun.
