What Is Qhaway?

Qhaway (Quechua: "to observe/watch") wraps every LLM call your agent makes and captures cost, latency, token usage, and model info. Export to OpenTelemetry, Prometheus, or MLflow. Works in Cloudflare Workers, Node.js, Deno, Bun.

import { QhawayTrace, ConsoleStorage } from '@carloscortezcloud/qhaway/trace';

const trace = new QhawayTrace(new ConsoleStorage(), { agent_id: 'my-agent' });

const wrapped = trace.wrap(myLlmCall, { model: 'gpt-4o', provider: 'openai', user_id: 'abc' });
const result = await wrapped(prompt);

// Console output:
// [Qhaway] ✓ gpt-4o (openai) | $0.00063 | 150→42 tok | 1234ms | user=abc

Install

npm install @carloscortezcloud/qhaway

Quick Start

import { QhawayTrace, ConsoleStorage } from '@carloscortezcloud/qhaway/trace';

const trace = new QhawayTrace(new ConsoleStorage(), { agent_id: 'my-agent' });
const wrapped = trace.wrap(myLlmCall, { model: 'gpt-4o', provider: 'openai', user_id: 'abc' });
const result = await wrapped(prompt);

Modules

Qhaway is a single package with subpath exports — import only what you need:

| Import path | What | |-------------|------| | @carloscortezcloud/qhaway | All-in-one entry | | @carloscortezcloud/qhaway/trace | Span wrapper + storage (D1/KV/Console) | | @carloscortezcloud/qhaway/cost | Pricing DB + cost attribution | | @carloscortezcloud/qhaway/otel | OTLP/HTTP JSON exporter | | @carloscortezcloud/qhaway/tinkuy | Auto-instrument TinkuyAgent | | @carloscortezcloud/qhaway/mlflow | MLflow metrics exporter | | @carloscortezcloud/qhaway/alerts | Threshold alerts (Slack/webhook/email/PagerDuty) | | @carloscortezcloud/qhaway/eval | Eval run tagging + promptfoo import + eval metrics | | @carloscortezcloud/qhaway/ui | Trace viewer tree + filters (feeds ui/index.html) |

Storage

| Adapter | Best for | Import | |---------|----------|--------| | D1Storage | Production (SQL, aggregation) | @carloscortezcloud/qhaway/trace (d1) | | KVStorage | High-scale write | @carloscortezcloud/qhaway/trace (kv) | | ConsoleStorage | Local dev / debug | @carloscortezcloud/qhaway/trace (console) |

Exporters

| Exporter | Destination | |----------|-------------| | OTEL | Any OTLP collector (Honeycomb, Grafana Tempo, Datadog, SigNoz) | | Prometheus | GET /metrics endpoint for Grafana dashboards | | MLflow | Log cost/latency metrics as MLflow experiment runs | | Alerts | Threshold rules → Slack/webhook/email/PagerDuty with cooldown |

Feedback Loop

Correlate user feedback (thumbs up/down) with cost, latency, and model. Attach a rating to any span:

import { QhawayTrace, MemoryStorage, aggregateRating, ratingStats } from '@carloscortezcloud/qhaway';

const trace = new QhawayTrace(new MemoryStorage());

const wrapped = trace.wrap(myLlmCall, {
  model: 'gpt-4o',
  provider: 'openai',
  session_id: 'ses-1',
  rating: 1, // thumbs up — or -1 for down, 0 for neutral
});

rating flows through every storage adapter (D1/KV/Console) and the Prometheus endpoint exposes qhaway_rating_total{model, rating} and qhaway_cost_by_rating_total{rating}. The Grafana dashboard includes a Satisfaction vs Cost scatter panel.

import { aggregateRating } from '@carloscortezcloud/qhaway';

const spans = await storage.query();
const byRating = aggregateRating(spans);
// [{ model: 'gpt-4o', rating: -1, calls: 3, costUsd: 0.15, avgLatencyMs: 412, ... }]

const stats = ratingStats(spans);
// { thumbsUp: 40, thumbsDown: 7, avgCostPerThumbsDown: 0.09, thumbsDownRate: 0.15 }

Wire it to a Tinkuy feedback hook: on thumbs down, record a span with rating: -1 (see examples/feedback-wiring.ts).

Agent Evaluations

Tag eval cases with an eval_run_id, then compare cost vs score across models and runs:

import { labelEvalRun, aggregateEvalRuns, generateEvalMetrics } from '@carloscortezcloud/qhaway/eval';

const tagged = labelEvalRun(span, 'run-7', { score: 0.9, pass: true });

const runs = aggregateEvalRuns(spans);
// [{ eval_run_id: 'run-7', model: 'gpt-4o', passRate: 0.85, costPerScorePoint: 0.02, ... }]

const prom = generateEvalMetrics(spans); // qhaway_eval_run_* metrics for Grafana

Import promptfoo output directly — parsePromptfooOutput(json) converts results into labeled spans, so you can drop eval suites from promptfoo/LangChain into the same dashboards (qhaway-eval-dashboard.json). See examples/eval-comparison.ts.

Trace Viewer UI

Standalone, dependency-free HTML viewer for agent traces — no Grafana Tempo or LangSmith required. Open ui/index.html with a spans endpoint:

open ui/index.html?endpoint=https://my-agent.example.com/spans

It renders session → iteration → tool-call trees, colors expensive spans red, and filters by model/agent/date/success. Programmatic API via @carloscortezcloud/qhaway/ui (buildTraceTree, filterSpans, getUiApi).

Python SDK

pip install qhaway-trace

from qhaway import QhawayTrace, console_storage
from qhaway.integrations import OpenAIPatch

trace = QhawayTrace(storage=console_storage)
OpenAIPatch.apply(trace)  # auto-instrument all OpenAI calls

See python/README.md for OpenAI, LangChain, Anthropic, and FastAPI examples.

Architecture

Your Agent (Tinkuy / LangChain / raw)
  │
  ▼
QhawayTrace.wrap(fn)
  │
  ├── D1/KV (storage)
  ├── OTLP (Honeycomb, Grafana, Datadog)
  └── GET /metrics → Prometheus → Grafana dashboard

Grafana Dashboard

Import qhaway-dashboard.json into Grafana (Cloud or OSS) to visualize:

• Cost by model and user
• Latency P99 over time
• Token usage (input vs output)
• Recent call log
• Daily spend summary

Ecosystem

| Package | Role | npm | |---------|------|-----| | Qhaway | Agent obs (this) | @carloscortezcloud/qhaway | | Styrr | LLM router | styrr | | Sayay | Cost guardrails | GitHub | | Tinkuy | Agent framework | @carloscortezcloud/tinkuy-agent | | TideRAG | Edge RAG pipeline | @carloscortezcloud/tiderag |

License

Apache 2.0 — see LICENSE.