Screenshots

Playground

Agent Mode with Tool Calling

Endpoint Proxy & Usage Guides

Dashboard

MCP Image Generation

MCP Image Understanding

Token Flow Sankey

Naming and Logo

I admit, the name "Waypoi" is proposed by LLM (shut out to Qwen3-vl-30B) after it reads my early repositoy. In the beginning, the work was just intended to be a simple proxy for ill-configured LLM endpoints. I am talking about PCAI.

At the time, way point was just a term, similar to "route point", providing access to the LLM endpoints.

As I learn more about the AI agents, I start to be convinced the name do has more meanings. AI agents are clearly a waypoi to the future AGI, superintelligence or singularity. I hope this project can be my stepping stone to that future.

But I feel I am not smart enough to do it by myself. In this sense, the project can provide a helping waypoi betwen me and a smarter me, what I projected myself in my tiny mind.

I asked the LLM (thanks again qwen3-vl-30B) to design a logo prompt and zimage to generate the logo. After several rounds of judging, the winner comes with the following prompt:

Minimal abstract gateway icon, two vertical parallel lines with a glowing orb passing through center, neon cyan accent on dark background, geometric precision, symbolizing data flow through a unified portal, clean vector style, icon design

Suprisingly, the logo looks like a layed down box plot.

Here is the ascii art version of the logo:

├─o─┤

Features

| Feature | Description | |---|---| | Reverse Proxy | Route LLM, diffusion, audio, and embedding requests to multiple backends behind a single OpenAI-compatible API | | Health-Based Failover | Automatic retry with circuit breaker, latency-aware routing, and per-endpoint TLS policy | | Smart Pools | Virtual model aliases (e.g. smart) that load-balance across providers matching capability requirements | | Web Playground | Chat interface with session history, image upload, streaming, voice call mode, and agentic tool use | | Agent Mode | Built-in MCP (Model Context Protocol) client — connect external servers or use the built-in /mcp endpoint; tools auto-connected and pre-selected on startup | | Built-in MCP Tools | generate_image (diffusion routing, file output to ~/.config/waypoi) and understand_image (vision model analysis with geometry metadata) | | Model Capability Matrix | Per-model input/output modality classification (text, image, audio, embedding, reasoning) with configured or inferred sources | | Provider Catalog | First-class provider and model management via waypoi providers / waypoi models CLI; supports openai, inference_v2, and custom adapters | | Responses API Shim | /v1/responses compatibility shim translates Responses-style requests to chat completions with SSE event mapping | | Benchmark Showcase | Live example replays and diagnostic suites via waypoi bench; supports file-driven scenarios, baselines, and per-model targeting | | Peek | Request-capture browser with calendar view, timeline inspection, media artifacts tab, and token-flow Sankey visualization | | Statistics | Per-model/endpoint request tracking with 7-day window, latency distribution, token usage charts, and CLI summaries | | Usage Guides | Dashboard-embedded copy-paste code snippets (cURL, Python, Node.js) for every proxied endpoint | | Hot Reload | Config changes apply without restart via filesystem watcher | | Auth Ready | Optional token-based authentication (disabled by default); MCP endpoint is always localhost-only |

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                           Waypoi Gateway                          │
├─────────────────────────────────────────────────────────────────────┤
│  Routes:                                                            │
│    /v1/chat/completions    → LLM backends                          │
│    /v1/embeddings          → Embedding backends                     │
│    /v1/images/*            → Diffusion backends                     │
│    /v1/audio/*             → Audio backends (TTS/STT)               │
│    /v1/models              → Aggregated model list                  │
│    /v1/responses           → Responses API shim                     │
│    /mcp                    → Built-in MCP tool server               │
│                                                                     │
│  Admin:                                                             │
│    /admin/endpoints        → Endpoint CRUD                          │
│    /admin/health           → Health status                          │
│    /admin/stats            → Request statistics                     │
│    /admin/sessions         → Chat session storage                   │
│    /admin/mcp/*            → MCP server management                  │
│                                                                     │
│  UI:                                                                │
│    /ui                     → React playground & dashboard           │
├─────────────────────────────────────────────────────────────────────┤
│  Storage (~/.config/waypoi):                                      │
│    config.yaml       │ Endpoint configuration                       │
│    health.json       │ Health state cache                           │
│    stats/*.jsonl     │ Request logs (30-day rotation)               │
│    sessions/*.json   │ Chat session history                         │
│    images/           │ AIGC image cache (LRU, 1GB default)          │
│    mcp-servers.yaml  │ MCP server registry                          │
└─────────────────────────────────────────────────────────────────────┘

Quickstart

# Install dependencies
npm install
cd ui && npm install && cd ..

# Build
npm run build:all

# Start server
npm run start

Open http://localhost:9469/ui for the web interface.

Add an Endpoint

# Via CLI
waypoi add \
  --name local-llm \
  --url http://localhost:11434 \
  --priority 1 \
  --type llm \
  --model local-default=llama3

# Diffusion endpoint
waypoi add \
  --name local-sd \
  --url http://localhost:7860 \
  --priority 1 \
  --type diffusion \
  --model sd-xl=stable-diffusion-xl

OpenAI-Compatible Endpoints

| Endpoint | Description | |----------|-------------| | POST /v1/chat/completions | Chat with streaming and tool calling | | POST /v1/embeddings | Text embeddings for RAG | | GET /v1/models | List available models | | POST /v1/images/generations | Image generation | | POST /v1/images/edits | Image editing | | POST /v1/audio/transcriptions | Speech-to-text | | POST /v1/audio/speech | Text-to-speech | | POST /v1/responses | Responses API shim | | POST /mcp | Built-in MCP service endpoint |

GET /v1/models includes both legacy endpoint_type and detailed capabilities metadata when available.

Responses API Shim

Waypoi exposes /v1/responses as a compatibility shim for clients that use Responses-style requests. Internally, requests are translated to chat completions and routed through the same failover path.

Key behavior:

• input_text / output_text content parts are normalized to text
• function call payloads are mapped to OpenAI tool-call shape
• developer role is normalized to system

For stream: true, Waypoi emits Responses-style SSE events (response.created, deltas, output items, and response.completed).

Admin Endpoints

| Endpoint | Description | |----------|-------------| | GET /admin/endpoints | List all endpoints | | POST /admin/endpoints | Add endpoint | | PATCH /admin/endpoints/:id | Update endpoint | | DELETE /admin/endpoints/:id | Remove endpoint | | GET /admin/health | Health status by endpoint | | GET /admin/stats | Aggregated statistics | | GET /admin/stats/latency | Latency distribution | | GET /admin/stats/tokens | Token usage over time | | GET/POST/DELETE /admin/sessions | Chat session CRUD | | GET/POST/DELETE /admin/mcp/servers | MCP server CRUD | | GET /admin/mcp/tools | List discovered tools | | POST /admin/mcp/tools/execute | Execute a tool | | GET /admin/providers | Provider catalog | | GET /admin/providers/:id | Provider details | | GET /admin/pools | Smart pool definitions | | POST /admin/pools/rebuild | Rebuild smart pools from providers |

Built-In MCP Service (/mcp)

Waypoi exposes a first-party MCP server for local agent-to-tool workflows without going through chat models.

• Endpoint: POST /mcp (Streamable HTTP MCP transport)
• Access: localhost only (localhost, 127.0.0.1, ::1)
• Auth: intentionally open on localhost, even when authEnabled=true

Client flow:

1. initialize
2. notifications/initialized
3. tools/list
4. tools/call

Initial built-in tool:

• generate_image
  • Generates image(s) using Waypoi's diffusion routing
  • Depends on at least one live diffusion-capable model
  • Always writes generated files to ~/.config/waypoi/generated-images by default
  • Override output location with WAYPOI_MCP_OUTPUT_ROOT (and optionally WAYPOI_MCP_OUTPUT_SUBDIR)
  • Returns structured result with model metadata plus file_path or file_paths relative to the output root; raw url/b64_json are optional via include_data
• understand_image
  • Performs image-to-text understanding with a vision-capable model
  • Supports image_path or image_url input with top-level text plus structured result output (ocr_text, objects, scene, details)
  • For local image_path inputs, preserves original image geometry and reports optional image_geometry metadata so coordinate-based tasks stay aligned with the source file

Agent defaults (summary):

1. Keep include_data=false unless inline image payload is explicitly required.
2. Set WAYPOI_MCP_OUTPUT_ROOT if you want images written to a specific location.

Example generate_image tool call:

{
  "name": "generate_image",
  "arguments": {
    "prompt": "Minimal icon with clean geometric shape"
  }
}

Example understand_image coordinate-sensitive prompt:

{
  "name": "understand_image",
  "arguments": {
    "image_path": "./assets/layout.png",
    "instruction": "Return the center point of the submit button as JSON with x and y in original image pixels."
  }
}

Canonical MCP governance and behavior contract: docs/mcp-guidelines.md.
Detailed MCP tool contract and examples: docs/mcp-service.md.

CLI Commands

# Endpoint management
waypoi ls                           # List endpoints
waypoi add --name --url --priority  # Deprecated (blocked in provider-first mode)
waypoi rm <id|name>                 # Deprecated (blocked in provider-first mode)
waypoi edit                         # Deprecated (blocked in provider-first mode)
waypoi stat                         # Run health check
waypoi test <model>                 # Test a model
waypoi acct                         # Token usage by endpoint

# Service management
waypoi service start                # Start background service
waypoi service stop                 # Stop service
waypoi service restart              # Restart service
waypoi service status               # Check if running

# Logs & stats
waypoi logs                         # Show last 50 log lines
waypoi logs -f                      # Follow log output
waypoi stats                        # Show 7-day statistics
waypoi stats --window=24h           # Custom time window
waypoi stats --json                 # JSON output

# MCP server management
waypoi mcp add --name --url         # Add MCP server
waypoi mcp list                     # List MCP servers
waypoi mcp rm <id|name>             # Remove MCP server
waypoi mcp enable <id|name>         # Enable server
waypoi mcp disable <id|name>        # Disable server

# Provider catalog + smart pools
waypoi providers                    # List providers (canonical)
waypoi providers import -f .env     # Import providers and credentials, rebuild pools
waypoi providers show <providerId>  # Show one provider
waypoi providers update <providerId> --insecure-tls|--strict-tls
waypoi providers update <providerId> --auto-insecure-domain ai-application.stjude.org
waypoi providers enable <providerId> # Enable provider
waypoi providers disable <providerId># Disable provider
waypoi providers migrate-endpoints --provider pcai --match-domain ai-application.stjude.org --protocol openai
waypoi providers pools              # List smart pools

# Models (one-hop by provider)
waypoi models                       # List models across providers
waypoi models pcai                  # List models for provider
waypoi models show pcai/gpt-4o      # Show one model
waypoi models add <providerId> --model-id <id> --upstream <name> --base-url <url>
waypoi models update <providerId> <modelRef> [patch options]
waypoi models rm <providerId> <modelRef>
waypoi models enable pcai/gpt-4o
waypoi models disable pcai/gpt-4o
waypoi models set-key pcai/gpt-4o --api-key <key>|--env-var <ENV>

# Benchmark showcase
waypoi bench                               # Default live showcase example suite
waypoi bench --list-examples               # List showcase examples
waypoi bench --example showcase-tinyqa-001
waypoi bench --suite showcase --model smart
waypoi bench --scenario file.json          # File-driven scenarios
waypoi bench --mode diagnostic --suite pool_smoke
waypoi bench --baseline ./bench-prev.json

Benchmark showcase examples are sourced from Hugging Face dataset vincentkoc/tiny_qa_benchmark (train split, 52 QA prompts).

Provider credentials imported with waypoi providers import -f .env are stored in plaintext at $WAYPOI_DIR/providers.json by design for local operation.

Canonical Provider-First Workflow

Use these commands as the primary operational path:

1. waypoi providers
2. waypoi providers show <providerId>
3. waypoi models
4. waypoi models <providerId>
5. waypoi models show <providerId>/<modelId>

Legacy waypoi provider ... and waypoi provider model ... forms are rewritten to canonical commands with a deprecation warning. Set WAYPOI_NO_WARN=1 to suppress legacy rewrite warnings in scripts.

TLS policy is provider-first:

• Provider insecureTls is the default for all provider models.
• Model insecureTls is an optional override (--clear-insecure-tls restores inheritance).
• Optional provider allowlist (autoInsecureTlsDomains) enables one-time TLS verify fallback and persists model override on successful retry.

Endpoint migrations use copy-then-disable semantics for rollback safety. Migrated source endpoints stay in config.yaml with disabled: true and can be re-enabled if needed.

Detailed benchmark format and assertions: docs/benchmark.md. Provider protocol adapter onboarding (including inference_v2): docs/providers.md.

External Client Integration (Opencode)

Waypoi is a local AI gateway for external clients. Configure Opencode (or other OpenAI-compatible clients) to use:

• Base URL: http://localhost:9469/v1
• API key: local-dev (or your configured token)
• Model: smart (recommended default free-model pool alias)
• Note: legacy smart-* aliases are removed; use smart or canonical provider/model IDs.

Protocol note: non-OpenAI upstreams are adapter-backed. inference_v2 is supported in sync mode for /v1/chat/completions (text + optional image), while keeping external OpenAI-compatible calls.

See docs/opencode.md for setup details.

Web UI

Access the playground at http://localhost:9469/ui:

• Playground — Chat interface with session history, image upload (VL models), and agent mode
• Dashboard — Real-time stats with latency charts, token usage, endpoint health, and usage guides with copy-paste code snippets (cURL, Python, Node.js)
• Peek — Calendar browser for captured requests, timeline inspection, media artifacts, and token-flow analysis
• Settings — Provider/model catalog management, MCP guidance, and image defaults

Endpoint Usage Guides

Each endpoint in the Dashboard includes an expandable "Usage Guide" dropdown showing:

• cURL — Command-line examples for quick testing
• Python — OpenAI SDK code with your proxy URL
• Node.js — TypeScript/JavaScript examples

All code snippets use localhost:PORT (your proxy) instead of the upstream endpoint, so you can copy-paste and run immediately.

Agent Mode

Toggle "Agent Mode" in the playground to enable tool calling:

1. Add MCP servers via UI or CLI
2. Connect to discover available tools
3. Select which tools to enable
4. Chat normally — the agent will use tools when appropriate

The agentic loop supports up to 10 tool iterations per message.

Peek

Peek is the request-capture browser for debugging prompts, routing, tool use, and media-heavy interactions:

• Calendar browse — move day by day through captured traffic
• Timeline view — inspect ordered request and response segments, including assistant reasoning/tool-call structure
• Request/Response tabs — compare raw payloads and normalized previews
• Media tab — review persisted images and other artifacts
• Token Flow Sankey — visualize where captured input/output tokens were attributed

Timeline inspection example:

Token-flow Sankey example:

Statistics

Waypoi exposes request statistics in both CLI and UI:

• waypoi stats for aggregated CLI summaries
• Dashboard cards and charts for latency, throughput, errors, and token usage
• /admin/stats for aggregate windows
• /admin/stats/latency for latency distribution buckets
• /admin/stats/tokens for token usage over time

Statistics are derived from the local JSONL request log store with retention-based rotation. Use them to validate routing changes, benchmark regressions, and slow/failing upstream behavior.

Configuration

config.yaml

endpoints:
  - name: openai
    baseUrl: https://api.openai.com
    apiKey: sk-...
    priority: 1
    type: llm
    models:
      - publicName: gpt-4
        upstreamModel: gpt-4-turbo
        capabilities:
          input: [text]
          output: [text]
      - publicName: qwen3-vl
        upstreamModel: qwen3-vl
        capabilities:
          input: [text, image]
          output: [text]
  
  - name: local-sd
    baseUrl: http://localhost:7860
    priority: 1
    type: diffusion
    models:
      - publicName: sd-xl
        upstreamModel: stable-diffusion-xl-base-1.0
        capabilities:
          input: [text]
          output: [image]

# Enable authentication (default: false)
authEnabled: false

capabilities is optional. If omitted, Waypoi infers capabilities from model metadata/name and endpoint type.

Environment Variables

| Variable | Default | Description | |----------|---------|-------------| | PORT | 9469 | Server port | | ADMIN_TOKEN | — | Bearer token for admin endpoints | | WAYPOI_DIR | ~/.config/waypoi | Storage directory | | WAYPOI_CONFIG | {dir}/config.yaml | Config file path | | WAYPOI_DEBUG_ERRORS | 0 | Set to 1 for verbose internal error logs |

Storage

All data is stored in ~/.config/waypoi (or $WAYPOI_DIR):

| File/Directory | Purpose | |----------------|---------| | config.yaml | Endpoint configuration | | health.json | Cached health state | | stats/ | JSONL stats files (30-day retention) | | sessions/ | Chat session JSON files | | images/ | AIGC image cache (1GB LRU) | | benchmarks/ | Benchmark reports | | mcp-servers.yaml | MCP server registry | | waypoi.pid | Service PID file | | waypoi.log | Service log file |

Development

# Run in development mode
npm run dev

# Build
npm run build

# Run tests
npm test

# Lint
npm run lint

Authentication

Authentication is disabled by default. To enable:

1. Set authEnabled: true in config.yaml
2. Restart (or wait for hot-reload)
3. Include Authorization: Bearer <token> header

The auth middleware protects /admin/* and /ui/* routes.

Manual Testing

# Test chat completions
curl http://localhost:9469/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "local-default", "messages": [{"role": "user", "content": "Hello"}]}'

# Test image generation
curl http://localhost:9469/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{"model": "sd-xl", "prompt": "A sunset over mountains"}'

# Run manual test script
BASE_URL=http://localhost:9469 MODEL=local-default npm run manual-test

License

MIT