@qianxude/ai

A TypeScript LLM client with provider-model-task architecture, designed for the Cechat knowledge base toolset.

Overview

This package provides a modular architecture for making LLM calls. It features:

1. Provider-Model-Task Architecture - Flexible configuration for multiple LLM providers
2. OpenAI-compatible API - Works with CloudEdge, SiliconFlow, OpenAI, and other compatible services

Installation

bun add @qianxude/ai

Architecture

Package Structure

┌─────────────────────────────────────────────────────────────────────────┐
│                         @qianxude/ai                                    │
├──────────────────────────┬──────────────────────────────────────────────┤
│    Client                │      Types                                   │
│  (@qianxude/ai/client)   │   (@qianxude/ai)                             │
├──────────────────────────┼──────────────────────────────────────────────┤
│ LLMClient                │                                              │
│ ConfigLoader             │   CompletionOptions                          │
│ OpenAICompatibleProvider │                                              │
│ Provider-Model-Task      │   LLMConfig                                  │
└──────────────────────────┴──────────────────────────────────────────────┘

Provider-Model-Task Architecture

The new LLM client uses a three-layer configuration architecture that separates concerns and enables flexible model management:

┌────────────────────────────────────────────────────────────────┐
│                         Task Layer                             │
│  ┌──────────────┐ ┌──────────────┐ ┌──────────────┐            │
│  │classification│ │  reasoning   │ │ generation   │            │
│  └──────┬───────┘ └──────┬───────┘ └──────┬───────┘            │
└─────────┼────────────────┼────────────────┼────────────────────┘
          │                │                │
          ▼                ▼                ▼
┌────────────────────────────────────────────────────────────────┐
│                        Model Layer                             │
│  ┌──────────────────┐        ┌──────────────────┐              │
│  │  deepseek-chat   │        │ deepseek-reasoner│              │
│  │  temperature: 0.3│        │ temperature: 0.1 │              │
│  │  maxTokens: 2000 │        │ maxTokens: 4096  │              │
│  └──────┬───────────┘        └──────┬───────────┘              │
└─────────┼───────────────────────────┼──────────────────────────┘
          │                           │
          └───────────┬───────────────┘
                      │
                      ▼
┌────────────────────────────────────────────────────────────────┐
│                       Provider Layer                           │
│  ┌──────────────────────────────────────────────────────┐      │
│  │                      ce                              │      │
│  │  apiKeyEnv: CE_API_KEY                               │      │
│  │  baseUrlEnv: CE_BASE_URL                             │      │
│  └──────────────────────────────────────────────────────┘      │
└────────────────────────────────────────────────────────────────┘

How It Works

1. Task Layer - Semantic abstraction for model selection

   • Tasks represent high-level use cases (e.g., classification, reasoning)
   • Tasks map to specific models in the configuration
   • Allows changing models without modifying code

2. Model Layer - Model-specific configuration

   • Each model defines its parameters (temperature, maxTokens, etc.)
   • Models reference a provider for API connectivity
   • Multiple models can use the same provider

3. Provider Layer - API endpoint configuration

   • Defines environment variable names for API keys and URLs
   • All providers use OpenAI-compatible API protocol
   • Easy to switch between different LLM services

Core Components

1. LLM Client (@qianxude/ai/client) - Provider-model-task based client with multi-provider support
2. Type Definitions (@qianxude/ai) - Shared TypeScript types and types

Configuration

Using llm.manifest.json (Required)

The LLM client requires a llm.manifest.json configuration file. You must set the LLM_MANIFEST environment variable to point to this file.

1. Create llm.manifest.json

{
  "version": "1.0",
  "providers": {
    "ce": {
      "name": "CloudEdge",
      "apiKeyEnv": "CE_API_KEY",
      "baseUrlEnv": "CE_BASE_URL"
    }
  },
  "models": {
    "deepseek-chat": {
      "provider": "ce",
      "model": "deepseek-v3.2-671b",
      "temperature": 0.3,
      "maxTokens": 2000
    },
    "deepseek-reasoner": {
      "provider": "ce",
      "model": "DeepSeek-R1-671B",
      "temperature": 0.1,
      "maxTokens": 4096
    }
  },
  "tasks": {
    "classification": "deepseek-chat",
    "generation": "deepseek-chat",
    "reasoning": "deepseek-reasoner",
    "summarization": "deepseek-chat"
  }
}

2. Set Environment Variables

| Variable | Required | Description | | --------------------- | -------- | -------------------------------------------------- | | LLM_MANIFEST | Yes | Path to your llm.manifest.json file | | CE_API_KEY | Yes* | API authentication key (required by ce provider) | | CE_BASE_URL | Yes* | Base URL for the LLM API (required by ce provider) |

* Required only if using the ce provider. Each provider defines its own required environment variables in llm.manifest.json via apiKeyEnv and baseUrlEnv.

Example

export LLM_MANIFEST=./llm.manifest.json
export CE_API_KEY=your_api_key
export CE_BASE_URL=https://api.ce.example.com

Configuration Schema

providers

Providers define API endpoints and credential sources:

{
  "providers": {
    "ce": {
      "name": "CloudEdge",
      "apiKeyEnv": "CE_API_KEY",
      "baseUrlEnv": "CE_BASE_URL"
    },
    "sf": {
      "name": "SiliconFlow",
      "apiKeyEnv": "SF_API_KEY",
      "baseUrlEnv": "SF_BASE_URL"
    }
  }
}

Each provider defines:

• name: Human-readable name for logging
• apiKeyEnv: Environment variable name containing the API key
• baseUrlEnv: Environment variable name containing the base URL

models

Models define LLM configurations and reference providers:

{
  "models": {
    "deepseek-chat": {
      "provider": "ce",
      "model": "deepseek-v3.2-671b",
      "temperature": 0.3,
      "maxTokens": 2000,
      "topP": 0.9
    },
    "deepseek-reasoner": {
      "provider": "ce",
      "model": "DeepSeek-R1-671B",
      "temperature": 0.1,
      "maxTokens": 4096
    }
  }
}

Each model defines:

• provider: Reference to provider key (must exist in providers)
• model: Actual model name/id for the API
• temperature: Default sampling temperature (0-2)
• maxTokens: Default maximum tokens per request
• topP: Optional nucleus sampling parameter (0-1)

tasks

Tasks provide semantic model selection:

{
  "tasks": {
    "classification": "deepseek-chat",
    "generation": "deepseek-chat",
    "reasoning": "deepseek-reasoner",
    "summarization": "deepseek-chat",
    "extraction": "deepseek-chat",
    "translation": "deepseek-chat"
  }
}

Task-to-model mappings:

• Key: Task name representing a use case
• Value: Either a simple model string or an object with model and API options

Simple format (backward compatible):

{
  "tasks": {
    "classification": "deepseek-chat"
  }
}

Object format with API options:

{
  "tasks": {
    "generation": {
      "model": "deepseek-chat",
      "temperature": 0.7,
      "maxTokens": 2000
    },
    "reasoning": {
      "model": "deepseek-reasoner",
      "temperature": 0.05
    }
  }
}

When using the object format:

• model (required): Model ID from the models section
• temperature, maxTokens, topK, topP (optional): Override model-level defaults for this specific task

This abstraction allows:

1. Model switching without code changes - Update the task mapping to use a different model
2. A/B testing - Route specific tasks to different models for comparison
3. Cost optimization - Use cheaper models for simple tasks, expensive ones for complex tasks
4. Fallback strategies - Define multiple tasks for the same use case with different model tiers

Advanced Configuration Examples

Multi-Provider Setup

{
  "version": "1.0",
  "providers": {
    "ce": {
      "name": "CloudEdge",
      "apiKeyEnv": "CE_API_KEY",
      "baseUrlEnv": "CE_BASE_URL"
    },
    "sf": {
      "name": "SiliconFlow",
      "apiKeyEnv": "SF_API_KEY",
      "baseUrlEnv": "SF_BASE_URL"
    }
  },
  "models": {
    "deepseek-chat": {
      "provider": "ce",
      "model": "deepseek-v3.2-671b",
      "temperature": 0.3,
      "maxTokens": 2000
    },
    "qwen-max": {
      "provider": "sf",
      "model": "qwen-max",
      "temperature": 0.3,
      "maxTokens": 2000
    }
  },
  "tasks": {
    "classification": "deepseek-chat",
    "generation": "qwen-max"
  }
}

Task-Based Model Selection by Complexity

{
  "version": "1.0",
  "providers": {
    "ce": {
      "name": "CloudEdge",
      "apiKeyEnv": "CE_API_KEY",
      "baseUrlEnv": "CE_BASE_URL"
    }
  },
  "models": {
    "deepseek-chat": {
      "provider": "ce",
      "model": "deepseek-v3.2-671b",
      "temperature": 0.3,
      "maxTokens": 1000
    },
    "deepseek-reasoner": {
      "provider": "ce",
      "model": "DeepSeek-R1-671B",
      "temperature": 0.1,
      "maxTokens": 4000
    }
  },
  "tasks": {
    "simple_classification": "deepseek-chat",
    "complex_reasoning": "deepseek-reasoner",
    "code_generation": "deepseek-reasoner",
    "text_summarization": "deepseek-chat"
  }
}

Usage

Basic LLM Client

import { createLLMClient } from '@qianxude/ai/client';

// Create client (uses LLM_MANIFEST env var)
const llm = createLLMClient();
await llm.initialize();

// Simple completion (uses first available model)
const response = await llm.complete('Explain TypeScript generics');
console.log(response.content);

// Task-based model selection
const result = await llm.complete('Classify this text', { task: 'classification' });

// Direct model selection
const result2 = await llm.complete('Write a function', {
  model: 'deepseek-reasoner',
  temperature: 0.7,
  maxTokens: 1000,
});

With Custom Config Path (Overrides Env Var)

const llm = createLLMClient({ manifestPath: './config/llm.manifest.json' });
await llm.initialize();

With Logger

const llm = createLLMClient({ logger: consoleLogger });
await llm.initialize();

Complete with Messages

import { LLMMessage } from '@qianxude/ai/client';

const messages: LLMMessage[] = [
  { role: 'system', content: 'You are a helpful assistant' },
  { role: 'user', content: 'Hello!' },
];

const response = await llm.complete(messages, { task: 'generation' });

Accessing Configuration

// Get model for a task
const modelId = llm.getModelForTask('reasoning'); // 'deepseek-reasoner'

// Get full config
const config = llm.getConfig();

// Get tasks using a specific model
const tasks = configLoader.getTasksForModel('deepseek-chat');
// Returns: ['classification', 'generation', 'summarization']

Runtime Model Resolution

When you call complete(), the client resolves the model using this priority:

1. Explicit model ID - If options.model is provided, use it directly
2. Task-based selection - If options.task is provided, look up the model in config
3. First available model - Use the first model defined in llm.manifest.json

// Priority 1: Direct model selection
await llm.complete('Hello', { model: 'deepseek-reasoner' });

// Priority 2: Task-based selection
await llm.complete('Hello', { task: 'classification' });

// Priority 3: First available model
await llm.complete('Hello');

This resolution happens at runtime, allowing dynamic model selection based on context.

Migrating from Legacy to New Architecture

If you were previously using the legacy client, here are the key changes:

Legacy approach (removed):

// This code no longer works - legacy client has been removed
const llm = createLegacyLLMClient(console);
const response = await llm.complete('Hello'); // Returns string

New approach:

import { createLLMClient } from '@qianxude/ai/client';

const llm = createLLMClient({ logger: console });
await llm.initialize();
const response = await llm.complete('Hello'); // Returns LLMResponse
console.log(response.content);

Key differences:

1. New client requires initialize() call before use
2. New client returns LLMResponse objects with content and usage
3. New client supports task-based model selection via llm.manifest.json
4. New client uses options object instead of logger as first parameter
5. New client requires a llm.manifest.json configuration file

API Reference

TaskBasedLLMClient (New Interface)

The primary interface for the new provider-model-task architecture:

interface TaskBasedLLMClient {
  initialize(): Promise<void>;
  complete(input: string | LLMMessage[], options?: CompletionOptions): Promise<LLMResponse>;
  stream(input: string | LLMMessage[], options?: CompletionOptions): AsyncGenerator<LLMStreamChunk>;
  getModelForTask(task: string): string;
  getConfig(): LLMConfig;
}

Methods:

• initialize() - Load and validate configuration from llm.manifest.json
• complete() - Single prompt completion or multi-turn conversation completion with task-based model selection. Accepts either a string prompt or an array of messages.
• stream() - Stream completion with task-based model selection. Accepts either a string prompt or an array of messages.
• getModelForTask() - Get model ID for a configured task
• getConfig() - Get full configuration object

LLMClient

Implementation of TaskBasedLLMClient:

class LLMClient implements TaskBasedLLMClient {
  constructor(options?: LLMClientOptions);

  initialize(): Promise<void>;
  complete(input: string | LLMMessage[], options?: CompletionOptions): Promise<LLMResponse>;
  stream(input: string | LLMMessage[], options?: CompletionOptions): AsyncGenerator<LLMStreamChunk>;
  getModelForTask(task: string): string;
  getConfig(): LLMConfig;
}

ConfigLoader

class ConfigLoader {
  constructor(configPath: string);

  load(): Promise<LLMConfig>;
  resolveModel(modelId: string): ResolvedModelConfig;
  getModelForTask(task: string): string;
  getTasksForModel(modelId: string): string[];
  getConfig(): LLMConfig;
}

OpenAICompatibleProvider

class OpenAICompatibleProvider implements ProviderClient {
  complete(config: ResolvedModelConfig, call: LLMRequest): Promise<LLMResponse>;
  stream(config: ResolvedModelConfig, call: LLMRequest): AsyncGenerator<LLMStreamChunk>;
}

Types

Core Types

interface CompletionOptions {
  task?: string; // Task-based model selection (e.g., 'classification')
  model?: string; // Direct model selection (e.g., 'deepseek-chat')
  systemPrompt?: string; // System prompt for the conversation
  temperature?: number; // Sampling temperature (0-2)
  topP?: number; // Nucleus sampling (0-1)
  maxTokens?: number; // Maximum tokens to generate
}

interface LLMResponse {
  content: string;
  usage?: {
    promptTokens: number;
    completionTokens: number;
    totalTokens: number;
  };
  model: string;
}

interface LLMConfig {
  version: string;
  providers: Record<string, LLMProvider>;
  models: Record<string, LLMModel>;
  tasks: Record<string, string>;
}

interface LLMProvider {
  name: string;
  apiKeyEnv: string;
  baseUrlEnv: string;
}

interface LLMModel {
  provider: string;
  model: string;
  temperature: number;
  maxTokens: number;
  topP?: number;
}

interface TaskBasedLLMClient {
  initialize(): Promise<void>;
  complete(input: string | LLMMessage[], options?: CompletionOptions): Promise<LLMResponse>;
  stream(input: string | LLMMessage[], options?: CompletionOptions): AsyncGenerator<LLMStreamChunk>;
  getModelForTask(task: string): string;
  getConfig(): LLMConfig;
}

interface LLMMessage {
  role: 'system' | 'user' | 'assistant';
  content: string;
}

Error Types

class LLMError extends Error {
  readonly code: string;
  readonly cause?: unknown;
}

class ConfigurationError extends LLMError {
  // Configuration-related errors
}

class ProviderError extends LLMError {
  readonly provider: string;
  // Provider API errors
}

Error Handling

import { createLLMClient, ConfigurationError, ProviderError } from '@qianxude/ai/client';

const llm = createLLMClient();

try {
  await llm.initialize();
  const response = await llm.complete('Hello');
} catch (error) {
  if (error instanceof ConfigurationError) {
    console.error('Config error:', error.message);
  } else if (error instanceof ProviderError) {
    console.error('Provider error:', error.message, 'Provider:', error.provider);
  } else {
    console.error('Unexpected error:', error);
  }
}

License

Private package for Cechat internal use.