@contentgrowth/llm-service

Unified LLM Service for Content Growth applications. This package provides a standardized interface for interacting with various LLM providers (OpenAI, Gemini) and supports "Bring Your Own Key" (BYOK) functionality via pluggable configuration.

Installation

npm install @contentgrowth/llm-service

Usage

Basic Usage

The service requires an environment object (usually from Cloudflare Workers) to access bindings.

import { LLMService } from '@contentgrowth/llm-service';

// In your Worker
export default {
  async fetch(request, env, ctx) {
    const llmService = new LLMService(env);
    
    // Chat
    const response = await llmService.chat('Hello, how are you?', 'tenant-id');
    console.log(response.text);
    
    // Chat Completion (with system prompt)
    const result = await llmService.chatCompletion(
      [{ role: 'user', content: 'Write a poem' }],
      'tenant-id',
      'You are a poetic assistant'
    );
    console.log(result.content);
  }
}

Configuration & BYOK

The service uses a ConfigManager to determine which LLM provider and API key to use for a given tenant.

Default Behavior (Cloudflare KV + Durable Objects)

By default, the service expects the env object passed to the constructor to contain:

• TENANT_LLM_CONFIG: A KV Namespace binding.
• TENANT_DO: A Durable Object Namespace binding.

It uses these to fetch tenant-specific configurations.

Custom Configuration (Pluggable Providers)

If your project stores tenant keys differently (e.g., in a SQL database, environment variables, or a different service), you can implement a custom ConfigProvider.

import { LLMService, ConfigManager, BaseConfigProvider } from '@contentgrowth/llm-service';

// 1. Define your custom provider
class MyDatabaseConfigProvider extends BaseConfigProvider {
  async getConfig(tenantId, env) {
    // Fetch config from your database or other source
    // You can use 'env' here if you need access to bindings
    const apiKey = await getApiKeyFromDB(tenantId);
    
    return {
      provider: 'openai', // or 'gemini'
      apiKey: apiKey,
      models: { 
        default: 'gpt-4o',
        // ... optional overrides
      },
      // Optional capabilities
      capabilities: { chat: true, image: true }
    };
  }
}

// 2. Register the provider at application startup
ConfigManager.setConfigProvider(new MyDatabaseConfigProvider());

// 3. Use LLMService as normal - it will now use your provider
const service = new LLMService(env);

Google Gemini & Vertex AI Configuration

The service provides unified support for Google's AI platforms through the @google/genai SDK:

1. Gemini AI Studio (Consumer API)

Uses your AI Studio API key directly.

LLM_PROVIDER=gemini
GEMINI_API_KEY=your-api-key-from-ai-studio
GEMINI_MODEL=gemini-3.5-flash

2. Vertex AI (Enterprise) — Two Authentication Modes

| Mode | Use Case | Required Config | |------|----------|-----------------| | ADC / IAM (Default) | Production, service accounts, Workload Identity | VERTEX_PROJECT, VERTEX_LOCATION | | Express Mode (API Key) | Quick setup, dev/test, no service account | VERTEX_API_KEY (project auto-resolved) |

Vertex Express Mode (API Key) — Recommended for quick setup:

LLM_PROVIDER=vertex
VERTEX_API_KEY=AIza...   # API key from Vertex AI Studio
# VERTEX_PROJECT and VERTEX_LOCATION are OPTIONAL — SDK resolves project from key

Vertex ADC/IAM Mode — For production with service accounts:

LLM_PROVIDER=vertex
VERTEX_PROJECT=your-gcp-project-id
VERTEX_LOCATION=us-central1
# VERTEX_API_KEY is NOT needed (uses Application Default Credentials)

Legacy aliases (also supported):

• GOOGLE_CLOUD_PROJECT → VERTEX_PROJECT
• GOOGLE_CLOUD_LOCATION → VERTEX_LOCATION

Model Overrides (All Providers)

VERTEX_MODEL=gemini-3.5-flash
VERTEX_MODEL_EDGE=gemini-3.5-pro
VERTEX_MODEL_FAST=gemini-3.5-flash
VERTEX_IMAGE_MODEL=gemini-nano-banana-2.1
VERTEX_VIDEO_MODEL=veo

Programmatic Configuration (No Environment Variables)

You can configure the provider directly via code by implementing a custom ConfigProvider. This is useful when you don't want to rely on environment variables or need dynamic configuration.

Vertex Express Mode (API Key) — Programmatic

import { LLMService, ConfigManager, BaseConfigProvider } from '@contentgrowth/llm-service';

class VertexExpressConfigProvider extends BaseConfigProvider {
  constructor(apiKey) {
    super();
    this.apiKey = apiKey;
  }

  async getConfig(tenantId, env) {
    return {
      provider: 'vertex',
      apiKey: this.apiKey,
      // project & location are OPTIONAL — SDK resolves from API key
      models: {
        default: 'gemini-3.5-flash',
        edge: 'gemini-3.5-pro',
        fast: 'gemini-3.5-flash',
        image: 'gemini-3.5-pro-image-preview',
        video: 'veo',
      },
      temperature: 0.7,
      maxTokens: 65536,
      capabilities: { chat: true, image: true, video: true },
    };
  }
}

// Register once at startup
ConfigManager.setConfigProvider(new VertexExpressConfigProvider('AIza...'));

// Use normally
const service = new LLMService(env);
const response = await service.chat('Hello Vertex!', 'tenant-id');

Vertex ADC/IAM Mode — Programmatic

class VertexADCConfigProvider extends BaseConfigProvider {
  constructor(project, location = 'us-central1') {
    super();
    this.project = project;
    this.location = location;
  }

  async getConfig(tenantId, env) {
    return {
      provider: 'vertex',
      project: this.project,
      location: this.location,
      // apiKey NOT needed — uses Application Default Credentials
      models: {
        default: 'gemini-3.5-flash',
        edge: 'gemini-3.5-pro',
        fast: 'gemini-3.5-flash',
        image: 'gemini-3.5-pro-image-preview',
        video: 'veo',
      },
      temperature: 0.7,
      maxTokens: 65536,
      capabilities: { chat: true, image: true, video: true },
    };
  }
}

ConfigManager.setConfigProvider(new VertexADCConfigProvider('my-gcp-project', 'us-central1'));

Gemini AI Studio — Programmatic

class GeminiStudioConfigProvider extends BaseConfigProvider {
  constructor(apiKey) {
    super();
    this.apiKey = apiKey;
  }

  async getConfig(tenantId, env) {
    return {
      provider: 'gemini',
      apiKey: this.apiKey,
      models: {
        default: 'gemini-3.5-flash',
        edge: 'gemini-3.5-pro',
        fast: 'gemini-3.5-flash',
        image: 'gemini-3.5-pro-image-preview',
        video: 'veo',
      },
      temperature: 0.7,
      maxTokens: 65536,
      capabilities: { chat: true, image: true, video: true },
    };
  }
}

ConfigManager.setConfigProvider(new GeminiStudioConfigProvider('your-gemini-api-key'));

Custom & OpenAI-Compatible Providers

Many third-party LLM proxies and gateways (LiteLLM, OpenRouter, custom corporate proxies) expose an OpenAI-compatible API. Because the OpenAI provider in this library accepts a baseURL override, you can point it at any of these services and use any model they expose — including Gemini, Claude, or your own fine-tuned models — without writing a custom provider.

How it works

Set LLM_PROVIDER=openai, point OPENAI_BASE_URL at your proxy, and set OPENAI_MODEL to whatever model name your proxy advertises. The library sends standard OpenAI-format requests; your proxy handles the translation.

Example: Gemini via an OpenAI-compatible proxy

LLM_PROVIDER=openai
OPENAI_API_KEY=your-proxy-api-key
OPENAI_BASE_URL=https://your-proxy.example.com/v1
OPENAI_MODEL=gemini-3.5-flash

No code changes needed. All features — chat, tool calling, JSON mode — work through the proxy using the OpenAI wire format.

Example: Programmatic configuration

import { LLMService, ConfigManager, BaseConfigProvider } from '@contentgrowth/llm-service';

class ProxyConfigProvider extends BaseConfigProvider {
  async getConfig(tenantId, env) {
    return {
      provider: 'openai',
      apiKey: 'your-proxy-api-key',
      baseURL: 'https://your-proxy.example.com/v1',
      models: {
        default: 'gemini-3.5-flash',
        fast: 'gemini-3.5-flash',
        edge: 'gemini-3.5-pro',
      },
      temperature: 0.7,
      maxTokens: 65536,
    };
  }
}

ConfigManager.setConfigProvider(new ProxyConfigProvider());
const service = new LLMService(env);

BYOK tenants via proxy

For multi-tenant setups, tenants can supply their own proxy endpoint alongside their API key:

{
  "provider": "openai",
  "api_key": "tenant-proxy-key",
  "baseURL": "https://tenant-proxy.example.com/v1"
}

baseURL support across all providers

Every provider respects a baseURL override, both from environment variables and from the tenant config object:

| Provider | Env var | Shared fallback | |----------|---------|-----------------| | openai | OPENAI_BASE_URL | LLM_BASE_URL | | gemini | GEMINI_BASE_URL | LLM_BASE_URL | | anthropic | ANTHROPIC_BASE_URL | LLM_BASE_URL | | vertex | VERTEX_BASE_URL | LLM_BASE_URL |

LLM_BASE_URL applies to whichever provider is active, useful when routing all traffic through a single proxy regardless of provider.

Caveat: JSON schema mode

If your proxy doesn't natively support OpenAI's response_format: { type: 'json_schema' } for the model it's proxying, strict schema mode may fall back to json_object or return an error. Proxies like LiteLLM handle this by translating to the target model's native format, but not all do. Test with a chatCompletionJson call first to confirm schema mode works end-to-end.

JSON Mode & Structured Outputs

The service supports native JSON mode for OpenAI and Gemini, guaranteeing valid JSON responses without escaping issues.

Basic JSON Mode

const response = await llmService.chatCompletion(
  messages,
  tenantId,
  'You are a helpful assistant. Always respond in JSON.',
  { responseFormat: 'json' }  // ← Enable JSON mode
);

// Response includes auto-parsed JSON
console.log(response.parsedContent); // Already parsed object
console.log(response.content);       // Raw JSON string

JSON Mode with Schema Validation (Structured Outputs)

Define a schema to guarantee the response structure:

const schema = {
  type: 'object',
  properties: {
    answer: { type: 'string' },
    confidence: { type: 'number' },
    sources: { 
      type: 'array', 
      items: { type: 'string' },
      nullable: true 
    }
  },
  required: ['answer', 'confidence']
};

const response = await llmService.chatCompletion(
  messages,
  tenantId,
  systemPrompt,
  {
    responseFormat: 'json_schema',
    responseSchema: schema,
    schemaName: 'question_answer'
  }
);

// Guaranteed to match schema
const { answer, confidence, sources } = response.parsedContent;

Convenience Method

For JSON-only responses, use chatCompletionJson() to get parsed objects directly:

// Returns parsed object directly (not response wrapper)
const data = await llmService.chatCompletionJson(
  messages,
  tenantId,
  systemPrompt,
  schema  // optional
);

console.log(data.answer);      // Direct access to fields
console.log(data.confidence);  // No .parsedContent needed

Flexible Call Signatures

The chatCompletion() method intelligently detects whether you're passing tools, options, or both:

// All these work!
await chatCompletion(messages, tenant, prompt);
await chatCompletion(messages, tenant, prompt, tools);
await chatCompletion(messages, tenant, prompt, { responseFormat: 'json' });
await chatCompletion(messages, tenant, prompt, tools, { responseFormat: 'json' });

Supported Options

• responseFormat: 'text' (default), 'json', or 'json_schema'
• responseSchema: JSON schema object (required for json_schema mode)
• schemaName: Name for the schema (optional, for json_schema mode)
• strictSchema: Enforce strict validation (default: true)
• autoParse: Auto-parse JSON responses (default: true)
• temperature: Override temperature
• maxTokens: Override max tokens
• tier: Model tier ('default', 'fast', 'smart')

Testing

Google Cloud Authentication Utilities

The package exports utilities for working with Google Cloud service accounts in edge runtime environments (Cloudflare Workers, Vercel Edge, etc.) where traditional ADC (Application Default Credentials) is not available.

import { 
  parseGoogleCredentials, 
  GoogleTokenManager,
  getGoogleOAuth2Token 
} from '@contentgrowth/llm-service/utils/google';

// Parse service account from various formats
const sa = parseGoogleCredentials(jsonString); // or base64 or object

// Create token manager for automatic refresh
const tokenManager = new GoogleTokenManager(sa);

// Get token (cached or fresh)
const token = await tokenManager.getToken();

// Check if token is still valid
if (tokenManager.isValid()) {
  console.log(`Token valid for ${tokenManager.getTimeToExpiry()} seconds`);
}

// Force refresh
tokenManager.invalidate();

// Or get token directly (no caching)
const freshToken = await getGoogleOAuth2Token(sa);

Using with llm-service

import { LLMService } from '@contentgrowth/llm-service';
import { parseGoogleCredentials, GoogleTokenManager } from '@contentgrowth/llm-service/utils/google';

// Backend manages token lifecycle
const sa = parseGoogleCredentials(serviceAccountJson);
const tokenManager = new GoogleTokenManager(sa);
const token = await tokenManager.getToken();

// Pass pre-generated token to llm-service
const llmService = new LLMService(env, {
  provider: 'vertex',
  accessToken: token,  // Pre-generated token
  project: sa.project_id,
  location: 'us-central1',
  models: {
    default: 'gemini-3.8-flash',
    image: 'gemini-nano-banana-2.1'
  }
});

// Use normally - llm-service will use the token
const response = await llmService.chat('Hello!', 'tenant-id');

Benefits of this approach:

• Backend controls token lifecycle and caching
• Service account credentials stay in backend
• llm-service receives only the token
• Works in edge runtime environments

Running JSON Mode Tests

1. Create .env file (copy from .env.example):

   cp .env.example .env

2. Add your API keys to .env:

   LLM_PROVIDER=openai  # or gemini
   OPENAI_API_KEY=sk-your-key-here
   GEMINI_API_KEY=your-gemini-key-here

3. Run tests:

   npm run test:json      # Run comprehensive test suite
   npm run examples:json  # Run interactive examples

See TESTING.md for detailed testing documentation.

Publishing

To publish this package to NPM:

1. Update Version: Update the version in package.json.

2. Login to NPM:

   npm login

3. Publish:

   # For public access
   npm publish --access public

Development

Directory Structure

• src/llm-service.js: Main service class.
• src/llm/config-manager.js: Configuration resolution logic.
• src/llm/config-provider.js: Abstract provider interfaces.
• src/llm/providers/: Individual LLM provider implementations.

Testing

Run the local test script to verify imports and configuration:

node test-custom-config.js