ngx-web-llm

Bringing the power of Large Language Models (LLMs) directly to your Angular applications with WebLLM.

ngx-web-llm is an Angular library that wraps the WebLLM JavaScript library, allowing you to run LLM inference entirely in the browser. This enables privacy-preserving, offline-capable AI features with no server-side dependencies for inference.

Current Version: 0.0.1

Features

• Easy Integration: Seamlessly integrate WebLLM into your Angular projects.
• Service-Based: Core functionalities exposed via the WebLlmService.
• Reactive: Uses RxJS Observables for model status, initialization progress, and streaming chat responses.
• Chat Completions: Supports both single-shot and streaming chat completions.
• Model Management: Load, reload, and unload models. Get a list of available prebuilt models.
• Configurable: Configure default models, engine settings, and preloading.
• Caching: Optional configuration for persistent model caching.
• (Optional) Web Worker Support: Can offload model inference to a Web Worker (requires manual setup of the worker script by the application).

Installation

Install the library and its peer dependencies:

npm install ngx-web-llm @mlc-ai/web-llm
# or
yarn add ngx-web-llm @mlc-ai/web-llm

Basic Usage

1. Import and Configure the Module:

   Import NgxWebLlmModule into your Angular application. For standalone applications, use importProvidersFrom. For NgModule-based applications, import into your desired module.

   For Standalone Applications (app.config.ts):

   import { ApplicationConfig, importProvidersFrom } from "@angular/core";
   import { provideRouter } from "@angular/router";
   import { NgxWebLlmModule } from "ngx-web-llm"; // Ensure this path is correct
   import { routes } from "./app.routes";
   
   export const appConfig: ApplicationConfig = {
     providers: [
       provideRouter(routes),
       importProvidersFrom(
         NgxWebLlmModule.forRoot({
           // Optional: Configure default model, preloading, caching etc.
           defaultModelId: "Llama-3.1-8B-Instruct-q4f32_1-MLC", // Example
           preload: true,
           // workerScriptUrl: 'path/to/your/web-llm.worker.js', // Only if using Web Worker
         })
       ),
     ],
   };

   For NgModule-based Applications (app.module.ts or feature module):

   import { NgModule } from "@angular/core";
   import { BrowserModule } from "@angular/platform-browser";
   import { NgxWebLlmModule } from "ngx-web-llm"; // Ensure this path is correct
   import { AppComponent } from "./app.component";
   
   @NgModule({
     declarations: [AppComponent],
     imports: [
       BrowserModule,
       NgxWebLlmModule.forRoot({
         // Optional: Configure default model, preloading, caching etc.
         defaultModelId: "Llama-3.1-8B-Instruct-q4f32_1-MLC", // Example
         preload: true,
         // workerScriptUrl: 'path/to/your/web-llm.worker.js', // Only if using Web Worker
       }),
     ],
     providers: [],
     bootstrap: [AppComponent],
   })
   export class AppModule {}

2. Inject and Use WebLlmService:

   import { Component, OnInit } from "@angular/core";
   import { FormsModule } from "@angular/forms"; // Needed for ngModel
   import { CommonModule } from "@angular/common"; // Needed for async pipe and new control flow
   import { WebLlmService, NgxMessage } from "ngx-web-llm"; // Ensure path is correct
   
   @Component({
     selector: "app-chat",
     standalone: true, // Assuming standalone component for modern Angular
     imports: [CommonModule, FormsModule], // Import necessary modules
     template: `
       <div>
         @if (webLlm.isLoading$ | async; as isLoading) {
         <div>Loading model... {{ (webLlm.initializationProgress$ | async)?.text }}</div>
         } @else if (webLlm.engineReady$ | async) {
         <div>Model Ready: {{ webLlm.currentModelId }}</div>
         } @else {
         <div>Engine not ready and not loading. Please initialize.</div>
         } @for (message of messages; track $index) {
         <div>
           <strong>{{ message.role }}:</strong> {{ message.content }}
         </div>
         } @empty {
         <div>No messages yet. Send one to start the chat!</div>
         }
   
         <input [(ngModel)]="userInput" placeholder="Type your message" (keyup.enter)="sendMessage()" />
         <button (click)="sendMessage()" [disabled]="isGenerating || !(webLlm.engineReady$ | async)">Send</button>
   
         @if (isGenerating) {
         <div>Generating response...</div>
         }
       </div>
     `,
   })
   export class ChatComponent implements OnInit {
     messages: NgxMessage[] = [];
     userInput: string = "";
     isGenerating: boolean = false;
   
     constructor(public webLlm: WebLlmService) {}
   
     async ngOnInit() {
       // Example: Explicit initialization if not relying on preload
       // if (!this.webLlm.isEngineReady && !(await (this.webLlm.isLoading$ | async))) { // Check isLoading correctly
       //   try {
       //     // Initialize with the default model (or specify one)
       //     // Set useWebWorker to false if not using a worker, or true if workerScriptUrl is configured
       //     await this.webLlm.initializeEngine(undefined, undefined, false);
       //     console.log('Engine initialized successfully!');
       //   } catch (e) {
       //     console.error('Engine initialization failed:', e);
       //   }
       // }
     }
   
     sendMessage(): void {
       if (!this.userInput.trim() || !this.webLlm.isEngineReady || this.isGenerating) return;
   
       const userMessage: NgxMessage = { role: "user", content: this.userInput };
       this.messages = [...this.messages, userMessage];
       const currentInput = this.userInput;
       this.userInput = "";
       this.isGenerating = true;
   
       let assistantResponse = "";
       const assistantMessagePlaceholder: NgxMessage = { role: "assistant", content: "..." };
       this.messages = [...this.messages, assistantMessagePlaceholder];
   
       this.webLlm.streamChatDeltas({ messages: [{ role: "user", content: currentInput }] }).subscribe({
         next: (deltaContent) => {
           assistantResponse += deltaContent;
           this.messages = this.messages.map((msg, index) => (index === this.messages.length - 1 ? { ...msg, content: assistantResponse } : msg));
         },
         error: (err) => {
           console.error("Error during chat streaming:", err);
           this.messages = this.messages.map((msg, index) => (index === this.messages.length - 1 ? { ...msg, content: "Error generating response." } : msg));
           this.isGenerating = false;
         },
         complete: () => {
           this.isGenerating = false;
         },
       });
     }
   }

Configuration (NgxWebLlmConfig)

When calling NgxWebLlmModule.forRoot(config), you can provide an NgxWebLlmConfig object:

export interface NgxWebLlmConfig {
  defaultModelId?: string; // e.g., 'Llama-3.1-8B-Instruct-q4f32_1-MLC'
  defaultEngineConfig?: webllm.MLCEngineConfig; // Advanced engine settings from WebLLM
  preload?: boolean; // If true, initializes with defaultModelId on service creation
  workerScriptUrl?: string; // Optional: Path to web-llm.worker.js if choosing to use Web Worker.
  // The application is responsible for serving this worker script.
  caching?: {
    usePersistentStorage?: boolean; // Request persistent storage for models
    storageQuotaBytes?: number; // (Currently informational) Desired quota
  };
}

WebLlmService API Highlights

• Properties & Observables:

  • currentModelId$: Observable<string | null>
  • engineReady$: Observable<boolean>
  • initializationProgress$: Observable<NgxInitProgressReport | null>
  • isLoading$: Observable<boolean>
  • get prebuiltAppConfig(): webllm.AppConfig

• Core Methods:

  • async initializeEngine(modelId?: string, config?: webllm.MLCEngineConfig, useWebWorker?: boolean): Promise<void>: Initializes the WebLLM engine. useWebWorker is false by default; set to true and provide workerScriptUrl in config if using a worker.
  • chatCompletions(request: NgxChatCompletionRequest): Observable<NgxChatCompletionChunk | NgxChatCompletion>
  • streamChatDeltas(request: Omit<NgxChatCompletionRequest, 'stream'>): Observable<string>
  • async reloadModel(modelId: string, chatOpts?: webllm.ChatOptions): Promise<void>
  • async unloadEngine(): Promise<void>
  • getAvailableModels(): NgxModelRecord[]
  • async interruptGeneration(): Promise<void>
  • async getRuntimeStats(): Promise<string>
  • async resetChat(keepSystemPrompt?: boolean): Promise<void>

Troubleshooting

• Ensure @mlc-ai/web-llm is correctly installed as a peer dependency.
• Model downloads can be large; ensure a stable internet connection during first initialization for a model. Check browser console for download progress.

Contributing

Contributions, issues, and feature requests are welcome! Please feel free to check the issues page (Update this link!).

License

This project is licensed under the MIT License. See the LICENSE file for details. (You'll need to create a LICENSE file).