chatbotaurus-nim-container-manager

NVIDIA NIM Container Manager for Chatbotaurus - Docker container management for NVIDIA NIM models.

Overview

This package provides Docker container management capabilities for NVIDIA NIM (NVIDIA Inference Microservices) models within Chatbotaurus. It replaces the external chatbotaurus-nim-container-manager dependency with a custom implementation designed specifically for Chatbotaurus infrastructure.

Features

• Docker Integration: Full Docker API integration via dockerode
• NVIDIA Runtime Support: Automatic detection and configuration of NVIDIA Docker runtime
• Container Lifecycle Management: Start, stop, and monitor NIM containers
• Image Management: Pull and manage NVIDIA NGC container images
• Event-Driven Architecture: Real-time events for container operations
• Authentication: NGC API key management for private repositories
• Resource Management: Memory and GPU resource configuration

Prerequisites

• Docker Engine with NVIDIA Container Toolkit
• NVIDIA GPU drivers
• NVIDIA NGC API key (for private models)

Installation

This package is part of the Chatbotaurus monorepo and is automatically installed as a workspace dependency.

pnpm install chatbotaurus-nim-container-manager

Usage

Basic Usage

import { NimContainerManager } from 'chatbotaurus-nim-container-manager'

// Get singleton instance
const nimManager = NimContainerManager.getInstance()

// Listen for events
nimManager.on('preload', (message) => console.log('Preload:', message))
nimManager.on('error', (error) => console.error('Error:', error))

// Initialize the environment
await nimManager.preload()

// Pull a NIM model image
await nimManager.pullImage('nvcr.io/nim/meta/llama3-8b-instruct:1.0.0', 'your-ngc-api-key')

// Start a container
const container = await nimManager.startContainer(
    'nvcr.io/nim/meta/llama3-8b-instruct:1.0.0',
    'your-ngc-api-key',
    8080, // host port
    1 // relax memory constraints
)

Static Methods (Compatibility)

import nimManager from 'chatbotaurus-nim-container-manager'

// Use static methods for compatibility with existing code
await nimManager.preload()
await nimManager.pullImage(imageTag, apiKey)
const container = await nimManager.startContainer(imageTag, apiKey, port)
const images = await nimManager.userImageLibrary()
const containers = await nimManager.listRunningContainers()
await nimManager.stopContainer(containerId)

API Reference

NimContainerManager Class

Methods

preload()

Initialize and verify the Docker environment with NVIDIA runtime support.

downloadInstaller()

Download NIM installer (placeholder implementation).

pullImage(imageTag, apiKey)

Pull a Docker image from NVIDIA NGC registry.

• imageTag: Full image tag (e.g., 'nvcr.io/nim/meta/llama3-8b-instruct:1.0.0')
• apiKey: NVIDIA NGC API key

startContainer(imageTag, apiKey, hostPort, nimRelaxMemConstraints?)

Start a new NIM container.

• imageTag: Docker image tag
• apiKey: NVIDIA NGC API key
• hostPort: Host port to bind to container port 8000
• nimRelaxMemConstraints: Memory constraint relaxation (optional, default: 0)

userImageLibrary()

Get list of available NIM images on the system.

listRunningContainers()

Get list of currently running NIM containers.

stopContainer(containerId)

Stop and remove a running container.

• containerId: Docker container ID

Events

The NimContainerManager extends EventEmitter and emits the following events:

• preload: Environment initialization messages
• download: Download progress messages
• pull: Image pull status and progress
• pull-progress: Detailed pull progress events
• start: Container start messages
• stop: Container stop messages
• error: Error messages

Example Event Handling

const nimManager = NimContainerManager.getInstance()

nimManager.on('pull-progress', (event) => {
    console.log(`Pull progress: ${event.status}`)
})

nimManager.on('error', (error) => {
    console.error('NIM Manager Error:', error)
})

Configuration

Environment Variables

• NGC_API_KEY: NVIDIA NGC API key for authentication
• NIM_RELAX_MEM_CONSTRAINTS: Memory constraint relaxation setting

Docker Requirements

The system must have:

• Docker Engine installed and running
• NVIDIA Container Toolkit installed
• NVIDIA runtime configured in Docker daemon

Troubleshooting

Common Issues

1. "NVIDIA Docker runtime not found"

   • Install nvidia-container-toolkit
   • Restart Docker daemon
   • Verify with: docker info | grep nvidia

2. "connect ENOENT //./pipe/docker_engine"

   • Ensure Docker Desktop is running (Windows)
   • Check Docker daemon status (Linux)

3. Authentication errors

   • Verify NGC API key is valid
   • Check network connectivity to nvcr.io

License

This software is provided under the Chatbotaurus Alpha Software License. See LICENSE file for details.

ALPHA VERSION - NOT FOR PRODUCTION USE

Support

For questions and support, contact: [email protected]