cactus-ai-sdk-provider

Cactus provider for Vercel AI SDK - enables on-device LLM inference in React Native applications.

Installation

npm install cactus-ai-sdk-provider ai

For model download capabilities (optional):

npm install react-native-fs
# iOS only:
cd ios && pod install

React Native Setup

Since AI SDK v5 uses Web Streams API, you need to install a polyfill for React Native:

npm install web-streams-polyfill

Then add this at the very top of your app's entry file (index.js or App.js):

import 'web-streams-polyfill/dist/polyfill';

Usage

Basic Usage (Direct Path)

import { cactus } from 'cactus-ai-sdk-provider';
import { generateText } from 'ai';

// Use a local model file
const model = cactus('/path/to/model.gguf');

// Initialize the model
await model.initialize();

// Generate text
const { text } = await generateText({
  model,
  prompt: 'What is the capital of France?',
});

Model Download (Requires react-native-fs)

import { cactus } from 'cactus-ai-sdk-provider';

// Create model from URL
const model = cactus('https://example.com/llama-2-7b.gguf');

// Download with progress tracking
await model.downloadModel({
  onProgress: (progress) => {
    console.log(`Download progress: ${(progress * 100).toFixed(2)}%`);
  }
});

// Initialize after download
await model.initialize();

// Use with AI SDK
const { text } = await generateText({
  model,
  prompt: 'Explain quantum computing',
});

Streaming

import { streamText } from 'ai';

const { textStream } = await streamText({
  model,
  prompt: 'Write a short story',
});

for await (const chunk of textStream) {
  console.log(chunk);
}

Model Management

// List downloaded models
const models = await model.listDownloadedModels();

// Get storage info
const { totalSize, modelCount } = await model.getStorageInfo();
console.log(`${modelCount} models using ${totalSize} bytes`);

// Delete a model
await model.deleteModel();

API Reference

Provider

import { cactus, createCactus } from 'cactus-ai-sdk-provider';

// Use default provider
const model = cactus(modelUrlOrPath);

// Create custom provider
const customProvider = createCactus({
  generateId: () => 'custom-id'
});

Model Lifecycle

// Model states
enum ModelStatus {
  IDLE,
  DOWNLOADING,
  INITIALIZING,
  READY,
  ERROR
}

// Check status
const status = model.getStatus();
const error = model.getLastError();

Requirements

• React Native >= 0.70.0
• AI SDK (ai) >= 5.0.0
• web-streams-polyfill (for React Native)
• react-native-fs >= 2.20.0 (optional, for downloads)

Platform Notes

• iOS: Models are stored in the app's documents directory
• Android: Models are stored in the app's files directory
• Model Format: Supports GGUF format models

Troubleshooting

TransformStream Error

If you see "Property 'TransformStream' doesn't exist", make sure you've added the web-streams-polyfill as shown in the React Native Setup section.

Model Download Issues

Ensure react-native-fs is properly linked:

• iOS: Run cd ios && pod install
• Android: Should auto-link with React Native 0.60+

License

MIT