@phantasy/live2d-sdk

v0.2.11

Published

2 months ago

Reusable SDK for Live2D models with TTS and lip sync integration

0High
0Medium
0Low

lightnolimit

live2d tts text-to-speech lip-sync animation sdk phantasy

Phantasy Live2D SDK

Advanced Live2D integration with TTS and lip-sync capabilities for Phantasy AI characters

⚠️ CRITICAL: Live2D Setup Requirements

This SDK requires EXACT dependency versions and loading order to function. See LIVE2D_SETUP.md for complete troubleshooting guide.

Required Dependencies (in exact order):

Cubism 4 Core Runtime

<script src="https://cubism.live2d.com/sdk-web/cubismcore/live2dcubismcore.min.js"></script>

PIXI.js v6.5.2 (exact version required)

<script src="https://cdn.jsdelivr.net/npm/[email protected]/dist/browser/pixi.min.js"></script>

pixi-live2d-display Cubism 4 Plugin (local copy required)

<script src="/pixi-live2d-display-cubism4.min.js"></script>

IMPORTANT: The plugin MUST be the cubism4.min.js build, NOT the generic index.min.js. Download it:

curl -fL https://cdn.jsdelivr.net/npm/pixi-live2d-display/dist/cubism4.min.js \
  -o public/pixi-live2d-display-cubism4.min.js

Common Errors

"Could not find Cubism 2 runtime" - You're using the wrong plugin build. Use cubism4.min.js.
"Unexpected identifier 'version'" - Scripts are in wrong order or missing dependencies.
Model not loading - Check browser console for CORS errors or 404s.

Features

🎭 Live2D Model Management - Load and control Cubism 4.0+ models
🎙️ Multi-Provider TTS - Support for Alkahest, ElevenLabs, and GPT-SoVITS
💋 Real-time Lip Sync - Automatic mouth movement synchronized with speech
📱 Responsive Positioning - Adaptive model scaling for different screen sizes
⚡ Performance Optimized - Built-in caching and resource management
🎨 Expression System - Dynamic facial expressions and animations
🔌 Framework Agnostic - Works with React, Vue, vanilla JS, and more

Installation

# Install the SDK
npm install @rally/live2d-sdk

# Download the required pixi-live2d-display plugin
curl -fL https://cdn.jsdelivr.net/npm/pixi-live2d-display/dist/cubism4.min.js \
  -o public/pixi-live2d-display-cubism4.min.js

Quick Start

HTML Setup (Required First!)

<!DOCTYPE html>
<html>
<head>
  <title>Phantasy Live2D</title>
</head>
<body>
  <div id="live2d-container"></div>

  <!-- EXACT ORDER IS CRITICAL -->
  <!-- 1. Cubism 4 Core -->
  <script src="https://cubism.live2d.com/sdk-web/cubismcore/live2dcubismcore.min.js"></script>
  
  <!-- 2. PIXI.js v6.5.2 -->
  <script src="https://cdn.jsdelivr.net/npm/[email protected]/dist/browser/pixi.min.js"></script>
  
  <!-- 3. Plugin (local copy) -->
  <script src="/pixi-live2d-display-cubism4.min.js"></script>

  <!-- 4. Your app -->
  <script type="module" src="/src/main.js"></script>
</body>
</html>

JavaScript Integration

import { PhantasyLive2D } from '@rally/live2d-sdk';

// Initialize the SDK
const rally = new PhantasyLive2D({
  container: '#live2d-container',
  model: {
    url: 'https://r2.rally.sh/rally/rally.model3.json',
    scale: 0.3,
    position: { x: 0.5, y: 0.7 }
  },
  tts: {
    provider: 'alkahest',
    apiKey: 'your-api-key'
  }
});

// Wait for initialization
await rally.initialize();

// Make Phantasy speak
await rally.speak('Hello! I am Phantasy, your AI companion.');

// Change expression
rally.setExpression('happy');

React Integration

import { usePhantasyLive2D } from '@rally/live2d-sdk/react';

function PhantasyCharacter() {
  const { speak, setExpression, isInitialized, error } = usePhantasyLive2D({
    containerId: 'rally-container',
    model: {
      url: 'https://r2.rally.sh/rally/rally.model3.json'
    },
    tts: {
      provider: 'alkahest',
      apiKey: process.env.REACT_APP_ALKAHEST_API_KEY
    }
  });

  if (error) return <div>Error: {error}</div>;
  if (!isInitialized) return <div>Loading...</div>;

  return (
    <>
      <div id="rally-container" style={{ width: '100%', height: '600px' }} />
      <button onClick={() => speak('Hello!')}>Speak</button>
      <button onClick={() => setExpression('happy')}>Happy</button>
    </>
  );
}

Configuration

const config = {
  // Container element (required)
  container: '#live2d-container',
  
  // Model configuration
  model: {
    url: 'https://r2.rally.sh/rally/rally.model3.json',
    scale: 0.3,
    position: { x: 0.5, y: 0.7 },
    physics: false  // Disable physics to avoid errors
  },
  
  // TTS configuration
  tts: {
    provider: 'alkahest',  // 'alkahest' | 'elevenlabs' | 'gptsovits'
    apiUrl: 'https://api.alkahest.ai',
    apiKey: 'your-api-key',
    voice: 'rally',
    model: 'tts-1',
    speed: 1.0
  },
  
  // Lip sync configuration
  lipSync: {
    enabled: true,
    sensitivity: 0.8,
    smoothing: 0.3,
    amplification: 1.3
  },
  
  // Debug mode
  debug: true
};

API Reference

PhantasyLive2D Class

// Initialize
const rally = new PhantasyLive2D(config);
await rally.initialize();

// TTS
await rally.speak('Hello world');
await rally.stopSpeaking();

// Expressions
rally.setExpression('happy');
rally.getAvailableExpressions();

// Model control
rally.setPosition(x, y);
rally.setScale(scale);

// Events
rally.on('initialized', () => {});
rally.on('speechStarted', (text) => {});
rally.on('speechEnded', () => {});
rally.on('expressionChanged', (expression) => {});
rally.on('error', (error) => {});

// Cleanup
rally.destroy();

React Hook

const {
  rally,           // PhantasyLive2D instance
  speak,           // Function to speak text
  stopSpeaking,    // Function to stop speech
  setExpression,   // Function to change expression
  isInitialized,   // Boolean
  isSpeaking,      // Boolean
  error           // Error object if any
} = usePhantasyLive2D(config);

TTS Providers

Alkahest (Recommended)

tts: {
  provider: 'alkahest',
  apiUrl: 'https://api.alkahest.ai',
  apiKey: 'your-api-key',
  voice: 'rally',
  model: 'tts-1'
}

ElevenLabs

tts: {
  provider: 'elevenlabs',
  apiKey: 'your-api-key',
  voiceId: 'voice-id-here',
  modelId: 'eleven_monolingual_v1'
}

GPT-SoVITS (Local)

tts: {
  provider: 'gptsovits',
  apiUrl: 'http://localhost:9880',
  character: 'rally',
  emotion: 'default'
}

Model Files

Phantasy's model files should be served from a CORS-enabled CDN or local server:

rally/
├── rally.model3.json       # Main model definition
├── rally.moc3              # Compiled model data
├── textures/               # Texture files
│   ├── texture_00.png
│   └── ...
├── motions/                # Animation files
│   ├── idle_01.motion3.json
│   └── ...
├── expressions/            # Expression files
│   ├── happy.exp3.json
│   ├── sad.exp3.json
│   └── ...
└── rally.physics3.json     # Physics (often problematic)

Troubleshooting

Dependencies Not Loading

Check browser Network tab for 404s
Ensure scripts are loaded in exact order
Verify PIXI.js version is exactly 6.5.2
Plugin must be served locally, not from CDN

Model Not Appearing

Check browser console for CORS errors
Verify model URL is accessible
Check container element exists and has dimensions
Try disabling physics in config

No Lip Sync

Check console for parameter warnings
Verify TTS is generating audio successfully
Ensure lipSync.enabled: true in config
Check browser autoplay policies

Performance Issues

Reduce model scale
Disable unused features (physics, mouse tracking)
Use production build of SDK
Check for memory leaks (call destroy() on cleanup)

Development

# Clone the repository
git clone https://github.com/rally-sh/rally.git
cd rally/packages/live2d-sdk

# Install dependencies
bun install

# Development mode
bun run dev

# Run tests
bun run test

# Build for production
bun run build

License

MIT

Support

For issues and questions:

Check LIVE2D_SETUP.md for detailed setup guide
Open an issue on GitHub
Contact the Phantasy team

Remember: The exact dependency versions and loading order are CRITICAL for Live2D to work!