Phantasy Live2D SDK

Reusable Live2D SDK for web apps that need:

• Cubism 4 model loading
• pluggable TTS providers
• real-time lip sync
• PIXI filter access
• a React hook on top of the core class

Package

npm install @phantasy/live2d-sdk

React users can import either from the package root or the explicit subpath:

import { usePhantasyLive2D } from '@phantasy/live2d-sdk';
// or
import { usePhantasyLive2D } from '@phantasy/live2d-sdk/react';

Runtime Requirements

By default, the SDK can auto-load the pinned Live2D browser runtime for you. The current compatibility stack is:

1. Cubism Core
2. [email protected]
3. pixi-live2d-display Cubism 4 build

Detailed setup and troubleshooting live in LIVE2D_SETUP.md.

Quick Start

Vanilla / Framework-Agnostic

import { PhantasyLive2D } from '@phantasy/live2d-sdk';

const phantasy = new PhantasyLive2D({
  container: '#live2d-container',
  model: {
    path: 'https://raw.githubusercontent.com/Live2D/CubismWebSamples/develop/Samples/Resources/Haru/Haru.model3.json',
    scale: 0.3,
    position: { x: 0.5, y: 0.7 },
    expressions: {
      F01: 'https://raw.githubusercontent.com/Live2D/CubismWebSamples/develop/Samples/Resources/Haru/expressions/F01.exp3.json',
      F02: 'https://raw.githubusercontent.com/Live2D/CubismWebSamples/develop/Samples/Resources/Haru/expressions/F02.exp3.json',
      F03: 'https://raw.githubusercontent.com/Live2D/CubismWebSamples/develop/Samples/Resources/Haru/expressions/F03.exp3.json',
      F04: 'https://raw.githubusercontent.com/Live2D/CubismWebSamples/develop/Samples/Resources/Haru/expressions/F04.exp3.json',
    },
  },
  runtime: {
    autoLoad: true,
    cubismCoreUrls: [
      '/live2d-sdk/live2dcubismcore.min.js',
      'https://cubism.live2d.com/sdk-web/cubismcore/live2dcubismcore.min.js',
    ],
    pixiUrls: [
      '/vendor/pixi-6.5.2.min.js',
      'https://cdn.jsdelivr.net/npm/[email protected]/dist/browser/pixi.min.js',
    ],
    pluginUrls: [
      '/pixi-live2d-display-cubism4.min.js',
      'https://cdn.jsdelivr.net/npm/pixi-live2d-display/dist/cubism4.min.js',
    ],
  },
  tts: {
    primary: {
      provider: 'agent',
      apiUrl: '/api/tts',
    },
    timeout: 30000,
  },
  lipSync: {
    enabled: true,
    sensitivity: 0.8,
    smoothing: 0.3,
    amplification: 1.3,
    parameterNames: ['ParamMouthOpenY'],
  },
  features: {
    breathing: true,
    mouseTracking: false,
    orbitControls: {
      enabled: true,
    },
  },
});

await phantasy.initialize();
await phantasy.speak('Hello from Phantasy.');
phantasy.setExpression('F01');

React

import { usePhantasyLive2D } from '@phantasy/live2d-sdk/react';

export function Character() {
  const {
    phantasy,
    speak,
    setExpression,
    stopSpeech,
    isInitialized,
    isLoading,
    error,
  } = usePhantasyLive2D({
    containerId: 'live2d-container',
    model: {
      path: 'https://raw.githubusercontent.com/Live2D/CubismWebSamples/develop/Samples/Resources/Haru/Haru.model3.json',
      scale: 0.3,
      position: { x: 0.5, y: 0.7 },
      expressions: {
        F01: 'https://raw.githubusercontent.com/Live2D/CubismWebSamples/develop/Samples/Resources/Haru/expressions/F01.exp3.json',
        F02: 'https://raw.githubusercontent.com/Live2D/CubismWebSamples/develop/Samples/Resources/Haru/expressions/F02.exp3.json',
      },
    },
    runtime: {
      autoLoad: true,
    },
    tts: {
      primary: {
        provider: 'agent',
        apiUrl: '/api/tts',
      },
      timeout: 30000,
    },
  });

  if (error) return <p>{error}</p>;

  return (
    <>
      <div id="live2d-container" style={{ width: 480, height: 640 }} />
      <button onClick={() => speak('Hello!')} disabled={!isInitialized || isLoading}>
        Speak
      </button>
      <button onClick={() => setExpression('F01')} disabled={!phantasy}>
        F01
      </button>
      <button onClick={stopSpeech} disabled={!phantasy}>
        Stop
      </button>
    </>
  );
}

The examples in this repository use Live2D's public Haru sample model so they can run without Rally-owned assets. Speech still requires a working TTS endpoint.

Public API

PhantasyLive2D

Core lifecycle:

• initialize()
• destroy()
• getStatus()
• getModel()
• getPixiApp()

Speech and lip sync:

• speak(text)
• stopSpeech()

Expressions:

• setExpression(expression)
• setExpressionStack(expressions)
• getAvailableExpressions()

Model transforms:

• setPosition(x, y)
• setScale(scale)

PIXI filter hooks:

• setModelFilters(filters)
• clearModelFilters()
• setStageFilters(filters)
• clearStageFilters()

Orbit controls:

• enableOrbitControls()
• disableOrbitControls()
• setZoom(zoom)
• setPan(x, y)
• resetOrbitControls()
• getOrbitControlsState()

Events:

• initialized
• destroyed
• modelLoaded
• expressionChanged
• speechStarted
• speechEnded
• speechError
• lipSyncData
• lipSyncStarted
• lipSyncEnded
• error
• warning

Event helpers come from the built-in emitter:

• on(event, callback)
• off(event, callback)
• once(event, callback)

usePhantasyLive2D

The React hook returns:

• phantasy
• isLoading
• isInitialized
• error
• speak(text)
• setExpression(expression)
• stopSpeech()
• initialize()
• destroy()

Configuration

type PhantasyLive2DConfig = {
  container: string | HTMLElement;
  model: string | {
    path: string;
    scale?: number;
    position?: { x: number; y: number };
    expressions?: Record<string, string>;
    responsive?: {
      mobile?: Partial<Live2DModelConfig>;
      tablet?: Partial<Live2DModelConfig>;
      desktop?: Partial<Live2DModelConfig>;
    };
  };
  tts?: {
    primary: TTSProviderConfig;
    fallback?: TTSProviderConfig;
    timeout: number;
  };
  lipSync?: {
    enabled: boolean;
    sensitivity: number;
    smoothing: number;
    amplification: number;
    parameterNames: string[];
  };
  features?: {
    mouseTracking?: boolean;
    breathing?: boolean;
    autoExpressions?: boolean;
    responsivePositioning?: boolean;
    blinking?: boolean;
    idleMotion?: boolean;
    speakingMotion?: boolean;
    orbitControls?: {
      enabled?: boolean;
      enableZoom?: boolean;
      enablePan?: boolean;
      zoomSpeed?: number;
      panSpeed?: number;
      minZoom?: number;
      maxZoom?: number;
    };
  };
  runtime?: {
    autoLoad?: boolean;
    verifyTimeoutMs?: number;
    exposeGlobals?: boolean;
    cubismCoreUrls?: string[];
    pixiUrls?: string[];
    pluginUrls?: string[];
  };
  debug?: boolean;
  onReady?: () => void;
  onError?: (error: Error) => void;
};

TTS Providers

Supported providers:

• agent
• alkahest
• elevenlabs
• gptsovits
• animalese

speak() only works if your chosen provider is reachable and configured correctly.

The SDK defaults to the agent provider with a relative /audio/speech endpoint, so local app integrations do not hardcode a third-party SaaS URL by default.

Examples

Vanilla example

Files live in examples/vanilla.

Run after building the SDK:

npm install
npm run build

# serve tmp/sdk-live2d from any static server, then open /examples/vanilla/

React example

A runnable Vite example lives in examples/react.

cd examples/react
npm install
npm run dev

Development

npm install
npm run type-check
npm run test:run
npm run build
npm run lint

Repository

• Issues: https://github.com/phantasy-bot/sdk-live2d/issues
• Source: https://github.com/phantasy-bot/sdk-live2d

License

MIT