@imgly/plugin-ai-apps-web

v0.2.12

Published

17 days ago

AI apps orchestration plugin for the CE.SDK editor

IMG.LY AI Apps for Web

A plugin for orchestrating all AI generation capabilities in CreativeEditor SDK.

Overview

The @imgly/plugin-ai-apps-web package provides a unified interface for accessing and organizing all AI generation features within the CreativeEditor SDK. It combines image, video, audio, and text generation capabilities into a single cohesive user experience.

Key Features

Unified AI Experience: Centralized access to all AI generation capabilities
AI Dock Component: Single entry point for all AI features with loading indicators
Smart Provider Management: Automatically organizes providers based on editor mode
Integrated History: Seamless integration with asset libraries for generated content
Mode-Aware Interface: Different UI layouts for Design and Video modes
Cross-Plugin Support: Works with all IMG.LY AI generation plugins

Installation

npm install @imgly/plugin-ai-apps-web

Basic Usage

To use the plugin, import it and configure it with your preferred providers:

import CreativeEditorSDK from '@cesdk/cesdk-js';
import AiApps from '@imgly/plugin-ai-apps-web';

// Import providers from individual AI generation packages
import { AnthropicProvider } from '@imgly/plugin-ai-text-generation-web/anthropic';
import FalAiImage from '@imgly/plugin-ai-image-generation-web/fal-ai';
import FalAiVideo from '@imgly/plugin-ai-video-generation-web/fal-ai';
import ElevenLabs from '@imgly/plugin-ai-audio-generation-web/elevenlabs';
import FalAiSticker from '@imgly/plugin-ai-sticker-generation-web/fal-ai';

// Initialize CreativeEditor SDK
CreativeEditorSDK.create(domElement, {
  license: 'your-license-key'
  // Other configuration options...
}).then(async (cesdk) => {
  // Add the AI Apps plugin
  cesdk.addPlugin(
    AiApps({
      providers: {
        // Text generation
        text2text: AnthropicProvider({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        }),

        // Image generation
        text2image: FalAiImage.RecraftV3({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        }),
        image2image: FalAiImage.GeminiFlashEdit({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        }),

        // Video generation (used in video mode)
        text2video: FalAiVideo.MinimaxVideo01Live({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        }),
        image2video: FalAiVideo.MinimaxVideo01LiveImageToVideo({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        }),

        // Audio generation (used in video mode)
        text2speech: ElevenLabs.MonolingualV1({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        }),
        text2sound: ElevenLabs.SoundEffects({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        }),

        // Sticker generation
        text2sticker: FalAiSticker.Recraft20b({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        })
      }
    })
  );

  // Position the AI dock button in the dock order
  cesdk.ui.setDockOrder(['ly.img.ai.apps.dock', ...cesdk.ui.getDockOrder()]);
});

Configuration Options

The plugin accepts the following configuration options:

| Option | Type | Description | | ----------- | ----------- | --------------------------------------------- | | providers | Providers | Object containing all AI providers to be used | | debug | boolean | Print debug messages |

Providers Configuration

The providers object can include the following provider functions:

| Provider | Type | Description | | ------------- | ------------------- | -------------------------------------------------------- | | text2text | Provider<'text'> | Provider for text generation and transformation | | text2image | Provider<'image'> | Provider for text-to-image generation | | image2image | Provider<'image'> | Provider for image-to-image transformation | | text2video | Provider<'video'> | Provider for text-to-video generation (video mode only) | | image2video | Provider<'video'> | Provider for image-to-video generation (video mode only) | | text2speech | Provider<'audio'> | Provider for text-to-speech generation (video mode only) | | text2sound | Provider<'audio'> | Provider for sound effects generation (video mode only) | | text2sticker | Provider<'sticker'> | Provider for sticker generation |

Provider Selection Strategy

The plugin intelligently selects which providers to use based on the current editor mode:

Design Mode

Uses: text2text, text2image, image2image, text2sticker
Focus: Image, text, and sticker generation for design workflows
UI: Shows AI apps cards for different generation types

Video Mode

Uses: All providers including text2video, image2video, text2speech, text2sound, text2sticker
Focus: Comprehensive media generation for video production
UI: Shows AI apps cards for different generation types

Advanced Configuration

Multiple Providers per Type

You can configure multiple providers for the same generation type by passing an array. When multiple providers are configured, a selection box will be rendered in the AI app interface allowing users to choose between different providers:

import FalAiImage from '@imgly/plugin-ai-image-generation-web/fal-ai';
import OpenAiImage from '@imgly/plugin-ai-image-generation-web/open-ai';

cesdk.addPlugin(
  AiApps({
    providers: {
      // Multiple image providers - selection box will be shown
      text2image: [
        FalAiImage.RecraftV3({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        }),
        FalAiImage.Recraft20b({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        }),
        OpenAiImage.GptImage1.Text2Image({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        })
      ],
      
      // Other providers...
    }
  })
);

Custom Headers and Configuration

Pass custom headers and configuration to providers:

cesdk.addPlugin(
  AiApps({
    providers: {
      text2image: FalAiImage.RecraftV3({
        proxyUrl: 'http://your-proxy-server.com/api/proxy',
        headers: {
          'x-client-version': '1.0.0',
          'x-request-source': 'cesdk-plugin',
          'x-user-id': 'user-123'
        },
        debug: true
      })
    },
    debug: true
  })
);

Property Configuration

Providers support configuring default values for their properties. These defaults can be static or dynamic based on context:

cesdk.addPlugin(
  AiApps({
    providers: {
      text2image: FalAiImage.RecraftV3({
        proxyUrl: 'http://your-proxy-server.com/api/proxy',
        properties: {
          // Static default
          image_size: { default: 'square_hd' },

          // Dynamic default based on locale
          style: {
            default: (context) => {
              return context.locale === 'ja' ? 'anime' : 'realistic';
            }
          }
        }
      })
    }
  })
);

UI Integration

The plugin adds the following UI components to CreativeEditor SDK:

AI Dock Button

The main entry point for all AI features, accessible from the dock:

ID: ly.img.ai.apps.dock
Functionality: Opens AI generation interface
Loading States: Shows progress indicators during generation
Mode Awareness: Adapts interface based on current editor mode

AI Apps Menu

The plugin shows a card-based interface for different AI capabilities:

Generate Image: Access to image generation providers
Generate Video: Access to video generation providers
Generate Audio: Access to audio generation providers
Edit Text: Access to text generation providers

Dock Integration

To position the AI dock button in your editor's dock, use the setDockOrder method:

// Add the AI dock component to the beginning of the dock
cesdk.ui.setDockOrder(['ly.img.ai.apps.dock', ...cesdk.ui.getDockOrder()]);

// Or add it at a specific position
const currentOrder = cesdk.ui.getDockOrder();
currentOrder.splice(2, 0, 'ly.img.ai.apps.dock');
cesdk.ui.setDockOrder(currentOrder);

Asset History Integration

The plugin automatically integrates generated assets into the appropriate asset libraries. For each generation type, both a history asset source and a corresponding asset library entry are automatically created with the same ID:

Image Generation History: ly.img.ai.image-generation.history
Video Generation History: ly.img.ai.video-generation.history
Audio Generation History: ly.img.ai.audio-generation.history
Sticker Generation History: ly.img.ai.sticker-generation.history

These asset library entries are automatically configured with:

Sorted by insertion date (newest first)
Square grid item height
Cover background type
Removable items

Integrating with Default Asset Libraries

To add these history sources to CE.SDK's default asset library entries, use the following approach:

// Add AI image history to the default image asset library
const imageEntry = cesdk.ui.getAssetLibraryEntry('ly.img.image');
if (imageEntry != null) {
  cesdk.ui.updateAssetLibraryEntry('ly.img.image', {
    sourceIds: [...imageEntry.sourceIds, 'ly.img.ai.image-generation.history']
  });
}

// Add AI video history to the default video asset library
const videoEntry = cesdk.ui.getAssetLibraryEntry('ly.img.video');
if (videoEntry != null) {
  cesdk.ui.updateAssetLibraryEntry('ly.img.video', {
    sourceIds: [...videoEntry.sourceIds, 'ly.img.ai.video-generation.history']
  });
}

// Add AI audio history to the default audio asset library
const audioEntry = cesdk.ui.getAssetLibraryEntry('ly.img.audio');
if (audioEntry != null) {
  cesdk.ui.updateAssetLibraryEntry('ly.img.audio', {
    sourceIds: [...audioEntry.sourceIds, 'ly.img.ai.audio-generation.history']
  });
}

// Add AI sticker history to the default sticker asset library
const stickerEntry = instance.ui.getAssetLibraryEntry('ly.img.sticker');
if (stickerEntry != null) {
  instance.ui.updateAssetLibraryEntry('ly.img.sticker', {
    sourceIds: [...stickerEntry.sourceIds, 'ly.img.ai.sticker-generation.history']
  });
}

This integration creates a seamless experience where users can easily find and reuse their AI-generated assets alongside other content.

Quick Actions Integration

The plugin automatically integrates with the quick actions system, providing context-sensitive AI operations directly in the canvas menu. You need to specify the children with the quick action order:

// Quick actions are automatically registered and will appear in canvas menus
cesdk.ui.setCanvasMenuOrder([
  {
    id: 'ly.img.ai.text.canvasMenu',
    children: [
      'ly.img.improve',
      'ly.img.fix',
      'ly.img.shorter',
      'ly.img.longer',
      'ly.img.separator',
      'ly.img.changeTone',
      'ly.img.translate',
      'ly.img.separator',
      'ly.img.changeTextTo'
    ]
  },
  {
    id: 'ly.img.ai.image.canvasMenu',
    children: [
      'ly.img.styleTransfer',
      'ly.img.artistTransfer',
      'ly.img.separator',
      'ly.img.editImage',
      'ly.img.swapBackground',
      'ly.img.createVariant',
      'ly.img.combineImages',
      'ly.img.separator',
      'ly.img.remixPage',
      'ly.img.separator',
      'ly.img.createVideo'
    ]
  },
  ...cesdk.ui.getCanvasMenuOrder()
]);

Quick actions provide:

Context-aware operations: Work with currently selected blocks
One-click transformations: Apply AI operations without opening panels
Cross-plugin functionality: Actions from one plugin can work with providers from another

Error Handling and Debugging

Debug Mode

Enable debug mode to get detailed logging information:

cesdk.addPlugin(
  AiApps({
    providers: {
      text2image: FalAiImage.RecraftV3({
        proxyUrl: 'http://your-proxy-server.com/api/proxy',
        debug: true // Provider-level debugging
      })
    },
    debug: true // Plugin-level debugging
  })
);

Common Issues and Solutions

Provider Not Loading

Check proxy URLs: Ensure all proxy URLs are correctly configured and accessible
Verify licenses: Make sure your CreativeEditor SDK license includes AI features
Check browser console: Look for network errors or API issues

Quick Actions Not Appearing

Verify canvas menu order: Ensure quick action menus are added to canvas menu order with proper children configuration
Check provider support: Verify that providers declare support for the quick actions
Scope permissions: Ensure blocks have the required scopes for quick actions

Generation Failures

API connectivity: Check that your proxy endpoints are working
Rate limiting: Verify you're not exceeding API rate limits
Input validation: Ensure inputs meet provider requirements

Plugin Architecture

How It Works

The AI Apps plugin acts as an orchestrator that:

Initializes Providers: Sets up all configured AI providers
Manages UI: Creates appropriate interface based on editor mode
Coordinates Actions: Integrates quick actions across different plugins
Handles Assets: Manages generated content and asset library integration

Provider Lifecycle

Registration: Providers are registered with the global ProviderRegistry
Initialization: Provider initialize methods are called
UI Setup: Panels and quick actions are registered
Event Handling: The plugin coordinates between providers and UI

TypeScript Support

The plugin is fully typed with TypeScript, providing excellent development experience:

import AiApps, { Providers } from '@imgly/plugin-ai-apps-web';

// Strongly typed provider configuration
const providers: Providers = {
  text2image: FalAiImage.RecraftV3({
    proxyUrl: 'http://your-proxy-server.com/api/proxy'
  }),
  // TypeScript will enforce correct provider types
};

cesdk.addPlugin(AiApps({ providers }));

Related Packages

This plugin works with the following IMG.LY AI generation packages:

@imgly/plugin-ai-generation-web - Core utilities for AI generation
@imgly/plugin-ai-image-generation-web - AI image generation
@imgly/plugin-ai-video-generation-web - AI video generation
@imgly/plugin-ai-audio-generation-web - AI audio generation
@imgly/plugin-ai-text-generation-web - AI text generation

Examples

Simple Setup for Design Mode

import CreativeEditorSDK from '@cesdk/cesdk-js';
import AiApps from '@imgly/plugin-ai-apps-web';
import FalAiImage from '@imgly/plugin-ai-image-generation-web/fal-ai';

CreativeEditorSDK.create(domElement, {
  license: 'your-license-key'
}).then(async (cesdk) => {
  cesdk.addPlugin(
    AiApps({
      providers: {
        text2image: FalAiImage.RecraftV3({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        })
      }
    })
  );

  cesdk.ui.setDockOrder(['ly.img.ai.apps.dock', ...cesdk.ui.getDockOrder()]);
});

Complete Setup for Video Mode

import CreativeEditorSDK from '@cesdk/cesdk-js';
import AiApps from '@imgly/plugin-ai-apps-web';
import FalAiImage from '@imgly/plugin-ai-image-generation-web/fal-ai';
import FalAiVideo from '@imgly/plugin-ai-video-generation-web/fal-ai';
import ElevenLabs from '@imgly/plugin-ai-audio-generation-web/elevenlabs';
import { AnthropicProvider } from '@imgly/plugin-ai-text-generation-web/anthropic';

CreativeEditorSDK.create(domElement, {
  license: 'your-license-key',
  ui: {
    elements: {
      panels: {
        settings: true
      }
    }
  }
}).then(async (cesdk) => {
  cesdk.addPlugin(
    AiApps({
      providers: {
        text2text: AnthropicProvider({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        }),
        text2image: FalAiImage.RecraftV3({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        }),
        image2image: FalAiImage.GeminiFlashEdit({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        }),
        text2video: FalAiVideo.MinimaxVideo01Live({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        }),
        image2video: FalAiVideo.MinimaxVideo01LiveImageToVideo({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        }),
        text2speech: ElevenLabs.MonolingualV1({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        }),
        text2sound: ElevenLabs.SoundEffects({
          proxyUrl: 'http://your-proxy-server.com/api/proxy'
        })
      }
    })
  );

  // Setup dock and canvas menus
  cesdk.ui.setDockOrder(['ly.img.ai.apps.dock', ...cesdk.ui.getDockOrder()]);
  cesdk.ui.setCanvasMenuOrder([
    {
      id: 'ly.img.ai.text.canvasMenu',
      children: [
        'ly.img.improve',
        'ly.img.fix',
        'ly.img.shorter',
        'ly.img.longer',
        'ly.img.separator',
        'ly.img.changeTone',
        'ly.img.translate',
        'ly.img.separator',
        'ly.img.changeTextTo'
      ]
    },
    {
      id: 'ly.img.ai.image.canvasMenu',
      children: [
        'ly.img.styleTransfer',
        'ly.img.artistTransfer',
        'ly.img.separator',
        'ly.img.editImage',
        'ly.img.swapBackground',
        'ly.img.createVariant',
        'ly.img.combineImages',
        'ly.img.separator',
        'ly.img.remixPage',
        'ly.img.separator',
        'ly.img.createVideo'
      ]
    },
    ...cesdk.ui.getCanvasMenuOrder()
  ]);
});

Translations

The AI Apps plugin uses translations from individual AI generation plugins. For customization, refer to the respective translation files:

Base translations - Core translation keys
Image generation translations - Image generation interfaces
Video generation translations - Video generation interfaces
Text generation translations - Text generation interfaces
Audio generation translations - Audio generation interfaces

License

This plugin is part of the IMG.LY plugin ecosystem for CreativeEditor SDK. Please refer to the license terms in the package.