Strapi Content Embeddings

A Strapi v5 plugin that creates vector embeddings from your content using OpenAI and stores them in Neon PostgreSQL with pgvector. Enables semantic search, RAG (Retrieval-Augmented Generation) chat, and MCP (Model Context Protocol) integration for AI assistants like Claude Desktop.

Features

• Vector Embeddings: Generate embeddings from your content using OpenAI's embedding models
• Neon PostgreSQL Storage: Store embeddings in Neon DB with pgvector for efficient similarity search
• RAG Chat Interface: Built-in chat widget to ask questions about your content
• MCP Server: Expose your embeddings to AI assistants via Model Context Protocol
• Content Manager Integration: Create embeddings directly from any content type's edit view
• Standalone Embeddings: Create embeddings independent of content types
• Multiple Embedding Models: Support for OpenAI's text-embedding-3-small, text-embedding-3-large, and text-embedding-ada-002
• Database Sync: Sync embeddings from Neon DB to Strapi via admin UI or API endpoints
• Automatic Chunking: Split large content into multiple embeddings with overlap for context preservation
• Content Preprocessing: Automatically strips HTML and Markdown formatting for cleaner embeddings

Requirements

• Strapi v5.x
• Node.js 18+
• OpenAI API key
• Neon PostgreSQL database with pgvector extension

Installation

npm install strapi-content-embeddings
# or
yarn add strapi-content-embeddings

Configuration

1. Enable the Plugin

Add the plugin to your config/plugins.ts (or config/plugins.js):

export default ({ env }) => ({
  "strapi-content-embeddings": {
    enabled: true,
    config: {
      openAIApiKey: env("OPENAI_API_KEY"),
      neonConnectionString: env("NEON_CONNECTION_STRING"),
      // Optional: Choose embedding model (default: "text-embedding-3-small")
      embeddingModel: env("EMBEDDING_MODEL", "text-embedding-3-small"),
    },
  },
});

2. Set Environment Variables

Add the following to your .env file:

OPENAI_API_KEY=sk-your-openai-api-key
NEON_CONNECTION_STRING=postgresql://user:[email protected]/dbname?sslmode=require
# Optional
EMBEDDING_MODEL=text-embedding-3-small

3. Get Your Neon Connection String

1. Sign up at Neon
2. Create a new project
3. Navigate to your project's Connection Details
4. Copy the connection string (it should look like postgresql://user:[email protected]/dbname?sslmode=require)

The plugin will automatically:

• Enable the pgvector extension
• Create the embeddings_documents table
• Set up HNSW indexes for fast similarity search

MCP Integration

This plugin exposes an MCP (Model Context Protocol) server that allows AI assistants like Claude Desktop to search your embeddings.

MCP Endpoint

POST /api/strapi-content-embeddings/mcp

Available MCP Tools

| Tool | Description | Trigger | |------|-------------|---------| | rag_query | Ask questions and get AI-generated answers from your content | /rag [question] | | semantic_search | Find semantically similar content | /rag search [query] | | list_embeddings | List all stored embeddings | - | | get_embedding | Get a specific embedding by ID | - | | create_embedding | Create a new embedding | - |

Claude Desktop Configuration

Add to your Claude Desktop config (~/Library/Application Support/Claude/claude_desktop_config.json):

{
  "mcpServers": {
    "strapi-content-embeddings": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "https://your-strapi-url.com/api/strapi-content-embeddings/mcp",
        "--header",
        "Authorization: Bearer YOUR_STRAPI_API_TOKEN"
      ]
    }
  }
}

Usage in Claude Desktop

Type /rag followed by your question to search your embeddings:

/rag What is Strapi?
/rag Who is Paul Bratslavsky?

Available Embedding Models

| Model | Dimensions | Description | |-------|------------|-------------| | text-embedding-3-small | 1536 | Fast, cost-effective (default) | | text-embedding-3-large | 3072 | Higher accuracy, more expensive | | text-embedding-ada-002 | 1536 | Legacy model |

Usage

Admin Panel

Create Embeddings Page

Navigate to Content Embeddings in the Strapi admin sidebar to:

• View all existing embeddings
• Create new standalone embeddings
• Delete embeddings
• Search and filter embeddings

Content Manager Integration

When editing any content type, you'll see an Embeddings panel in the right sidebar:

• Create Embedding: Generate an embedding from the current content
• View Embedding: Navigate to the embedding details
• Update Embedding: Update the embedding with current content changes

Chat Widget

Click the robot icon in the bottom-right corner to open the RAG chat interface:

• Ask questions about your embedded content
• View source documents used to generate answers
• Navigate to source embeddings

Programmatic Usage

Create an Embedding

const result = await strapi
  .plugin("strapi-content-embeddings")
  .service("embeddings")
  .createEmbedding({
    data: {
      title: "My Document",
      content: "This is the content to embed...",
      collectionType: "api::article.article", // optional
      fieldName: "content", // optional
      metadata: { customField: "value" }, // optional
    },
  });

Query Embeddings (RAG)

const response = await strapi
  .plugin("strapi-content-embeddings")
  .service("embeddings")
  .queryEmbeddings("What is this document about?");

// response.text - The AI-generated answer
// response.sourceDocuments - The documents used for context

Similarity Search

const documents = await strapi
  .plugin("strapi-content-embeddings")
  .service("embeddings")
  .similaritySearch("search query", 4); // returns top 4 similar documents

API Endpoints

All endpoints require admin authentication.

| Method | Endpoint | Description | |--------|----------|-------------| | POST | /strapi-content-embeddings/embeddings/create-embedding | Create a new embedding | | PUT | /strapi-content-embeddings/embeddings/update-embedding/:id | Update an existing embedding | | DELETE | /strapi-content-embeddings/embeddings/delete-embedding/:id | Delete an embedding | | GET | /strapi-content-embeddings/embeddings/find | List all embeddings | | GET | /strapi-content-embeddings/embeddings/find/:id | Get a single embedding | | GET | /strapi-content-embeddings/embeddings/embeddings-query?query=... | RAG query |

Database Sync (Neon to Strapi)

The plugin provides endpoints to sync embeddings from Neon DB (source of truth) to Strapi. These endpoints are designed to be triggered manually or via cron jobs.

Sync Endpoints

| Method | Endpoint | Description | |--------|----------|-------------| | GET/POST | /api/strapi-content-embeddings/sync | Sync embeddings from Neon to Strapi | | GET | /api/strapi-content-embeddings/sync/status | Check sync status without making changes |

Query Parameters

| Parameter | Type | Default | Description | |-----------|------|---------|-------------| | dryRun | boolean | false | Preview changes without applying them | | removeOrphans | boolean | false | Remove Strapi entries that don't exist in Neon |

Usage Examples

Check sync status:

curl "http://localhost:1337/api/strapi-content-embeddings/sync/status" \
  -H "Authorization: Bearer YOUR_API_TOKEN"

Response:

{
  "neonCount": 150,
  "strapiCount": 145,
  "inSync": false,
  "missingInStrapi": 5,
  "missingInNeon": 0,
  "contentDifferences": 2
}

Dry run (preview changes):

curl "http://localhost:1337/api/strapi-content-embeddings/sync?dryRun=true" \
  -H "Authorization: Bearer YOUR_API_TOKEN"

Run sync:

curl "http://localhost:1337/api/strapi-content-embeddings/sync" \
  -H "Authorization: Bearer YOUR_API_TOKEN"

Sync and remove orphans:

curl "http://localhost:1337/api/strapi-content-embeddings/sync?removeOrphans=true" \
  -H "Authorization: Bearer YOUR_API_TOKEN"

Sync Response

{
  "success": true,
  "timestamp": "2024-01-07T12:00:00.000Z",
  "neonCount": 150,
  "strapiCount": 150,
  "actions": {
    "created": 5,
    "updated": 2,
    "orphansRemoved": 0
  },
  "details": {
    "created": ["doc1 (Title 1)", "doc2 (Title 2)"],
    "updated": ["doc3 (Title 3)"],
    "orphansRemoved": []
  },
  "errors": []
}

Cron Job Example

# Sync every hour
0 * * * * curl -s "https://your-strapi.com/api/strapi-content-embeddings/sync" \
  -H "Authorization: Bearer YOUR_API_TOKEN" >> /var/log/embeddings-sync.log

Content Chunking

For large content that exceeds the recommended size for embeddings (~4000 characters / ~1000 tokens), the plugin supports automatic chunking.

How Chunking Works

1. Smart Splitting: Content is split at natural boundaries (paragraphs, sentences, words) to preserve meaning
2. Overlap: Chunks include overlapping content (default: 200 chars) to maintain context between chunks
3. Metadata: Each chunk stores metadata linking it to the original content and other chunks
4. Titles: Chunk titles include part numbers (e.g., "My Document [Part 1/3]")

Configuration

Add chunking options to your plugin config:

// config/plugins.ts
export default ({ env }) => ({
  "strapi-content-embeddings": {
    enabled: true,
    config: {
      openAIApiKey: env("OPENAI_API_KEY"),
      neonConnectionString: env("NEON_CONNECTION_STRING"),
      // Chunking options
      chunkSize: 4000,      // Max characters per chunk (default: 4000)
      chunkOverlap: 200,    // Overlap between chunks (default: 200)
      autoChunk: false,     // Auto-chunk large content globally (default: false)
    },
  },
});

Using Chunking

Via MCP Tool

{
  "tool": "create_embedding",
  "arguments": {
    "title": "My Long Document",
    "content": "... very long content ...",
    "autoChunk": true
  }
}

Programmatic Usage

// Create with automatic chunking
const result = await strapi
  .plugin("strapi-content-embeddings")
  .service("embeddings")
  .createChunkedEmbedding({
    data: {
      title: "My Long Document",
      content: "... very long content ...",
    },
  });

console.log(result);
// {
//   entity: { ... first chunk ... },
//   chunks: [ ... all chunks ... ],
//   totalChunks: 5,
//   wasChunked: true
// }

// Or use createEmbedding with autoChunk flag
const embedding = await strapi
  .plugin("strapi-content-embeddings")
  .service("embeddings")
  .createEmbedding({
    data: {
      title: "My Document",
      content: "... long content ...",
      autoChunk: true,  // Enable chunking
    },
  });

Chunk Metadata

Each chunk embedding includes metadata:

{
  "isChunk": true,
  "chunkIndex": 0,
  "totalChunks": 5,
  "startOffset": 0,
  "endOffset": 4200,
  "originalTitle": "My Long Document",
  "parentDocumentId": "abc123",
  "estimatedTokens": 1050
}

Content Preprocessing

The plugin automatically preprocesses content before creating embeddings to improve semantic search quality. This is enabled by default.

What Gets Cleaned

• HTML tags: Stripped while preserving text content
• Markdown syntax: Headers (#), bold (**), italic (*), links, lists, code blocks
• Whitespace: Normalized (multiple spaces/newlines collapsed)

Why Preprocess?

Raw markdown/HTML formatting adds noise to embeddings without adding semantic meaning:

Input:  "## Features\n- **Fast** search\n- <b>Reliable</b>"
Output: "Features: Fast search. Reliable"

Both produce the same semantic meaning, but the cleaned version creates better embeddings for search.

Configuration

Preprocessing is enabled by default. To disable:

// config/plugins.ts
export default ({ env }) => ({
  "strapi-content-embeddings": {
    enabled: true,
    config: {
      openAIApiKey: env("OPENAI_API_KEY"),
      neonConnectionString: env("NEON_CONNECTION_STRING"),
      preprocessContent: false,  // Disable preprocessing
    },
  },
});

Note: The original content is always preserved in Strapi. Preprocessing only affects the text sent to OpenAI for embedding generation.

Admin Sync UI

The plugin includes a built-in sync interface accessible from the admin panel. Click the Sync button in the Content Embeddings page header.

Available Operations

| Operation | Description | |-----------|-------------| | Check Status | Compare Neon and Strapi databases, shows counts and differences | | Sync from Neon | Import embeddings from Neon to Strapi (with preview option) | | Recreate All | Delete all Neon embeddings and recreate from Strapi data |

Sync Workflow

1. Click Sync button to open the sync modal
2. View current sync status (Neon vs Strapi counts)
3. Select Sync from Neon operation
4. Click Preview Sync to see what changes would be made
5. Review the results (Created/Updated/Removed counts)
6. Click Apply Changes to execute the sync

Sync Options

• Dry Run: Preview changes without applying them (enabled by default)
• Remove Orphans: Delete Strapi entries that don't exist in Neon

How It Works

1. Embedding Creation: When you create an embedding, the content is sent to OpenAI's embedding API to generate a vector representation (1536 or 3072 dimensions depending on the model).

2. Storage: The embedding vector is stored in Neon PostgreSQL using the pgvector extension, along with the content and metadata.

3. Similarity Search: When querying, the search query is converted to an embedding and compared against stored embeddings using cosine similarity via pgvector's HNSW index.

4. RAG Response: For chat queries, the most relevant documents are retrieved and passed to GPT-4o-mini as context to generate an accurate response.

Database Schema

The plugin creates an embeddings_documents table in your Neon database:

CREATE TABLE embeddings_documents (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  content TEXT,
  metadata JSONB,
  embedding vector(1536)  -- or 3072 for text-embedding-3-large
);

Indexes:

• HNSW index on embedding for fast similarity search
• GIN index on metadata for filtering

Permissions

The plugin registers the following RBAC permissions:

• plugin::strapi-content-embeddings.read - View embeddings
• plugin::strapi-content-embeddings.create - Create embeddings
• plugin::strapi-content-embeddings.update - Update embeddings
• plugin::strapi-content-embeddings.delete - Delete embeddings
• plugin::strapi-content-embeddings.chat - Use the RAG chat feature

Configure these in Settings > Roles for each admin role.

Troubleshooting

Embeddings not being created

1. Check that OPENAI_API_KEY is set correctly
2. Check that NEON_CONNECTION_STRING is valid
3. Look for errors in the Strapi console

Chat returns "cannot find the answer"

1. Ensure embeddings exist in the database
2. Try creating more specific content
3. Check that the embedding model matches between creation and query

Connection errors

1. Verify your Neon connection string includes ?sslmode=require
2. Check that your Neon project is active (not paused)
3. Ensure the pgvector extension is enabled

MCP not connecting

1. Verify the MCP endpoint URL is correct
2. Check the Authorization header has a valid Strapi API token
3. Ensure the plugin is properly configured and Strapi is running

License

MIT