payload-plugin-rag-chatbot

Self-hosted RAG chatbot plugin for Payload CMS 3 — crawl your site, answer visitor questions, no external vector store required. It's a thin adapter around rag-core-engine (crawl → chunk → embed → store → retrieve → answer) that wires it into a Payload config: a chat endpoint with multi-turn conversation memory, site-crawl/file-upload/manual-text ingestion, and admin dashboard widgets for both status and adding content.

If your app doesn't actually run Payload CMS 3, this package has nothing to attach to — use rag-core-engine directly instead (see its README for plain Node.js / Express / NestJS examples).

Requirements

• Payload CMS 3.x already set up in your app (payload in your package.json)
• React 18 or 19 (Payload's own peer requirement)
• A Postgres database with the pgvector extension available — the plugin creates its own table and enables the extension on first use (this is the only vector store shipped today; see Provider selection for how to point at a different one once you register it)
• An API key for whichever LLM provider you use for answer generation — Anthropic by default
• An API key for whichever embedding provider you use — Voyage AI by default

Install

pnpm add payload-plugin-rag-chatbot

(npm install / yarn add work the same way — payload and react are peer dependencies, expected to already be in your project.)

Setup

Define your options in a dedicated rag-chatbot.config.ts file, typed with the plugin's exported RagChatbotPluginOptions, then pass it to the plugin in payload.config.ts. There's nothing plugin-specific about this split — it's the same module composition you'd use to keep any large config object out of payload.config.ts — but it keeps this plugin's (fairly large) options surface in one place and fully type-checked. The plugin doesn't generate or expect this file; it's plain code you write yourself.

Only siteUrl is required — every other key below is optional and falls back to an environment variable, then a built-in default. Shown here with every possible key so you can see the full shape in one place (see Options below for the detailed reference table):

// rag-chatbot.config.ts
import type { RagChatbotPluginOptions } from "payload-plugin-rag-chatbot";

export const ragChatbotConfig: RagChatbotPluginOptions = {
  enabled: true, // default: true — set false to disable the plugin without removing it from config

  siteUrl: process.env.RAG_SITE_URL!, // required — root URL the crawl job starts from
  databaseUri: process.env.DATABASE_URL, // default: process.env.DATABASE_URI

  // provider/apiKey below are shown explicitly for clarity — in practice you
  // can usually omit them and rely on $RAG_LLM_PROVIDER/$RAG_LLM_API_KEY (or
  // the built-in providers' own $ANTHROPIC_API_KEY/$VOYAGE_API_KEY fallback)
  // instead. See "Provider selection" below.
  llm: {
    provider: process.env.RAG_LLM_PROVIDER, // default: $RAG_LLM_PROVIDER, else "anthropic"
    apiKey: process.env.RAG_LLM_API_KEY, // default: $RAG_LLM_API_KEY, else $ANTHROPIC_API_KEY
    model: process.env.RAG_LLM_MODEL, // default: "claude-opus-5" for "anthropic"
    maxTokens: process.env.RAG_LLM_MAX_TOKENS ? Number(process.env.RAG_LLM_MAX_TOKENS) : undefined, // default: unset (AI SDK/model default)
    options: {}, // default: {} — extra fields merged into the factory config, for provider-specific settings
  },

  pdfExtraction: {
    apiKey: process.env.ANTHROPIC_API_KEY, // default: llm.apiKey when llm.provider is "anthropic", else $ANTHROPIC_API_KEY
    model: process.env.RAG_PDF_EXTRACTION_MODEL, // default: llm.model when llm.provider is "anthropic"
    maxTokens: process.env.RAG_PDF_EXTRACTION_MAX_TOKENS
      ? Number(process.env.RAG_PDF_EXTRACTION_MAX_TOKENS)
      : undefined, // default: 8192
  },

  embedding: {
    provider: process.env.RAG_EMBEDDING_PROVIDER, // default: $RAG_EMBEDDING_PROVIDER, else "voyage"
    apiKey: process.env.RAG_EMBEDDING_API_KEY, // default: $RAG_EMBEDDING_API_KEY, else $VOYAGE_API_KEY
    model: process.env.RAG_EMBEDDING_MODEL, // default: "voyage-3" for "voyage"
    options: {
      dimensions: process.env.RAG_EMBEDDING_DIMENSIONS ? Number(process.env.RAG_EMBEDDING_DIMENSIONS) : undefined, // default: 1024 for "voyage"
      // baseUrl, retryOnRateLimit — see "Rate-limit retry (Voyage 429s)" below
    },
  },

  vectorStore: {
    provider: process.env.RAG_VECTOR_STORE_PROVIDER, // default: $RAG_VECTOR_STORE_PROVIDER, else "pgvector"
    tableName: process.env.RAG_TABLE_NAME, // default: "rag_chatbot_chunks"
    options: {}, // default: {} — extra fields merged into the factory config, for provider-specific settings
  },

  crawl: {
    maxDepth: 3, // default: 3
    maxPages: 200, // default: 200
    userAgent: "MySiteChatbotCrawler/1.0", // default: a rag-core-engine identifier
  },

  chunking: {
    bufferSize: 1, // default: 1
    breakpointPercentile: 95, // default: 95
    minChunkChars: 200, // default: 200
    maxChunkChars: 2000, // default: 2000
  },

  chat: {
    topK: 5, // default: 5
    historyLimit: 5, // default: 5
    persona: "a friendly, knowledgeable member of the Acme Corp team", // default: a neutral, site-agnostic identity
    conversationMemory: true, // default: true
    noContextMessage: "I don't have anything indexed on that topic yet.", // default: a built-in "nothing indexed yet" message
  },
};

// payload.config.ts
import { buildConfig } from "payload";
import { ragChatbotPlugin } from "payload-plugin-rag-chatbot";
import { ragChatbotConfig } from "./rag-chatbot.config";

export default buildConfig({
  // ...your existing config
  plugins: [ragChatbotPlugin(ragChatbotConfig)],
});

Required: regenerate your admin import map

This plugin adds custom admin components — the "RAG Chatbot" dashboard widget (with the re-crawl button), the Crawled Pages/Knowledge Base chunk browsers, and the conversation transcript viewer. Payload resolves these through app/(payload)/admin/importMap.js, a static file that's only written by the generate:importmap CLI command — next dev does not regenerate it for you. After adding the plugin (or upgrading to a version that adds new admin components), run:

payload generate:importmap

Skip this and the components silently fail to render — no error, the "RAG Chatbot" widget and the re-crawl button just won't show up.

rag-core-engine itself still never reads environment variables (see its design boundary) — this plugin is the one layer that does, purely as a convenience on top of explicit options, which always win when set. See Provider selection for the full resolution order and how to switch providers.

Required: exclude the chunks table from Postgres schema push

The plugin's vector store table (rag_chatbot_chunks by default) isn't a Payload collection — pgVectorStore creates and manages it directly, outside Payload's schema. In local dev, Payload's postgresAdapter auto-syncs your schema on every run (push mode) by diffing the database against its own collections; since it doesn't recognize this table, it will interactively prompt to drop it on every schema diff — which blocks the entire server if nothing is attached to answer the prompt (e.g. running non-interactively, or just easy to miss in a terminal). Exclude it:

db: postgresAdapter({
  pool: { connectionString: process.env.DATABASE_URL },
  tablesFilter: ["!rag_chatbot_chunks"], // update if you set vectorStore.tableName to something else
}),

This isn't needed in production, where Payload uses migrations instead of push — but it's required for local dev with the default Postgres adapter setup.

Options

All options are defined in src/types.ts. Only siteUrl is required — everything else falls back to a provider-specific default, an environment variable, or rag-core-engine's built-in defaults, in that order (see Provider selection for the embedding/LLM/ vector-store slots specifically).

Top-level

| Option | Type | Required | Default | Notes | | --- | --- | --- | --- | --- | | enabled | boolean | no | true | Set false to disable the plugin without removing it from config. | | siteUrl | string | yes | — | Root URL the crawl job starts from. | | databaseUri | string | no | process.env.DATABASE_URI | Postgres connection string for the vector store. A dedicated pg.Pool is opened for this — it does not reuse Payload's internal adapter connection. Throws at plugin-init time if neither this nor DATABASE_URI is set. Set DATABASE_URI regardless of what you pass here — see Environment variables, a few admin components can only read that env var directly. |

embedding — which embedding provider to use, and its settings

| Option | Type | Default | Notes | | --- | --- | --- | --- | | embedding.provider | string | $RAG_EMBEDDING_PROVIDER, else "voyage" | Name of a provider registered in rag-core-engine's embeddingProviders registry. | | embedding.apiKey | string | $RAG_EMBEDDING_API_KEY, else $VOYAGE_API_KEY when provider is "voyage" | API key passed to the provider's factory. | | embedding.model | string | "voyage-3" for "voyage" | Embedding model name. | | embedding.options | object | {} | Extra fields merged into the factory config — for "voyage": dimensions (default 1024), baseUrl (default "https://api.voyageai.com/v1"), retryOnRateLimit (see Rate-limit retry below). |

llm — which LLM provider to use for chat answers, and its settings

| Option | Type | Default | Notes | | --- | --- | --- | --- | | llm.provider | string | $RAG_LLM_PROVIDER, else "anthropic" | Name of a provider registered in rag-core-engine's llmProviders registry. | | llm.apiKey | string | $RAG_LLM_API_KEY, else $ANTHROPIC_API_KEY when provider is "anthropic" | API key passed to the provider's factory. | | llm.model | string | "claude-opus-5" for "anthropic" | Chat model name. | | llm.maxTokens | number | unset (AI SDK/model default) | Max output tokens for chat answers. | | llm.options | object | {} | Extra fields merged into the factory config, for provider-specific settings. |

pdfExtraction — PDF text extraction (Claude's native document support; currently Anthropic-specific regardless of llm.provider — see Provider selection)

| Option | Type | Default | Notes | | --- | --- | --- | --- | | pdfExtraction.apiKey | string | llm.apiKey when llm.provider is "anthropic", else $ANTHROPIC_API_KEY | | | pdfExtraction.model | string | llm.model when llm.provider is "anthropic", else "claude-opus-5" | | | pdfExtraction.maxTokens | number | 8192 | A multi-page PDF's extracted text can run long. |

vectorStore

| Option | Type | Default | Notes | | --- | --- | --- | --- | | vectorStore.provider | string | $RAG_VECTOR_STORE_PROVIDER, else "pgvector" | Name of a provider registered in rag-core-engine's vectorStores registry. | | vectorStore.tableName | string | "rag_chatbot_chunks" | Postgres table name for chunks + embeddings — used by "pgvector". | | vectorStore.options | object | {} | Extra fields merged into the factory config, for provider-specific settings. |

crawl

| Option | Type | Default | Notes | | --- | --- | --- | --- | | crawl.maxDepth | number | 3 | Max link depth from siteUrl. | | crawl.maxPages | number | 200 | Max pages indexed per crawl. | | crawl.userAgent | string | a rag-core-engine identifier | User-Agent header sent while crawling. |

chunking — semantic chunker tuning (all four optional; see rag-core-engine's chunking docs for what each does)

| Option | Type | Default | | --- | --- | --- | | chunking.bufferSize | number | 1 | | chunking.breakpointPercentile | number | 95 | | chunking.minChunkChars | number | 200 | | chunking.maxChunkChars | number | 2000 |

chat

| Option | Type | Default | Notes | | --- | --- | --- | --- | | chat.topK | number | 5 | Number of chunks retrieved per question. | | chat.historyLimit | number | 5 | Prior conversation turns fed back into the answer prompt. | | chat.persona | string | a neutral, site-agnostic identity | Short identity description inserted into the chat system prompt, e.g. "a friendly, knowledgeable member of the Acme Corp team". | | chat.conversationMemory | boolean | true | Persist conversation history so follow-up questions have context (adds two collections — see Conversation memory). Set false to opt out. | | chat.noContextMessage | string | a built-in "nothing indexed yet" message | Returned as-is (no LLM call) when nothing relevant is indexed for a question. |

Every option at once, to see the full shape in one place: see the rag-chatbot.config.ts example in Setup above — it shows every key from this reference table together in a single file.

Rate-limit retry (Voyage 429s)

A Voyage account with no payment method on file is capped at 3 requests per minute — easy to hit while crawling a real site (each page costs at least one embedding call, sometimes two — see Chunking strategy), and even a single chat question can fail outright on the very first 429.

By default, embedding.options.retryOnRateLimit is true whenever NODE_ENV !== "production" — a 429 gets retried with exponential backoff (20s base, doubling each attempt, capped at 90s, up to 8 attempts — long enough to ride out the 3-requests/minute ceiling) instead of failing the crawl/chat request outright. In production it defaults to false — the same fail-fast behavior as before this option existed — since a live chat request silently hanging for minutes on a rate limit is usually worse than just erroring; you're also expected to have billing set up in production, where this ceiling doesn't apply. Set embedding.options.retryOnRateLimit explicitly to override either default:

// rag-chatbot.config.ts
export const ragChatbotConfig: RagChatbotPluginOptions = {
  // ...
  embedding: { options: { retryOnRateLimit: true } }, // force it on even in production, or false to force it off in dev
};

This flag is specific to the built-in "voyage" provider; it's read regardless of which embedding provider is active, but only "voyage"'s factory does anything with it.

This applies to every Voyage call through the shared embedding provider — crawling, file/text indexing, and the chat endpoint's query embedding all go through the same retry logic, matching how this plugin's Laravel counterpart (VoyageEmbedder) handles it. A rate-limited retry logs a warning (console.warn) with the attempt number and wait time each time it backs off.

What the plugin adds

Registering ragChatbotPlugin(...) in plugins: [] mutates your Payload config (see src/index.ts) to add these:

1. A chat endpoint

POST /api/rag-chatbot/chat (path exported as CHAT_ENDPOINT_PATH, see src/endpoints/chat.ts):

curl -X POST https://your-app.com/api/rag-chatbot/chat \
  -H "Content-Type: application/json" \
  -d '{"question": "What does this site do?"}'

{
  "answer": "...",
  "conversationId": "018f...",
  "sources": [{ "ref": "https://example.com/about", "content": "..." }]
}

Returns 400 if question is missing or empty. If nothing relevant is indexed yet, it returns a canned "nothing indexed" message without calling the LLM at all.

Pass conversationId back in on follow-up questions to continue the same conversation (see Conversation memory — omit it, or leave chat.conversationMemory: false, for stateless single-shot Q&A):

curl -X POST https://your-app.com/api/rag-chatbot/chat \
  -H "Content-Type: application/json" \
  -d '{"question": "Can you say more about that?", "conversationId": "018f..."}'

2. File upload and manual text endpoints — backed by background jobs

Besides crawling a whole site, you can index content one item at a time — either an uploaded file or pasted text. Unlike the chat endpoint, these don't index synchronously inline in the request: parsing a PDF/docx and calling the embedding API for every chunk can take longer than an HTTP request should block for, so both endpoints just validate the input, queue a Payload job, and return immediately. The actual indexing (chunk → embed → store) happens when that job runs, same as the crawl job below.

POST /api/rag-chatbot/upload-file (path exported as UPLOAD_FILE_ENDPOINT_PATH, see src/endpoints/uploadFile.ts) — accepts multipart/form-data with a file field. Supported types: text/plain, text/markdown, application/pdf, and .docx (application/vnd.openxmlformats-officedocument.wordprocessingml.document). Validates the file is present and of a supported type, then queues task rag-chatbot-index-file (exported as UPLOAD_FILE_TASK_SLUG, see src/jobs/uploadFileJob.ts) with the file content base64-encoded as job input. PDFs are extracted by handing them to Claude directly (reusing llm's API key/model when llm.provider is "anthropic" — see pdfExtraction in Options — no extra config needed in the default setup) rather than a parsing library, so scanned pages and complex layouts extract cleanly; .docx/.txt/.md still use plain library-based extraction.

curl -X POST https://your-app.com/api/rag-chatbot/upload-file \
  -F "[email protected]"

{ "queued": true, "filename": "handbook.pdf" }

POST /api/rag-chatbot/index-text (path exported as INDEX_TEXT_ENDPOINT_PATH, see src/endpoints/indexText.ts) — accepts JSON { "id"?: string, "content": string }. id is optional and auto-generated (UUID) if omitted; passing your own id lets you re-index the same logical document later (it upserts into the same source). Queues task rag-chatbot-index-text (exported as INDEX_TEXT_TASK_SLUG, see src/jobs/indexTextJob.ts).

curl -X POST https://your-app.com/api/rag-chatbot/index-text \
  -H "Content-Type: application/json" \
  -d '{"id": "faq-refunds", "content": "Refunds are processed within 5 business days."}'

{ "queued": true, "id": "faq-refunds" }

Both return 202 (not 200) on success — the response only confirms the job was queued, not that indexing finished — and 400 if the file/content is missing or the file type is unsupported.

3. Background jobs

Three Payload Jobs Queue tasks are registered:

| Task slug | Constant | Input | What it does | | --- | --- | --- | --- | | rag-chatbot-crawl-site | CRAWL_TASK_SLUG | { siteUrl?: string } | Crawls a site and indexes every page. Queued directly by you (see below) — not queued by an endpoint. | | rag-chatbot-index-file | UPLOAD_FILE_TASK_SLUG | { filename, mimeType, contentBase64 } | Parses and indexes one uploaded file. Queued by the upload endpoint above. | | rag-chatbot-index-text | INDEX_TEXT_TASK_SLUG | { id, content } | Indexes one block of raw text. Queued by the index-text endpoint above. |

Unlike the file/text tasks, nothing queues the crawl task for you — trigger it yourself using Payload's Jobs Queue API, e.g. from a cron job, an admin action, or on deploy:

await payload.jobs.queue({
  task: "rag-chatbot-crawl-site",
  input: { siteUrl: "https://example.com" }, // optional — omit to use the plugin's configured siteUrl
});

For all three tasks, queuing only creates the job record — it still has to run. Run pending jobs however you already run Payload jobs in your app (a cron-triggered payload.jobs.run(), the /api/payload-jobs/run endpoint, or your queue's runner) — see the Payload Jobs Queue docs for how job execution is wired up in your environment. If nothing ever runs pending jobs, queued uploads/text/crawls will sit unindexed indefinitely.

4. Conversation memory

When chat.conversationMemory isn't set to false (the default), the plugin registers two collections (see src/collections/Conversations.ts and src/collections/ConversationMessages.ts):

| Collection | Slug | Purpose | | --- | --- | --- | | Conversations | rag-chatbot-conversations | One row per conversation — just an id and timestamps. | | Conversation Messages | rag-chatbot-conversation-messages | One row per turn (role: user/assistant, content, a conversation relationship). |

The chat endpoint reads/writes these via src/conversationStore.ts (a ConversationStore implementation from rag-core-engine, backed by Payload's Local API) — see Conversation memory in the endpoint section above for the request/response shape. Both collections default to admin-only read/delete access (Boolean(req.user)) and block create/update via the public API entirely — the endpoint writes through the Local API, which bypasses access control, so locking down the public API doesn't block anything internal.

Set chat.conversationMemory: false to skip registering these collections and keep the chat endpoint stateless (no conversationId handling, no history fed into answers).

Browsing conversations in the admin UI

You browse conversations through Payload's own collection UI — no separate custom routes. The sidebar gets a single "RAG Chatbot" group containing one link, "Rag Chatbot Conversations" (/admin/collections/rag-chatbot-conversations).

The list itself previews each conversation without opening it — two virtual (not stored, computed on read) columns, "Last message from visitor" and "Last message from assistant", show the most recent message of each role (truncated to 140 characters), so you can tell what a conversation was about at a glance. Opening a conversation shows its full transcript as chat bubbles (visitor right-aligned, assistant left-aligned) via a transcript UI field on the document's edit page (see src/admin/ConversationTranscriptField.tsx) — a Server Component reading directly via the Local API (payload.find).

ConversationMessages (the raw per-message rows the transcript reads from) stays hidden from the nav (admin.hidden: true) — there's no need to browse it directly since the transcript field already shows everything, and hiding it keeps this to one clean menu entry instead of two.

A fully custom top-level admin route (its own /admin/rag-chatbot/... page, wrapped in Payload's DefaultTemplate for the normal nav/sidebar chrome) was tried first, but DefaultTemplate turns out to need request context that's only established inside Payload's own internal view rendering pipeline — even a trivial call crashes when invoked from a plugin-registered admin.components.views route. A field component embedded in Payload's own already-correct document page, as done here, has no such requirement.

5. "RAG Chatbot" admin nav group — Crawled Pages & Knowledge Base

Two more real Payload collections are registered, both grouped under "RAG Chatbot" in the sidebar alongside Conversations — so opening the admin UI gives you three pages: Crawled Pages, Knowledge Base, Conversations.

Like Conversations, these have no fields of their own beyond a stored identity field (url / sourceId) — chunkCount and lastCrawledAt/lastUpdatedAt are virtual, computed on read from rag_chatbot_chunks, and each has a chunks/content UI field on its edit page showing the actual chunk text for that source (see src/admin/CrawledPageChunksField.tsx / src/admin/KnowledgeBaseChunksField.tsx). Deleting a row cascades to its chunks via vectorStore.deleteBySourceId (an afterDelete hook), so removing a page/document here actually removes it from what the chatbot can answer from.

• Crawled Pages (src/collections/CrawledPages.ts, slug rag-chatbot-crawled-pages) — one row per crawled URL, scoped to source_type = 'url' in the chunks table (uploaded files and manual text live in the same table under a different source_type, so they're excluded here). Rows are created by the crawl job itself: after a successful crawl, it finds-or-creates a row for every distinct URL currently in rag_chatbot_chunks — nothing to keep in sync manually.
• Knowledge Base (src/collections/KnowledgeBase.ts, slug rag-chatbot-knowledge-base) — one row per manually-added document (uploaded file or pasted text), matching what the upload-file/index-text jobs already index. Adding content still happens via the dashboard's Ingest panel below (or the endpoints directly) — this collection is a browse/delete view onto what those jobs created, populated the same find-or-create way, keyed by the same source_id core computes (`file:${filename}` for uploads, the given id for text).

Both collections' read/delete access requires an authenticated admin user; create/update are blocked entirely (population only happens through the Local API from the job handlers, which bypasses access control).

6. Admin dashboard widgets

Two beforeDashboard components are added to your Payload admin UI:

• CrawlStatusView (see src/admin/CrawlStatusView.tsx) — shows how many chunks are indexed and when the last crawl ran, plus a "Re-crawl site" button (see below). Reads directly from the rag_chatbot_chunks table via process.env.DATABASE_URI — note this is a different env var than databaseUri/DATABASE_URL, see Environment variables below; if it isn't set, or the table doesn't exist yet, the widget just shows "No crawl data yet" without erroring.

• IngestPanel (see src/admin/IngestPanel.tsx) — a client-side form with two sections, right on the dashboard:

  • Upload file — pick a .txt/.md/.pdf/.docx file, posts it to upload-file.
  • Add text — paste content with an optional ID, posts it to index-text.

  Both show inline success/error status after submitting, so editors can add content without touching the API directly.

Re-crawl site button (in CrawlStatusView, implemented in src/admin/CrawlTriggerButton.tsx) — queues the crawl job (CRAWL_TASK_SLUG) and immediately kicks off payload.jobs.run() for it — fire-and-forget, not awaited — so it starts processing right away instead of waiting for the next jobs.autoRun tick, which might be minutes away or not configured at all. The button then polls a status endpoint every 3 seconds until the job completes or errors. The fire-and-forget run only works as long as the server process stays alive after the request — true for a normal dev/Node server, not guaranteed on serverless; there, jobs.autoRun (or your own cron hitting the job runner) is what actually processes a queued crawl.

Endpoints backing the button — both require an authenticated admin user (req.user), unlike the public chat/ingest endpoints above:

| Endpoint | Method | Purpose | | --- | --- | --- | | /api/rag-chatbot/crawl | POST | Queue + immediately run the crawl job, returns { jobId } | | /api/rag-chatbot/crawl?jobId= | GET | Poll that job's status for a known job id | | /api/rag-chatbot/crawl | GET (no jobId) | Auto-discover: is a crawl queued from an earlier page load still running? Used to resume the progress display after a page refresh. |

Status response shape (both GET forms): { active, jobId, completed, hasError, errorMessage?, pagesIndexed?, progress? }. active: false means nothing is currently running in this process (nothing queued, or an earlier job whose worker died with a server restart before finishing — see the endpoint's own comments for why that's treated as "nothing to resume" rather than stuck forever). progress is either { phase: "crawling", pagesFound } or { phase: "indexing", completed, total } — see rag-core-engine's Progress reporting.

Chat widget for your frontend

The quickest way to add chat to your site: ChatWidget, a self-contained floating launcher + panel exported from payload-plugin-rag-chatbot/client (a plain React client component — separate from the ./rsc subpath, which is specifically for Payload's own admin-panel import map). It talks to the chat endpoint above, persists the conversation id across reloads (localStorage), and follows the visitor's light/dark preference automatically. Live preview, light + dark, full prop table.

// e.g. app/(frontend)/layout.tsx
import { ChatWidget } from "payload-plugin-rag-chatbot/client";

export default function Layout({ children }: { children: React.ReactNode }) {
  return (
    <>
      {children}
      <ChatWidget title="Acme Support" subtitle="Usually replies in seconds" accentColor="#6d5bf6" />
    </>
  );
}

| Prop | Default | Notes | | --- | --- | --- | | apiPath | "/api/rag-chatbot/chat" | Override if your Payload routes.api isn't the default. | | title / subtitle | "Chat" / none | Header text. | | welcomeMessage | a generic greeting | Shown before the visitor sends anything. | | placeholder | "Type your question…" | Input placeholder. | | accentColor | "#6d5bf6" | Any CSS color — drives the launcher, header, and user bubbles. | | theme | "auto" | "light" \| "dark" \| "auto" — "auto" follows prefers-color-scheme. | | position | "bottom-right" | "bottom-right" \| "bottom-left". | | defaultOpen | false | Render the panel open on mount instead of collapsed to the launcher. | | storageKey | "rag-chat-conversation-id" | Base localStorage key — persists both the conversation id (under this key) and the visible transcript (under `${storageKey}-messages`, capped at the last 50 messages) across page reloads. Pass null to disable persistence (fresh conversation every load). |

Prefer to build your own UI instead? The widget is just a thin client over the same public endpoint — call it from anywhere:

let conversationId: string | undefined;

async function ask(question: string) {
  const res = await fetch("/api/rag-chatbot/chat", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ question, conversationId }),
  });
  const { answer, sources, conversationId: newConversationId } = await res.json();
  conversationId = newConversationId; // carry forward so the next call continues this conversation
  return { answer, sources };
}

Provider selection

rag-core-engine exposes three registries — embeddingProviders, llmProviders, vectorStores — each mapping a string name to a factory function (see src/providers/registry.ts). Today each registry ships exactly one entry: "voyage", "anthropic", and "pgvector". This plugin picks a provider for each slot by name, so adding support for a new provider is a two-step, core-first change:

1. In rag-core-engine, implement the provider (matching EmbeddingProvider / LLMProvider / VectorStore from src/types.ts) and register it — embeddingProviders.register("openai", openaiEmbeddingProvider), for example.
2. Here, set embedding.provider (or llm.provider / vectorStore.provider) to that name — in config or via the matching env var — plus whatever apiKey/model/options it needs. No other plugin code changes.

You can also .register() a provider of your own (private to your app, or one core doesn't ship) the same way, from your own payload.config.ts, before the plugin builds its pipeline — core's registries are plain mutable exports, not sealed to core's own providers.

Resolution order, per slot (embedding / llm / vectorStore), applied in src/pipeline.ts's resolveProviderConfig:

1. The explicit option (embedding.provider, embedding.apiKey, ...).
2. A slot-wide env var: $RAG_EMBEDDING_PROVIDER / $RAG_EMBEDDING_API_KEY (and $RAG_LLM_*, $RAG_VECTOR_STORE_PROVIDER for the other slots).
3. For apiKey only: a provider-specific env var, from a small built-in map ("voyage" → $VOYAGE_API_KEY, "anthropic" → $ANTHROPIC_API_KEY) — extend DEFAULT_API_KEY_ENV_VARS in pipeline.ts when you register a new provider, if you want the same zero-config pickup for it.
4. A built-in default: "voyage" for embedding, "anthropic" for llm, "pgvector" for vectorStore.

Switching to a registered "openai" embedding provider, purely via env vars, with no code change:

RAG_EMBEDDING_PROVIDER=openai
RAG_EMBEDDING_API_KEY=sk-...
RAG_EMBEDDING_MODEL=text-embedding-3-small

Environment variables

| Variable | Used by | | --- | --- | | DATABASE_URI | Fallback for databaseUri plugin option, and read directly (required, no fallback) by three admin components: CrawlStatusView, CrawledPageChunksField, KnowledgeBaseChunksField | | RAG_EMBEDDING_PROVIDER, RAG_EMBEDDING_API_KEY, RAG_EMBEDDING_MODEL | Fallbacks for embedding.provider/.apiKey/.model — see Provider selection. | | VOYAGE_API_KEY | Fallback for embedding.apiKey specifically when the embedding provider is (or defaults to) "voyage". | | RAG_LLM_PROVIDER, RAG_LLM_API_KEY, RAG_LLM_MODEL | Fallbacks for llm.provider/.apiKey/.model. | | ANTHROPIC_API_KEY | Fallback for llm.apiKey when the LLM provider is (or defaults to) "anthropic", and for pdfExtraction.apiKey under the same condition. | | RAG_VECTOR_STORE_PROVIDER | Fallback for vectorStore.provider. |

Set DATABASE_URI even if you already set databaseUri explicitly in code (e.g. databaseUri: process.env.DATABASE_URL). Endpoints, jobs, and collection hooks all receive the plugin's resolved options and work fine either way — but the three admin components listed above are registered as static admin.components string paths with no way to receive that resolved config, so they can only read process.env.DATABASE_URI directly, under exactly that name. If your app's own Postgres adapter uses a different env var name (DATABASE_URL is Payload's own convention, and what every example in this README uses), set DATABASE_URI too, to the same value — otherwise those three components silently show "no data yet" / "could not load" instead of erroring, which is easy to miss.

Every env var above is a fallback — an explicit option always wins, and none of this reaches into rag-core-engine itself, which still never reads process.env (see its design boundary). This plugin is the one layer that reads env vars, purely as an opt-in convenience on top of explicit config.

Relationship to rag-core-engine

This package is a thin adapter — all crawling/chunking/embedding/retrieval logic lives in rag-core-engine (a normal published dependency, see USING_CORE_PACKAGE.md for how this package calls into it). If you need to use the same RAG engine outside of Payload, see rag-core-engine's README.

Development

npm run build
npm run typecheck

This repo is standalone — rag-core-engine is installed from npm like any other dependency, no separate build step needed. To develop against an unpublished local rag-core-engine change, use pnpm link (or npm link) to point node_modules/rag-core-engine at your local checkout instead.