@bernierllc/contentful-sync-service

Full export + incremental webhook sync to local storage for Contentful content.

Overview

ContentfulSyncService syncs Contentful content to local storage for ML training pipelines. It performs an initial full export using contentful-export, then maintains sync via CDA Sync API + webhook processing. Provides a structured content mirror for training without hitting Contentful APIs during training.

Features

• 🔄 Initial Full Export - Complete space export to local storage
• 📦 Incremental Sync - Delta sync using CDA Sync API
• 🪝 Webhook Processing - Real-time updates via webhooks
• 💾 Abstract Storage - Pluggable storage adapters
• 📊 Sync Statistics - Track sync performance
• 🔌 NeverHub Integration - Optional integration with NeverHub
• 📝 Comprehensive Logging - Detailed operation logs

Installation

npm install @bernierllc/contentful-sync-service

Quick Start

import {
  ContentfulSyncService,
  InMemoryContentStorage,
  InMemorySyncStateStorage
} from '@bernierllc/contentful-sync-service';

// Create storage instances
const contentStorage = new InMemoryContentStorage();
const syncStateStorage = new InMemorySyncStateStorage();

// Create service
const service = new ContentfulSyncService({
  spaceId: 'your-space-id',
  environmentId: 'master',
  accessToken: 'your-cda-token',
  contentStorage,
  syncStateStorage,
  webhookSecret: 'your-webhook-secret' // Optional
});

// Initialize
await service.initialize();

// Perform initial sync
const result = await service.initialSync('your-management-token');
console.log(`Synced ${result.stats.entriesSynced} entries`);

// Start auto-sync
service.startAutoSync();

Storage Adapters

Built-in Adapters

The package provides in-memory adapters for testing:

import {
  InMemoryContentStorage,
  InMemorySyncStateStorage
} from '@bernierllc/contentful-sync-service';

const contentStorage = new InMemoryContentStorage();
const syncStateStorage = new InMemorySyncStateStorage();

Custom Storage Adapters

Implement the storage interfaces for your database:

import type { ContentStorage, SyncStateStorage } from '@bernierllc/contentful-sync-service';
import type { Entry, Asset, ContentType } from '@bernierllc/contentful-types';

class PostgreSQLContentStorage implements ContentStorage {
  async storeContentTypes(contentTypes: ContentType[]): Promise<void> {
    // Store in PostgreSQL
  }

  async storeEntries(entries: Entry[]): Promise<void> {
    // Store in PostgreSQL
  }

  async storeAssets(assets: Asset[]): Promise<void> {
    // Store in PostgreSQL
  }

  async updateEntry(entry: Entry): Promise<void> {
    // Update in PostgreSQL
  }

  async updateAsset(asset: Asset): Promise<void> {
    // Update in PostgreSQL
  }

  async deleteEntry(entryId: string): Promise<void> {
    // Delete from PostgreSQL
  }

  async deleteAsset(assetId: string): Promise<void> {
    // Delete from PostgreSQL
  }

  async getEntry(entryId: string): Promise<Entry | null> {
    // Retrieve from PostgreSQL
  }

  async getAsset(assetId: string): Promise<Asset | null> {
    // Retrieve from PostgreSQL
  }

  async clearSpace(spaceId: string): Promise<void> {
    // Clear space data from PostgreSQL
  }
}

class PostgreSQLSyncStateStorage implements SyncStateStorage {
  async getSyncToken(spaceId: string): Promise<string | null> {
    // Get from PostgreSQL
  }

  async setSyncToken(spaceId: string, token: string): Promise<void> {
    // Store in PostgreSQL
  }

  async clearSyncToken(spaceId: string): Promise<void> {
    // Clear from PostgreSQL
  }
}

API Reference

ContentfulSyncService

Constructor

new ContentfulSyncService(config: SyncServiceConfig)

Config Options:

• spaceId (string) - Contentful space ID
• environmentId (string) - Contentful environment ID
• accessToken (string) - CDA access token
• contentStorage (ContentStorage) - Content storage implementation
• syncStateStorage (SyncStateStorage) - Sync state storage implementation
• webhookSecret (string, optional) - Webhook validation secret
• syncInterval (number, optional) - Auto-sync interval in ms (default: 300000)
• neverhubEnabled (boolean, optional) - Enable NeverHub integration

Methods

initialize(): Promise<void>

Initialize the service and optionally connect to NeverHub.

await service.initialize();

initialSync(managementToken: string): Promise<SyncResult>

Perform initial full export from Contentful.

const result = await service.initialSync('cma-token');
console.log(result.stats.entriesSynced); // Number of entries synced

incrementalSync(): Promise<SyncResult>

Perform incremental sync using the CDA Sync API.

const result = await service.incrementalSync();
console.log(result.stats.itemsDeleted); // Number of items deleted

processWebhook(body: string, headers: Record<string, string>): Promise<void>

Process a webhook event from Contentful.

app.post('/webhook', async (req, res) => {
  await service.processWebhook(
    JSON.stringify(req.body),
    req.headers
  );
  res.sendStatus(200);
});

startAutoSync(): void

Start automatic incremental sync at the configured interval.

service.startAutoSync();

stopAutoSync(): void

Stop automatic sync.

service.stopAutoSync();

shutdown(): Promise<void>

Shutdown the service cleanly.

await service.shutdown();

Usage Examples

Basic Sync

import {
  ContentfulSyncService,
  InMemoryContentStorage,
  InMemorySyncStateStorage
} from '@bernierllc/contentful-sync-service';

const service = new ContentfulSyncService({
  spaceId: 'my-space',
  environmentId: 'master',
  accessToken: 'cda-token',
  contentStorage: new InMemoryContentStorage(),
  syncStateStorage: new InMemorySyncStateStorage()
});

await service.initialize();
await service.initialSync('cma-token');
await service.incrementalSync();

Auto-Sync

// Start auto-sync every 5 minutes
service.startAutoSync();

// Later...
service.stopAutoSync();

Webhook Processing

import express from 'express';

const app = express();

const service = new ContentfulSyncService({
  spaceId: 'my-space',
  environmentId: 'master',
  accessToken: 'cda-token',
  contentStorage: new InMemoryContentStorage(),
  syncStateStorage: new InMemorySyncStateStorage(),
  webhookSecret: 'webhook-secret'
});

await service.initialize();

app.post('/contentful-webhook', express.json(), async (req, res) => {
  try {
    await service.processWebhook(
      JSON.stringify(req.body),
      req.headers as Record<string, string>
    );
    res.sendStatus(200);
  } catch (error) {
    console.error('Webhook processing failed:', error);
    res.sendStatus(500);
  }
});

app.listen(3000);

NeverHub Integration

const service = new ContentfulSyncService({
  spaceId: 'my-space',
  environmentId: 'master',
  accessToken: 'cda-token',
  contentStorage: new InMemoryContentStorage(),
  syncStateStorage: new InMemorySyncStateStorage(),
  neverhubEnabled: true // Enable NeverHub
});

await service.initialize();
// NeverHub registration happens automatically if available

TypeScript Support

Full TypeScript support with comprehensive type definitions:

import type {
  ContentStorage,
  SyncStateStorage,
  SyncServiceConfig,
  SyncResult,
  SyncStats
} from '@bernierllc/contentful-sync-service';

Error Handling

const result = await service.initialSync('cma-token');

if (!result.success) {
  console.error('Sync failed:', result.error);
} else {
  console.log('Sync succeeded:', result.stats);
}

License

Copyright (c) 2025 Bernier LLC

This file is licensed to the client under a limited-use license. The client may use and modify this code only within the scope of the project it was delivered for. Redistribution or use in other products or commercial offerings is not permitted without written consent from Bernier LLC.