Content Store

SDK for fetching CMS and translation bundles from Amazon S3 and, for CMS data, querying them locally from the filesystem. CMS bundles come from syncs of Contentful or Sanity; translation bundles come from Lingohub projects synced to S3 (see the Server & CLI README for the upload pipeline).

For the Express API, server CLI, scheduling, and deployment, see the Server & CLI README.

Package entry points

• @tandem-language-exchange/content-store (default) — types only at runtime. Safe to import from shared code that Next.js, Vite, or Turbopack may bundle for the browser.

• @tandem-language-exchange/content-store/node — ContentStoreSDK, fetchCmsBundles, fetchTranslationBundles, fetchMergedTranslationBundles, queryCmsBundle, ContentStore, getDefaultS3RetryConfig, and trimDepth. Real implementations use the filesystem and S3 and run only under the Node (node) export condition (Route Handlers, getServerSideProps, CLI, etc.).

  For browser bundles (including anything imported from _app.tsx, client components, or shared modules that reach the client graph), bundlers should resolve the browser / edge-light conditions to a stub that does not import fs. That stub throws if you call server-only APIs; trimDepth is fully implemented and safe on the client.

  Prefer not importing /node from files that _app or layouts load: keep SDK usage in server-only modules and pass data in as props. If the stub throws at runtime, move the import to server-only code.

Installation

npm install @tandem-language-exchange/content-store

Use the same scoped name in package.json dependencies and in npx / import paths.

Initialisation

import { ContentStoreSDK } from '@tandem-language-exchange/content-store/node';

const sdk = new ContentStoreSDK({
  s3: {
    bucket: 'beta-content-store',
    region: 'eu-central-1',
    accessKeyId: process.env.CONTENT_STORE_AWS_ACCESS_KEY!,
    secretAccessKey: process.env.CONTENT_STORE_AWS_SECRET_ACCESS_KEY!,
  },
  outputDir: './content-cache',
});

| Option | Description | | --- | --- | | s3.bucket | S3 bucket where content bundles are stored | | s3.region | AWS region of the bucket | | s3.accessKeyId | AWS IAM access key | | s3.secretAccessKey | AWS IAM secret key | | outputDir | Local directory where bundle JSON files are written |

S3 config via environment variables

The CLI commands (fetch-content-bundles, query-cms) read S3 credentials automatically from the following environment variables — no code needed:

| Variable | Description | | --- | --- | | CONTENT_STORE_S3_BUCKET | S3 bucket name | | CONTENT_STORE_S3_REGION | AWS region (default: eu-central-1) | | CONTENT_STORE_AWS_ACCESS_KEY | AWS IAM access key | | CONTENT_STORE_AWS_SECRET_ACCESS_KEY | AWS IAM secret key |

When using the SDK class directly, pass the values explicitly as shown above. You can load them from env vars yourself:

const sdk = new ContentStoreSDK({
  s3: {
    bucket: process.env.CONTENT_STORE_S3_BUCKET!,
    region: process.env.CONTENT_STORE_S3_REGION!,
    accessKeyId: process.env.AWS_ACCESS_KEY!,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
  },
  outputDir: './content-cache',
});

fetchCmsBundles(options)

Downloads the latest content bundles from S3 and saves them as JSON files to outputDir.

const files = await sdk.fetchCmsBundles({
  cms: 'contentful',
  contentTypes: ['gridLayout', 'iconWithText', 'page'],
});

Parameters:

| Field | Type | Description | | --- | --- | --- | | cms | 'contentful' \| 'sanity' | Which CMS the bundles were synced from | | contentTypes | string[] | Content types to download | | retry | S3RetryConfig | Optional. Overrides S3 download retries. |

Returns: Record<string, string> — a map of content type to absolute file path.

{
  gridLayout: '/abs/path/content-contentful-gridLayout.json',
  iconWithText: '/abs/path/content-contentful-iconWithText.json',
  page: '/abs/path/content-contentful-page.json'
}

Files are written to outputDir with the naming pattern {cms}-{contentType}.json.

Staging content refresh endpoint (Next.js web-site / web-app)

On staging (pages rebuild on request), content-store can tell your app to pull fresh bundles from S3 without an Azure pipeline build. Your Next.js app exposes an HTTP endpoint; content-store calls it after a successful CMS sync (or via POST /notifyContentRefresh on content-store).

Request (from content-store):

POST /api/internal/content-refresh
Authorization: Basic <base64(username:password)>
Content-Type: application/json

{"scope":"cms","cms":"contentful","content_types":["page","banner"]}

The handler runs fetchCmsBundles / fetchTranslationBundles and returns paths written under outputDir. You can also call sdk.refreshContent(body, defaults) directly (no HTTP).

Import @tandem-language-exchange/content-store/node only from server-only code (pages/api/*, app/api/**/route.ts, or modules they import) — not from _app, layouts, or client components.

Shared SDK setup (use in both examples below):

import { ContentStoreSDK } from '@tandem-language-exchange/content-store/node';

export const contentStoreSdk = new ContentStoreSDK({
  s3: {
    bucket: process.env.CONTENT_STORE_S3_BUCKET!,
    region: process.env.CONTENT_STORE_S3_REGION ?? 'eu-central-1',
    accessKeyId: process.env.CONTENT_STORE_AWS_ACCESS_KEY!,
    secretAccessKey: process.env.CONTENT_STORE_AWS_SECRET_ACCESS_KEY!,
  },
  outputDir: process.env.CONTENT_CACHE_DIR ?? './content-cache',
});

export const contentRefreshDefaults = {
  scope: 'cms' as const,
  cms: 'contentful' as const,
  contentTypes: ['page', 'banner', 'cookieBanner', 'downloadPage'],
  translationProjects: ['tandem-(website)'],
};

Pages Router (pages/api)

pages/api/internal/content-refresh.ts

import type { NextApiRequest, NextApiResponse } from 'next';
import { createNextPagesApiContentRefreshHandler } from '@tandem-language-exchange/content-store/node';
import { contentRefreshDefaults, contentStoreSdk } from '../../lib/content-store';

const handler = createNextPagesApiContentRefreshHandler({
  sdk: contentStoreSdk,
  basicAuthBase64: process.env.CONTENT_REFRESH_BASIC_AUTH!,
  defaults: contentRefreshDefaults,
});

export default async function contentRefresh(
  req: NextApiRequest,
  res: NextApiResponse,
) {
  return handler(req, res);
}

App Router (app/api/.../route.ts)

app/api/internal/content-refresh/route.ts

import { handleNextAppRouterContentRefresh } from '@tandem-language-exchange/content-store/node';
import { contentRefreshDefaults, contentStoreSdk } from '@/lib/content-store';

export async function POST(request: Request) {
  return handleNextAppRouterContentRefresh(request, {
    sdk: contentStoreSdk,
    basicAuthBase64: process.env.CONTENT_REFRESH_BASIC_AUTH!,
    defaults: contentRefreshDefaults,
  });
}

Point content-store staging at the deployed URL, e.g.
CONTENT_REFRESH_WEB_SITE_URL=https://staging.example.com/api/internal/content-refresh

After refresh, load bundles from outputDir in getStaticProps, getServerSideProps, or Server Components on the next request.

Configure content-store (staging instance)

| Variable | Description | |--------------------------------| --- | | CONTENT_REFRESH_WEB_SITE_URL | Full URL to web-site refresh endpoint | | CONTENT_REFRESH_WEB_APP_URL | Full URL to web-app refresh endpoint | | CONTENT_REFRESH_BASIC_AUTH | Base64 of staging site basic-auth username:password (no Basic prefix). Generate with encodeBasicAuthCredentials from the package or printf '%s' 'user:pass' \| base64 |

After POST /syncCmsContent completes successfully on a beta/staging instance, content-store notifies every configured URL automatically.

fetchTranslationBundles(options)

Downloads translation objects from S3. The sync stores each Lingohub file verbatim (raw UTF-8); this call parses each file (JSON / .strings / Android XML per src/shared/lingohub.ts) and writes normalized JSON under outputDir (see file naming below).

const files = await sdk.fetchTranslationBundles({
  projects: {
    'tandem': [],                          // all resources
    'tandem-(website)': ['main', 'ai'],    // only "main" and "ai" resources
  },
  locales: ['en', 'de'], // omit or leave empty to use the package default locale list
});

Parameters (extends TranslationFilterConfig):

| Field | Type | Description | | --- | --- | --- | | projects | Record<string, string[]> | Map of Lingohub project id to resource keys. An empty array fetches all resources for that project; a populated array fetches only the listed resources (matched against the resource field in src/shared/lingohub.ts). | | locales | string[] | Optional. Locale codes to fetch (e.g. pt-br, zh-hans). If omitted or empty, a built-in default list is used. | | structure | 'flat' \| 'nested' | Optional (default 'flat'). When 'nested', dotted keys in translation files are expanded into nested objects (e.g. "General.tryAgain" becomes { "General": { "tryAgain": … } }). | | retry | S3RetryConfig | Optional. Overrides S3 download retries. |

Returns: TranslationBundleInfo — nested map project → S3 object key → absolute file path on disk.

{
  'tandem-(website)': {
    'lingohub-tandem-(website).en.json': '/abs/path/to/content-cache/lingohub-tandem-(website).en.json',
    'lingohub-tandem-(website).AI.en.json': '/abs/path/to/...'
  }
}

S3 object keys follow lingohub-{project}.{fileName} where {fileName} is the Lingohub resource template with [locale] replaced by the mapped locale when a resource defines localeMapping (same rules as the server sync). On disk, non-.json keys gain a trailing .json (e.g. …en.strings → …en.strings.json) containing the parsed structure as JSON.

Downloads are run in parallel (per project); S3 download retries apply by default.

fetchMergedTranslationBundles(options)

Same projects, locales, and retry as fetchTranslationBundles. Downloads and parses every matching resource, flattens each file to string key/value pairs, then merges all pairs per catalog locale into a single file {locale}.json in outputDir (e.g. en.json). Duplicate keys across resources or projects: last wins (order: projects → resources → locales loop).

Returns: Record<string, string> — locale code → absolute path of the merged file.

const mergedPaths = await sdk.fetchMergedTranslationBundles({
  projects: {
    'tandem-(new-website)': [],
    'tandem-(website)': ['main'],
  },
  locales: ['en'],
});
// mergedPaths.en → path to one big en.json

S3 download retries

fetchCmsBundles, fetchTranslationBundles, and fetchMergedTranslationBundles use retry + exponential backoff on transient S3/network failures (for example HTTP 503 “Slow Down”, throttling, timeouts). They do not retry clear client errors such as 404 (missing key).

Default limits are read from the environment (highest precedence first):

| Variable | Fallback | Purpose | | --- | --- | --- | | S3_RETRY_MAX_RETRIES | RETRY_MAX_RETRIES (default 5) | Maximum retry attempts after the first try | | S3_RETRY_BASE_DELAY_MS | RETRY_BASE_DELAY_MS (default 1000) | Base delay for exponential backoff | | S3_RETRY_MAX_DELAY_MS | RETRY_MAX_DELAY_MS (default 60000) | Cap on backoff delay |

Override per call with retry: { maxRetries, baseDelayMs, maxDelayMs }, or import getDefaultS3RetryConfig() to merge with your own defaults.

queryCmsBundle(cms, contentType, options?)

Reads a previously fetched bundle from the local filesystem and returns a filtered, shaped result set.

const results = await sdk.queryCmsBundle('contentful', 'gridLayout', {
  fields: { columns: '2' },
  select: ['title', 'bodyBefore'],
  limit: 10,
  include: 2,
});

Parameters:

| Field | Type | Description | | --- | --- | --- | | cms | 'contentful' \| 'sanity' | CMS provider | | contentType | string | Content type to query | | options | QueryOptions | Optional filtering/shaping (see below) |

QueryOptions

All fields are optional.

| Field | Type | Description | | --- | --- | --- | | fields | Record<string, unknown> | Filter by top-level properties (see Filtering) | | select | string[] | Properties to include in each result object | | limit | number | Maximum number of items to return | | include | number | Depth of nested references to include (see Include depth) |

Filtering

The fields option matches items by their top-level properties.

Exact match — value must be strictly equal:

{ fields: { columns: '2' } }

IN match — value must be one of the provided options:

{ fields: { variant: ['A', 'B', 'E'] } }

Multiple fields are combined with AND logic:

{ fields: { columns: '2', refsType: 'Icon with Text' } }

Include depth

The include option controls how many levels of nested referenced objects are returned. Omit it to get the full depth.

Given this bundle item:

{
  "title": "Page Title",
  "columns": "2",
  "refs": [
    {
      "heading": "Child heading",
      "icon": {
        "title": "Icon title",
        "file": { "url": "//images.ctfassets.net/..." }
      }
    }
  ]
}

include: 1 — scalar properties only, all nested objects/refs are null:

{
  "title": "Page Title",
  "columns": "2",
  "refs": null
}

include: 2 — the item including its direct refs, but refs' own nested objects are null:

{
  "title": "Page Title",
  "columns": "2",
  "refs": [
    {
      "heading": "Child heading",
      "icon": null
    }
  ]
}

include: 3 — three levels deep; icon is included but icon.file is null:

{
  "title": "Page Title",
  "columns": "2",
  "refs": [
    {
      "heading": "Child heading",
      "icon": {
        "title": "Icon title",
        "file": null
      }
    }
  ]
}

Processing order

Query options are applied in this order:

1. fields — filter the full item set
2. limit — cap the result count
3. include — trim nested depth
4. select — pick output properties

Standalone functions

The same operations are available as standalone imports (no ContentStoreSDK wrapper):

import {
  fetchCmsBundles,
  fetchTranslationBundles,
  fetchMergedTranslationBundles,
  queryCmsBundle,
  getDefaultS3RetryConfig,
  ContentStore,
} from '@tandem-language-exchange/content-store/node';

const store = new ContentStore({
  bucket: 'beta-content-store',
  region: 'eu-central-1',
  accessKeyId: process.env.CONTENT_STORE_AWS_ACCESS_KEY!,
  secretAccessKey: process.CONTENT_STORE_env.AWS_SECRET_ACCESS_KEY!,
});

await fetchCmsBundles(store, './content-cache', {
  cms: 'contentful',
  contentTypes: ['gridLayout'],
});

await fetchTranslationBundles(store, './content-cache', {
  projects: { 'tandem-(website)': [] },
  locales: ['en', 'fr'],
  retry: getDefaultS3RetryConfig(),
});

await fetchMergedTranslationBundles(store, './content-cache/merged', {
  projects: { 'tandem-(new-website)': [], 'tandem-(website)': ['main'] },
  locales: ['en'],
});

const results = await queryCmsBundle('./content-cache', 'contentful', 'gridLayout', {
  fields: { columns: '2' },
  limit: 5,
});

Full example

import { ContentStoreSDK } from '@tandem-language-exchange/content-store/node';

const sdk = new ContentStoreSDK({
  s3: {
    bucket: process.env.CONTENT_STORE_S3_BUCKET!,
    region: process.env.CONTENT_STORE_S3_REGION!,
    accessKeyId: process.env.AWS_ACCESS_KEY!,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
  },
  outputDir: './.content-cache',
});

// 1. Pull latest CMS bundles from S3 to disk
await sdk.fetchCmsBundles({
  cms: 'contentful',
  contentTypes: ['page', 'gridLayout'],
});

// Optional: pull translation bundles written by the Lingohub → S3 sync
await sdk.fetchTranslationBundles({
  projects: { 'tandem-(website)': [] },
});

// 2. Query CMS bundle locally — no further network calls
const grids = await sdk.queryCmsBundle('contentful', 'gridLayout', {
  fields: { columns: '2' },
  select: ['title', 'refs'],
  include: 2,
  limit: 5,
});

console.log(grids);

CLI

The package ships fetch-content-bundles, fetch-translation-bundles, fetch-merged-translation-bundles, list-projects, and list-resources as bin commands. The query-cms command lives at dist/client/query-cms.js; run with node node_modules/@tandem-language-exchange/content-store/dist/client/query-cms.js or via npm scripts.

fetch-content-bundles — download bundles from S3

Downloads the latest version of each requested content type bundle from S3 and writes them as JSON files to a local directory. Reads S3 credentials from environment variables (see S3 config via environment variables).

npx fetch-content-bundles --cms contentful --types gridLayout,page

| Flag | Required | Default | Description | | --- | --- | --- | --- | | --cms <provider> | Yes | | contentful or sanity | | --types <types> | Yes | | Comma-separated content types | | --output <dir> | No | ./content-cache | Local directory to write bundle files to |

Files are written as {cms}-{contentType}.json inside the output directory.

Typical use in a host app's package.json:

"scripts": {
  "fetch-content": "fetch-content-bundles --cms contentful --types gridLayout,iconWithText,page --output ./content-cache"
}

All bin tools read S3 settings from S3 config via environment variables.

fetch-translation-bundles

Both translation CLI commands accept a --config flag pointing to a JSON config file that defines which projects, resources, and locales to fetch.

Config file format (TranslationFilterConfig):

{
  "projects": {
    "tandem": [],
    "tandem-(website)": ["main", "ai"]
  },
  "locales": ["en", "de", "it"],
  "structure": "nested"
}

• projects — map of Lingohub project id to resource keys. An empty array ([]) fetches all resources for that project. A populated array fetches only the listed resources (matched against the resource field in src/shared/lingohub.ts).
• locales — optional. Omit or leave empty to use the built-in default locale list.
• structure — optional. "flat" (default) keeps dotted keys as-is; "nested" expands them into nested objects.

npx fetch-translation-bundles --config ./translation-config.json --output ./content-cache

| Flag | Required | Default | Description | | --- | --- | --- | --- | | --config <path> | Yes | | Path to a JSON config file (TranslationFilterConfig) | | --output <dir> | No | ./content-cache | Output directory |

fetch-merged-translation-bundles

Writes merged {locale}.json files (string key/value map; duplicate keys: last wins).

Uses the same config file format as fetch-translation-bundles.

npx fetch-merged-translation-bundles --config ./translation-config.json --output ./content-cache/merged

Alternatively call fetchTranslationBundles / fetchMergedTranslationBundles from a Node script, or use the server’s POST /getTranslationBundles API (see Server & CLI README).

query-cms — query a local bundle

Reads a previously fetched bundle from disk and prints JSON to stdout. This command is not a separate bin; run the built client CLI (after npm install of this package):

node node_modules/@tandem-language-exchange/content-store/dist/client/query-cms.js \
  --cms contentful --type gridLayout \
  --fields '{"columns":"2"}' \
  --select title,bodyBefore \
  --limit 5 \
  --include 2

Typical package.json shortcut:

"scripts": {
  "query:cms": "node ./node_modules/@tandem-language-exchange/content-store/dist/client/query-cms.js"
}

Then: npm run query:cms -- --cms contentful --type gridLayout …

| Flag | Required | Default | Description | | --- | --- | --- | --- | | --cms <provider> | Yes | | contentful or sanity | | --type <type> | Yes | | Content type to query | | --output <dir> | No | ./content-cache | Directory where bundles are stored | | --fields <json> | No | | JSON filter object (e.g. '{"columns":"2"}') | | --select <props> | No | | Comma-separated properties to include in results | | --limit <n> | No | | Maximum number of results | | --include <n> | No | | Depth of nested references to include |

list-projects — list available Lingohub projects

npx list-projects

Prints every project name registered in the package's Lingohub configuration.

list-resources — list resources for a project

npx list-resources 'tandem-(website)'

Prints every resource name (and its file name template) for the given project. If the project argument is omitted, an interactive numbered list of projects is presented for selection.