navu-catalog

Navu service for harvesting, indexing, and querying customer-specific product catalogs.

Shared infrastructure (configuration, logging, MongoDB, HTTP, lifecycle, message bus) lives in @navu-co/navu-core. This repo holds catalog-domain code: site harvesters, manifest-driven indexing, OpenSearch query, and intrasystem REST for navu-server AI tools.

Dependencies: @hivepoint/navu-apis, @navu-co/navu-core, @opensearch-project/opensearch, openai. Publish navu-apis before deploying; then npm update @hivepoint/navu-apis here (no file: tarballs in production).

Documentation

| Document | Purpose | |----------|---------| | docs/catalog-pipeline-guidelines.md | Primary — site onboarding, harvest/index rules, Phase D audits | | docs/catalog-search-architecture.md | Target search model (manifest, nested variants, three tools) | | docs/catalog-overview.md | End-to-end overview for engineers and integrators | | docs/catalog-site-discovery-playbook.md | Phase 0 upstream discovery from homepage URL | | .cursor/skills/catalog-site-review/SKILL.md | Agent onboarding workflow (Phases 0–D) | | docs/catalog-rebuild-audit.md | Historical Phase 0 rebuild audit (pre-v2 engine) |

Legacy EAV pipeline code was removed from this repo. Active v2 site pipelines live under src/site-catalogs/<site>/.

Development

| Script | Purpose | |--------|---------| | npm run build | Compile TypeScript (src/ → dist/) | | npm run test | Unit tests (no Mongo/OpenSearch required) | | npm run start:standalone:v2 | Long-running service with file-based catalog sites (v2 DB, port 21224) | | npm run harvest:<site>:v2 | Harvest one site → catalog.harvestProducts | | npm run build-index-documents:<site>:v2 | Build Mongo index mirrors (no OpenSearch required) | | npm run rebuild-index:<site>:v2 | Mirror + OpenSearch bulk index (--skip-embeddings supported) | | npm run bootstrap:<site>:v2 | Harvest + build index documents |

Per-site script suffixes: vimvigr, hamilton, ciscoeagle, atlascopco, gantrade, customtruck, customtruck-parts, tpcwire, alpsupply, atkore. See package.json for audit and reprocess variants.

Standalone configuration

v2 (current engine): use standalone.config.v2.json and standalone.catalog-sites.v2.json (Mongo database navu-catalog-standalone-v2, port 21224).

v1 (legacy): standalone.config.json and standalone.catalog-sites.json remain for older EAV-era sites only. Prefer v2 for all active development.

Override catalog-sites files — pass a different --catalog-sites-file for site-specific harvest modes:

| File | Purpose | |------|---------| | standalone.catalog-sites.atlascopco-pdp.v2.json | Atlas Copco with pdpEnrichment: true (harvest:atlascopco:pdp:v2) | | standalone.catalog-sites.atkore-sample.v2.json | Atkore sample harvest (maxFamilies: 500) | | standalone.catalog-sites.atkore-delta.v2.json | Atkore delta harvest (deltaHarvestByModifiedOn: true) |

Copy secrets.json.example to secrets.json (gitignored). Platform secrets (elastic, openai, …) resolve __SECRET(...) placeholders in config files. Per-catalog credentials in standalone.catalog-sites.v2.json use the same __SECRET(...) pattern. OpenSearch indexing is optional in dev — Mongo mirrors are always written.

npm run bootstrap:vimvigr:v2

MongoDB collections (v2 engine)

| Collection | Purpose | |------------|---------| | catalog.sites | Per-site harvest coordination + language | | catalog.siteCatalogs | Per-catalog locks, status, catalogMeta.toolName | | catalog.harvestProducts | Harvest output — CatalogHarvestProduct with typed fields + nested variants[] | | catalog.manifests | Per-site CatalogManifest (properties, scope, facets) | | catalog.openSearchDocuments | Mirror of indexed docs (document field; no narrative_embedding) | | catalog.toolDefinitions | AI tool schemas (find-products-discovery, -query, -get-details) |

Query time uses OpenSearch when credentials are configured. Inspect harvest payloads in catalog.harvestProducts and indexed JSON in catalog.openSearchDocuments.

Intrasystem API

Base path: /d/navu-catalog/int/ (see @hivepoint/navu-apis INTRASYSTEM_NAVU_CATALOG_SERVICE_API_SPECS).

| Endpoint | Purpose | |----------|---------| | POST …/discovery | Facet discovery (scoped aggregations) | | POST …/query | Hybrid search — product or variant rows | | POST …/get-details | Hydrate full product + variants by id/SKU | | POST …/rebuild-catalog-index | Rebuild mirrors + OpenSearch from harvest | | POST …/list-site-catalogs | Catalog visibility and counts | | POST …/get-catalog-tools | Tool definitions for navu-server |

Default HTTP port: 21223 (v2 standalone: 21224).

Migrated sites (v2 engine)

All production site pipelines are registered in standalone.catalog-sites.v2.json and wired in src/site-catalogs/site-catalog-module-registry.ts:

| Site | Code | Module | Upstream | |------|------|--------|----------| | Vim & Vigr | S10VVGR | vim-vigr/ | Shopify | | Hamilton | S10HAMT | hamilton/ | Algolia + PDP SSR | | Cisco-Eagle | S10CEGL | cisco-eagle/ | XML feed | | Atlas Copco | S10ATCO | atlascopco/ | Algolia + Next __NEXT_DATA__ PDP | | Gantrade | S10GNTR | gantrade/ | HubDB | | Custom Truck (products) | S10CTOS | customtruck/ | SSR HTML | | Custom Truck (parts) | S10CTOS | customtruck/parts/ | SSR HTML | | Alp Supply | S10ALPS | alpsupply/ | Sitemap + SSR | | TPC Wire | S10TPCW | tpcwire/ | Local JSON (sources/tpcwire/vol8b/) | | Atkore | S10ATKR | atkore/ | HubDB |

Shared upstream clients: src/reusable/ (Shopify, Algolia, HubDB, Next.js, harvest staging).

Site-specific recon notes: docs/sites/<site>/RECON.md (where available).