@holokai/holo-etl

ETL pipelines for the Holo model catalog. Fetches model metadata from external sources, matches against the internal dictionary, generates aliases, and publishes canonical model entries.

Installation

npm install @holokai/holo-etl

Subpath Exports

| Subpath | Description | |-----------------------------|---------------------------------------------------------------| | @holokai/holo-etl | Root — re-exports everything | | @holokai/holo-etl/sources | External data source fetchers | | @holokai/holo-etl/sync | Dictionary sync, alias generation, canonicalization, pipeline |

Data Sources

The sources subpath fetches model metadata from external catalogs:

import {fetchOpenRouterModels, fetchLiteLLMModels, fetchModelsDevModels} from '@holokai/holo-etl/sources';

const openRouterModels = await fetchOpenRouterModels();
const liteLLMModels = await fetchLiteLLMModels();
const modelsDevModels = await fetchModelsDevModels();

Each source returns raw API data that can be converted to a unified SourceModel format:

import {toSourceModels, toLiteLLMSourceModels, toModelsDevSourceModels} from '@holokai/holo-etl/sources';

Sync Pipeline

The sync subpath provides the full ETL pipeline:

import {runPipeline} from '@holokai/holo-etl/sync';

const result = await runPipeline(deps);
// result: { fetched, matched, enriched, published, aliases }

Individual steps:

| Function | Description | |-------------------------------------|--------------------------------------------------------| | syncDictionary(deps) | Match source observations against the model dictionary | | enrichUnmatchedObservations(deps) | Enrich unmatched models with metadata | | generateAliases(models) | Generate model name aliases for fuzzy matching | | buildCanonicalCandidates(models) | Build canonical model entries from matched data | | publishCanonicalCandidates(deps) | Publish finalized entries to the database |

License

MIT