AssistantIngest

Content ingestion and document-processing pipeline for StrongAI.

Overview

AssistantIngest/ converts source material into formats the StrongAI platform can search, validate, and load into downstream systems.

Primary responsibilities:

• Convert documents such as PDF, DOCX, and DOC into Markdown.
• Extract programming, rotations, events, and related structured content.
• Count tokens and prepare chunks for LLM workflows.
• Load or reconcile data with Fluid/Caucus-backed operational stores.
• Provide container-management and migration utilities.

Runtime and Tooling

• Node.js 22.x
• npm
• make for the preferred ingestion workflow
• API credentials for the specific pipeline you are running

Preferred Workflow

For normal StrongAI development, use the Makefile targets first. The Makefile captures the supported end-to-end flows and keeps the steps reproducible.

make all
make count-tokens
make help

Useful validation targets include:

make validate-rotations-local
make validate-rotations-prod
make validate-programming-local
make validate-programming-prod

Installation

From the AssistantIngest repo:

npm install
npm run build

Common npm Commands

The repo also exposes targeted npm scripts for advanced or one-off tasks:

npm test
npm run test:ci
npm run test:integration
npm run validate-rotations -- --personality kCoachNotesAssistant --local-container
npm run validate-programming -- --personality kCoachNotesAssistant --production
npm run cleanup-containers -- --help

Use these direct scripts when you need a focused operation; prefer make when you are running a documented ingestion flow.

Environment Variables

The exact variables depend on the task you are running. Common examples:

• LLAMA_CLOUD_API_KEY for document parsing
• LLAMA_CLOUD_BASE_URL when using a non-default LlamaCloud region
• GOOGLE_CLIENT_EMAIL and GOOGLE_PRIVATE_KEY for Google Docs or Sheets extraction
• FLUID_TENANT_ID and FLUID_TENANT_KEY for container-management workflows
• Azure subscription and resource variables for management operations when required

If a command touches StrongAI server-side systems, make sure your shared workspace env is loaded as described in AssistantBuild/README.md.

Common Flows

Document Parsing and Markdown Output

The Makefile coordinates the standard content-processing flow. For ad hoc parsing, package scripts such as parse-rotations-xlsx, extract-events, and process-workout-classifications are available, but they should be treated as lower-level tools.

Week Extraction

The week extractor downloads a Google Doc, parses it with LlamaParse, and extracts week tables while preserving day-level notes.

Relevant inputs:

• googleFileId
• targetWeekQuery
• siteHint
• downloadFormat

Container Management

This repo also owns operational tools for:

• container cleanup
• backup and restore
• date migrations
• local-to-production synchronization tasks

Use the npm scripts under src/container-management/ for those workflows.

Testing

This repo follows the StrongAI testing standard and uses Mocha with expect.

npm run test:ci
npm run test:integration

Troubleshooting

• Region mismatch errors from LlamaCloud usually mean LLAMA_CLOUD_BASE_URL does not match the key you are using.
• Validation commands that appear to find the wrong week often indicate inconsistent document date formats.
• Container-management commands typically need both Fluid credentials and Azure management credentials when deleting or listing production resources.

For date-format guidance, see docs/DATE-FORMAT.md.

Related Docs

• Root README
• Workspace AGENTS
• AssistantIngest AGENTS