Ingest Pipeline

A Node.js CLI tool for traversing files recursively based on glob patterns and sending file contents to a knowledge base API. This tool is designed to be integrated into CI/CD pipelines for automating file ingestion processes.

Features

• Traverses files recursively based on glob patterns
• Respects .gitignore rules automatically to avoid processing unwanted files
• Securely authenticates with API token
• Provides detailed logging of the ingestion process
• Optimized for CI/CD pipeline integration

Installation

# Install globally
npm install -g @trendyol-js/ingest

# Or use directly with npx (recommended for CI/CD pipelines)
npx @trendyol-js/ingest [options]

Usage

Basic Usage

npx @trendyol-js/ingest --matcher "**/*.js" --identifier "your-identifier" --api-token "your-api-token" --host "https://your-api-host.com"

With Shorthand Options

npx @trendyol-js/ingest -m "**/*.js" -i "your-identifier" -t "your-api-token" -h "https://your-api-host.com"

Options Reference

| Option | Shorthand | Description | Default | Required | | ----------------------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | -------- | | --matcher <pattern> | -m | Glob pattern(s) for matching files to process. Supports standard glob patterns like **/*.js or src/**/*.{js,ts}. | - | Yes | | --identifier <string> | -i | Unique identifier for your knowledge base. This is used to route the files to the correct destination in the API. | - | Yes | | --api-token <token> | -t | Authentication token for the API. Keep this secure and use CI/CD secrets when possible. | - | Yes | | --host <url> | -h | Host URL for the API endpoint. Change this if you're using a custom endpoint or environment. | - | Yes | | --agent <name> | -a | Agent name that will be passed in the x-tyai-agent header. Useful for tracking or filtering API requests. If not provided, defaults to "project-{CI_PROJECT_ID}" in CI environments or "project-local" otherwise. | Auto-detected | No |

CI/CD Integration Examples

GitHub Actions

jobs:
  ingest:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Set up Node.js
        uses: actions/setup-node@v3
        with:
          node-version: "18"

      - name: Run Ingest Pipeline
        run: npx @trendyol-js/ingest -m "**/*.{js,ts}" -i "${{ secrets.KB_IDENTIFIER }}" -t "${{ secrets.API_TOKEN }}" -h "${{ secrets.API_HOST }}"

GitLab CI

ingest-job:
  image: node:18-alpine
  script:
    - npx @trendyol-js/ingest --matcher "**/*.{js,ts}" --identifier "$KB_IDENTIFIER" --api-token "$API_TOKEN" --host "$API_HOST"
  variables:
    KB_IDENTIFIER: $KB_IDENTIFIER
    API_TOKEN: $API_TOKEN
    API_HOST: $API_HOST

Jenkins Pipeline

pipeline {
    agent {
        docker {
            image 'node:18-alpine'
        }
    }

    stages {
        stage('Ingest') {
            steps {
                sh 'npx @trendyol-js/ingest -m "**/*.{js,ts}" -i "${KB_IDENTIFIER}" -t "${API_TOKEN}" -h "${API_HOST}"'
            }
        }
    }
}

Advanced Usage

Multiple File Types

To process multiple file types, use glob pattern sets:

npx @trendyol-js/ingest -m "**/*.{js,ts,jsx,tsx}" -i "your-identifier" -t "your-api-token" -h "https://your-api-host.com"

Custom Host

For targeting a specific environment:

npx @trendyol-js/ingest -m "**/*.js" -i "your-identifier" -t "your-api-token" -h "https://your-custom-api-host.com"

Troubleshooting

• If you encounter authentication errors, verify that your API token is correct and not expired
• For file access issues, ensure your CI/CD runner has the necessary permissions to read the files
• To debug which files are being processed, examine the console output for the list of matched files
• Check for potential .gitignore conflicts if expected files are not being processed

Development

# Clone the repository
git clone https://github.com/Trendyol/ingest-pipeline.git
cd ingest-pipeline

# Install dependencies
npm install

# Link the package locally for testing
npm link

# Run the application
npm start -- --matcher "**/*.js" --identifier "test" --api-token "token" --host "https://your-api-host.com"

License

ISC