@partnersync/conduit

A modern, type-safe ETL framework for building data integration pipelines with TypeScript.

Overview

Conduit provides a declarative, configuration-driven approach to data integration with:

• Type-safe configurations - Full TypeScript support with compile-time validation
• Modular architecture - Pluggable extractors, transformers, and loaders
• Built-in resilience - Automatic retries, error handling, and transaction support
• Multiple data sources - APIs, databases, files with authentication handling
• Production-ready - Event-driven monitoring, atomic commits, and data integrity tracking

Installation

npm install @partnersync/conduit

Quick Start

1. Initialize the database:

   npx conduit init

2. Create an integration:

   // integrations/my-api/config.ts
   import { IntegrationConfig } from '@partnersync/conduit/core';
   
   export default {
     id: 'my-api',
     name: 'My API Integration',
     entities: [{
       id: 'users',
       name: 'Users',
       source: {
         type: 'api',
         url: 'https://api.example.com/users'
       },
       destination: {
         type: 'database',
         tableName: 'Users'
       },
       fields: [
         { source: 'id', target: 'Id', type: 'integer', isPrimaryKey: true },
         { source: 'name', target: 'Name', type: 'string' }
       ]
     }]
   } as IntegrationConfig;

3. Deploy and run:

   npx conduit deploy my-api
   npx conduit run my-api

Documentation

• Getting Started Guide - Detailed setup instructions
• Configuration Reference - Complete guide to all configuration options
• Architecture Documentation - Deep dive into framework design
• API Reference - Generated TypeDoc documentation

Environment Variables

The Conduit CLI uses system environment variables exclusively and does NOT load .env files. This follows the pattern of professional CLI tools like kubectl and aws-cli.

Setting Environment Variables

Linux/Mac:

# Required environment variables
export DATABASE_CONNECTION_STRING="Server=localhost;Database=conduit;User Id=sa;Password=YourPassword;TrustServerCertificate=true"
export STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=https;AccountName=youraccount;AccountKey=..."

# Integration-specific variables (examples based on your authentication type)
# For an integration with id: "salesforce-api" using OAuth2:
export SALESFORCE_API_CLIENT_SECRET="your-client-secret"
export SALESFORCE_API_REFRESH_TOKEN="your-refresh-token"

# For an integration with id: "stripe" using API Key:
export STRIPE_API_KEY="your-api-key"

Windows Command Prompt:

rem Required environment variables
set DATABASE_CONNECTION_STRING=Server=localhost;Database=conduit;User Id=sa;Password=YourPassword;TrustServerCertificate=true
set STORAGE_CONNECTION_STRING=DefaultEndpointsProtocol=https;AccountName=youraccount;AccountKey=...

rem Integration-specific variables (examples based on your authentication type)
rem For an integration with id: "salesforce-api" using OAuth2:
set SALESFORCE_API_CLIENT_SECRET=your-client-secret
set SALESFORCE_API_REFRESH_TOKEN=your-refresh-token

rem For an integration with id: "stripe" using API Key:
set STRIPE_API_KEY=your-api-key

Windows PowerShell:

# Required environment variables
$env:DATABASE_CONNECTION_STRING="Server=localhost;Database=conduit;User Id=sa;Password=YourPassword;TrustServerCertificate=true"
$env:STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=https;AccountName=youraccount;AccountKey=..."

# Integration-specific variables (examples based on your authentication type)
# For an integration with id: "salesforce-api" using OAuth2:
$env:SALESFORCE_API_CLIENT_SECRET="your-client-secret"
$env:SALESFORCE_API_REFRESH_TOKEN="your-refresh-token"

# For an integration with id: "stripe" using API Key:
$env:STRIPE_API_KEY="your-api-key"

For Library Usage Only

If you're using Conduit as a library (not the CLI), you can create a .env file in your project root. See .env.template for an example. The .env file is ONLY loaded when using Conduit programmatically, never by the CLI.

Required Variables by Authentication Type

Based on your integration's authentication configuration:

OAuth2:

• {INTEGRATION_ID}_CLIENT_SECRET
• {INTEGRATION_ID}_REFRESH_TOKEN

API Key:

• {INTEGRATION_ID}_API_KEY

Bearer Token:

• {INTEGRATION_ID}_BEARER_TOKEN

Form Login:

• {INTEGRATION_ID}_USERNAME
• {INTEGRATION_ID}_PASSWORD

Note: Replace hyphens with underscores and use uppercase. For example, api-service becomes API_SERVICE.

Troubleshooting

Environment Variable Issues

If you see errors about missing environment variables:

1. Check the error message - The CLI provides platform-specific instructions
2. Verify the variable is set - Use echo $VAR_NAME (Linux/Mac) or echo %VAR_NAME% (Windows)
3. Remember: The CLI does NOT read .env files - you must set system environment variables
4. For persistent variables:
   • Linux/Mac: Add to ~/.bashrc, ~/.zshrc, or ~/.profile
   • Windows: Use System Properties → Environment Variables

CLI Commands

• npx conduit init - Initialize the database with operational schema
• npx conduit deploy <integration> - Deploy integration tables and procedures
• npx conduit run <integration> - Execute an integration
• npx conduit health - Check database health and configuration
• npx conduit list - List available integrations

Contributing

See CONTRIBUTING.md for guidelines.

License

MIT - see LICENSE for details.