Data Bridge Backend API

A robust backend API server for the Data Bridge application that handles database connections, file parsing, and external function calls.

🚀 Quick Start

Prerequisites

• Node.js 16+ installed
• Database server (PostgreSQL, MySQL, or SQL Server)
• npm or yarn package manager

Installation

1. Install dependencies:

   npm install

2. Configure environment:

   cp .env.example .env
   # Edit .env with your configuration

3. Start the server:

   # Development mode
   npm run dev
   
   # Production mode
   npm start

4. Verify installation:

   curl http://localhost:3001/health

📚 API Endpoints

Health Check

• GET /health - Server health status

Database Connector

• POST /api/db-connector - Main database operations
• POST /api/db-connector/credentials - Store database credentials (testing)

File Parser

• POST /api/parse-file/upload - Upload file
• POST /api/parse-file - Parse uploaded file
• POST /api/parse-file/excel - Parse Excel file specifically
• POST /api/parse-file/csv - Parse CSV file specifically
• DELETE /api/parse-file/cleanup - Clean up old files

External Functions

• POST /api/external-function - Execute external function
• POST /api/external-function/register - Register new external function
• GET /api/external-function/list - List all functions
• GET /api/external-function/:id - Get function details
• PUT /api/external-function/:id - Update function
• DELETE /api/external-function/:id - Delete function
• POST /api/external-function/:id/test - Test function

🔧 Configuration

Environment Variables

# Server Configuration
NODE_ENV=development
PORT=3001

# Database Examples
POSTGRES_HOST=localhost
POSTGRES_PORT=5432
POSTGRES_DB=databridge
POSTGRES_USER=your_username
POSTGRES_PASSWORD=your_password

MYSQL_HOST=localhost
MYSQL_PORT=3306
MYSQL_DB=databridge
MYSQL_USER=your_username
MYSQL_PASSWORD=your_password

# File Upload
UPLOAD_DIR=./uploads
MAX_FILE_SIZE=50MB

# External API
EXTERNAL_API_TIMEOUT=30000
EXTERNAL_API_RETRIES=3

# Security
RATE_LIMIT_WINDOW_MS=900000
RATE_LIMIT_MAX_REQUESTS=100

📖 Usage Examples

Database Connector

// List tables
const response = await fetch('/api/db-connector', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    credentialId: 'your-credential-id',
    action: 'list_tables'
  })
});

// Get table columns
const response = await fetch('/api/db-connector', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    credentialId: 'your-credential-id',
    action: 'get_columns',
    payload: { tableName: 'users' }
  })
});

// Insert data
const response = await fetch('/api/db-connector', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    credentialId: 'your-credential-id',
    action: 'insert_data',
    payload: {
      tableName: 'users',
      data: [
        { name: 'John', email: '[email protected]' },
        { name: 'Jane', email: '[email protected]' }
      ],
      columnMappings: {
        name: 'A',
        email: 'B'
      }
    }
  })
});

File Parser

// Upload file
const formData = new FormData();
formData.append('file', fileInput.files[0]);

const uploadResponse = await fetch('/api/parse-file/upload', {
  method: 'POST',
  body: formData
});

// Parse file
const parseResponse = await fetch('/api/parse-file', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    file_url: uploadResponse.file_url,
    getFullData: false // true to get all data
  })
});

External Functions

// Register external function
const registerResponse = await fetch('/api/external-function/register', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    id: 'custom-processor',
    name: 'Custom Data Processor',
    description: 'Processes data with custom logic',
    endpoint: 'https://your-api.com/process',
    method: 'POST',
    timeout: 30000
  })
});

// Execute external function
const executeResponse = await fetch('/api/external-function', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    externalFunctionId: 'custom-processor',
    payload: {
      tableName: 'users',
      data: [...],
      columnMappings: {...}
    }
  })
});

🛡️ Security Features

• CORS Protection - Configurable origin restrictions
• Rate Limiting - Prevents API abuse
• Helmet - Security headers
• Input Validation - Joi schema validation
• File Type Restrictions - Only allowed file types
• File Size Limits - Configurable upload limits

🗂️ Project Structure

backend/
├── middleware/          # Custom middleware
│   ├── errorHandler.js # Global error handling
│   └── logger.js       # Request/response logging
├── routes/             # API route handlers
│   ├── db-connector.js # Database operations
│   ├── parse-file.js   # File parsing
│   └── external-function.js # External function calls
├── services/           # Business logic
│   └── databaseFactory.js # Database connection factory
├── uploads/            # File upload directory
├── .env.example        # Environment template
├── package.json        # Dependencies and scripts
└── server.js          # Main application entry point

🔍 Monitoring & Debugging

The server includes comprehensive logging:

• Request/response logging
• Error tracking with stack traces
• Database connection monitoring
• External API call tracking

Check console output for detailed information about all operations.

🚨 Error Handling

All endpoints return consistent error responses:

{
  "error": {
    "message": "Error description",
    "status": 400,
    "timestamp": "2024-01-01T00:00:00.000Z",
    "details": ["Additional error details"]
  }
}

🤝 Integration with Frontend

This backend is designed to work with your Data Bridge frontend application. Make sure to:

1. Set the correct VITE_API_URL in your frontend environment
2. Configure CORS origins for your frontend domain
3. Handle file uploads through the provided endpoints
4. Use the consistent response format across all endpoints

📞 Support

For issues or questions, check the console logs for detailed error information.