hyperconn

v1.0.7

Published

16 days ago

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

0High
0Medium
0Low

princetcgre

HyperConn

Multi-tenant MongoDB and PostgreSQL connection management with AWS Secrets Manager integration

HyperConn simplifies multi-tenant database management for NestJS and TypeScript services. It retrieves tenant database URLs from AWS Secrets Manager, manages reusable connection pools, and keeps each organization's queries isolated.

🚀 Features

Multi-tenant Architecture: Connect to multiple databases with organization-specific schemas
AWS Secrets Manager Integration: Secure credential management with automatic rotation support
Schema Isolation: Each organization operates in its own database schema namespace
Connection Pooling: Optimized database connections with configurable pool settings
Type Safety: Full TypeScript support with comprehensive interfaces
NestJS Compatible: Built specifically for NestJS applications
Flexible Configuration: Support for multiple services and collections per organization
PostgreSQL + Knex: Schema-scoped Knex clients with per-organization connection pools
Production Ready: Built with enterprise-scale applications in mind

📦 Installation

# Using npm
npm install hyperconn

# Using yarn
yarn add hyperconn

# Using pnpm
pnpm add hyperconn

🔧 Prerequisites

1. AWS Secrets Manager Setup

Ensure AWS Secrets Manager is configured with database credentials for each organization. Format for secret name must be: -

Service name will be the name of the microservice. Each secret should contain below JSON in stringified format:

{
  "<microserviceName>-mongodb": "db_url",
}

Example - Name can be - martech-orgId1 Stringified JSON can be - '{"corMicroservice-mongodb": "db_url"}'

2. Environment Variables

Create a .env file in the project root directory with your AWS credentials:

AWS_SECRETS_MANAGER_ACCESS_KEY_ID=your_access_key_id
AWS_SECRETS_MANAGER_SECRET_ACCESS_KEY=your_secret_access_key
AWS_SECRETS_MANAGER_REGION=your_aws_region

PostgreSQL / Knex multi-tenancy

Use KnexPgDbManagerService when tenant data lives in PostgreSQL. HyperConn creates one reusable Knex pool per organization and returns a client that automatically targets that organization's PostgreSQL schema.

AWS Secrets Manager convention

Create one secret for each organization using this name:

<projectName>-<orgId>

Its value must be a JSON object. The PostgreSQL URL is read from the key configured as dbKeyName:

{
  "read-postgresql": "postgresql://username:password@host:5432/database"
}

For example, if projectName is sirrusprod and the organization ID is 5b22b97c-9637-43f3-9476-fcf80c1d49f0, use the secret name:

sirrusprod-5b22b97c-9637-43f3-9476-fcf80c1d49f0

The AWS identity used by the service needs permission to list and read these secrets: secretsmanager:ListSecrets and secretsmanager:GetSecretValue.

Configure once during application startup

Create one manager instance for the application. Call connectToDatabases() before the service begins handling requests; it discovers and warms all secrets whose names begin with <projectName>-.

import {
  KnexPgDbManagerService,
  requiredKnexParamsInterface,
} from "hyperconn";

export const tenantPostgres = new KnexPgDbManagerService();

export async function connectTenantPostgres(): Promise<void> {
  const config: requiredKnexParamsInterface = {
    secretManager: {
      secretsManagerAccessKeyId: process.env.AWS_SECRETS_MANAGER_ACCESS_KEY_ID!,
      secretsManagerSecretAccessKey:
        process.env.AWS_SECRETS_MANAGER_SECRET_ACCESS_KEY!,
      secretsManagerRegion: process.env.AWS_SECRETS_MANAGER_REGION!,
    },
    projectName: "sirrusprod",
    knexServiceManager: {
      // Only add organizations that use a schema different from their org ID.
      tenantSchemaMap: {
        "5b22b97c-9637-43f3-9476-fcf80c1d49f0": "public",
        "522a6f97-39e0-4a6e-b42f-80879cafd579": "public",
      },
      dbKeyName: "read-postgresql",
    },
    knexConnectionOptions: {
      pool: { min: 2, max: 10 },
      // Optional. Enables TLS when the file exists and is readable.
      caFilePath: process.env.PG_SSL_CA_FILE,
      // Optional; defaults to true.
      debug: process.env.NODE_ENV !== "production",
    },
  };

  await tenantPostgres.connectToDatabases(config);
}

For an organization not listed in tenantSchemaMap, HyperConn uses the organization ID itself as the PostgreSQL schema name. This keeps the configuration small when each tenant follows the same naming convention.

Query a tenant database

getConnection(orgId) returns a Promise<Knex>. The returned client is cached for that organization and automatically scopes normal table queries to its schema.

async function findCampaigns(orgId: string) {
  const db = await tenantPostgres.getConnection(orgId);

  return db("campaigns")
    .select("id", "name", "status")
    .where({ status: "active" })
    .orderBy("created_at", "desc");
}

Each call creates an independent Knex query builder, so concurrent requests for the same organization do not mix query state or results. They reuse that tenant's pool; requests beyond the configured pool limit wait for an available database connection.

The returned client supports the normal Knex APIs, including transaction, queryBuilder, and raw. In a transaction callback, explicitly select the tenant schema because Knex supplies its base transaction client:

const db = await tenantPostgres.getConnection(orgId);
const schema = "public"; // Use the value from tenantSchemaMap, or orgId when unmapped.

await db.transaction(async (trx) => {
  await trx
    .withSchema(schema)
    .table("campaigns")
    .where({ id: campaignId })
    .update({ status: "archived" });
});

Raw SQL: db.raw() executes on the tenant connection but does not add a schema automatically. Qualify table names yourself, for example db.raw('select * from "public"."campaigns"'), or use your application's schema-aware raw-SQL helper.

Lifecycle and public API

| Method | Purpose | | --- | --- | | connectToDatabases(config) | Validates configuration, discovers project secrets, and warms available tenant pools. A single tenant warm-up failure is logged without preventing other tenants from initializing. | | getConnection(orgId) | Returns the organization’s schema-scoped Knex client. If it was not warmed at startup, HyperConn loads its secret and creates the pool on demand. | | getOrgIds() | Lists organization IDs discovered from secrets matching <projectName>-. | | destroyAll() | Closes every PostgreSQL pool. Call it during graceful shutdown. |

Graceful shutdown

process.once("SIGTERM", async () => {
  await tenantPostgres.destroyAll();
  process.exit(0);
});

PostgreSQL connection options

knexConnectionOptions.pool is passed to Knex. Use it to control database concurrency per tenant. If caFilePath points to a readable CA certificate, HyperConn connects with TLS and rejectUnauthorized: true; otherwise it falls back to the URL-only connection configuration.

knexConnectionOptions: {
  pool: {
    min: 2,
    max: 10,
  },
  caFilePath: "/path/to/postgres-ca.pem", // optional
  debug: false, // optional; true by default
}

For capacity planning, remember that every active organization can own a separate pool. For example, 20 active tenants with max: 10 can open up to 200 PostgreSQL connections.

🚦 Quick Start

1. Make a file with name "connectHypercon" and Import Required Components

import { MongoDbManagerService, requiredParamsInterface } from 'hyperconn';
import eventsSchema from "./yourpath/eventsSchema"
import userSchema from "./yourpath/userSchema"
import profileSchema from "./yourpath/profileSchema"

2. Initialize and Configure

export class DatabaseService {
  private dbManager: MongoDbManagerService;

  constructor() {
    this.dbManager = new MongoDbManagerService();
  }

  async initializeDatabases() {
    const params: requiredParamsInterface = {
      secretManager: {
        secretsManagerAccessKeyId: process.env.AWS_SECRETS_MANAGER_ACCESS_KEY_ID,
        secretsManagerSecretAccessKey: process.env.AWS_SECRETS_MANAGER_SECRET_ACCESS_KEY,
        secretsManagerRegion: process.env.AWS_SECRETS_MANAGER_REGION
      },
      mongoDbServiceManager: [
        {
          serviceName: "service1",
          collections: [
            {
              name: "events",
              schema: eventsSchema
            },
            {
              name: "users",
              schema: userSchema
            }
          ]
        },
        {
          serviceName: "service2",
          collections: [
            {
              name: "events",
              schema: eventsSchema
            },
            {
              name: "profiles",
              schema: profileSchema
            }
          ]
        }
      ],
      projectName: "martech",
      mongoConnectionOptions: {
        maxPoolSize: 10,
        connectTimeoutMS: 10000,
        socketTimeoutMS: 45000,
        retryWrites: true,
        readPreference: 'primary'
      }
    };

    await this.dbManager.connectToDatabasesAndModels(params);
  }

  // Get model for specific organization
  getEventModel(orgId: string, serviceName, modelName) {
    return this.dbManager.getModel(serviceName, orgId, modelName);
  }

  // Get all organization IDs
  getAllOrganizations() {
    return await this.dbManager.getOrgIds();
  }
}

3. In main.ts, initialize connectHypercon.ts

import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';
import { connectMongoWithHypercon } from './database/connectHyperconn';

async function bootstrap() {
  // Initialize HyperConn before creating the app
  await connectMongoWithHypercon.connect();
  
  const app = await NestFactory.create(AppModule);
  await app.listen(3000);
  
  console.log('🚀 Application is running on: http://localhost:3000');
}

bootstrap();

4. Use in Your NestJS Service

@Injectable()
export class EventsService {
  constructor(private readonly databaseService: DatabaseService) {}

  async createEvent(orgId: string, eventData: any) {
    const EventModel = this.databaseService.getEventModel(orgId, 'service1', 'events');
    const event = new EventModel(eventData);
    return await event.save();
  }

  async getEventsByOrganization(orgId: string, filters: any = {}) {
    const EventModel = this.databaseService.getEventModel(orgId, 'service1', 'users');
    return await EventModel.find(filters).exec();
  }

  async getOrganizationStats() {
    const organizations = await this.databaseService.getAllOrganizations();
    const stats = [];

    for (const orgId of organizations) {
      const EventModel = this.databaseService.getEventModel(orgId, 'service1', 'events');
      const count = await EventModel.countDocuments();
      stats.push({ orgId, eventCount: count });
    }

    return stats;
  }
}

📚 API Reference

`MongoDbManagerService`

The main service class for managing multi-tenant database connections.

Methods

`connectToDatabasesAndModels(params: requiredParamsInterface): Promise<void>`

Establishes connections to all configured databases and initializes models.

Parameters:

params - Configuration object containing all required parameters

`getEventModel(orgId: string, serviceName: string, modelName: string): Model<any>`

Retrieves a Mongoose model for a specific organization and schema.

Parameters:

serviceName - Name of the service (e.g., 'core', 'segment')
orgId - Organization identifier
modelName - Name of the collection/schema

Returns: Mongoose Model instance

`getOrgIds(): Promise<string[]>`

Retrieves organisation ids.

Returns: All the organisation ids for which db connections are made

Array of all available organization IDs.

`destroyAll(): Promise<void>`

Closes every cached MongoDB connection and clears the model registry. Call this during graceful application shutdown.

const mongoDbManager = new MongoDbManagerService();
await mongoDbManager.connectToDatabasesAndModels(params);

process.once('SIGTERM', async () => {
  await mongoDbManager.destroyAll();
  process.exit(0);
});

`requiredParamsInterface`

Configuration interface for database connections.

interface requiredParamsInterface {
  secretManager: {
    secretsManagerAccessKeyId: string;
    secretsManagerSecretAccessKey: string;
    secretsManagerRegion: string;
  };
  mongoDbServiceManager: Array<{
    serviceName: string;
    collections: Array<{
      name: string;
      schema: Schema;
    }>;
  }>;
  projectName: string;
  mongoConnectionOptions?: ConnectOptions;
}

🔧 Configuration Options

MongoDB Connection Options

mongoConnectionOptions: {
  // Connection pool settings
  maxPoolSize: 10,                    // Maximum number of connections in pool
  minPoolSize: 5,                     // Minimum number of connections in pool
  
  // Timeout settings
  connectTimeoutMS: 10000,            // Connection timeout (10 seconds)
  socketTimeoutMS: 45000,             // Socket timeout (45 seconds)
  serverSelectionTimeoutMS: 5000,     // Server selection timeout
  
  // Retry and reliability
  retryWrites: true,                  // Enable retry writes
  retryReads: true,                   // Enable retry reads
  
  // Read preferences
  readPreference: 'primary',          // Read from primary by default
  
  // Buffer settings
  bufferMaxEntries: 0,                // Disable mongoose buffering
  bufferCommands: false,              // Disable mongoose buffering
  
  // Authentication
  authSource: 'admin',                // Authentication database
  
  // SSL/TLS
  ssl: true,                          // Enable SSL
  sslValidate: true,                  // Validate SSL certificates
}

🏗️ Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        HyperConn Package                        │
├─────────────────────────────────────────────────────────────────┤
│  MongoDbManagerService                                          │
│  ├── AWS Secrets Manager Integration                           │
│  ├── Connection Pool Management                                │
│  └── Model Registry (Org-specific)                             │
└─────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                    AWS Secrets Manager                          │
├─────────────────────────────────────────────────────────────────┤
│  org-1-db-credentials │ org-2-db-credentials │ org-n-db-creds   │
└─────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                     MongoDB Clusters                            │
├─────────────────────────────────────────────────────────────────┤
│ Org 1 DB    │    Org 2 DB     │    Org N DB                    │
│ └─ Schema A │    └─ Schema A   │    └─ Schema A                 │
│ └─ Schema B │    └─ Schema B   │    └─ Schema B                 │
└─────────────────────────────────────────────────────────────────┘

🔒 Security Best Practices

1. AWS Credentials Management

Never hardcode AWS credentials in your code
Use IAM roles when running on AWS infrastructure
Rotate AWS access keys regularly
Follow the principle of least privilege for IAM permissions

2. Database Security

Enable MongoDB authentication and authorization
Use SSL/TLS for database connections
Implement network-level security (VPC, Security Groups)
Regularly update database versions and security patches

3. Environment Variables

# Use strong, unique credentials
AWS_SECRETS_MANAGER_ACCESS_KEY_ID=AKIA...
AWS_SECRETS_MANAGER_SECRET_ACCESS_KEY=...
AWS_SECRETS_MANAGER_REGION=us-east-1

## 🚀 Advanced Usage

### Custom Error Handling

```typescript
import { MongoDbManagerService } from 'hyperconn';

class CustomDatabaseService {
  private dbManager: MongoDbManagerService;

  async safeGetModel(serviceName: string, orgId: string, modelName: string) {
    try {
      return this.dbManager.getModel(serviceName, orgId, modelName);
    } catch (error) {
      console.error(`Failed to get model for ${orgId}:${serviceName}:${modelName}`, error);
      throw new Error(`Database model not available for organization ${orgId}`);
    }
  }
}

Health Checks

@Controller('health')
export class HealthController {
  constructor(private readonly databaseService: DatabaseService) {}

  @Get('database')
  async checkDatabaseHealth() {
    const organizations = await this.databaseService.getAllOrganizations();
    const healthStatus = {};

    for (const orgId of organizations) {
      try {
        const EventModel = this.databaseService.getEventModel(orgId, 'service1', "events");
        await EventModel.findOne().limit(1).exec();
        healthStatus[orgId] = 'healthy';
      } catch (error) {
        healthStatus[orgId] = 'unhealthy';
      }
    }

    return { status: 'ok', organizations: healthStatus };
  }
}

Batch Operations

async bulkCreateEvents(orgId: string, events: any[]) {
  const EventModel = this.databaseService.getEventModel(orgId, 'service1', "events");
  return await EventModel.insertMany(events, { ordered: false });
}

async aggregateEventsByType(orgId: string) {
  const EventModel = this.databaseService.getEventModel(orgId, 'service1', "events");
  return await EventModel.aggregate([
    { $group: { _id: '$eventType', count: { $sum: 1 } } },
    { $sort: { count: -1 } }
  ]);
}

🧪 Testing

Unit Testing Example

import { Test, TestingModule } from '@nestjs/testing';
import { MongoDbManagerService } from 'hyperconn';

describe('DatabaseService', () => {
  let service: DatabaseService;
  let dbManager: MongoDbManagerService;

  beforeEach(async () => {
    const module: TestingModule = await Test.createTestingModule({
      providers: [
        DatabaseService,
        {
          provide: MongoDbManagerService,
          useValue: {
            connectToDatabasesAndModels: jest.fn(),
            getModel: jest.fn(),
            orgIds: ['org1', 'org2']
          }
        }
      ],
    }).compile();

    service = module.get<DatabaseService>(DatabaseService);
    dbManager = module.get<MongoDbManagerService>(MongoDbManagerService);
  });

  it('should be defined', () => {
    expect(service).toBeDefined();
  });
});

📊 Performance Considerations

Connection Pooling

Default Pool Size: 10 connections per database
Recommended: Monitor connection usage and adjust based on load
Scaling: Consider connection limits per MongoDB instance

Memory Usage

Each organization connection maintains its own model registry
Monitor memory usage with multiple organizations
Implement model caching strategies for frequently accessed schemas

Query Optimization

// Use indexes for better query performance
const eventSchema = new Schema({
  userId: { type: String, required: true, index: true },
  eventType: { type: String, required: true, index: true },
  timestamp: { type: Date, default: Date.now, index: true }
});

// Compound indexes for complex queries
eventSchema.index({ userId: 1, timestamp: -1 });
eventSchema.index({ eventType: 1, timestamp: -1 });

🐛 Troubleshooting

Common Issues

Connection Timeout

Error: MongoServerSelectionError: connection timeout

Solution: Increase connectTimeoutMS and serverSelectionTimeoutMS values.

AWS Credentials Error

Error: Missing AWS credentials

Solution: Verify environment variables and IAM permissions.

Schema Not Found

Error: Schema not found for organization

Solution: Ensure the organization exists in AWS Secrets Manager and the schema is properly registered.

Debug Mode

// Enable debug logging
const params: requiredParamsInterface = {
  // ... other params
  mongoConnectionOptions: {
    // ... other options
    debug: true, // Enable Mongoose debug mode
  }
};

🤝 Contributing

We welcome contributions! Please follow these steps:

Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

Development Setup

# Clone the repository
git clone https://github.com/TCG-LS/HyperConn.git
cd hyperconn

# Install dependencies
npm install

# Run tests
npm test

# Build the package
npm run build

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

📞 Support

Documentation: GitHub Wiki
Issues: GitHub Issues
Discussions: GitHub Discussions

🎯 Roadmap

Version 1.1.0

[ ] Redis caching integration
[ ] Connection health monitoring
[ ] Automatic failover support

Version 1.2.0

[x] PostgreSQL and Knex multi-tenancy
[ ] GraphQL integration
[ ] Advanced monitoring dashboard

Version 2.0.0

[ ] Microservices architecture support
[ ] Kubernetes operator
[ ] Real-time synchronization

🏆 Acknowledgments

Built with NestJS
Powered by Mongoose
Secured by AWS Secrets Manager

Made with ❤️ by the HyperConn Team