Docker Compose Merger

A powerful tool for merging multiple Docker Compose files into a single unified compose file. Perfect for microservices architectures where you want to run multiple projects together with shared infrastructure.

Features

• Project Merging: Group multiple Docker Compose files to share service names without renaming
• Service Merging: Combine duplicate services (kafka, databases, etc.) into single instances
• Network Consolidation: Merge networks across projects for seamless inter-service communication
• Smart Renaming: Only renames services when conflicts exist, preserving original dependencies
• Port Conflict Resolution: Automatically resolves port conflicts by incrementing host ports
• Service Exclusion: Filter out unwanted services (test, CI/CD services, etc.)
• Profile Management: Automatically removes Docker Compose profiles for simplified deployment
• Environment Variable Rewriting: Update service references in environment variables
• Provenance Tracking: Generates mapping files showing which services came from which projects

The Problem This Solves

Before: When running multiple compose files together, conflicting service names got renamed everywhere:

• db → db1, db2 (environment variables like DB_HOST: db break)
• redis → redis1, redis2 (dependencies like depends_on: [redis] break)
• You had to manually update hostnames in every compose file

After: With project-level merging:

• Group related compose files (app + database)
• Services keep their original names
• DB_HOST: postgres works without modification
• depends_on: [postgres] works without changes
• Only renames when actual conflicts exist

Installation

Global Installation

npm install -g

After installation, the dc-merge command will be available globally.

Local Installation

npm install

Use with node merge.js or through npm scripts.

Quick Start

1. Create a configuration file (e.g., config.yml):

version: 1

# Merge projects together so they share service names
merge:
  app_stack:
    - api
    - database

projects:
  api:
    file: ../api/docker-compose.yml
    naming:
      strategy: suffix
      value: "1"
  
  database:
    file: ../database/docker-compose.yml
    naming:
      strategy: suffix
      value: "1"

networks:
  merge:
    shared-network:
      - api.default
      - database.default

2. Run the merge:

dc-merge config.yml

3. Start the services:

docker compose -f docker-compose.merged.yml up -d

Or combine steps 2 and 3:

dc-merge config.yml up -d

Configuration Reference

Basic Structure

version: 1

# Optional: Merge projects together (keeps original service names)
merge:
  <group_name>:
    - <project_1>
    - <project_2>

projects:
  <project_name>:
    file: <path_to_docker_compose_file>
    naming:
      strategy: suffix|prefix
      value: <string>

Project Merging (NEW!)

Group multiple Docker Compose files together so they can share service names and communicate seamlessly:

version: 1

merge:
  app_stack:
    - frontend
    - backend
    - database

projects:
  frontend:
    file: ../frontend/docker-compose.yml
    naming:
      strategy: suffix
      value: "1"
  backend:
    file: ../backend/docker-compose.yml
    naming:
      strategy: suffix
      value: "1"
  database:
    file: ../database/docker-compose.yml
    naming:
      strategy: suffix
      value: "1"

How it works:

• Services in merged projects keep their original names (e.g., api, db, redis)
• Dependencies work without modification (e.g., depends_on: [db] just works)
• Environment variables reference the correct services (e.g., DB_HOST: db)
• Only rename services when there's an actual conflict with services outside the group
• Networks and volumes also preserve names within merged groups

Benefits:

• No hostname changes needed: Add an app to a database without changing DB_HOST
• Dependencies work out-of-the-box: Service references remain valid
• Easier development: Test multiple related services together as a unit
• Less configuration: No need to update environment variables for merged stacks

Example use case: You have an API service that depends on PostgreSQL and Redis. Instead of renaming everything:

# Without project merge:
# api1 → depends_on: [postgres1, redis1]
# DB_HOST: postgres1 (need to change this!)

# With project merge:
# api → depends_on: [postgres, redis]
# DB_HOST: postgres (works as-is!)

Projects Section

Defines which Docker Compose files to merge.

projects:
  gvrn_intake:
    file: ../gvrn-intake/docker-compose.yaml
    naming:
      strategy: suffix
      value: "1"

Naming Strategies:

• suffix: Appends value to service names (e.g., kafka becomes kafka1)
• prefix: Prepends value to service names (e.g., kafka becomes 1_kafka)

Networks Section

Merges networks from different projects into shared networks.

networks:
  merge:
    shared-network:
      - project_a.network-a
      - project_b.network-b

This creates a single shared-network that combines both source networks.

Services Section

Service Merging

Combine multiple services into a single instance:

services:
  merge:
    kafka1:
      - project_a.kafka
      - project_b.kafka
    postgres1:
      - project_a.db
      - project_b.database

Service Exclusion

Exclude services from the merged output:

services:
  exclude:
    - project_a.test-service
    - project_b.ci-runner

Extends Section

Add additional configuration to services before merging:

extends:
  services:
    project_b.api:
      networks:
        - project_a.shared-network

Overrides Section

Override service properties after loading but before merging:

overrides:
  services:
    project_a.app:
      image: my-app:latest
      environment:
        DATABASE_URL: postgres://localhost:5432/db

Environment Section

Configure environment variable rewriting:

environment:
  rewriteServiceNames: true
  patterns:
    - "*_HOST"
    - "DB_HOST"
    - "KAFKA_BROKER"

Remove Section

Remove fields from all services:

remove:
  build: true
  profiles: true

Run Section

Default arguments for docker compose commands:

run:
  up:
    - "-d"
  logs:
    - "-f"

Command Line Usage

Basic Commands

# Generate merged compose file
dc-merge config.yml

# Generate and start services
dc-merge config.yml up -d

# Check service status
dc-merge config.yml ps

# View logs
dc-merge config.yml logs -f

# Stop services
dc-merge config.yml down

Direct Node.js Usage

node merge.js config.yml
node merge.js config.yml up -d

Output Files

The merge process generates two files:

docker-compose.merged.yml

The merged Docker Compose file ready for deployment.

docker-compose.mapping.yml

A mapping file showing the provenance of each service:

services:
  kafka1:
    source: project_a.kafka
    originalName: kafka
    containerName: kafka1
ports:
  kafka1:
    "29092":
      original: 29092
      container: 9092

Examples

Example 1: Merge App with Database (NEW!)

The most common use case - merge an app and its database together:

version: 1

# Merge app and database so they can reference each other by original names
merge:
  app_stack:
    - app
    - database

projects:
  app:
    file: ../app/docker-compose.yml
    naming:
      strategy: suffix
      value: "1"
  
  database:
    file: ../database/docker-compose.yml
    naming:
      strategy: suffix
      value: "1"

networks:
  merge:
    shared:
      - app.default
      - database.default

environment:
  rewriteServiceNames: true
  patterns:
    - "*_HOST"

Result:

• App services can reference database by original name (DB_HOST: postgres)
• No need to change hostnames or service references
• All services share the same network

Example 2: Multiple Microservices with Conflicts

version: 1

# Merge service1 with database, but keep service2 separate
merge:
  stack1:
    - service1
    - database

projects:
  service1:
    file: ../service1/docker-compose.yml
    naming:
      strategy: suffix
      value: "1"
  
  service2:
    file: ../service2/docker-compose.yml
    naming:
      strategy: suffix
      value: "2"
  
  database:
    file: ../database/docker-compose.yml
    naming:
      strategy: suffix
      value: "1"

# If service1 and service2 both have a 'redis' service,
# only service2's redis will be renamed to 'redis2'

Example 3: Merge Two Microservices

version: 1

projects:
  api:
    file: ../api/docker-compose.yml
    naming:
      strategy: suffix
      value: "1"
  
  worker:
    file: ../worker/docker-compose.yml
    naming:
      strategy: suffix
      value: "2"

networks:
  merge:
    app-network:
      - api.default
      - worker.default

services:
  merge:
    redis1:
      - api.redis
      - worker.redis

Example 2: Shared Kafka Cluster

version: 1

projects:
  service_a:
    file: ../service-a/docker-compose.yml
    naming:
      strategy: suffix
      value: "1"
  
  service_b:
    file: ../service-b/docker-compose.yml
    naming:
      strategy: suffix
      value: "2"

services:
  merge:
    kafka1:
      - service_a.kafka
      - service_b.kafka
    zookeeper1:
      - service_a.zookeeper
      - service_b.zookeeper

extends:
  services:
    service_b.api:
      networks:
        - service_a.shared-network

Example 3: Development Environment

version: 1

projects:
  backend:
    file: ../backend/docker-compose.yml
    naming:
      strategy: suffix
      value: "1"
  
  frontend:
    file: ../frontend/docker-compose.yml
    naming:
      strategy: suffix
      value: "2"

services:
  exclude:
    - backend.test-db
    - backend.test-runner
    - frontend.e2e-tests

networks:
  merge:
    dev-network:
      - backend.default
      - frontend.default

overrides:
  services:
    backend.api:
      environment:
        DEBUG: "true"
        LOG_LEVEL: "debug"

How It Works

1. Load Projects: Reads all specified Docker Compose files
2. Apply Extensions: Adds networks or other config to services
3. Build Project Groups: Groups projects that should be merged together
4. Collect Dependencies: Analyzes service dependencies within each group
5. Build Mappings: Creates service, network, and volume name mappings
   • Services in merged project groups keep their original names
   • Only renames when conflicts exist
   • Services in standalone projects are renamed only if conflicts occur
6. Process Services: For each service:
   • Skip if excluded
   • Skip if already processed (merged)
   • Apply naming based on conflicts, not blindly
   • Rewrite references (networks, volumes, depends_on)
   • Resolve port conflicts
   • Remove profiles (optional)
7. Apply Overrides: Final service modifications
8. Generate Output: Write merged compose and mapping files

Port Conflict Resolution

When multiple services use the same host port, the merge tool automatically increments the port number:

Original:
  service1: 8080:8080
  service2: 8080:8080

After merge:
  service1: 8080:8080
  service2: 8081:8080

The mapping file records these changes.

Troubleshooting

Services not starting

Problem: no service selected

Solution: The merge tool removes profiles by default. If services still have profiles, regenerate the merged file.

Network not found errors

Problem: network X not found

Solution: Ensure all referenced networks are either:

• Defined in the source compose files
• Merged in the config under networks.merge
• Added via extends.services

Service dependency errors

Problem: service X depends on undefined service Y

Solution:

• Use project-level merge to group related compose files together
• Add projects to the same merge group so they share service names
• For services outside the group, update depends_on to use renamed names
• Or use service-level merge for specific shared dependencies

Port conflicts

Problem: Services fail to start due to port conflicts

Solution: The merge tool should handle this automatically. If not, check:

• Port mappings in the mapping file
• Ensure all projects are using the merge tool

Development

Running Tests

npm test

Building from TypeScript

npm run build

This compiles merge.ts to merge.js.

Project Structure

.
├── bin/
│   └── dc-merge.js        # CLI wrapper
├── merge.ts               # TypeScript source
├── merge.js               # Compiled JavaScript
├── config.yml             # Example configuration
├── package.json
└── README.md

Advanced Configuration

Custom Network Names

Use the name property to set explicit network names:

networks:
  merge:
    shared-network:
      - project_a.default
      - project_b.default

In the merged file, this becomes:

networks:
  shared-network:
    name: shared-network

Environment Variable Patterns

Control which environment variables get rewritten:

environment:
  rewriteServiceNames: true
  patterns:
    - "*_HOST"      # Matches DB_HOST, API_HOST, etc.
    - "REDIS_URL"   # Exact match
    - "KAFKA_*"     # Matches KAFKA_BROKER, KAFKA_PORT, etc.

Conditional Merging

Use extends to conditionally add services to networks:

extends:
  services:
    project_a.worker:
      networks:
        - project_b.processing-network
      environment:
        WORKER_MODE: distributed

Contributing

Contributions are welcome. Please ensure:

1. Code follows existing style
2. TypeScript types are properly defined
3. Tests pass
4. Documentation is updated

License

MIT

Support

For issues, questions, or contributions, please open an issue in the repository.