GFTD CLI - Kafka & ksqlDB Schema Management

Drizzle-like schema management tool for Kafka topics and ksqlDB streams/tables with organization support.

🚀 Quick Start

Installation

npm install -g @gftdcojp/gftd-cli
# or
pnpm add -g @gftdcojp/gftd-cli

Initialize Project

gftd init --org-id=myorg --project=myproject

Configure Credentials

Edit gftd.config.yml:

orgId: "myorg"
project: "myproject" # optional

kafka:
  endpoint: ${KAFKA_ENDPOINT}
  apiKey: ${KAFKA_API_KEY}
  apiSecret: ${KAFKA_API_SECRET}

ksqlDB:
  endpoint: ${KSQLDB_ENDPOINT}
  apiKey: ${KSQLDB_API_KEY}
  apiSecret: ${KSQLDB_API_SECRET}

Run Migrations

gftd migrate

📋 Commands

Core Commands

| Command | Description | |---------|-------------| | gftd init | Initialize new project | | gftd migrate | Run schema migrations | | gftd status | Check current schema status | | gftd validate | Validate schema definition | | gftd generate | Generate SQL files only | | gftd info | Show project information |

Database Management (Supabase-like + Drizzle-style)

| Command | Description | |---------|-------------| | gftd db:init | Initialize database project with schema structure | | gftd migration:generate | Generate migration from schema changes | | gftd migration:up | Apply pending migrations | | gftd migration:down | Rollback migrations | | gftd migration:list | List migration status | | gftd generate:types | Generate TypeScript types from schema | | gftd db:pull | Pull schema from remote database | | gftd db:push | Push schema changes to database |

Development Tools

| Command | Description | |---------|-------------| | gftd dev | Start local development server with proxy |

🏗️ Schema Definition

Define your schemas in gftd.schema.yml:

orgId: "myorg"
project: "myproject"
version: "1.0.0"

topics:
  users:
    partitions: 3
    replicationFactor: 3
    configEntries:
      - name: "cleanup.policy"
        value: "compact"
      - name: "retention.ms"
        value: "-1"

streams:
  users:
    valueFormat: "JSON"
    keyFormat: "KAFKA"
    columns:
      - name: "ID"
        type: "VARCHAR"
        key: true
      - name: "NAME"
        type: "VARCHAR"
      - name: "EMAIL"
        type: "VARCHAR"

tables:
  users:
    aggregation: |
      SELECT ID,
             LATEST_BY_OFFSET(NAME) AS NAME,
             LATEST_BY_OFFSET(EMAIL) AS EMAIL
      FROM {sourceStream}
      GROUP BY ID

📖 Naming Convention

Resources are created with the following naming pattern:

Simple Environment-based Naming (env = orgId)

• Topic: {topicName}
  • Example: users
• Stream: {ORGID}_{STREAM_NAME}
  • Example: MYORG_USERS
• Table: {ORGID}_{TABLE_NAME}
  • Example: MYORG_USERS

This simplified approach treats the organization ID as the environment, making resource names cleaner and easier to manage.

🔧 Configuration

Environment Variables

Create .env file:

KAFKA_ENDPOINT=your-kafka-endpoint
KAFKA_API_KEY=your-kafka-api-key
KAFKA_API_SECRET=your-kafka-api-secret

KSQLDB_ENDPOINT=your-ksqldb-endpoint
KSQLDB_API_KEY=your-ksqldb-api-key
KSQLDB_API_SECRET=your-ksqldb-api-secret

Configuration File

gftd.config.yml:

• orgId: Organization identifier (required)
• project: Project name (optional)
• kafka: Kafka connection settings
• ksqlDB: ksqlDB connection settings
• schemaPath: Path to schema file (default: ./gftd.schema.yml)
• migrationPath: Migration output directory (default: ./migrations)

📊 Schema Validation

The CLI validates your schema for:

• Required fields
• Column type compatibility
• Topic/stream/table relationships
• Naming convention compliance

🎯 Example Workflow

# 1. Initialize project
gftd init --org-id=myorg --project=myapp

# 2. Edit gftd.schema.yml with your schema definition

# 3. Validate schema
gftd validate

# 4. Run migrations (dry run first)
gftd migrate --dry-run

# 5. Apply changes
gftd migrate

# 6. Check status
gftd status

🔗 Integration

Use with your favorite CI/CD:

# GitHub Actions example
- name: Validate Schema
  run: gftd validate

- name: Run Migrations
  run: gftd migrate
  env:
    KAFKA_ENDPOINT: ${{ secrets.KAFKA_ENDPOINT }}
    KAFKA_API_KEY: ${{ secrets.KAFKA_API_KEY }}
    KAFKA_API_SECRET: ${{ secrets.KAFKA_API_SECRET }}

🚨 Migration Safety

• Always run with --dry-run first
• Migrations are executed in dependency order
• Use --force flag to continue on failures (use with caution)
• Generated SQL files are saved in migration directory

🗄️ Database Management (Supabase-like + Drizzle-style)

GFTD CLI provides a comprehensive database management system that combines the best of Supabase CLI and Drizzle's migration approach.

Features

• 🏗️ Supabase-like CLI: Familiar commands for database operations
• 🔄 Drizzle-style Migrations: TypeScript-first schema management
• 📝 Type Generation: Automatic TypeScript types from database schema
• 🛠️ Multi-database Support: PostgreSQL, MySQL, SQLite, MongoDB
• 🔐 Schema Validation: Built-in schema validation and safety checks
• 🌱 Seeding: Database seeding capabilities

Quick Start

# Initialize database project
gftd db:init --type postgresql --database myapp_dev

# Generate migration from schema changes
gftd migration:generate "add_user_table"

# Apply migrations
gftd migration:up

# Generate TypeScript types
gftd generate:types --output types/database.ts

# List migration status
gftd migration:list

Database Configuration

GFTD uses gftd.json for database configuration:

{
  "database": {
    "type": "postgresql",
    "host": "localhost",
    "port": 5432,
    "database": "myapp_dev",
    "username": "postgres",
    "password": "postgres",
    "migrationsPath": "migrations",
    "seedsPath": "seeds"
  },
  "migrations": {
    "path": "migrations",
    "tableName": "_gftd_migrations",
    "schemaName": "public",
    "createSchema": true
  }
}

Schema-First Development

Define your database schema in TypeScript:

// schemas/schema.ts
export interface User {
  id: number;
  email: string;
  name?: string;
  createdAt: Date;
  updatedAt: Date;
}

export interface Post {
  id: number;
  title: string;
  content: string;
  authorId: number;
  published: boolean;
  createdAt: Date;
  updatedAt: Date;
}

export type DatabaseSchema = {
  users: User;
  posts: Post;
};

Migration Workflow

1. Define Schema: Update your TypeScript schema files
2. Generate Migration: gftd migration:generate "description"
3. Review: Check generated SQL files
4. Apply: gftd migration:up
5. Generate Types: gftd generate:types

Advanced Usage

# Dry run migrations
gftd migration:up --dry

# Rollback migrations
gftd migration:down --steps 2

# Pull schema from existing database
gftd db:pull --force

# Push schema changes (dangerous!)
gftd db:push --drop --seed

# Watch for schema changes
gftd generate:types --watch

🚀 Local Development Proxy

GFTD CLI includes a powerful local development proxy system that allows you to access multiple projects via custom subdomains instead of remembering port numbers.

Features

• 🌐 Access projects via {id}.gftd.ai.dev URLs
• 🔄 Automatic port allocation and management
• 🚀 Built-in project process management
• 📋 Easy project listing and control
• 🛑 Graceful shutdown and restart

Quick Start

# Start development server in current directory
gftd dev

# Start with custom settings
gftd dev --name myproject --subdomain myapp --command "pnpm dev"

# Start proxy server only
gftd dev --proxy

# List all projects
gftd dev --list

# Stop a specific project
gftd dev --stop <project-id>

# Stop all projects
gftd dev --stop-all

DNS Setup

To access projects via *.gftd.ai.dev domains, you need to set up DNS resolution:

Option 1: /etc/hosts (Simple)

# Add to /etc/hosts
127.0.0.1 *.gftd.ai.dev

Option 2: dnsmasq (Recommended)

# Install dnsmasq
brew install dnsmasq

# Configure wildcard DNS
echo "address=/.gftd.ai.dev/127.0.0.1" | sudo tee /etc/dnsmasq.d/gftd-dev.conf

# Start dnsmasq
sudo brew services start dnsmasq

# Configure DNS resolver
sudo mkdir -p /etc/resolver
echo "nameserver 127.0.0.1" | sudo tee /etc/resolver/gftd.ai.dev

Option 3: Built-in DNS Management (New!)

# GFTD CLI automatically manages your hosts file
gftd dev --name "My App" --subdomain "myapp"

# DNS entries are automatically added/removed
# No manual setup required!

# Check DNS status
gftd dns --status

# Initialize DNS management
gftd dns --init

Example Workflow

# 1. Start your Next.js project
cd my-nextjs-app
gftd dev --name "My Next App" --subdomain "myapp"

# 2. Start another project
cd ../my-react-app
gftd dev --name "My React App" --subdomain "reactapp"

# 3. Access your projects
# - http://myapp.gftd.ai.dev
# - http://reactapp.gftd.ai.dev

# 4. List all running projects
gftd dev --list

# 5. Stop all projects when done
gftd dev --stop-all

Configuration

The proxy system stores configuration in gftd.json:

{
  "port": 8080,
  "domain": "gftd.ai.dev",
  "dnsEnabled": true,
  "dnsAutoManage": true,
  "projects": {
    "project-id": {
      "id": "project-id",
      "name": "My Project",
      "path": "/path/to/project",
      "port": 3000,
      "subdomain": "myproject",
      "command": "pnpm dev",
      "status": "running"
    }
  }
}

DNS Management

GFTD CLI includes automatic DNS management that eliminates the need for manual DNS setup:

Features

• 🔧 Automatic hosts file management: No manual DNS configuration required
• 🛡️ Safe backup & restore: Automatic backup before any changes
• 🔄 Auto cleanup: DNS entries are removed when projects are deleted
• 🚀 Zero configuration: Works out of the box with proper permissions
• 🌐 Cross-platform: Works on macOS, Linux, and Windows

DNS Commands

| Command | Description | |---------|-------------| | gftd dns --status | Show DNS management status | | gftd dns --init | Initialize DNS management | | gftd dns --cleanup | Remove all GFTD DNS entries | | gftd dns --restore | Restore DNS from backup | | gftd dns --add <hostname> | Add DNS entry manually | | gftd dns --remove <hostname> | Remove DNS entry manually |

How It Works

1. Automatic Setup: When you start a project, GFTD automatically adds a DNS entry to your hosts file
2. Safe Management: Creates backup before making any changes
3. Clean Removal: Removes DNS entries when projects are stopped or deleted
4. Permission Handling: Uses sudo when needed for hosts file access

Example Usage

# Start a project (DNS automatically managed)
gftd dev --name "My App" --subdomain "myapp"
# DNS entry added: 127.0.0.1 myapp.gftd.ai.dev

# Check DNS status
gftd dns --status

# Remove project (DNS automatically cleaned up)
gftd dev --remove <project-id>
# DNS entry removed automatically

# Manual DNS management
gftd dns --add custom.gftd.ai.dev
gftd dns --remove custom.gftd.ai.dev

📝 License

MIT

🤝 Contributing

1. Fork the repository
2. Create your feature branch
3. Commit your changes
4. Push to the branch
5. Create a Pull Request

🔬 統合設定オプション (NEW!)

圏論・写像の概念を活用した数理的に簡潔な統合設定が利用可能です。

統合設定の特徴

• 単一ファイル管理: gftd.yml で設定とスキーマを統合
• YAML参照: & と * でDRY原則の徹底
• コンパクト記法: スキーマ定義が50%短縮
• 圏論的構造:数学的一貫性のある設計

使用方法

# 統合設定でプロジェクト初期化
gftd init --org-id=myorg --project=myproject --unified

# 統合設定でマイグレーション
gftd migrate --unified

# 自動検出（gftd.ymlが存在する場合）
gftd migrate

統合設定例

# gftd.yml - 設定とスキーマを統合
org: "myorg"
project: "myproject"

connections:
  kafka:
    endpoint: ${KAFKA_ENDPOINT}
    credentials: [${KAFKA_API_KEY}, ${KAFKA_API_SECRET}]
  ksqldb:
    endpoint: ${KSQLDB_ENDPOINT}
    credentials: [${KSQLDB_API_KEY}, ${KSQLDB_API_SECRET}]

schema:
  version: "1.0.0"
  topics:
    users:
      partitions: 3
      replication: 3
      config:
        cleanup.policy: compact
        retention.ms: -1

  streams:
    users:
      format: [JSON, KAFKA]
      schema:
        ID: VARCHAR(PK)    # コンパクト記法
        NAME: VARCHAR
        EMAIL: VARCHAR

  tables:
    users:
      source: users
      aggregation: |
        SELECT ID,
               LATEST_BY_OFFSET(NAME) AS NAME,
               LATEST_BY_OFFSET(EMAIL) AS EMAIL
        FROM {stream}
        GROUP BY ID

naming:
  topic: "{name}"
  stream: "{ORG}_{NAME}"
  table: "{ORG}_{NAME}"

詳細情報

統合設定の詳細は CATEGORICAL.md を参照してください。

🎯 クイックデモ

内蔵のDNS管理機能を試してみましょう：

# 1. テストプロジェクトに移動
cd test-project

# 2. 依存関係をインストール
npm install

# 3. GFTD CLI でプロジェクトを開始（DNS自動管理）
gftd dev --name "Test Project" --subdomain "test"

# 4. ブラウザで確認
# http://test.gftd.ai.dev でアクセス可能！

# 5. DNS状態を確認
gftd dns --status

# 6. プロジェクト一覧を確認
gftd dev --list

# 7. プロジェクトを停止
gftd dev --stop <project-id>

🚀 従来の方法との比較

従来のやり方（dnsmasq使用）:

# dnsmasqのインストール
brew install dnsmasq

# 設定ファイルの作成
echo "address=/.gftd.ai.dev/127.0.0.1" | sudo tee /etc/dnsmasq.d/gftd-dev.conf

# dnsmasqの起動
sudo brew services start dnsmasq

# DNS resolverの設定
sudo mkdir -p /etc/resolver
echo "nameserver 127.0.0.1" | sudo tee /etc/resolver/gftd.ai.dev

# プロジェクトの起動
npm run dev

GFTD CLI（DNS自動管理）:

# これだけ！
gftd dev --name "My Project" --subdomain "myproject"

🎉 特徴

• ✅ dnsmasq不要: 外部ツールのインストール不要
• ✅ 自動管理: プロジェクトの追加・削除で自動的にDNSエントリを更新
• ✅ 安全: 変更前に自動バックアップ、エラー時は復元可能
• ✅ クロスプラットフォーム: macOS、Linux、Windows対応
• ✅ ゼロ設定: 追加設定不要で即座に使用可能