rustybytes-cloudflare-seeder

Published as the npm package cf-seeder (CLI: cf-seeder).

A CLI tool to seed local D1 databases from remote Cloudflare environments. This tool reads your wrangler.toml, exports databases from a specified environment, and saves them to your local Miniflare storage for development.

No installation required! Use it directly with npx, bunx, or pnpm dlx.

Features

• ✅ Single command seeding — Seed your local D1 databases with one command
• ✅ Multi-environment support — Seed from seed, staging, or production environments
• ✅ Automatic conversion — Converts SQL exports to SQLite format automatically
• ✅ Miniflare integration — Saves databases directly to the correct Miniflare storage location
• ✅ Safe operations — Read-only exports, never modifies source environments

Getting Started

Prerequisites

Before using this tool, ensure you have:

1. Wrangler CLI installed — Install Wrangler if you haven't already
2. Cloudflare authentication — Run npx wrangler login to authenticate with your Cloudflare account
3. A Cloudflare Workers project — Your project must have a wrangler.toml file with D1 database configurations
4. Package manager — One of: Node.js (npm/npx), Bun (bunx), or pnpm (pnpm dlx)

Installation

No installation required! This tool is designed to be used directly via package manager execution tools. This keeps your project dependencies clean and ensures you're always using the latest version.

Recommended: Use Directly (No Installation)

Run the tool directly without installing:

# Using npx (Node.js/npm)
npx cf-seeder --from staging

# Using bunx (Bun)
bunx cf-seeder --from staging

# Using pnpm dlx (pnpm)
pnpm dlx cf-seeder --from staging

Benefits:

• ✅ No installation needed
• ✅ Always uses the latest version
• ✅ Keeps your project dependencies clean
• ✅ Works from any directory

Alternative: Install as a Development Dependency

If you prefer to pin a specific version, you can install it locally:

# Using Bun
bun add -d cf-seeder

# Using npm
npm install --save-dev cf-seeder

# Using pnpm
pnpm add -D cf-seeder

Then run with:

bun run cf-seed --from staging
# or
npx cf-seed --from staging

Alternative: Global Installation

For global installation (not recommended):

# Using Bun
bun install -g cf-seeder

# Using npm
npm install -g cf-seeder

# Using pnpm
pnpm add -g cf-seeder

Then use the cf-seed command directly:

cf-seed --from staging

Quick Start Guide

1. Navigate to your Cloudflare Workers project directory:

   cd your-workers-project

2. Ensure you're authenticated with Cloudflare:

   npx wrangler login

3. Verify your wrangler.toml has D1 database configurations:

   [[d1_databases]]
   binding = "DB"
   database_name = "my-database"
   database_id = "abc123..."

4. Run the seeder to import data from a remote environment:

   # Recommended: Use directly without installation
   npx cf-seeder --from staging
   # or
   bunx cf-seeder --from staging
   # or
   pnpm dlx cf-seeder --from staging

5. Start your local development server:

   bun run dev
   # or
   npx wrangler dev

   Your local environment now has the data from the remote environment!

Usage

Command Syntax

The basic command structure is:

# Recommended: Use directly without installation
npx cf-seeder --from <environment> [options]
# or
bunx cf-seeder --from <environment> [options]
# or
pnpm dlx cf-seeder --from <environment> [options]

If you've installed it locally or globally:

cf-seed --from <environment> [options]
# or
npx cf-seed --from <environment> [options]

Available Environments

You can seed from any environment defined in your wrangler.toml:

• seed — Development/seed environment
• staging — Staging environment
• production — Production environment (use with caution)
• default — Base configuration (no environment prefix)
• Any custom environment name defined in [env.<name>] section

Command Options

| Option | Description | Required | Default | | ----------------- | --------------------------------- | -------- | --------------- | | --from <env> | Source environment to seed from | Yes | - | | --config <path> | Path to wrangler.toml file | No | wrangler.toml | | --force | Overwrite existing database files | No | false | | --help | Display help message | No | - | | --version | Show version number | No | - |

Usage Examples

Basic Usage

Seed from staging environment:

# Recommended: Use directly without installation
npx cf-seeder --from staging
# or
bunx cf-seeder --from staging
# or
pnpm dlx cf-seeder --from staging

Seed from production (with caution):

npx cf-seeder --from production

Seed from base/default configuration:

npx cf-seeder --from default

Advanced Usage

Force overwrite existing database files:

npx cf-seeder --from staging --force

This will overwrite any existing local database files without prompting.

Use a custom wrangler.toml path:

npx cf-seeder --from seed --config ./config/wrangler.toml

Seed from a custom environment:

If you have a custom environment like [env.develop] in your wrangler.toml:

npx cf-seeder --from develop

Alternative: If Installed Locally

If you've installed the package as a dependency:

# Using Bun
bun run cf-seed --from staging

# Using npm/npx
npx cf-seed --from staging

# Using pnpm
pnpm exec cf-seed --from staging

Alternative: If Installed Globally

If you've installed globally:

cf-seed --from staging

Configuration

wrangler.toml

The tool automatically reads your wrangler.toml to discover D1 database configurations. The seeder will:

1. Parse your wrangler.toml file
2. Identify all D1 database bindings
3. Match them to the specified environment
4. Export and seed each database found

Basic Configuration Example

name = "my-worker"

[[d1_databases]]
binding = "DB"
database_name = "my-database"
database_id = "abc123..."

[env.seed]
name = "my-worker-seed"

[env.staging]
name = "my-worker-staging"

[env.production]
name = "my-worker-production"

Environment-Specific Databases

You can define environment-specific D1 databases that override the base configuration:

# Base configuration
[[d1_databases]]
binding = "DB"
database_name = "my-database"
database_id = "base-db-id"

# Staging environment with its own database
[env.staging]
name = "my-worker-staging"

[[env.staging.d1_databases]]
binding = "DB"
database_name = "my-database-staging"
database_id = "staging-db-id"

# Production environment with its own database
[env.production]
name = "my-worker-production"

[[env.production.d1_databases]]
binding = "DB"
database_name = "my-database-production"
database_id = "production-db-id"

When you run cf-seed --from staging, the tool will:

• Use the database defined in [env.staging.d1_databases] if it exists
• Fall back to the base [[d1_databases]] configuration if no environment-specific database is found

Multiple Database Bindings

You can seed multiple databases in a single run:

[[d1_databases]]
binding = "USERS_DB"
database_name = "users-database"
database_id = "users-db-id"

[[d1_databases]]
binding = "PRODUCTS_DB"
database_name = "products-database"
database_id = "products-db-id"

[[d1_databases]]
binding = "ORDERS_DB"
database_name = "orders-database"
database_id = "orders-db-id"

Running cf-seed --from staging will export and seed all three databases.

How It Works

The seeding process follows these steps:

Workflow

┌─────────────────────────────────────────────────────────────┐
│  1. Parse wrangler.toml                                      │
│     → Discover D1 databases and environment configurations  │
└────────────────────┬────────────────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────────────────────┐
│  2. For each database:                                       │
│     → Determine database_id for the specified environment   │
│     → Check if local dev database_id exists (for matching)  │
└────────────────────┬────────────────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────────────────────┐
│  3. Export from Cloudflare                                   │
│     → Run: wrangler d1 export <database_name> --env <env>   │
│     → Downloads SQL dump file to temporary location         │
└────────────────────┬────────────────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────────────────────┐
│  4. Convert SQL to SQLite                                    │
│     → Parse SQL dump file                                    │
│     → Execute SQL statements against new SQLite database     │
│     → Create .sqlite file in temporary location              │
└────────────────────┬────────────────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────────────────────┐
│  5. Save to Miniflare Storage                                │
│     → Determine correct local database_id (dev environment)  │
│     → Copy .sqlite file to:                                 │
│       .wrangler/state/v3/d1/miniflare-D1DatabaseObject/     │
│       <local-database-id>.sqlite                             │
└────────────────────┬────────────────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────────────────────┐
│  6. Cleanup                                                  │
│     → Remove temporary SQL and SQLite files                 │
│     → Display success message                               │
└─────────────────────────────────────────────────────────────┘

Detailed Process

1. Read Configuration

   • Parses wrangler.toml using TOML parser
   • Extracts all [[d1_databases]] entries
   • Matches environment-specific configurations if --from is specified
   • Validates that required fields (database_name, database_id) are present

2. Export Data

   • For each database, executes: wrangler d1 export <database_name> --env <environment>
   • The export creates a SQL dump file containing all tables, data, and schema
   • Export files are saved to a temporary directory

3. Convert Format

   • Reads the SQL dump file
   • Creates a new SQLite database file
   • Executes all SQL statements (CREATE TABLE, INSERT, etc.) against the SQLite database
   • Handles SQLite-specific syntax requirements

4. Save Locally

   • Determines the local development database ID (from [env.local] or base config)
   • Creates the Miniflare storage directory structure if it doesn't exist
   • Copies the SQLite file to: .wrangler/state/v3/d1/miniflare-D1DatabaseObject/<database_id>.sqlite
   • Uses the local dev database ID so it matches what Wrangler uses during wrangler dev

5. Cleanup

   • Removes all temporary SQL dump files
   • Removes temporary SQLite files
   • Displays summary of seeded databases

Output Location

Database Storage

Databases are saved to the Miniflare storage directory:

.wrangler/state/v3/d1/miniflare-D1DatabaseObject/<database_id>.sqlite

Where <database_id> is determined by:

1. If [env.local] exists in your wrangler.toml with a d1_databases entry, it uses that database_id
2. Otherwise, it uses the database_id from the base [[d1_databases]] configuration
3. This ensures the local database file matches what Wrangler expects when running wrangler dev

Example

If your wrangler.toml has:

[[d1_databases]]
binding = "DB"
database_name = "my-database"
database_id = "abc123..."

[env.local]
[[env.local.d1_databases]]
binding = "DB"
database_id = "local-dev-id"

The seeded database will be saved as:

.wrangler/state/v3/d1/miniflare-D1DatabaseObject/local-dev-id.sqlite

This matches the database ID that Wrangler uses during local development.

Authentication

The tool uses the same authentication as Wrangler:

1. Via Wrangler Login (Recommended):

   npx wrangler login

2. Via Environment Variables:

   export CLOUDFLARE_API_TOKEN="your-api-token"
   export CLOUDFLARE_ACCOUNT_ID="your-account-id"

Troubleshooting

Common Issues and Solutions

Error: "wrangler.toml not found"

Problem: The tool cannot find your wrangler.toml file.

Solutions:

• Make sure you're running the command from your Cloudflare Workers project root directory
• Specify the path explicitly: npx cf-seeder --from staging --config ./path/to/wrangler.toml
• Verify the file exists and is named correctly (case-sensitive)

Error: "Environment 'X' not found"

Problem: The specified environment doesn't exist in your wrangler.toml.

Solutions:

• Check that the environment exists in your wrangler.toml under [env.<name>]
• Verify the environment name spelling (case-sensitive)
• Use default if you want to use the base configuration without an environment prefix
• List available environments by checking your wrangler.toml file

Error: "Database file already exists"

Problem: A database file already exists at the target location.

Solutions:

• Use the --force flag to overwrite: npx cf-seeder --from staging --force
• Manually delete the existing file: rm .wrangler/state/v3/d1/miniflare-D1DatabaseObject/<database_id>.sqlite
• Stop your dev server before seeding to avoid file lock issues

Error: "Failed to export database"

Problem: The export from Cloudflare failed.

Solutions:

• Check authentication: Run npx wrangler login to ensure you're authenticated
• Verify database name: Ensure the database_name in your wrangler.toml matches the actual database name in Cloudflare
• Check permissions: Verify you have access to the database in your Cloudflare account
• Test manually: Try exporting manually: npx wrangler d1 export <database_name> --env <environment>
• Check network: Ensure you have a stable internet connection

Error: "No D1 databases found in configuration"

Problem: No D1 databases are configured in your wrangler.toml.

Solutions:

• Add at least one [[d1_databases]] entry to your wrangler.toml
• Verify the TOML syntax is correct (check for missing brackets or quotes)
• Ensure the environment you're targeting has database configurations

Local data not updating

Problem: Changes made by the seeder aren't reflected when running wrangler dev.

Solutions:

1. Stop your dev server completely (Ctrl+C or kill the process)
2. Clear the state directory:

   rm -rf .wrangler/state/

3. Re-run the seeder:

   npx cf-seeder --from staging

4. Restart your dev server:

   npx wrangler dev

Note: Wrangler may cache database connections. Always stop the dev server before seeding.

Conversion errors

Problem: Errors occur during SQL to SQLite conversion.

Solutions:

• Check SQL export validity: The exported SQL file might be corrupted. Try re-exporting manually
• Review SQL syntax: Some Cloudflare D1-specific SQL might not be compatible. Check the SQL dump file
• Check dependencies: Ensure all required dependencies are installed (better-sqlite3 is used internally)
• Large databases: Very large databases might timeout. Consider exporting smaller subsets

Authentication errors

Problem: Getting authentication or permission errors.

Solutions:

• Re-authenticate: Run npx wrangler login again
• Check API token: If using environment variables, verify CLOUDFLARE_API_TOKEN is set correctly
• Verify account: Ensure account_id in wrangler.toml matches your Cloudflare account
• Check permissions: Ensure your Cloudflare account has D1 database access permissions

Database ID mismatch

Problem: The seeded database doesn't match what Wrangler expects.

Solutions:

• Check local environment config: Ensure [env.local] has the correct database_id if you're using it
• Verify base config: The base [[d1_databases]] should have a valid database_id
• Match binding names: Ensure the binding name matches between environments

Getting Help

If you encounter issues not covered here:

1. Check the logs: The tool provides detailed error messages. Read them carefully
2. Verify configuration: Double-check your wrangler.toml syntax and values
3. Test manually: Try running wrangler d1 export manually to isolate the issue
4. Check Wrangler version: Ensure you're using a recent version of Wrangler
5. Review documentation: Check the Cloudflare D1 documentation

Development

Project Structure

rustybytes-cloudflare-seeder/
├── src/
│   ├── cli.ts           # Main CLI entry point and command definitions
│   ├── parser.ts        # wrangler.toml parser and configuration reader
│   ├── wrangler.ts      # Wrangler command execution (d1 export)
│   ├── converter.ts     # SQL to SQLite conversion logic
│   ├── filesystem.ts    # File operations and Miniflare storage management
│   ├── dev-detector.ts  # Local development environment detection
│   ├── types.ts         # TypeScript type definitions
│   └── errors.ts        # Custom error classes
├── docs/                # Documentation files
│   ├── guide.mdx        # Extended guide documentation
│   └── d1-seeding-plan.md
├── index.ts             # Package entry point (executable)
├── package.json         # Package configuration and dependencies
├── tsconfig.json        # TypeScript configuration
├── wrangler.toml        # Example wrangler.toml (for testing)
└── README.md            # This file

Technology Stack

• Runtime: Bun (TypeScript-first runtime)
• Language: TypeScript
• CLI Framework: Commander.js
• TOML Parser: @iarna/toml
• Process Execution: execa
• SQLite: better-sqlite3 (used internally for conversion)

Running Locally

This project uses Bun and TypeScript. No build step is required for development:

# Run the CLI directly
bun run index.ts --from staging

# Or if you have a test wrangler.toml
bun run index.ts --from staging --config ./wrangler.toml

Development Workflow

1. Clone the repository:

   git clone <repository-url>
   cd rustybytes-cloudflare-seeder

2. Install dependencies:

   bun install

3. Make changes to the source files in src/

4. Test your changes:

   bun run index.ts --from staging

Contributing

When contributing to this project:

1. Follow the existing code style and patterns
2. Add appropriate error handling
3. Update documentation if adding new features
4. Test with different wrangler.toml configurations
5. Ensure all temporary files are cleaned up properly

Common Use Cases

Setting Up a New Developer Environment

When onboarding a new team member:

# 1. Clone the repository
git clone <your-repo>
cd <your-project>

# 2. Install dependencies
bun install
# or: npm install / pnpm install

# 3. Authenticate with Cloudflare
npx wrangler login

# 4. Seed local databases from staging (no installation needed!)
npx cf-seeder --from staging
# or: bunx cf-seeder --from staging
# or: pnpm dlx cf-seeder --from staging

# 5. Start development
bun run dev

Refreshing Local Data

When you need to update your local database with the latest data:

# Stop your dev server first
# Then run (no installation needed):
npx cf-seeder --from staging --force

Testing with Production-like Data

For testing features that require production-like data (be cautious with sensitive data):

npx cf-seeder --from production --force

Working with Multiple Environments

Switch between different environments easily:

# Work with staging data
npx cf-seeder --from staging --force
bun run dev

# Later, switch to seed environment
npx cf-seeder --from seed --force
bun run dev

CI/CD Integration

You can integrate this into your CI/CD pipeline to ensure test environments have fresh data:

# In your CI script (no installation needed)
npx cf-seeder --from staging --force
npm test

Best Practices

Environment Selection

1. Prefer staging/seed environments — Use production data only when absolutely necessary
2. Use appropriate environments — Match the environment to your development needs:
   • seed for initial setup and testing
   • staging for pre-production testing
   • production only for critical debugging (use with extreme caution)

Data Management

1. Regular updates — Re-seed periodically to keep local data fresh and relevant
2. Use --force carefully — Only overwrite when you're sure you want to replace local data
3. Stop dev server before seeding — Always stop wrangler dev before running the seeder to avoid file locks
4. Backup important local changes — If you've made local database changes, back them up before re-seeding

Configuration

1. Keep wrangler.toml updated — Ensure database IDs and names are correct and match your Cloudflare account
2. Use environment-specific configs — Define separate database configurations for each environment
3. Document custom environments — If you use custom environment names, document them for your team

Security

1. Never commit seeded data — Ensure .wrangler/ is in your .gitignore
2. Protect API tokens — Never commit wrangler.toml with API tokens or sensitive credentials
3. Review database access — Ensure team members only have access to appropriate environments
4. Sanitize sensitive data — Consider data sanitization if seeding from production (future feature)

Workflow

1. Seed before major development — Start with fresh data when beginning new features
2. Seed after schema changes — Re-seed after database migrations to test with updated schemas
3. Coordinate with team — Communicate when you're seeding to avoid conflicts in shared environments

Security Considerations

Data Protection

• ⚠️ Never commit exported data — The .wrangler/ directory should be in .gitignore. Seeded databases may contain sensitive information
• ⚠️ Protect API tokens — Never commit wrangler.toml with API tokens or sensitive credentials. Use environment variables instead
• ⚠️ Review database access — Ensure you have appropriate permissions for the databases you're exporting. Don't export databases you don't have access to

Production Data

• ⚠️ Use production data cautiously — Production databases may contain sensitive user data, PII, or business-critical information
• ⚠️ Follow data handling policies — Ensure you comply with your organization's data handling and privacy policies
• ⚠️ Secure local storage — Protect your local machine and ensure .wrangler/ directory has appropriate file permissions

Authentication

• ⚠️ Protect Cloudflare credentials — Never share or commit your Cloudflare API tokens
• ⚠️ Use least privilege — Grant only necessary permissions to service accounts or API tokens
• ⚠️ Rotate credentials regularly — Update API tokens periodically for security

Quick Reference

No Installation Required!

Use directly with your package manager:

# Using npx (Node.js/npm)
npx cf-seeder --from staging

# Using bunx (Bun)
bunx cf-seeder --from staging

# Using pnpm dlx (pnpm)
pnpm dlx cf-seeder --from staging

Basic Commands

# Seed from staging
npx cf-seeder --from staging

# Seed from production (with caution)
npx cf-seeder --from production

# Force overwrite existing databases
npx cf-seeder --from staging --force

# Use custom config file
npx cf-seeder --from staging --config ./custom/wrangler.toml

# Show help
npx cf-seeder --help

Alternative: If Installed

If you've installed the package:

# Local installation
bun add -d cf-seeder
# or
npm install --save-dev cf-seeder

# Then use
bun run cf-seed --from staging

npx cf-seed --from staging

### File Locations

-   **Configuration:** `wrangler.toml` (project root)
-   **Output:** `.wrangler/state/v3/d1/miniflare-D1DatabaseObject/<database_id>.sqlite`
-   **Temporary files:** Automatically cleaned up after seeding

### Environment Options

-   `seed` — Development/seed environment
-   `staging` — Staging environment
-   `production` — Production environment
-   `default` — Base configuration
-   Any custom environment from `[env.<name>]`

## Related Tools

-   [Wrangler](https://developers.cloudflare.com/workers/wrangler/) — Cloudflare Workers CLI
-   [Miniflare](https://miniflare.dev/) — Local Cloudflare Workers runtime
-   [Cloudflare D1 Documentation](https://developers.cloudflare.com/d1/) — D1 database documentation

## License

MIT