Data Migrator

Migrate data from one Dataverse environment to another with intelligent auto-mapping and smart operations.

Overview

The Data Migrator is a React-based tool designed exclusively for Power Platform ToolBox (PPTB) that enables seamless data transfer between Dataverse environments with advanced features:

• Auto-Mapping: Automatically map users, teams, and business units between environments
• Smart Operations: Choose between create, update, or upsert operations
• Field Selection: Select which fields to migrate
• Lookup Handling: Intelligent mapping of lookup fields and references
• Progress Tracking: Real-time progress monitoring with detailed status for each record
• Batch Processing: Efficient batch processing for large data sets

Features

Auto-Mapping

The tool automatically maps system entities between source and target environments:

• Users: Matched by domain name, email address, or full name
• Teams: Matched by name and team type
• Business Units: Matched by name

Each mapping includes a confidence level (high, medium, low) based on the matching criteria used.

Smart Migration Operations

Choose the operation that fits your scenario:

• Create: Insert new records only (fails if record exists)
• Update: Update existing records by primary key (requires records to exist)
• Delete: Delete records from target environment by matching primary key

Preview Before Migration

• Preview data before migration to verify the operation
• Shows action column (CREATE, UPDATE, DELETE)
• Displays primary ID, primary name, and selected fields
• Limited to first 100 records for quick review
• Confirm before starting the actual migration

Field and Lookup Management

• Select which fields to include in the migration
• Fields are loaded on-demand after entity selection for better performance
• Configure lookup field mapping strategies:
  • Auto-Map: Automatically map system entities (users, teams, business units)
  • Skip: Exclude the lookup field from migration

Flexible Filtering

• OData Filters: Use OData syntax for simple filtering
  • Example: statecode eq 0 and createdon gt 2024-01-01
• FetchXML Queries: Use complete FetchXML for complex queries
  • Supports advanced filtering, joins, and aggregations
• Toggle between filter types with a modern selector

Advanced Options

• Filter Query: Apply OData or FetchXML filters to select specific records
• Batch Size: Control batch size for optimal performance (1-100 records per batch)

Modern, Fluid UI

• Step-based workflow: Clear progression through configuration steps
• Collapsible sections: Minimize scrolling with expandable step cards
• Modern design: Gradient headers, card-based layout, smooth transitions
• Responsive: Works well on different screen sizes
• Visual feedback: Color-coded badges, progress indicators, and status displays

Progress Tracking

Real-time monitoring with:

• Overall progress bar
• Statistics (total, successful, failed, skipped)
• Batch processing status
• Detailed record-by-record status with error messages

Technical Stack

• React 18 with TypeScript
• Fluent UI React Components for modern UI
• Vite for fast development and optimized builds
• PPTB API for all Dataverse operations
• @pptb/types v1.0.16 - Latest PPTB type definitions

Installation

This tool is distributed as part of the PPTB-Tools monorepo and can be installed in Power Platform ToolBox.

Prerequisites

• Node.js >= 18.0.0
• npm >= 9.0.0

Build

cd tools/data-migrator
npm install
npm run build

The build output will be in the dist directory.

Usage in PPTB

The tool follows a step-by-step workflow:

1. Install the tool in Power Platform ToolBox
2. Connect to your source environment (primary connection)
3. Select a secondary connection as the target environment
4. Open the Data Migrator tool
5. The tool will display both connections:
   • Source (Primary): Your source environment
   • Target (Secondary): Your target environment

Migration Workflow

6. Select an entity to migrate
   • Entity data (without fields) loads initially for faster performance
7. Wait for fields to load automatically after entity selection
8. Select fields to include in the migration
9. Add filter (optional) to select a subset of records using OData syntax
10. Choose migration operation from settings:
    • Create: Insert new records only
    • Update: Update existing records by primary key
    • Delete: Delete records from target environment
11. Set batch size (max 100 records per batch)
12. Auto-map system entities (optional):
    • Click "Auto-Map System Entities" to map users, teams, and business units
    • Review the auto-mapping results
13. Preview the data to be migrated:
    • Shows Action column (CREATE, UPDATE, DELETE)
    • Displays primary ID and primary name
    • Shows selected fields
    • Limited to first 100 records for preview
14. Start Migration from the preview and monitor progress

Important Notes

• Secondary Connection Required: This tool requires both a primary (source) and secondary (target) connection to be configured in PPTB
• Source Environment: Data is read from the primary connection
• Target Environment: Data is written to the secondary connection
• Auto-Mapping: Users, teams, and business units are mapped between source and target environments
• Preview First: Always preview data before starting migration to verify the operation and data
• Batch Limit: Maximum batch size is 100 records for optimal performance

Use Cases

• Environment Refresh: Migrate configuration or transactional data after environment refresh
• Dev to Test Migration: Move test data from development to test environments
• Cross-Tenant Migration: Transfer data between different tenants with user/team mapping
• Partial Data Migration: Use filters to migrate specific records
• Data Deletion: Remove specific records from target environment based on source data

Architecture

Components

• App.tsx: Main application component and state management
• EntitySelector: Entity selection dropdown
• OperationSelector: Migration operation selection
• FieldSelector: Field selection with bulk actions
• LookupMapper: Lookup field mapping configuration
• MigrationProgress: Real-time progress display
• AutoMappingPanel: Auto-mapping results modal

Utilities

• DataverseClient: Handles all Dataverse API interactions via PPTB API
• MigrationEngine: Core migration logic with auto-mapping and transformation

Design Philosophy

The UI follows a modern minimalist approach:

• No header or unnecessary chrome
• Compact layout to minimize scrolling
• Clear visual hierarchy
• Progressive disclosure (options appear as needed)
• Real-time feedback
• Clean, professional appearance

Reference

This tool is inspired by Colso.Xrm.DataTransporter with enhancements for modern React, PPTB integration, and improved user experience.

Limitations

• Primary key-based operations only (for update and upsert)
• Lookup auto-mapping limited to system entities (users, teams, business units)
• Requires same metadata schema in source and target environments
• Large data sets may take time to migrate (monitor batch progress)

Best Practices

1. Test First: Always test with a small data set first
2. Use Filters: Use OData filters to migrate specific records
3. Check Mappings: Review auto-mapping results before starting migration
4. Monitor Progress: Watch for errors during migration
5. Backup Data: Always backup target environment before migration
6. Same Metadata: Ensure source and target have matching entity/field schemas

Troubleshooting

Migration Fails

• Check that target environment has the same entity/field structure
• Verify required fields have values
• Review error messages in the progress panel

Lookup Mapping Issues

• Run auto-mapping before starting migration
• Verify users/teams/business units exist in target environment
• Check that names match between environments

Performance Issues

• Reduce batch size for better stability
• Use filters to migrate fewer records at once
• Check network connectivity

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the GPL-2.0 License - see the LICENSE file for details.

Support

For issues, questions, or contributions, please visit the GitHub repository.