@flatfile/mapping

A powerful data transformation engine for structured data records. This package provides a flexible system for transforming data using a comprehensive set of mapping rules that can handle simple assignments, complex transformations, conditional logic, and data restructuring.

Installation

npm install @flatfile/mapping

Quick Start

import { Run, MappingRule, MappingRuleType } from '@flatfile/mapping';

// Sample data
const records = [
  { firstName: 'John', lastName: 'Doe', email: '[email protected]' },
  { firstName: 'Jane', lastName: 'Smith', email: '[email protected]' }
];

// Define mapping rules
const rules: MappingRule[] = [
  {
    type: MappingRuleType.ASSIGN,
    sourceField: 'firstName',
    destinationField: 'first_name'
  },
  {
    type: MappingRuleType.CONCATENATE,
    sourceFields: ['firstName', 'lastName'],
    destinationField: 'fullName',
    separator: ' '
  }
];

// Create and run mapping
const mapper = new Run(rules);
const result = mapper.applyRules(records);

Core Concepts

Mapping Rules

All transformations are defined using MappingRule objects. Each rule specifies:

• type: The transformation operation to perform
• sourceField(s): Input field(s) from the source record
• destinationField(s): Output field(s) in the transformed record
• Additional configuration: Rule-specific options

Base Rule Properties

Every mapping rule extends the base interface:

interface BaseMappingRule {
  type: MappingRuleType;
  filter?: string;        // FFQL filter to conditionally apply rule
  name?: string;          // Optional rule name for debugging
  description?: string;   // Optional rule description
  timestamp?: number;     // Optional timestamp for rule differentiation
}

FFQL Filters

Rules can be conditionally applied using FFQL (Flatfile Query Language) filters:

// Apply rule only to records where status is 'active'
filter: "status eq 'active'"

// Reference destination fields with 'destination!' prefix
filter: "destination!processedStatus eq 'pending'"

// Reference source fields with 'source!' prefix  
filter: "source!originalValue eq 'test'"

Rule Types

1. ASSIGN - Direct Field Assignment

Assigns a source field value to a destination field.

{
  type: MappingRuleType.ASSIGN,
  sourceField: 'email',
  destinationField: 'emailAddress'
}

2. CONSTANT - Assign Static Values

Assigns a constant value to a destination field.

{
  type: MappingRuleType.CONSTANT,
  destinationField: 'status',
  value: 'active'
}

3. TRANSFORM - Text Transformations

Applies simple text transformations.

{
  type: MappingRuleType.TRANSFORM,
  sourceField: 'name',
  destinationField: 'name_upper',
  transform: 'uppercase' // or 'lowercase'
}

4. CONCATENATE - Join Multiple Fields

Combines multiple source fields into a single destination field.

{
  type: MappingRuleType.CONCATENATE,
  sourceFields: ['firstName', 'lastName'],
  destinationField: 'fullName',
  separator: ' ' // optional, defaults to empty string
}

5. INTERPOLATE - Template-Based Concatenation

Creates formatted strings using template syntax.

{
  type: MappingRuleType.INTERPOLATE,
  sourceFields: ['firstName', 'lastName'],
  destinationField: 'greeting',
  output: 'Hello {0} {1}!' // {0} = firstName, {1} = lastName
}

6. COALESCE - First Non-Empty Value

Takes the first non-null, non-undefined, non-empty value from a list of fields.

{
  type: MappingRuleType.COALESCE,
  sourceFields: ['preferredName', 'firstName', 'displayName'],
  destinationField: 'name',
  defaultValue: 'Unknown' // optional fallback
}

7. ARRAY - Create Arrays

Creates an array from multiple source fields.

{
  type: MappingRuleType.ARRAY,
  sourceFields: ['phone1', 'phone2', 'phone3'],
  destinationField: 'phoneNumbers'
}

8. REGEX_EXTRACT - Pattern Extraction

Extracts values using regular expressions.

{
  type: MappingRuleType.REGEX_EXTRACT,
  sourceField: 'fullName',
  destinationFields: ['firstName', 'lastName'],
  regex: ['(\\w+)', '(\\w+)$'] // or single regex string
}

9. FIND_REPLACE - Value Substitution

Replaces specific values with other values.

{
  type: MappingRuleType.FIND_REPLACE,
  destinationField: 'country',
  values: [
    { find: 'US', replace: 'United States' },
    { find: 'UK', replace: 'United Kingdom' }
  ]
}

10. NORMALIZE_LIST - List Processing

Converts delimited strings into normalized arrays.

{
  type: MappingRuleType.NORMALIZE_LIST,
  sourceField: 'tags',
  destinationField: 'tagList',
  separator: ',',
  values: [ // optional value normalization
    { find: 'js', replace: 'JavaScript' }
  ],
  unique: true,  // remove duplicates (default: true)
  trim: true     // trim whitespace (default: true)
}

11. ARITHMETIC - Mathematical Operations

Performs mathematical calculations on numeric fields.

{
  type: MappingRuleType.ARITHMETIC,
  sourceFields: ['price', 'tax', 'discount'],
  destinationField: 'total',
  equation: 'price + tax - discount'
}

12. NEST - Create Nested Objects

Groups related fields into nested objects using regex patterns.

{
  type: MappingRuleType.NEST,
  destinationField: 'addresses',
  subfields: [
    {
      sourceRegex: 'address\\.([0-9]+)\\.street',
      destinationSubfield: 'street'
    },
    {
      sourceRegex: 'address\\.([0-9]+)\\.city', 
      destinationSubfield: 'city'
    }
  ]
}
// Transforms: { 'address.1.street': '123 Main', 'address.1.city': 'NYC' }
// Into: { addresses: [{ street: '123 Main', city: 'NYC', __control: '1' }] }

13. FUNCTION - Custom Transformations

Executes custom JavaScript functions for complex transformations.

{
  type: MappingRuleType.FUNCTION,
  sourceFields: ['birthDate'],
  destinationFields: ['age'],
  function: 'calculateAge' // Function name to execute
}

14. SPLIT - Field Splitting

Splits single fields into multiple destination fields.

{
  type: MappingRuleType.SPLIT,
  sourceFields: ['fullName'],
  destinationFields: ['firstName', 'lastName'],
  function: 'splitName'
}

15. SUBPROGRAM - Conditional Rule Groups

Groups multiple rules to be applied conditionally.

{
  type: MappingRuleType.SUBPROGRAM,
  filter: "userType eq 'premium'",
  rules: [
    { type: MappingRuleType.CONSTANT, destinationField: 'priority', value: 'high' },
    { type: MappingRuleType.CONSTANT, destinationField: 'support', value: 'premium' }
  ]
}

16. UNPIVOT - Reshape Data

Transforms wide-format data into long-format.

{
  type: MappingRuleType.UNPIVOT,
  destinationFields: ['metric', 'value'],
  columnMapping: [
    { 'sales_q1': 'Q1 Sales', 'sales_q2': 'Q2 Sales' },
    { 'sales_q1': 'revenue', 'sales_q2': 'revenue' }
  ]
}

17. Utility Rules

Additional utility rules for data management:

• DELETE: Remove fields from destination
• IGNORE: Explicitly ignore source fields
• ADD_FIELD: Add empty fields to destination
• APPLY_CHANGES: Apply batch modifications
• DEDUPE: Remove duplicate records

Advanced Features

Custom VM Functions

For FUNCTION and SPLIT rules, provide custom JavaScript execution:

const customFunctions = {
  calculateAge: (birthDate: string) => {
    const birth = new Date(birthDate);
    const today = new Date();
    return today.getFullYear() - birth.getFullYear();
  },
  
  splitName: (fullName: string) => {
    const parts = fullName.split(' ');
    return [parts[0], parts.slice(1).join(' ')];
  }
};

const mapper = new Run(rules, {
  getVM: (functionName: string) => customFunctions[functionName]
});

Calculations in Function Rules

Function rules support aggregation calculations:

{
  type: MappingRuleType.FUNCTION,
  sourceFields: ['scores'],
  destinationFields: ['totalScore', 'avgScore'],
  function: 'processScores',
  calculations: [
    { key: 'total', type: CalculationType.SUM, fieldKey: 'scores', value: null },
    { key: 'average', type: CalculationType.AVERAGE, fieldKey: 'scores', value: null }
  ]
}

Entry Processing

For individual record processing, use the Entry class:

import { Entry } from '@flatfile/mapping';

const entry = new Entry(singleRecord, rules, { vm: customVMProvider });
const transformedRecord = entry.getTransformed();

API Reference

Run Class

Main class for bulk record processing.

Constructor

new Run(rules: MappingRule[], vm?: { getVM?: (code: string) => Function })

Methods

• applyRules(records: Record<string, any>[]): Record<string, any>[]
• getKeyHierarchy(options?: { includeUnused?: boolean }): KeyNode[]

Entry Class

Class for individual record processing.

Constructor

new Entry(record: Record<string, any>, rules: MappingRule[], options?: EntryOptions)

Methods

• getTransformed(): Record<string, any>
• getOriginal(): Record<string, any>

Error Handling

The mapping engine includes comprehensive error handling:

import { parseAsMappingRule } from '@flatfile/mapping';

try {
  const rule = parseAsMappingRule(ruleObject);
} catch (error) {
  console.error('Invalid rule format:', error.message);
}

Performance Considerations

• Rules are processed in the order they appear in the array
• Use filters to conditionally apply expensive transformations
• Consider using SUBPROGRAM rules to group related transformations
• Custom VM functions should be optimized for performance

Examples

Complete Transformation Example

import { Run, MappingRule, MappingRuleType } from '@flatfile/mapping';

const sourceData = [
  {
    first_name: 'john',
    last_name: 'doe', 
    email_address: '[email protected]',
    phone_primary: '555-1234',
    phone_secondary: '555-5678',
    user_type: 'premium',
    birth_date: '1990-01-15'
  }
];

const rules: MappingRule[] = [
  // Basic assignments with transformations
  {
    type: MappingRuleType.TRANSFORM,
    sourceField: 'first_name',
    destinationField: 'firstName',
    transform: 'uppercase'
  },
  {
    type: MappingRuleType.ASSIGN,
    sourceField: 'last_name', 
    destinationField: 'lastName'
  },
  
  // Create full name
  {
    type: MappingRuleType.CONCATENATE,
    sourceFields: ['first_name', 'last_name'],
    destinationField: 'fullName',
    separator: ' '
  },
  
  // Create phone array
  {
    type: MappingRuleType.ARRAY,
    sourceFields: ['phone_primary', 'phone_secondary'],
    destinationField: 'phoneNumbers'
  },
  
  // Conditional processing for premium users
  {
    type: MappingRuleType.SUBPROGRAM,
    filter: "user_type eq 'premium'",
    rules: [
      {
        type: MappingRuleType.CONSTANT,
        destinationField: 'priority',
        value: 'high'
      },
      {
        type: MappingRuleType.FUNCTION,
        sourceFields: ['birth_date'],
        destinationFields: ['age'],
        function: 'calculateAge'
      }
    ]
  }
];

// Custom function for age calculation
const customFunctions = {
  calculateAge: (birthDate: string) => {
    const birth = new Date(birthDate);
    const today = new Date();
    let age = today.getFullYear() - birth.getFullYear();
    const monthDiff = today.getMonth() - birth.getMonth();
    if (monthDiff < 0 || (monthDiff === 0 && today.getDate() < birth.getDate())) {
      age--;
    }
    return age;
  }
};

const mapper = new Run(rules, {
  getVM: (functionName: string) => customFunctions[functionName]
});

const result = mapper.applyRules(sourceData);
console.log(JSON.stringify(result, null, 2));

Contributing

This package is part of the Flatfile Platform monorepo. See the main repository for contribution guidelines.

License

ISC License