@flatfile/implementation-utils-dedupe-worker

Provides a job worker for deduping records in a sheet as well as utility functions for deduping records.

Features

• deduplicateRecords: A function for deduping records in a sheet based on specified dedupe keys.
• getDedupeActionInputForm: A function for getting the input form for the dedupe action.
• DedupeJobWorker: An abstract job worker for deduping records in a sheet, compatible with both @flatfile/safe-api's FlatfileRecord and the legacy Item class from the embedded "safe" client.

Usage

To use the DedupeJobWorker, you need to create a concrete class that extends it and implements the fetchRecords and writeRecords methods. This allows you to define how records are retrieved from and written back to your sheet.

Example Implementation

Here's a complete example of how to set up a deduplication action on a sheet.

1. Create your Dedupe Worker

   Create a file, for example my-dedupe.worker.ts, and define your worker by extending DedupeJobWorker. This implementation uses @flatfile/safe-api for easier record handling.

   import type { Collection } from "collect.js";
   import { DedupeJobWorker } from "@flatfile/dedupe-worker";
   import api from "@flatfile/safe-api";
   
   export class MyDedupeWorker extends DedupeJobWorker {
     /**
      * Fetches all records from the sheet using a stream.
      */
     async fetchRecords() {
       return await api.records.stream({ sheetId: this.sheetId });
     }
   
     /**
      * Writes the updated, deleted, and new records back to the sheet.
      */
     async writeRecords(records: Collection) {
       return await api.records.write({ sheetId: this.sheetId }, records);
     }
   }

2. Register the Action on a Sheet

   When defining your sheet, add an action with the input form from getDedupeActionInputForm.

   import { getDedupeActionInputForm } from "@flatfile/dedupe-worker";
   
   const mySheetFields = [
     { key: "firstName", type: "string", label: "First Name" },
     { key: "lastName", type: "string", label: "Last Name" },
     { key: "email", type: "string", label: "Email" },
   ];
   
   export const MY_SHEET = {
     name: "My Sheet",
     slug: "my-sheet",
     fields: mySheetFields,
     actions: {
       dedupe: {
         operation: ACTION_DEDUPE,
         label: "Dedupe",
         description:
           "Dedupe records in this sheet based on the selected field.",
         mode: "foreground",
         primary: false,
         inputForm: getDedupeActionInputForm(mySheetFields),
       },
     },
   };

3. Register the Listener

   In your main listener file (e.g., index.ts), import your worker and use the worker() utility from @flatfile/implementation-utils-job-worker to simplify listener setup.

   import { FlatfileListener } from "@flatfile/listener";
   import { worker } from "@flatfile/implementation-utils-job-worker";
   import { getDedupeActionInputForm } from "@flatfile/dedupe-worker";
   import { MyDedupeWorker } from "./my-dedupe.worker";
   
   export default function (listener: FlatfileListener) {
     listener.use(worker(MyDedupeWorker));
   }