@flatfile/safe-api

A safer, more efficient version of @flatfile/api that adds streaming capabilities while maintaining all original functionality.

Installation

npm install @flatfile/safe-api

Usage

This package provides a safer version of @flatfile/api with additional streaming capabilities while maintaining all existing functionality and the familiar API interface.

Basic Usage

import api from '@flatfile/safe-api';

// Use exactly like the original @flatfile/api
const records = await api.records.get(workbookId);

Streaming Capabilities

The streaming API provides four different methods for working with records:

1. Get Records (Full Data with Collection Support)

import api from '@flatfile/safe-api';
import { Collection, Item } from '@flatfile/safe-api';

// Get all records at once (default behavior)
const records = await api.records.stream.get({ 
  workbookId: 'your-workbook-id',
  includeMetadata: true,
  includeConfig: true 
});

// Or stream records in chunks for large datasets
for await (const collection of api.records.stream.get({ 
  workbookId: 'your-workbook-id',
  includeMetadata: true,
  includeConfig: true,
  stream: true
})) {
  // collection is a Collection<Item> with change tracking
  collection.each((item: Item<any>) => {
    console.log('Full record:', item.toJSON());
  });
}

2. Simple Records (Clean Data)

import api from '@flatfile/safe-api';

// Get all records at once in simplified format
const records = await api.records.stream.simple({ 
  workbookId: 'your-workbook-id' 
});

// Or stream records one by one for large datasets
for await (const record of api.records.stream.simple({ 
  workbookId: 'your-workbook-id',
  stream: true
})) {
  console.log('Clean record:', record);
  // record has no internal __k, __m prefixes
}

3. Update Records (With Change Tracking)

import api from '@flatfile/safe-api';
import { Collection, Item } from '@flatfile/safe-api';

// Create a collection of records
const collection = new Collection([
  new Item({ id: '1', name: 'John' }),
  new Item({ id: '2', name: 'Jane' })
]);

// Make changes
collection.each((item: Item<any>) => {
  item.toJSON().name = item.toJSON().name.toUpperCase();
});

// Update records with change tracking
await api.records.stream.update(
  { 
    workbookId: 'your-workbook-id',
    truncate: false,
    snapshot: true
  },
  collection
);

4. Raw Upsert (Direct JSONL)

import api from '@flatfile/safe-api';

// Raw upsert of records
const records = [
  { id: '1', name: 'John' },
  { id: '2', name: 'Jane' }
];

await api.records.stream.raw(
  { 
    workbookId: 'your-workbook-id',
    truncate: true
  },
  records
);

API Reference

api.records.stream.get(options?: StreamRecordsQuery)

Gets or streams records as Collection with full metadata and change tracking support.

Options:

• workbookId: The ID of the workbook
• sheetId: The ID of the sheet (alternative to workbookId)
• stream: When true, returns AsyncGenerator for streaming records. When false (default), returns all records at once
• includeMetadata: Include record metadata
• includeConfig: Include configuration
• includeSheet: Include sheet information
• includeSheetSlug: Include sheet slug
• noContext: Exclude context information

Returns:

• When stream: false (default): Promise<Collection<Item>>
• When stream: true: AsyncGenerator<Collection<Item>>

api.records.stream.simple(options?: StreamRecordsQuery)

Gets or streams records in a simplified format without internal metadata.

Options:

• workbookId: The ID of the workbook
• sheetId: The ID of the sheet (alternative to workbookId)
• stream: When true, returns AsyncGenerator for streaming records. When false (default), returns all records at once

Returns:

• When stream: false (default): Promise<SimpleRecord[]>
• When stream: true: AsyncGenerator

api.records.stream.update(options: StreamRecordsPatchQuery, records: Collection<Item>)

Updates records using JSONL format with change tracking support.

Options:

• workbookId: The ID of the workbook
• sheetId: The ID of the sheet (alternative to workbookId)
• truncate: Whether to truncate existing records (default: false)
• snapshot: Create a snapshot before update (default: false)
• silent: Suppress notifications (default: false)

Returns: Promise<{ success: true }>

api.records.stream.raw(options: StreamRecordsPatchQuery, records: Array<Record<string, Primitive>>)

Raw upsert of records using JSONL format with GZIP compression.

Options:

• workbookId: The ID of the workbook
• sheetId: The ID of the sheet (alternative to workbookId)
• truncate: Whether to truncate existing records (default: false)
• snapshot: Create a snapshot before update (default: false)
• silent: Suppress notifications (default: false)

Returns: Promise<{ success: true }>

Types

Item

A wrapper class for records that provides change tracking and record management.

class Item<T> {
  constructor(data: T);
  toJSON(): T;                 // Get current state
  isDeleted(): boolean;        // Check if marked for deletion
  delete(): void;              // Mark for deletion
  changeset(): T;              // Get changes
  commit(): void;              // Commit changes
}

SimpleRecord

interface SimpleRecord {
  id?: string;
  metadata?: Record<string, any>;
  [key: string]: string | number | boolean | null | undefined | Record<string, any>;
}

StreamRecordsQuery

interface StreamRecordsQuery {
  sheetId?: string;
  workbookId?: string;
  stream?: boolean;           // When true, returns AsyncGenerator for streaming
  includeMetadata?: boolean;
  includeConfig?: boolean;
  includeSheet?: boolean;
  includeSheetSlug?: boolean;
  noContext?: boolean;
}

Change Tracking

The Item class and Collection support provide robust change tracking:

import { Collection, Item } from '@flatfile/safe-api';

// Create a collection
const collection = new Collection([
  new Item({ id: '1', name: 'John' })
]);

// Make changes
collection.each((item: Item<any>) => {
  const record = item.toJSON();
  record.name = 'Johnny';
  // Changes are tracked automatically
});

// Get only changed records
const changes = collection.changes();

// Commit changes after successful update
changes.each((item: Item<any>) => item.commit());

Error Handling

All streaming methods will throw errors in the following cases:

1. Network errors (failed requests)
2. Invalid workbook or sheet IDs
3. Permission issues
4. Invalid record format in updates
5. Empty updates when not using truncate mode
6. Temporary record deletion issues

Example error handling:

try {
  for await (const record of api.records.stream.simple({ workbookId })) {
    await processRecord(record);
  }
} catch (error) {
  console.error('Streaming error:', error);
}

License

MIT