@flatfile/safe-api

A safer, more efficient version of @flatfile/api that adds streaming capabilities while maintaining all original functionality.

Installation

npm install @flatfile/safe-api

Usage

This package provides a safer version of @flatfile/api with additional streaming capabilities while maintaining all existing functionality and the familiar API interface. It integrates with @flatfile/records for robust record handling and change tracking.

Basic Usage

import api from '@flatfile/safe-api';

// Use exactly like the original @flatfile/api
const records = await api.records.get(workbookId);

Streaming Capabilities

The streaming API provides four different methods for working with records:

1. Get Records (Full Data with Collection Support)

import api from '@flatfile/safe-api';
import { FlatfileRecord, Collection } from '@flatfile/records';

// Get all records at once (default behavior)
const records = await api.records.stream.get({ 
  workbookId: 'your-workbook-id',
  includeMetadata: true,
  includeConfig: true 
});

// Or stream records in chunks for large datasets
for await (const collection of api.records.stream.get({ 
  workbookId: 'your-workbook-id',
  includeMetadata: true,
  includeConfig: true,
  stream: true
})) {
  // collection is a Collection<FlatfileRecord> with change tracking
  collection.each((record: FlatfileRecord<any>) => {
    console.log('Full record:', record.toJSON());
    console.log('Name field:', record.get('name'));
  });
}

2. Simple Records (Clean Data)

import api from '@flatfile/safe-api';

// Get all records at once in simplified format
const records = await api.records.stream.simple({ 
  workbookId: 'your-workbook-id' 
});

// Or stream records one by one for large datasets
for await (const record of api.records.stream.simple({ 
  workbookId: 'your-workbook-id',
  stream: true
})) {
  console.log('Clean record:', record);
  // record has no internal __k, __m prefixes
}

3. Update Records (With Change Tracking)

import api from '@flatfile/safe-api';
import { FlatfileRecord, Collection } from '@flatfile/records';

// Create a collection of records
const collection = new Collection([
  new FlatfileRecord({ id: '1', name: 'John' }),
  new FlatfileRecord({ id: '2', name: 'Jane' })
]);

// Make changes
collection.each((record: FlatfileRecord<any>) => {
  record.set('name', record.get('name').toUpperCase());
});

// Update records with change tracking
await api.records.stream.update(
  { 
    workbookId: 'your-workbook-id',
    truncate: false,
    snapshot: true
  },
  collection
);

4. Raw Upsert (Direct JSONL)

import api from '@flatfile/safe-api';

// Raw upsert of records
const records = [
  { id: '1', name: 'John' },
  { id: '2', name: 'Jane' }
];

await api.records.stream.raw(
  { 
    workbookId: 'your-workbook-id',
    truncate: true
  },
  records
);

API Reference

Records API

The records API provides comprehensive record handling capabilities:

Basic Operations

// Insert records
await api.records.insert(sheetId, records);

// Get record counts
await api.records.counts(sheetId, options);

// Get all records
const simpleRecords = await api.records.all.simple(sheetId, options);
const objectRecords = await api.records.all.object(sheetId, options);

Streaming Operations

// Stream full records with metadata
for await (const collection of api.records.stream.get({ 
  workbookId, stream: true 
})) {
  // Handle collection
}

// Stream simplified records
for await (const record of api.records.stream.simple({ 
  workbookId, stream: true 
})) {
  // Handle record
}

// Update records
await api.records.stream.update(options, collection);

// Raw upsert
await api.records.stream.raw(options, records);

Workbooks API

The workbooks API provides workbook management capabilities:

// Basic operations
await api.workbooks.get(workbookId);
await api.workbooks.list(options);
await api.workbooks.update(workbookId, data);
await api.workbooks.create(config);

// Sheet operations
const sheets = api.workbooks.sheets(workbook);
const sheet = api.workbooks.sheet(workbook, nameOrSlug);
const records = api.workbooks.records(sheetId);

// Lock operations
await api.workbooks.lock(workbookId, options);
await api.workbooks.unlock(workbookId, options);

// Bulk operations
await api.workbooks.bulk.create(config);
await api.workbooks.bulk.lock(workbookIds, options);
await api.workbooks.bulk.unlock(workbookIds, options);

Sheets API

The sheets API provides sheet management capabilities:

// Basic operations
await api.sheets.get(sheetId);
await api.sheets.list(options);
await api.sheets.validate(sheetId);

// Find sheet by name or slug
await api.sheets.find(nameOrSlug, options);

Files API

The files API provides file management capabilities:

// Basic operations
await api.files.get(fileId);
await api.files.update(fileId, data);

Jobs API

The jobs API provides job management capabilities:

// Basic operations
await api.jobs.get(jobId);
await api.jobs.list(options);
await api.jobs.create(config);
await api.jobs.update(jobId, data);

// Job lifecycle
await api.jobs.ack(jobId, data);
await api.jobs.complete(jobId, data);
await api.jobs.fail(jobId, data);

Spaces API

The spaces API provides space management capabilities:

// Basic operations
await api.spaces.get(spaceId);
await api.spaces.list(options);
await api.spaces.update(spaceId, data);

Configuration

The API client includes a configuration system that allows you to control debug logging and retry behavior. You can access the configuration through the api.config object:

// Enable or disable debug logging
api.config.debug = true;

// Configure retry behavior globally
api.config.retryConfig = {
  retry: true,        // Enable/disable retries
  maxAttempts: 5,     // Maximum number of retry attempts
  timeoutDelay: 500   // Base delay (in ms) between retries
};

// You can also update individual retry settings
api.config.retryConfig.retry = false;
api.config.retryConfig.maxAttempts = 3;
api.config.retryConfig.timeoutDelay = 1000;

The retry configuration follows this priority order:

1. Options passed to individual calls
2. Global configuration
3. Default values (retry: true, maxAttempts: 5, timeoutDelay: 500)

Example with per-call options:

// Use global config
const { data: sheets1 } = await api.sheets.list({ workbookId });

// Override retry settings for this call only
const { data: sheets2 } = await api.sheets.list(
  { workbookId },
  { retry: true, maxAttempts: 3, timeoutDelay: 1000 }
);

When debug logging is enabled, you'll see detailed information about:

• Request attempts and URLs
• Rate limit responses
• Retry delays and backoff calculations
• Error handling and recovery

API Rate Limiting

This package includes built-in rate limit handling that automatically manages retries when rate limits are hit. Each request independently handles rate limiting based on the API's response headers:

• Uses response headers for intelligent retry timing:
  • retry-after: Primary source for retry delay timing
  • x-ratelimit-reset: Fallback for calculating retry delay
  • Adds jitter to prevent thundering herd problems

The retry system works automatically when you enable it on your requests:

// Enable retries with default max attempts (5)
await api.sheets.list({ workbookId }, { retry: true });

// Enable retries with custom max attempts
await api.sheets.list({ workbookId }, { retry: true, maxAttempts: 10 });

When a rate limit (429) is hit:

1. The request will automatically calculate the appropriate retry delay using response headers
2. It will wait for the specified time (plus some jitter)
3. The request will retry automatically up to the specified max attempts
4. If max attempts are reached, it will throw a RetryError

This system is designed to handle parallel requests efficiently, with each request managing its own retry state independently. You can safely make multiple concurrent requests and the system will handle rate limits appropriately for each one.

Types

StreamRecordsQuery

interface StreamRecordsQuery {
  sheetId?: string;
  workbookId?: string;
  stream?: boolean;           // When true, returns AsyncGenerator for streaming
  includeMetadata?: boolean;
  includeMessages?: boolean;
  includeConfig?: boolean;
  includeSheet?: boolean;
  includeSheetSlug?: boolean;
  noContext?: boolean;
}

LockOptions

interface LockOptions {
  sheetId?: string;
  spaceId?: string;
}

MappedRecord

type MappedRecord = {
  __id: string;
  [key: string]: any;
};

Change Tracking

This package uses the FlatfileRecord class from @flatfile/records for change tracking. Here's an example:

import { FlatfileClient } from '@flatfile/api';
import { FlatfileRecord, Collection } from '@flatfile/records';

// Create a collection with a record
const collection = new Collection([
  new FlatfileRecord({ id: '1', name: 'John' })
]);

// Make changes
collection.each((record: FlatfileRecord<any>) => {
  record.set('name', 'Johnny');
});

// Get only changed records
const changes = collection.changes();

// Commit changes after successful update
changes.each((record: FlatfileRecord<any>) => record.commit());

Error Handling

All API methods will throw errors in the following cases:

1. Network errors (failed requests)
2. Invalid resource IDs (workbook, sheet, job, etc.)
3. Permission issues
4. Invalid data format in updates
5. Missing required parameters
6. Resource not found
7. Rate limit exceeded (will automatically retry)

Example error handling:

try {
  const workbook = await api.workbooks.get(workbookId);
  const sheet = api.workbooks.sheet(workbook, 'main');
  await api.workbooks.lock(workbookId, { sheetId: sheet.id });
} catch (error) {
  console.error('Operation failed:', error);
}

License

MIT