lazy-api-paginator

A TypeScript module for lazily fetching paginated API data using async generators. Features include exponential backoff retry logic and lifecycle hooks.

Features

• Lazy loading of paginated API data using async generators
• Iterate over items one-by-one without loading all pages into memory
• Built-in strategies for cursor, offset, page number, link header, and keyset pagination
• Exponential backoff retry with configurable jitter
• Lifecycle hooks: onBeforeFetch, onAfterFetch, onError, onData
• SSRF protection for secure server-to-server calls (via ssrf-agent-guard)
• Full TypeScript support
• Works with both ESM and CommonJS

Installation

npm install lazy-api-paginator

Usage

Basic Usage

import { createPaginator } from 'lazy-api-paginator';

interface ApiResponse {
  data: User[];
  nextCursor: string | null;
}

interface User {
  id: number;
  name: string;
}

const paginator = createPaginator<ApiResponse, User>({
  initialUrl: 'https://api.example.com/users',
  extractItems: (response) => response.data,
  getNextPageUrl: (response) =>
    response.nextCursor
      ? `https://api.example.com/users?cursor=${response.nextCursor}`
      : null,
});

// Iterate lazily - pages are fetched on-demand
for await (const user of paginator) {
  console.log(user.name);
}

With Lifecycle Hooks

const paginator = createPaginator<ApiResponse, User>({
  initialUrl: 'https://api.example.com/users',
  extractItems: (response) => response.data,
  getNextPageUrl: (response) => response.nextCursor,
  hooks: {
    onBeforeFetch: ({ url, pagination }) => {
      console.log(`Fetching page ${pagination.page}: ${url}`);
    },
    onAfterFetch: ({ response, duration }) => {
      console.log(`Fetched ${response.data.data.length} items in ${duration}ms`);
    },
    onError: ({ error, attempt, willRetry }) => {
      console.error(`Error (attempt ${attempt}): ${error.message}`);
      if (willRetry) console.log('Retrying...');
    },
    onData: ({ item, globalIndex }) => {
      console.log(`Processing item ${globalIndex}: ${item.name}`);
    },
  },
});

Custom Retry Configuration

const paginator = createPaginator<ApiResponse, User>({
  initialUrl: 'https://api.example.com/users',
  extractItems: (response) => response.data,
  getNextPageUrl: (response) => response.nextCursor,
  retry: {
    maxRetries: 5,
    initialDelay: 1000,      // 1 second
    maxDelay: 30000,         // 30 seconds
    backoffMultiplier: 2,    // Exponential factor
    jitter: 0.1,             // 10% randomness
    retryableStatusCodes: [408, 429, 500, 502, 503, 504],
    isRetryable: (error, statusCode) => {
      // Custom retry logic
      return statusCode === 418; // Retry teapot errors
    },
  },
});

Request Configuration

const paginator = createPaginator<ApiResponse, User>({
  initialUrl: 'https://api.example.com/users',
  extractItems: (response) => response.data,
  getNextPageUrl: (response) => response.nextCursor,
  requestConfig: {
    method: 'POST',
    headers: {
      'Authorization': 'Bearer token123',
      'Content-Type': 'application/json',
    },
    body: { filter: 'active' },
    timeout: 10000,
  },
});

Helper Methods

// Get first N items
const firstTen = await paginator.take(10);

// Get all items (use with caution for large datasets)
const allUsers = await paginator.toArray();

Built-in Pagination Strategies

Use pre-built strategies to eliminate boilerplate for common API patterns:

import { createPaginator, strategies } from 'lazy-api-paginator';

// Cursor-based (Slack, Stripe, Notion)
const cursorPaginator = createPaginator({
  initialUrl: 'https://api.slack.com/users.list',
  ...strategies.cursor({
    dataPath: 'members',
    cursorPath: 'response_metadata.next_cursor',
  }),
});

// Offset-based (traditional REST APIs)
const offsetPaginator = createPaginator({
  initialUrl: 'https://api.example.com/items?offset=0&limit=100',
  ...strategies.offset({
    dataPath: 'items',
    totalPath: 'total',
    pageSize: 100,
  }),
});

// Page number-based (Laravel, Django)
const pagePaginator = createPaginator({
  initialUrl: 'https://api.example.com/items?page=1',
  ...strategies.pageNumber({
    dataPath: 'results',
    totalPagesPath: 'total_pages',
  }),
});

// Link header (GitHub API)
const linkStrategy = strategies.linkHeader({ dataPath: '' });
const githubPaginator = createPaginator({
  initialUrl: 'https://api.github.com/repos/owner/repo/issues',
  ...linkStrategy,
  hooks: {
    onAfterFetch: ({ response }) => {
      const link = response.headers['link'];
      if (link) linkStrategy.setNextFromHeader(link);
    },
  },
});

// Keyset/Seek (efficient for large datasets)
const keysetPaginator = createPaginator({
  initialUrl: 'https://api.example.com/items',
  ...strategies.keyset({
    dataPath: 'data',
    keyPath: 'id',
    hasMorePath: 'has_more',
  }),
});

SSRF Protection

For server-to-server calls, enable SSRF (Server-Side Request Forgery) protection to block requests to internal networks, cloud metadata endpoints, and other potentially dangerous destinations.

First, install the optional dependency:

npm install ssrf-agent-guard

Then enable SSRF protection in your paginator:

import { createPaginator } from 'lazy-api-paginator';

const paginator = createPaginator({
  initialUrl: 'https://api.example.com/items',
  extractItems: (r) => r.items,
  getNextPageUrl: (r) => r.next,
  ssrfProtection: {
    enabled: true,
    options: {
      // Optional: customize ssrf-agent-guard options
      mode: 'block', // 'block' | 'report' | 'allow'
    },
  },
});

You can also use the standalone createSecureFetch utility:

import { createSecureFetch, createPaginator } from 'lazy-api-paginator';

const secureFetch = await createSecureFetch({ enabled: true });

const paginator = createPaginator({
  initialUrl: 'https://api.example.com/items',
  extractItems: (r) => r.items,
  getNextPageUrl: (r) => r.next,
  fetchFn: secureFetch,
});

API Reference

For complete API documentation including all types, interfaces, error classes, and usage patterns, see API.md.

createPaginator<TResponse, TItem>(config)

Creates a new lazy paginator instance.

Config Options

| Option | Type | Required | Description | |--------|------|----------|-------------| | initialUrl | string | Yes | The URL of the first page to fetch | | extractItems | (response: TResponse) => TItem[] | Yes | Function to extract items from API response | | getNextPageUrl | (response: TResponse, pagination: PaginationState) => string \| null | Yes | Function to get next page URL (return null to stop) | | requestConfig | RequestConfig | No | HTTP request configuration | | retry | RetryConfig | No | Retry configuration | | hooks | PaginatorHooks | No | Lifecycle hooks | | fetchFn | typeof fetch | No | Custom fetch function | | ssrfProtection | SsrfProtectionConfig | No | SSRF protection settings |

Hooks

| Hook | Context | Description | |------|---------|-------------| | onBeforeFetch | { url, config, pagination } | Called before each request | | onAfterFetch | { url, response, pagination, duration } | Called after successful request | | onError | { error, url, attempt, maxRetries, willRetry, pagination } | Called on error | | onData | { item, indexInPage, globalIndex, pagination } | Called for each item yielded |

Error Types

• MaxRetriesExceededError - Thrown when max retries are exceeded
• FetchTimeoutError - Thrown when a request times out
• HttpError - Thrown for non-2xx HTTP responses

License

MIT © Swapnil Srivastava