infinity-fetch 🚀

Configurable recursive fetch for paginated APIs. Works in Node.js and browsers.

Automatically re-invokes a fetch function across pages until a stop condition is met — accumulating all results into a single array. Zero dependencies.

How it works

                    ┌─────────────────────────────────────────────────────┐
                    │                   infinityFetch                     │
                    └────────────────────────┬────────────────────────────┘
                                             │
                         initialParams = { start: 0, limit: 100 }
                                             │
                                             ▼
                    ┌────────────────────────────────────────┐
                    │          fetcher(params)               │
                    │   api.project('x').repo('y').commits() │
                    └────────────────┬───────────────────────┘
                                     │
                    ┌────────────────▼───────────────────────┐
                    │             Response                    │
                    │  { values, isLastPage, nextPageStart,  │
                    │    size, limit, start }                 │
                    └──────┬─────────────────────┬───────────┘
                           │                     │
                    isLastPage?                  no
                           │                     │
                          yes            getNextParams()
                           │          { start: nextPageStart }
                           │                     │
                           │                     ▼
                           │        ┌────────────────────────┐
                           │        │   fetcher(nextParams)  │  ← repeats
                           │        └────────────────────────┘
                           │
                           ▼
              ┌──────────────────────────────────────┐
              │   return { items[], pages: number }  │
              │   items  → all pages accumulated     │
              │   pages  → total iterations done     │
              └──────────────────────────────────────┘

Installation

npm install infinity-fetch

Usage

pagedFetch — for Bitbucket-style paginated APIs

If your API returns { values, isLastPage, nextPageStart, size, limit, start }, use the built-in helper:

import { pagedFetch } from 'infinity-fetch';

const { items, pages } = await pagedFetch({
  fetcher: (params) => api.project('my-project').repo('my-repo').commits(params),
  limit: 100, // items per page, defaults to 100
});

console.log(`${items.length} commits fetched across ${pages} pages`);

With loading state, progress tracking, and a delay between pages:

const { items, pages } = await pagedFetch({
  fetcher: (params) => api.project('my-project').repo('my-repo').commits(params),
  limit: 100,
  maxPages: 20,
  delay: 200, // wait 200ms between each page fetch
  onStart: () => setLoading(true),
  onEnd:   () => setLoading(false),
  onPage: (pageItems, _response, pageIndex) => {
    console.log(`Page ${pageIndex + 1}: ${pageItems.length} commits`);
  },
});

infinityFetch — generic, fully configurable

Use this when your API has a different response shape:

import { infinityFetch } from 'infinity-fetch';

const { items, pages } = await infinityFetch({
  // The function that performs a single page request
  fetcher: (params) => github.issues.list(params),

  // Parameters for the very first request
  initialParams: { page: 1, per_page: 50 },

  // When to stop — return true on the last page
  isLastPage: (response) => response.data.length < 50,

  // How to compute params for the next page
  getNextParams: (_response, currentParams) => ({
    ...currentParams,
    page: currentParams.page + 1,
  }),

  // Which field holds the items
  getItems: (response) => response.data,

  // Optional: safety cap on number of pages
  maxPages: 100,

  // Optional: milliseconds to wait between each page fetch
  delay: 200,

  // Optional: retry failed page fetches
  retry: {
    maxRetries: 3,
    delay: (attempt) => attempt * 500,
    retryWhen: (error) => {
      return error instanceof Response && error.status >= 500;
    },
  },

  // Optional: called once before the first fetch
  onStart: () => setLoading(true),

  // Optional: called once after all pages are done — receives the final result
  onEnd: ({ items, pages }) => {
    setLoading(false);
    console.log(`Done: ${items.length} items across ${pages} pages`);
  },

  // Optional: called after each individual page
  onPage: (pageItems, _response, pageIndex) => {
    console.log(`Page ${pageIndex + 1}: ${pageItems.length} items`);
  },
});

console.log(`${items.length} issues fetched across ${pages} pages`);

API Reference

pagedFetch<TItem>(config)

| Option | Type | Default | Description | |---|---|---|---| | fetcher | (params: PagedParams) => Promise<PagedResponse<TItem>> | required | Function that fetches one page | | limit | number | 100 | Items per page | | maxPages | number | Infinity | Maximum pages to fetch (safety limit) | | delay | number | — | Milliseconds to wait between each page fetch | | retry | InfinityFetchRetryConfig | — | Retry failed page fetches | | onStart | () => void | — | Called once before the first fetch | | onEnd | (result: InfinityFetchResult<TItem>) => void | — | Called once after all pages are done | | onPage | (items, response, pageIndex) => void | — | Called after each individual page |

PagedParams

{ start: number; limit: number }

PagedResponse<TItem> — expected response shape:

{
  values: TItem[];
  isLastPage: boolean;
  nextPageStart?: number;
  size: number;
  limit: number;
  start: number;
}

Returns: Promise<InfinityFetchResult<TItem>>

infinityFetch<TResponse, TParams, TItem>(config)

| Option | Type | Default | Description | |---|---|---|---| | fetcher | (params: TParams) => Promise<TResponse> | required | Function that fetches one page | | initialParams | TParams | required | Parameters for the first request | | isLastPage | (response: TResponse) => boolean | required | Returns true to stop iteration | | getNextParams | (response: TResponse, currentParams: TParams) => TParams | required | Computes params for the next page | | getItems | (response: TResponse) => TItem[] | required | Extracts items from a response | | maxPages | number | Infinity | Maximum pages to fetch (safety limit) | | delay | number | — | Milliseconds to wait between each page fetch | | retry | InfinityFetchRetryConfig | — | Retry failed page fetches | | onStart | () => void | — | Called once before the first fetch | | onEnd | (result: InfinityFetchResult<TItem>) => void | — | Called once after all pages are done | | onPage | (items, response, pageIndex) => void | — | Called after each individual page |

Returns: Promise<InfinityFetchResult<TItem>>

type InfinityFetchResult<TItem> = {
  items: TItem[];  // all items collected across every page
  pages: number;   // total number of pages fetched
};

type InfinityFetchRetryConfig = {
  maxRetries?: number;
  delay?: number | ((attempt: number, error: unknown) => number);
  retryWhen?: (error: unknown, attempt: number) => boolean | Promise<boolean>;
};

Compatibility

| Environment | Support | |---|---| | Node.js 18+ | ✅ | | Node.js 20+ | ✅ | | Modern browsers | ✅ | | Deno / Bun | ✅ | | ESM | ✅ | | TypeScript | ✅ (types included) |

Contributing

Commits must follow the Conventional Commits spec — this drives automatic versioning and changelog generation via semantic-release.

| Commit prefix | Triggers | |---|---| | fix: | Patch release (0.0.x) | | feat: | Minor release (0.x.0) | | feat!: / BREAKING CHANGE: | Major release (x.0.0) | | chore:, docs:, test: | No release |

git commit -m "feat: add onPage callback to pagedFetch"
git commit -m "fix: handle missing nextPageStart gracefully"
git commit -m "feat!: rename items field to data"

Setting up repository secrets

For the release pipeline to work, add these secrets in Settings → Secrets and variables → Actions:

| Secret | Description | |---|---| | NPM_TOKEN | npm automation token (npm token create --type automation) |

GITHUB_TOKEN is provided automatically by GitHub Actions — no setup needed.

Enabling GitHub Pages

Go to Settings → Pages and set the source to GitHub Actions.

Changelog

See CHANGELOG.md for the full release history.

License

MIT