firestore-iterator

Concurrent, memory-bounded iteration over Firestore collections, queries, and collection groups — with a bounded worker pool, streaming pagination, and built-in file logging.

Built for maintenance work against large Firestore datasets: backfills, migrations, field updates, validations, and cleanups. You hand it a query (or a set of document references) and a per-document processor; it streams pages, runs your processor across a fixed number of concurrent workers, and returns an aggregated result. Memory stays proportional to the page size, not the size of the collection.

Installation

npm install firestore-iterator

firebase-admin is a peer dependency:

npm install firebase-admin

Usage

import { iterateFirestore } from "firestore-iterator";
import { getFirestore } from "firebase-admin/firestore";

const db = getFirestore();

const result = await iterateFirestore({
  query: db.collection("users"),
  concurrency: 20,
  processDocument: async (doc) => {
    // doc.data holds the document — hydrate defaults to true
    console.log(doc.data);
  },
});

console.log(`Processed: ${result.totalProcessed}`);
console.log(`Failed:    ${result.totalFailed}`);
console.log(`Log file:  ${result.logPath}`);

The processor receives an IterationDoc and a context (logPath, log()). doc.data holds the full document by default (hydrate defaults to true). Narrow it to a projection (hydrate: ["a", "b"]) or turn it off entirely (hydrate: false) for the id-only cases described below.

Collection groups

A collection-group query works like any other query, and because the processor gets doc.ref / doc.path (not just an id), documents stay unambiguous across parents:

await iterateFirestore({
  query: db.collectionGroup("castingTalent"),
  processDocument: (doc) => {
    console.log(doc.path); // teams/T1/phases/P1/sprints/S1/castingTalent/X
  },
});

Document data (hydrate)

doc.data holds the full document by default — hydrate defaults to true. Narrow what gets fetched when you don't need everything:

// Projection — only the fields you need (same read cost, smaller payload for
// wide documents):
await iterateFirestore({
  query: db.collection("users"),
  hydrate: ["email", "status"],
  processDocument: (doc) => console.log(doc.data?.email),
});

hydrate: false (ids only, doc.data is undefined) is the exception, not the rule — reach for it only when you genuinely don't need the scanned document's fields:

// Ref-only work — you need doc.ref, not the data:
await iterateFirestore({
  query: db.collection("staleSessions"),
  hydrate: false,
  processDocument: (doc) => doc.ref.delete(),
});

Other cases where hydrate: false is the right call:

• Transactional read-modify-write — you must read inside the transaction, so scanning the data up front is wasted:

  processDocument: (doc) =>
    db.runTransaction(async (tx) => {
      const snap = await tx.get(doc.ref);
      tx.update(doc.ref, { hits: (snap.data()?.hits ?? 0) + 1 });
    });

• Freshness on a live collection — if the value may change between the scan and processing, re-read at process time with doc.ref.get().

• A rare subset of wide documents — scan ids only, then doc.ref.get() just the few you actually need; less bandwidth than pulling every document's data.

Note: with hydrate: false or a projection, paging is by document name (__name__). If your query has an .orderBy() on a field that isn't fetched, include that field in the projection — otherwise the page cursor can't be built.

An explicit set of documents

await iterateFirestore({
  documentRefs: [db.doc("users/a"), db.doc("users/b")],
  processDocument: (doc) => {
    /* ... */
  },
});

Nesting (subcollections)

Use one call per level, and pass the parent's ctx.logPath as the child's log file so every level writes to the same log:

await iterateFirestore({
  query: db.collection("teams"),
  processDocument: async (team, ctx) => {
    await iterateFirestore({
      query: db.collection(`teams/${team.id}/phases`),
      logging: { file: ctx.logPath, level: "silent" },
      processDocument: async (phase) => {
        /* ... */
      },
    });
  },
});

Logging

Every run writes a timestamped log file. By default it lands at logs/firestore-iterator_<timestamp>.log, resolved against the current working directory (never the installed package location). That path comes back as result.logPath and is also on the processor context as ctx.logPath.

Configure it with logging, or pass logging: false to turn file logging off entirely (then logPath is undefined):

// Custom directory, file name, and verbosity:
await iterateFirestore({
  query: db.collection("users"),
  logging: { dir: "tmp/logs", fileName: "backfill_users", level: "verbose" },
  processDocument: (doc) => {
    /* ... */
  },
});

// No file logging:
await iterateFirestore({
  query: db.collection("users"),
  logging: false,
  processDocument: (doc) => {
    /* ... */
  },
});

level controls verbosity: "silent" logs only failures and the final summary, "info" (default) adds run-level start/finish lines, and "verbose" also logs every processed document.

Append your own lines to the same file with ctx.log(message, data?):

await iterateFirestore({
  query: db.collection("users"),
  processDocument: (doc, ctx) => {
    if (!doc.data?.email) {
      ctx.log(`missing email: ${doc.path}`);
    }
  },
});

To make nested runs share one log file, pass the parent's ctx.logPath as the child's file — see Nesting.

API

iterateFirestore<TResult>(params): Promise<IterationResult<TResult>>

| Param | Type | Default | Description | | ----------------- | ------------------------------------ | ------- | ------------------------------------------------------------------------------------------------------------------------ | | query | Query | — | Query, collection, or collection-group query. One of query / documentRefs is required. | | documentRefs | DocumentReference[] | — | Explicit set of documents to process instead of a query. | | processDocument | (doc, ctx) => Promise<TResult> | — | Required. Called once per document. | | concurrency | number | 20 | Maximum documents processed concurrently. | | pageSize | number | 500 | Documents fetched per page while scanning a query. | | hydrate | boolean \| string[] | true | doc.data contents: true = full document, string[] = only those fields, false = none (doc.data is undefined). | | collectResults | boolean | false | Retain each processor's return value in result.results. | | logging | LoggingOptions \| false | enabled | Built-in file logging (keys below); pass false to disable (logPath becomes undefined). | | onDocument | (doc, { result?, error? }) => void | — | Called after each document settles. |

logging is either false (disable) or a LoggingOptions object — every key optional:

| Key | Type | Default | Description | | ---------- | --------------------------------- | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | dir | string | "logs" | Directory for the log file, resolved against process.cwd(). | | fileName | string | "firestore-iterator" | Base file name; a timestamp and .log are appended. | | file | string | — | Append to this exact existing file instead of creating one (e.g. a nested run reusing the parent's ctx.logPath). Overrides dir / fileName. | | level | "silent" \| "info" \| "verbose" | "info" | Verbosity: silent = failures + summary only; info = adds run-level lines; verbose = logs every processed document. |

Resolves with an IterationResult<TResult>: processed, failed, totalProcessed, totalFailed, results? (path→value, only with collectResults), errors, startTime, endTime, durationMs, and logPath. Every option and field ships with TSDoc — hover any symbol in your editor for details.

forEachConcurrent(source, concurrency, worker)

The bounded-concurrency pool that powers the iterator, exported for reuse: runs worker over any (async) iterable, at most concurrency at a time, pulling items lazily so a streaming source only advances as workers free up.

License

MIT © Michael Katskiy