@fantasticfour/world-firestore-tasks

GCP-native workflow backend using Cloud Firestore for storage and Cloud Tasks for queue management. Features best-in-class real-time streaming via native Firestore listeners.

Why Use This Package

• Real-Time Excellence: Native Firestore listeners for true push-based streaming
• Cost-Effective: 30-50% cheaper than AWS for many workloads
• Auto-Scaling: Serverless infrastructure with multi-region support
• ACID Transactions: Strong consistency guarantees
• Rich Querying: Composite indexes for complex queries
• Developer Experience: Excellent GCP tooling and documentation

Best for GCP deployments requiring real-time updates, cost optimization, and serverless NoSQL storage.

Installation

pnpm add @fantasticfour/world-firestore-tasks

Prerequisites

• GCP Account: Active Google Cloud Platform account
• Firestore: Native mode database enabled
• Cloud Tasks: API enabled in your project
• Service Account: Credentials with Firestore and Cloud Tasks permissions

Usage

import { Firestore } from '@google-cloud/firestore';
import { CloudTasksClient } from '@google-cloud/tasks';
import { createFirestoreTasksWorld } from '@fantasticfour/world-firestore-tasks';

const firestore = new Firestore({
  projectId: 'my-project',
});

const tasksClient = new CloudTasksClient();

const world = createFirestoreTasksWorld({
  firestore,
  tasksClient,
  project: 'my-project',
  location: 'us-central1',
  queueName: 'workflow-queue',
  targetUrl: 'https://my-app.run.app',
  deploymentId: 'production',
});

// Create a workflow run
const run = await world.runs.create({
  runId: ulid(),
  workflowName: 'my-workflow',
  status: 'pending',
  input: ['arg1', 'arg2'],
  deploymentId: 'production',
  executionContext: {},
  createdAt: new Date(),
  updatedAt: new Date(),
});

Architecture

• Storage: Cloud Firestore (serverless NoSQL document database)
• Queue: Cloud Tasks (HTTP-based task queue)
• Streaming: Firestore real-time listeners (native push updates)
• Indexes: Composite indexes for efficient querying
• IDs: ULID-based for sortable identifiers

Firestore Collections

• workflow_runs - Main workflow state
• workflow_runs/{runId}/events - Event subcollection
• workflow_runs/{runId}/steps - Step subcollection
• workflow_runs/{runId}/hooks - Hook subcollection
• workflow_streams/{streamId}/chunks - Stream chunks with listeners

Composite indexes required:

• workflowName ASC, createdAt DESC
• status ASC, createdAt DESC

Real-Time Streaming

Firestore provides the best streaming experience among all world implementations:

// True push-based updates, no polling required
const unsubscribe = firestore
  .collection('workflow_streams')
  .doc(streamId)
  .collection('chunks')
  .orderBy('sequence', 'asc')
  .onSnapshot(snapshot => {
    snapshot.docChanges().forEach(change => {
      if (change.type === 'added') {
        const chunk = change.doc.data();
        controller.enqueue(chunk.data);
      }
    });
  });

Streaming Comparison:

• Firestore: <50ms latency, native listeners, excellent efficiency
• Redis/Postgres: <100ms, pub/sub, good efficiency
• Others: 100-200ms+, polling-based, fair efficiency

Testing

Tests run against the Firestore emulator.

Setup

Install Firebase CLI (includes emulator):

npm install -g firebase-tools

Run Tests

1. Start emulator (separate terminal):

   firebase emulators:start --only firestore --project=test-project

2. Run tests:

   pnpm --filter @fantasticfour/world-firestore-tasks test

Test Coverage

• storage.test.ts: CRUD operations, status transitions, filtering
• spec.test.ts: @workflow/world interface compliance
• realtime.test.ts: Real-time listeners, transactions, lifecycle

Total: ~1,100 lines ensuring production reliability.

Cost Estimate

Firestore:

• Document reads: $0.06 per 100K
• Document writes: $0.18 per 100K
• Storage: $0.18/GB-month

Cloud Tasks:

• First 1M tasks/month: Free
• Additional: $0.40 per million

Estimated Monthly Cost:

• Low usage (100K workflows/month): $5-10
• Medium usage (1M workflows/month): $15-30
• High usage (10M workflows/month): $80-150

Typically 30-50% cheaper than AWS DynamoDB + SQS.

Configuration Options

| Option | Type | Description | |--------|------|-------------| | firestore | Firestore | Firestore client instance | | tasksClient | CloudTasksClient | Cloud Tasks client instance | | project | string | GCP project ID | | location | string | Cloud Tasks location (e.g., 'us-central1') | | queueName | string | Cloud Tasks queue name | | targetUrl | string | HTTP endpoint for task delivery | | deploymentId | string | Deployment identifier |

Environment Variables

# GCP credentials (or use service account key file)
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account.json"

# Application config
export GCP_PROJECT_ID="my-project"
export GCP_LOCATION="us-central1"
export CLOUD_TASKS_QUEUE="workflow-queue"
export TARGET_URL="https://my-app.run.app"

When to Choose This Package

Use world-firestore-tasks when:

• GCP-native architecture required
• Real-time streaming is important
• Cost optimization desired
• NoSQL document model fits your data
• Already using GCP ecosystem

Consider alternatives when:

• SQL queryability required → use @fantasticfour/world-postgres-redis
• AWS ecosystem → use @fantasticfour/world-dynamodb-sqs
• Multi-cloud strategy → use @fantasticfour/world-postgres-upstash
• Edge deployment → use @fantasticfour/world-cloudflare

Performance Characteristics

• Latency: 10-50ms per operation (GCP region-dependent)
• Streaming: <50ms real-time updates (native listeners)
• Throughput: Auto-scales with traffic
• Consistency: Strong consistency with ACID transactions

License

Apache License