@render-lab/tasks-pinecone

⚠️ Experimental: proof of concept. This package is part of the Render Tasks POC and is published for testing only. It is not fully tested or production ready. Task names, inputs, outputs, and behavior can change or break in any release. Pin exact versions and expect breaking changes.

Durable Pinecone vector and bulk-import tasks for Render Workflows.

This pack wraps the official @pinecone-database/pinecone SDK, but constructs it with maxRetries: 0 so vendor retries are off. The SDK's internal retry and backoff would collapse several attempts into one opaque call and hide them from Render Workflows. With vendor retries disabled, the baked-in durable retry policy is the single source of truth: every poll and every attempt is one observable durable run (ADR-0005). Only JSON-serializable DTOs cross a task boundary — no SDK object, Date, or vendor error ever escapes (ADR-0003); createdAt/finishedAt cross as ISO-8601 strings.

Installation

pnpm add @render-lab/tasks-pinecone

@renderinc/sdk is a peer dependency pinned to exact 0.6.0 — install it once in your workflow service so every pack registers against the same TaskRegistry (ADR-0001). @pinecone-database/pinecone is a regular dependency, fully encapsulated inside the tasks.

import {
  provisionIndex,
  deleteIndex,
  createNamespace,
  listNamespaces,
  deleteNamespace,
  upsertVectors,
  fetchVectors,
  deleteVectors,
  search,
  startImport,
  awaitImport,
  cancelImport,
} from "@render-lab/tasks-pinecone";

Tasks

| Task | Input | Output | Retry | Idempotency | | --- | --- | --- | --- | --- | | pinecone.provisionIndex | ProvisionIndexInput | IndexDTO | provision poll (every 15s up to 20m) | find-or-create; re-runs converge on the ready index | | pinecone.deleteIndex | DeleteIndexInput | DeleteIndexResult | general (3× backoff) | a missing index normalizes to success | | pinecone.createNamespace | CreateNamespaceInput | NamespaceDTO | general (3× backoff) | a same-name conflict is described back | | pinecone.listNamespaces | ListNamespacesInput | ListNamespacesResult | general (3× backoff) | read-only | | pinecone.deleteNamespace | DeleteNamespaceInput | DeleteNamespaceResult | general (3× backoff) | a missing namespace normalizes to success | | pinecone.upsertVectors | UpsertVectorsInput | UpsertVectorsResult | general (3× backoff) | stable IDs overwrite the same records | | pinecone.fetchVectors | FetchVectorsInput | FetchVectorsResult | general (3× backoff) | read-only | | pinecone.deleteVectors | DeleteVectorsInput | DeleteVectorsResult | general (3× backoff) | a missing namespace normalizes to success | | pinecone.search | SearchInput | SearchResult | general (3× backoff) | read-only | | pinecone.startImport | StartImportInput | ImportDTO | trigger (0 retries) | no stable identity — never retried | | pinecone.awaitImport | AwaitImportInput | ImportDTO | import poll (every 30s up to 24h) | one describeImport per attempt | | pinecone.cancelImport | CancelImportInput | CancelImportResult | general (3× backoff) | absent or already-terminal is success |

Reads, deterministic upserts (stable vector IDs overwrite the same records), find-or-create, convergent updates, deletes, and cancels use the general policy because replay converges. startImport gets zero retries: it has no stable recoverable identity, so a retried start could launch a second import — resume with awaitImport on the returned id instead. The provisionIndex and awaitImport polls are durable waits: SDK 0.6.0 has no native sleep, so they poll by throwing — still working → throw a progress error the fixed-interval retry re-runs; terminal failure (Terminating for an index, Failed/Cancelled for an import) → throw a distinct terminal error so a failure is never retried as ordinary progress; done → return.

Serverless-only namespaces

Namespace tasks (createNamespace, listNamespaces, deleteNamespace) target serverless indexes. listNamespaces returns one bounded page with an opaque nextPaginationToken; pass it back to page. A namespace's recordCount and schema are null when the API does not report them.

Stable IDs and index names

• Index names are the find-or-create key for provisionIndex and the routing key for every data-plane call. Keep them stable across runs.
• Vector IDs are the upsert idempotency key. upsertVectors rejects duplicate IDs within a batch and requires nonempty, dimension-consistent dense values (and matching sparse indices/values lengths). Because a stable ID overwrites the same record, a retried upsert converges.

Provisioning: find-or-create and immutable validation

provisionIndex reads the index first and creates only when it is absent. If an index with the same name already exists, its immutable settings — dimension, metric, cloud, region — must match the request; a mismatch throws:

Pinecone index "docs" exists with incompatible immutable settings: dimension 768 != 1536

Once the index (existing or newly created) is Ready, the ready IndexDTO is returned.

const index = await provisionIndex({
  name: "docs",
  dimension: 1536,
  metric: "cosine",
  cloud: "aws",
  region: "us-east-1",
});

Bulk import: trigger, await, cancel

// 1. Trigger — zero retries; the returned id is your handle.
const started = await startImport({ indexName: "docs", uri: "s3://my-bucket/vectors/" });

// 2. Await — a durable poll every 30s for up to 24h; throws until Completed.
const done = await awaitImport({ indexName: "docs", importId: started.id });

// 3. Cancel — idempotent; an absent or already-terminal import is success.
await cancelImport({ indexName: "docs", importId: started.id });

Limits: 1,000 items and 4 MB

Every list, fetch, and search is bounded. upsertVectors caps at 1,000 vectors, fetchVectors at 1,000 IDs, search at topK 1,000, and listNamespaces rejects a limit above 1,000. Each serialized result is measured before it is returned and rejected at or above the Render Workflows 4 MB task-result cap, with guidance to lower the limit (fewer IDs, a smaller topK, or a tighter page).

Environment

| Variable | Required | Purpose | | --- | --- | --- | | PINECONE_API_KEY | Yes | Authenticates the first Pinecone call |

PINECONE_API_KEY is read lazily on the first port call, never at import (ADR-0007), so importing this pack for one task never requires the key until a task actually runs. The default client is constructed with maxRetries: 0 — vendor retries are disabled and the durable task retry is the single source of truth.

No webhook adapter

Pinecone publishes no signed event contract for imports, so this pack ships no ./webhooks subpath. Import progress is discovered by polling awaitImport, not by inbound webhooks.

Testing

pnpm -C packages/tasks-pinecone test        # Tier 1: hermetic unit tests (no secrets, no network)
pnpm -C packages/tasks-pinecone build
pnpm -C packages/tasks-pinecone typecheck
RUN_LIVE=1 PINECONE_API_KEY=... pnpm -C packages/tasks-pinecone test:live   # Tier 2: opt-in live

Tier 1 tests inject a fake port at the boundary and never touch the network. The live suite (tasks.live.test.ts) is guarded by describe.skipIf(!process.env.RUN_LIVE) and provisions, uses, and tears down a real serverless index.