Pinecone Import Helper

Turn a folder of raw files into a completed Pinecone bulk import — from one command, in your browser, without your data ever leaving your machine.

npx @pinecone-field/import-helper

Bulk import is the cheapest way to get large datasets into Pinecone ($0.25/GB vs. roughly 18× that via upserts), but it asks a lot of you up front: pre-computed vectors, a strict Parquet schema, and an exact bucket directory layout. This tool does all of that through a guided web interface:

drop files / point at a bucket → assign namespaces → chunk → embed → shape metadata
          → Parquet (id, values LIST<FLOAT>, metadata JSON)
          → s3://bucket/import-dir/<namespace>/*.parquet
          → start_import

Quick start

One command (recommended)

npx @pinecone-field/import-helper

The app starts on http://localhost:3210 (or the next free port) and opens your browser. No build step — it launches in seconds.

• Have a .env.local in the directory you run it from? Its keys are loaded automatically.
• --no-open skips the browser launch; PORT=4000 npx ... prefers a different port.

Docker

docker build -t pinecone-import-helper .
docker run --rm -p 3210:3210 \
  -v ~/.aws:/home/node/.aws:ro \
  -v ~/.config/gcloud:/home/node/.config/gcloud:ro \
  --env-file .env.local \
  pinecone-import-helper

Mount only the credential directories for the provider(s) you use, then open http://localhost:3210 yourself.

From source

git clone https://github.com/pinecone-field/import-helper.git
cd import-helper
cp .env.local.example .env.local   # optional — keys can also be pasted in the UI
npm install
npm run dev                        # http://localhost:3210

What you'll need

| | Required? | Notes | | --- | --- | --- | | Node.js 20+ | ✅ | For npx (Docker route doesn't need it) | | Pinecone API key | ✅ | Everything beyond dry runs — get one here | | A cloud bucket you can write to | ✅ | S3, GCS, or Azure Blob — where the Parquet is staged | | Pinecone storage integration | Usually | How Pinecone reads your (non-public) bucket; created once in the console | | OpenAI API key | Optional | Only for OpenAI embedding models | | Anthropic API key | Optional | Powers the "describe it in words" boxes |

Keys can go in .env.local (see .env.local.example) or be pasted into the in-app setup panel, where they're held in the local server's memory for the session only.

How it works

The interface walks you through six stages; a live sidebar always shows the exact bucket tree Pinecone will read plus a running cost estimate.

1. Add files — drag and drop files/folders, or point at s3:// / gs:// / https://<account>.blob.core.windows.net/... raw data and pull it in (up to 200 files / 500 MB per pull, folder structure preserved). Documents (pdf docx md txt html rst log) are text-extracted; datasets (csv tsv json jsonl ndjson) parse row-by-row with a guessed, overridable text column.
2. Namespaces — each namespace becomes a directory in the import layout (__default__ for the default namespace). Quick-assign by top folder, one-namespace-per-file, or file type — or name them per file.
3. Chunking — recursive / paragraph / sentence / fixed / none, with token size and overlap. Or describe what you want in plain words ("about 150 words, never split a sentence") and Claude translates it into a config. Live preview with exact token counts, from the start of a file or a random spot.
4. Model & index — Pinecone-hosted (llama-text-embed-v2, multilingual-e5-large), OpenAI, or a no-API dry-run mode. Point at an existing index (it checks serverless-ness, dimension alignment, and namespace collisions, and samples existing records so your metadata matches) or create a new one with the dimension locked to your model.
5. Metadata — fields mapped to sources (chunk text, filename, path, chunk index, doc id, file date, static values), renamable inline. Claude can suggest a schema from samples of your actual files or translate a plain-English description. Live example-record preview before you commit.
6. Process & ship — chunks, then embeds with several provider requests in flight at once (default 6, tune with EMBED_CONCURRENCY; 429s retry with backoff), writes Snappy Parquet per namespace, uploads to your bucket, then starts the import and polls status until Pinecone reports completion. Prefer to upload yourself? A copy-paste aws s3 sync / gsutil rsync / az storage blob upload-batch command is provided instead.

Cloud storage credentials

Bucket reads and Parquet uploads use your machine's existing cloud credentials — the tool stores nothing. The Cloud storage connections panel shows what's detected per provider, includes setup instructions, and has a Test button that makes a real API call and reports the authenticated identity (or an actionable fix). Before shipping, the upload card's Test access button verifies list + write on your exact bucket/prefix using a tiny probe object it deletes immediately.

| Provider | Credential source | Minimum permission | | --- | --- | --- | | Amazon S3 | AWS credential chain (aws configure, SSO, env vars) | s3:ListBucket + s3:Get/Put/DeleteObject on the prefix | | Google Cloud Storage | ADC (gcloud auth application-default login or service-account JSON) | roles/storage.objectAdmin on the bucket | | Azure Blob Storage | AZURE_STORAGE_CONNECTION_STRING or DefaultAzureCredential (az login) | "Storage Blob Data Contributor" on the account/container |

Separately, Pinecone itself reads the staged files during the import. That's authorized by a storage integration created in the Pinecone console — paste its ID in the import step — or by making the bucket public.

Privacy & security

• Your files never leave your machine except to go to your bucket and your chosen embedding provider.
• Cloud credentials are read from the local environment (AWS profile, gcloud ADC, az login) and are never stored, logged, or transmitted by the tool.
• API keys pasted in the UI are held in the local server's memory for the session and are never written to disk or echoed back to the browser.
• Scratch data (uploads, generated Parquet) lives in a temp directory (npx/Docker) or .data/ (source), overridable with PC_IMPORT_DATA_DIR.
• LLM assistance is capped: inputs truncated (~2k chars for descriptions, ~6k chars for file samples), 1k max output tokens, 40 calls per server session.

Costs

The sidebar estimates before you ship: embedding spend for your chosen model, bulk-import cost ($0.25/GB), what the same data would cost via upserts (1 WU/KB at ~$4.25/M WUs), and the savings between them. Standard and Enterprise plans currently include a one-time $250 import credit (≈1 TB).

Limits & compatibility

Bulk import targets serverless indexes only, and namespaces being imported must not already exist in the index. Source bucket ↔ index cloud compatibility (enforced in the UI):

| Import from… | AWS index | GCP index | Azure index | | -------------------- | :-------: | :-------: | :---------: | | Amazon S3 | ✅ | ❌ | ❌ | | Google Cloud Storage | ✅ | ✅ | ✅ | | Azure Blob Storage | ✅ | ✅ | ✅ |

Other limits: 10 GB per file, 100k files and 10k namespaces per import, 1 TB per import on on-demand billing. Duplicate IDs within an import are deduplicated by Pinecone. S3 Express One Zone isn't supported. Dense vectors only for now (no sparse/hybrid).

The Parquet written matches the dense-index contract exactly: id: STRING, values: LIST<FLOAT>, metadata: STRING (JSON, null when empty).

Troubleshooting

• "Pinecone API key is not set" — add it in the setup panel at the top of the page, or set PINECONE_API_KEY in .env.local before launching.
• A provider shows "—" in the connections panel — expand Cloud storage connections for per-provider setup commands, then hit Test.
• Import stuck at 100% "InProgress" — normal; Pinecone typically takes ~10 more minutes to finalize before reporting Completed.
• "unreachable from s3" next to an index — that's the compatibility matrix above; S3 sources can only import into AWS indexes.
• Port already in use — the CLI automatically walks forward from 3210 to find a free port; check the printed URL.

Development

npm run dev     # dev server with hot reload on :3210
npm run build   # production build (also runs automatically on npm publish)
npx tsc --noEmit

Releasing: bump version in package.json, then npm publish (builds via prepublishOnly). The published package ships the prebuilt .next so npx users never compile.

License

Apache-2.0