hf-supabase-sync

CLI tool to sync HuggingFace datasets to Supabase.

Installation

npm install -g hf-supabase-sync

Usage

# Set environment variables
export SUPABASE_URL=https://your-project.supabase.co
export SUPABASE_SERVICE_ROLE_KEY=your-service-role-key
export HF_TOKEN=your-huggingface-token  # Optional, for private datasets

# Optional: Set proxy
export HTTPS_PROXY=http://your-proxy:port

# Run sync
hf-supabase-sync <dataset> <config> <split>

# Example
hf-supabase-sync Salesforce/wikitext wikitext-2-raw-v1 test

Environment Variables

| Variable | Required | Description | |----------|----------|-------------| | SUPABASE_URL | Yes | Your Supabase project URL | | SUPABASE_SERVICE_ROLE_KEY | Yes | Supabase service role key | | HF_TOKEN | No | HuggingFace token for private datasets | | HTTPS_PROXY | No | HTTP proxy URL |

Supabase Table Schema

CREATE TABLE public.hf_rows (
  id BIGSERIAL PRIMARY KEY,
  hf_repo_id TEXT NOT NULL,
  config TEXT NOT NULL DEFAULT '',
  split TEXT NOT NULL,
  source_index INT NOT NULL,
  data JSONB NOT NULL,
  checksum TEXT NOT NULL,
  imported_at TIMESTAMPTZ DEFAULT NOW(),
  updated_at TIMESTAMPTZ DEFAULT NOW(),
  UNIQUE (hf_repo_id, config, split, source_index)
);

License

MIT