geoparquet-extractor

v0.2.2

Published

11 days ago

Extract and convert spatial data from remote GeoParquet files. Supports bbox filtering, multiple output formats, and pluggable metadata providers.

0High
0Medium
0Low

ramseraph

geoparquet parquet geospatial gis extract download duckdb wasm geopackage shapefile geojson kml csv

geoparquet-extractor

Extract and convert spatial data from remote GeoParquet files entirely in the browser, using OPFS (Origin Private File System) for temporary storage and spill-over. Supports bbox filtering, multiple output formats, and pluggable metadata providers.

Browser-only — requires Origin Private File System (OPFS), Web Workers, and Web Locks APIs.

How It Works

The library queries remote GeoParquet files using HTTP range requests — only the data matching the requested bounding box is transferred. All processing happens client-side with no backend involved.

Browser
  ├─ DuckDB-WASM
  │   ├─ HTTP range requests → remote GeoParquet
  │   ├─ Spatial filtering by bounding box
  │   ├─ OPFS temp directory for spill-over
  │   └─ COPY TO OPFS (intermediate or final output)
  └─ GeoPackage Worker (for .gpkg only)
      ├─ hyparquet (reads intermediate parquet from OPFS)
      └─ wa-sqlite/sqwab (writes .gpkg with R-tree index to OPFS)

Key Internal Dependencies

DuckDB-WASM with OPFS temp directory — uses duckdb-wasm-opfs-tempdir which supports SET temp_directory = 'opfs://...' for processing datasets larger than available memory. DuckDB's spatial extension is loaded at runtime for geometry operations (ST_AsWKB, ST_AsGeoJSON, ST_Hilbert, etc.).
sqwab — wa-sqlite with R-tree support for GeoPackage output. Runs in a dedicated Web Worker using OPFSAdaptiveVFS for file I/O.
hyparquet — pure-JS parquet reader used in the GeoPackage worker to read intermediate files from OPFS.
Apache Arrow — columnar data handling.

Used By

Indian Open Maps — extract Indian geospatial datasets
OSM Layercake Extract — extract OpenStreetMap data from OSMUS Layercake datasets

Installation

npm install geoparquet-extractor

Quick Start

import { GeoParquetExtractor, createDuckDBClient } from 'geoparquet-extractor';

// You initialize DuckDB yourself
import * as duckdb from 'duckdb-wasm-opfs-tempdir';

const db = /* your initialized AsyncDuckDB instance */;
const client = await createDuckDBClient(db, {
  extensions: ['spatial', 'httpfs'],
});

const extractor = new GeoParquetExtractor({ duckdb: client });

await extractor.extract({
  urls: ['https://example.com/data.parquet'],
  bbox: [77.5, 12.9, 77.7, 13.1],
  format: 'geoparquet',
  baseName: 'my-data',
  onProgress: (pct) => console.log(`${pct}%`),
  onStatus: (msg) => console.log(msg),
});

Features

9 output formats: GeoParquet (v1.1 & v2.0), GeoPackage, Shapefile, CSV, GeoJSON, GeoJSONSeq, KML, DXF
Spatial filtering: Bbox intersection with per-partition and per-row-group optimization
Pluggable metadata: Override how partition URLs and bboxes are resolved
Extent visualization data: Fetch partition/row-group bboxes as GeoJSON for map display
DuckDB-powered: Spatial SQL queries via DuckDB WASM (you provide the instance)
Self-contained GeoPackage worker: wa-sqlite-rtree bundled into the worker — no CDN needed

Formats

| Format | Value | Extension | Notes | |--------|-------|-----------|-------| | GeoPackage | geopackage | .gpkg | Requires GeoPackage worker | | GeoJSON | geojson | .geojson | FeatureCollection | | GeoJSONSeq | geojsonseq | .geojsonl | Newline-delimited | | GeoParquet v1.1 | geoparquet | .parquet | With Hilbert spatial sort | | GeoParquet v2.0 | geoparquet2 | .parquet | Native geometry encoding | | CSV | csv | .csv | WKT geometry column | | Shapefile | shapefile | .shp | 2 GB limit per component | | KML | kml | .kml | XML format | | DXF | dxf | .dxf | AutoCAD R14, UTM projection |

DuckDB Setup

The library does NOT bundle DuckDB WASM. You initialize it yourself and pass it in:

import { createDuckDBClient } from 'geoparquet-extractor';
import * as duckdb from 'duckdb-wasm-opfs-tempdir';

// Standard duckdb-wasm-opfs-tempdir init
const MANUAL_BUNDLES = { /* your bundle config */ };
const bundle = await duckdb.selectBundle(MANUAL_BUNDLES);
const worker = new Worker(bundle.mainWorker);
const logger = new duckdb.ConsoleLogger();
const db = new duckdb.AsyncDuckDB(logger, worker);
await db.instantiate(bundle.mainModule, bundle.pthreadWorker);

// Wrap it for the library
const client = await createDuckDBClient(db, {
  extensions: ['spatial', 'httpfs'],
});

Custom DuckDB Builds

The duckdb-wasm-opfs-tempdir package supports SET temp_directory = 'opfs://...' for large downloads that exceed browser memory limits. The library's createDuckDBClient adapter works with any DuckDB WASM build that provides AsyncDuckDB.

GeoPackage Worker

The GeoPackage format requires a Web Worker for wa-sqlite. The library ships a self-contained worker with wa-sqlite-rtree bundled in:

// Option 1: URL to hosted worker
const extractor = new GeoParquetExtractor({
  duckdb: client,
  gpkgWorkerUrl: '/workers/gpkg_worker.js',
});

// Option 2: Worker instance (import.meta.url resolves to dist/gpkg_worker.js)
const worker = new Worker(new URL('geoparquet-extractor/gpkg-worker', import.meta.url), { type: 'module' });
const extractor = new GeoParquetExtractor({
  duckdb: client,
  gpkgWorker: worker,
});

Note: The worker requires wa-sqlite-async.wasm to be served from the same directory as gpkg_worker.js. Both files are included in the dist/ directory.

Custom Metadata Provider

Override how partition URLs and bboxes are resolved:

import { MetadataProvider, GeoParquetExtractor } from 'geoparquet-extractor';

class MyMetadataProvider extends MetadataProvider {
  async getParquetUrls(sourceUrl) {
    const meta = await fetch(sourceUrl + '.meta.json').then(r => r.json());
    const baseUrl = sourceUrl.replace(/[^/]+$/, '');
    return Object.keys(meta.extents).map(f => baseUrl + f);
  }

  async getExtents(sourceUrl) {
    const meta = await fetch(sourceUrl + '.meta.json').then(r => r.json());
    return meta.extents; // { "file.parquet": [minx, miny, maxx, maxy] }
  }

  async getBbox(sourceUrl, duckdb) {
    const extents = await this.getExtents(sourceUrl);
    // Compute overall bbox from all partition extents
    let bbox = [Infinity, Infinity, -Infinity, -Infinity];
    for (const ext of Object.values(extents)) {
      bbox[0] = Math.min(bbox[0], ext[0]);
      bbox[1] = Math.min(bbox[1], ext[1]);
      bbox[2] = Math.max(bbox[2], ext[2]);
      bbox[3] = Math.max(bbox[3], ext[3]);
    }
    return bbox;
  }
}

const extractor = new GeoParquetExtractor({
  duckdb: client,
  metadataProvider: new MyMetadataProvider(),
});

Extent Visualization

Fetch partition and row-group bboxes as GeoJSON for map display:

import { ExtentData, MetadataProvider } from 'geoparquet-extractor';

const extentData = new ExtentData({
  metadataProvider: new MetadataProvider(),
  duckdb: client,
});

const { dataExtents, rgExtents } = await extentData.fetchExtents({
  sourceUrl: 'https://example.com/data.mosaic.json',
  partitioned: true,
});
// dataExtents: { filename: [minx, miny, maxx, maxy] } or null
// rgExtents: { filename: { rg_N: [minx, miny, maxx, maxy] } } or null

API