@dataset.sh/server

@dataset.sh/server is a simple server implement the following endpoints

# health check
GET /api/health

# list categories (pagination via query params, return CategoryListingResult) 
GET /api/category

# list datasets under category (pagination via query params, return DatasetListingResult)
GET /api/dataset/:categoryName


# download dataset file based on version
GET /api/dataset/:categoryName/:datasetName/version/:versionHex/file

# resolve version tag to versionHex
GET /api/dataset/:categoryName/:datasetName/tag/:tagName

type CategoryListingResult = {
    items: {
        category: string
    }[]
    page: number
    pageSize: number
}
type DatasetListingResult = {
    items: {
        category: string,
        datasetName: string
    }[]
    page: number
    pageSize: number
}

DIRECTORY routes

The following html routes (human-readable) will be available if ENABLE_DIRECTORY is true,

## Listing pages (browser friendly, simple pure html files, similar to apache server)

# list of categories (links to `/dataset/:categoryName`, support pagination via query params)
GET /dataset 

# list of datasets (links to `/dataset/:categoryName/:datasetName`, support pagination via query params)
GET /dataset/:categoryName

# list of tag and versions (contains links to `/dataset/:categoryName/:datasetName/tag/:tagName` and `/dataset/:categoryName/:datasetName/version/:versionHex`)
GET /dataset/:categoryName/:datasetName

# download dataset file by tag
GET /dataset/:categoryName/:datasetName/tag/:tagName

# download dataset file by versionHex
GET /dataset/:categoryName/:datasetName/version/:versionHex

Commands

npx @dataset.sh/server -p 3111 -h 127.0.0.1

Configs

# if AUTH_KEY is set, 
AUTH_KEY=

# where is the data located 
BASE_DIR=

# enable directory routes, default true. 
ENABLE_DIRECTORY=

Storage folder structure

BASE_DIR
|--dataset
|  |--<category>
|  |   |--<datasetName>
|  |   |   |--tag
|  |   |   |  |--<tagName>
|  |   |   |--version        
|  |   |   |  |--<version_hex>
|  |   |   |     |--file.dataset

Tag Reference (dataset/<category>/<datasetName>/tag/<tagName>)

file dataset/<category>/<datasetName>/tag/<tagName> looks like this:

version: hex_value_of_version