PHORBAS Store

A PHORBAS store is an associative 1:1 key to binary content data storage abstraction. Key must be a string, although often hex encoded from cryptographic hashing via PHORBAS Opaque. Content is binary (ArrayBuffer), direclty supporting encrypted content. The store handles the principles of Persistent, Binary content, Addressable, and Store.

• PHORBAS Opaque handles the principle of Hashed, Opaque, and Addressable.
• PHORBAS HAMT handles the principle of Replicable.

API & Use

import {kbc_js_map} from '@phorbas/store'

// use one of the many backend storage adapters
let stg = kbc_js_map()

// keys are opaque strings; some backend adapters may route based on parts of keys
let some_key = `some_key_${Date.now().toString(36)}`

await stg.bkc_exists(some_key) // 0

// store binary content addressed by key
let binary_body = crypto.getRandomValues(new Uint32Array(1024))
await stg.bkc_store(some_key, binary_body)


// key now exists
await stg.bkc_exists(some_key) // 1

// and we can fetch the binary content, returned in an ArrayBuffer
let body_result = await stg.bkc_fetch(some_key)

The backend storage API is fundamentally asynchronous, designed for use with @phorbas/opaque. The kbc prefix is for "keyed binary content".

• async kbc_exists(key) returns 1 if exists, otherwise 0

• async kbc_fetch(key) returns ArrayBuffer if exists, otherwise null|undefined

• async kbc_store(key, body) returns error | null, where error is null upon success.

• async * kbc_stream_exists(key_aiter, {signal}) is an async iterable streaming version of kbc_stream_exists

• async * kbc_stream_fetch(key_aiter, {signal}) is an async iterable streaming version of kbc_stream_fetch

• async * kbc_stream_store(key_body_aiter, {signal}) is an async iterable streaming version of kbc_store

Storage Implementations

JavaScript, Browser, and WinterCG

| Unit Test | Integ Test | Technology / Library | Implementation | |-----------|------------|----------------------|----------------| | ✅ | ✅ | JS Map new Map() | impl: code/js_map.jsy test_js_map.jsy integ--core/integ-core.test.mjs | ✅ | ✅ | JS Fetch API globalThis.fetch() | impl: code/web/fetch.jsy integ_fetch.jsy integ--core/integ-web.test.mjs | ✅ | | Browser DOM Storage globalThis.localStorage or globalThis.sessionStorage | impl: code/web/web_storage.jsy test_web_storage.jsy | ✅ | | Browser DOM Element document.createElement() | impl: code/web/web_dom.jsy test_web_dom.jsy | ✅ | | Browser IndexedDB API globalThis.indexedDB.open() | impl: code/web/web_db.jsy test_web_db.jsy | ✅ | ✅ | Browser Service Worker Cache API globalThis.caches.open() | impl: code/web/web_cache.jsy test_web_cache.jsy integ_web_cache_fetch.jsy

Local

| Unit Test | Integ Test | Technology / Library | Implementation | |-----------|------------|----------------------|----------------| | | ✅ | LMDB node-lmdb | impl: code/local/lmdb.jsy integ--level/integ-lmdb.test.mjs | | ✅ | LevelDB Level ecosystem | impl: code/adapter/level.jsy integ--level/integ-level.test.mjs | ✅ | ✅ | Node FS Promises API import * as fsp from 'node:fs/promises' | impl: code/local/fsp.jsy test_local_fsp.jsy integ--local/integ-fs-promises.test.mjs | ✅ | ✅ | Node FS API import * as fs from 'node:fs' | impl: code/local/fs.jsy test_local_fs.jsy integ--local/integ-fs.test.mjs | ✅ | ✅ | Web FS & FS Promises via lightning-fs from isomorphic-git | impl: code/local/fsp.jsy integ_web_lightning-fs.js

NoSQL

| Integ Test | Technology / Library | Implementation | |------------|------------|----------------------|----------------| | ⚠️ | ArangoDB arangojs | impl: code/nosql/arangojs.jsy integ--arangodb/integ-arango.test.mjs note: Binary values encode in base64 | ✅ | CouchDB nano | impl: code/nosql/couchdb.jsy integ--couchdb/integ-couchdb.test.mjs | ✅ | PouchDB pouchdb | impl: code/nosql/pouchdb.jsy test/nosql/integ_pouchdb.jsy integ--couchdb/integ-pouchdb.test.mjs | ✅ | Consul KV node-consul | impl: code/nosql/consulkv.jsy integ--consulkv/integ-consul.test.mjs | ✅ | AWS DynamoDB aws-sdk | impl: code/nosql/dynamodb.jsy integ--dynamodb/integ-dynamodb.test.mjs | ✅ | Memcache memjs | impl: code/nosql/memjs.jsy integ--memcache/integ-memjs.test.mjs | ✅ | MongoDB mongojs | impl: code/nosql/mongojs.jsy integ--mongodb/integ-mongodb.test.mjs | ✅ | FerretDB mongojs | impl: code/nosql/mongojs.jsy integ--mongodb/integ-mongodb.test.mjs | ✅ | Redis ioredis | impl: code/nosql/ioredis.jsy integ--redis/integ-ioredis.test.mjs | ✅ | Valkey ioredis | impl: code/nosql/ioredis.jsy integ--redis/integ-ioredis.test.mjs | ✅ | RethinkDB rethinkdb | impl: code/nosql/rethinkdb.jsy integ--rethinkdb/integ-rethinkdb.test.mjs

S3 NoSQL

| Integ Test | Technology / Library | Implementation | |------------|----------------------|----------------| | ✅ | MinIO minio | impl: code/nosql/minio.jsy integ--core/integ-minio.test.mjs | ✅ | AWS S3 or compatible aws4fetch | impl: code/nosql/s3_aws4fetch.jsy test/unittest.web.mjs integ--core/integ-aws4fetch.test.mjs

SQL and Knex adapter

| Integ Test | Technology / Library | Implementation | |------------|----------------------|----------------| | ✅ | SQLite import {DatabaseSync} from 'node:sqlite' | impl: code/sql/better_sqlite3.jsy integ--sqlite/integ-sqlite-nodejs.test.mjs | ✅ | SQLite better-sqlite3 | impl: code/sql/better_sqlite3.jsy integ--sqlite/integ-better-sqlite3.test.mjs | ✅ | SQLite node-sqlite3 | impl: code/sql/sqlite.jsy integ--sqlite/integ-sqlite3.test.mjs | ✅ | SQLite knex.js | impl: code/sql/knex.jsy integ--sqlite3/integ-knex-sqlite.test.mjs | ✅ | PostgreSQL knex.js | impl: code/sql/knex.jsy integ--postgres/integ-postgres-knex.test.mjs | ✅ | CockroachDB knex.js | impl: code/sql/knex.jsy integ--postgres/integ-cockroach-knex.test.mjs | ❌ | CrateDB knex.js | impl: code/sql/knex.jsy note: Crate SQL lacks BLOB in-table support | ✅ | MariaDB knex.js | impl: code/sql/knex.jsy integ--maria-mysql/integ-knex-maria-mysql.test.mjs note: Uses MEDIUMBLOB | ✅ | MySQL knex.js | impl: code/sql/knex.jsy integ--maria-mysql/integ-knex-maria-mysql.test.mjs note: Uses MEDIUMBLOB | ✅ | Microsoft SQL Server knex.js | impl: code/sql/knex.jsy integ--mssql-server/integ-mssql.test.mjs

Keyv adapter

| Integ Test | Technology / Library | Implementation | |------------|----------------------|----------------| | ✅ | JavaScript Map keyv and new Map() | impl: code/adapter/keyv.jsy integ--local-filesystem/integ-keyv-map.test.jsy | ⚠️ | One file keyv and keyv-file | impl: code/adapter/keyv.jsy integ--local-filesystem/integ-keyv-file.test.jsy Stored in a single JSON file with binary values encode in base64 | ✅ | Memcache keyv and keyv-memcache | impl: code/adapter/keyv.jsy integ--memcache/integ-keyv-memjs.test.mjs | ✅ | MongoDB keyv and @keyv/mongo | impl: code/adapter/keyv.jsy integ--mongodb/integ-keyv-mongodb.test.mjs | ✅ | SQLite keyv and @keyv/sqlite | impl: code/adapter/keyv.jsy integ--sqlite3/integ-keyv-sqlite.test.mjs

About PHORBAS

The PHORBAS project implements a persistent binary data storage scheme that accomodates opaque (encrypted) information. It is inspired by binary content addressable data storage, replication features of CouchDB, the opaque data storage of Tahoe-LAFS, the eventual consistency of CRDTs, and persistent Hash array mapped trie data structures.

Phorbas appears in in Greek mythology and is connected to "giving pasture" -- in this case, safe pasture to for data storage.

P - Persistent
H - Hashed
O - Opaque
R - Replicable
B - Binary content
A - Addressable
S - Store

Due to the opaque nature of the data storage, the storage is not queryable like a database. It is similar to IPFS with a less ambitious goal of being a pratical data storage scheme for data for embedding into other projects.

License

2-Clause BSD