Run-DB

Crawls the blockchain and indexes RUN state.

Using Run-DB, you can self-host the State APIs that Run uses to work well.

Use Run-DB to:

• Operate a State Server to improve RUN performance by pre-loading jigs
• Query balances, volume, history, and other information across many users and contracts
• Blacklist individual transactions and their descendants in your app
• Create your own local database of transactions your app uses

Requirements

Node 10+

Getting started

1. Install npm run install
2. Download a db snapshot: wget https://run.network/run-db-snapshots/main/latest -O run.db (optional)
3. Run npm run start
4. Install a trustlist: curl -s https://api.run.network/v1/main/trust | curl -H "Content-Type: application/json" -X POST -d @- http://localhost:8000/trust (optional)

Note: For testnet, you may use test in place of main in the above commands.

Use with your App Server

Setup your server's Run instance as follows:

const client = true
const state = new Run.plugins.RunDB('http://localhost:8000')
const trust = ['state']
const run = new Run({ client, state, trust })

Client mode makes Run-DB the source of truth for your server for all jig information. RUN will not load jigs that are not in your database, and your inventory will only be populated by jig UTXOs known to your database.

Setting trust to 'state' makes Run use your database for its trustlist too. This means you only have to setup trust in one place using:

curl -X POST localhost:8000/trust/<txid>

You may also want to run additional instance of Run-DB in SERVE_ONLY mode. That allows you to have an writer that crawls transactions and puts data into the database, and multiple readers that serve your application servers.

Use with a Browser or Mobile Client

The same approach taken for servers can be used to improve performance of client Run instances. You should expose your Run-DB endpoints on a public or private domain rather than connect to localhost. If your client connections are not authenticated, be sure to only expose the GET endpoints and never the POST or DELETE endpoints, and use HTTPS to prevent MITM attacks.

Configuration

Create a .env file or set the following environment variables before running to configure the DB.

| Name | Description | Default | | ---- | ----------- | ------- | | API| mattercloud, planaria, bitcoin-node, run, or none | mattercloud | MATTERCLOUD_KEY | Mattercloud API key | undefined | PLANARIA_TOKEN | Planaria API key | undefined | ZMQ_URL | Only for bitcoin-node. ZMQ tcp url | null | RPC_URL | Only for bitcoin-node. bitcoin RPC http url | null | NETWORK | Bitcoin network (main or test) | main | DB | Database file | run.db | PORT | Port used for the REST server | randomly generated | WORKERS | Number of threads used to index | 4 | FETCH_LIMIT | Number of parallel downloads | 20 | START_HEIGHT | Block height to start indexing | block shortly before sep 2020 | TIMEOUT | Network timeout in milliseconds | 10000 | MEMPOOL_EXPIRATION | Seconds until transactions are removed from the mempool | 86400 | DEFAULT_TRUSTLIST | Comma-separated values of trusted txids | predefined trustlist | SERVE_ONLY | Whether to only serve data and not index transactions | false

Connecting with a bitcoin node

During development is useful to connect to a local node. In order to do this you need to provide RUN-db with access to a bitcoin node trough RPC and ZMQ.

export API="bitcoin-node"
export ZMQ_URL="tcp://your-node-uri:port"
export RPC_URL="http://user:password@your-node-uri:port"

The only zmq message is needed is rawtx. ZMQ is only used to get the new transactions in the mempool.

Direct connection with the node is tested in regtest and testnet, but it's not recommeded for production environments in mainnet at the moment.

Endpoints

• GET /jig/:location - Gets the state for a jig at a particular location

• GET /berry/:location - Gets the state for a berry at a particular location

• GET /tx/:txid - Gets the raw transaction hex for an added transaction

• GET /time/:txid - Gets the block or mempool time of a transaction in seconds since unix epoch

• GET /spends/:location - Gets the spending txid for an output at a particular location

• GET /unspent - Gets the locations of all unspent jigs that are trusted. You may optionally pass in the following query params: class to filter by contract origin, address to filter by owner address, pubkey to filter by owner pubkey, scripthash to filter by hash of the owner script, lock to filter by lock class origin.

• GET /trust/:txid? - Gets whether a particular txid is trusted, or the entire trust list

• GET /ban/:txid? - Gets whether a particular txid is banned, or the entire ban list

• GET /status - Prints status information

• POST /trust/:txid? - Trusts a transaction to execute its code, as well as any untrusted ancestors. To trust multiple transactions at once, you may add an array of txids in the body as application/json.

• POST /ban/:txid - Bans a transaction from being executed, and unindexes it and its descendents

• POST /tx/:txid? - Indexes a transaction and any ancestors. You may optionally add the raw hex data for the transaction in the body as text/plain.

• DELETE /trust/:txid - Removes trust for a transaction, and unindexes it and its descendents

• DELETE /ban/:txid - Removes a transaction ban, and reindexes it and its descendents

• DELETE /tx/:txid - Removes a transaction, its descendents, and any connected state

Performing Custom Queries

Run-DB uses SQLite as its underlying database in WAL mode. SQLite and WAL allows multiple connections to the database so long as there is only one writer, which should be Run-DB. Alternatively, forking Run-DB to create new endpoints for your application may be simpler.

Example Queries

For some of these queries, you will need the JSON1 SQLite extension.

Calculate SHUA supply

SELECT SUM(amount) as supply
FROM (
    SELECT
        json_extract(jig.state, '$.props.amount') AS amount
    FROM jig JOIN spends ON jig.location = spends.location
    WHERE spends.spend_txid IS NULL
    AND jig.class = 'ce8629aa37a1777d6aa64d0d33cd739fd4e231dc85cfe2f9368473ab09078b78_o1')

Calculate SHUA token balances by owner

SELECT owner, SUM(amount) as amount
FROM (SELECT
        json_extract(jig.state, '$.props.owner') AS owner,
        json_extract(jig.state, '$.props.amount') AS amount
    FROM jig JOIN spends ON jig.location = spends.location
    WHERE spends.spend_txid IS NULL
    AND jig.class = 'ce8629aa37a1777d6aa64d0d33cd739fd4e231dc85cfe2f9368473ab09078b78_o1')
GROUP BY owner
ORDER BY amount DESC

Get transaction hex

SELECT HEX(bytes) AS hex
FROM tx
WERE txid = 'ce8629aa37a1777d6aa64d0d33cd739fd4e231dc85cfe2f9368473ab09078b78'

Re-execute all transactions

UPDATE tx SET executed = 0; DELETE FROM jig; DELETE FROM berry;

Database Schema

There are currently 8 tables updated by Run-DB.

jig

Stores jig and code states at output locations or destroyed locations.

| Column | Type | Description | | ------ | ---- | ----------- | | location | TEXT | Jig or code location | | state | TEXT | JSON string describing the object state | | class | TEXT | Contract origin if this state is a jig | | scripthash | TEXT | Hex string of the reversed sha256 of the owner script | | lock | TEXT | Lock class origin if this state has a custom lock |

tx

Stores all transactions known by Run-DB and their indexing state.

| Column | Type | Description | | ------ | ---- | ----------- | | txid | TEXT | Hex string for the transaction hash | | height | INTEGER | Block height for this transaction, or -1 for mempool, or NULL for unknown | | time | INTEGER | Transaction or bock time in seconds since the unix epoch | | bytes | BLOB | Raw transaction data, or NULL if not downloaded | | has_code | INTEGER | 1 if this transaction deployed or upgraded code and requires trust, 0 otherwise | | executable | INTEGER | 1 if this transaction is a valid RUN transaction, 0 otherwise | | executed | INTEGER | 1 if this transaction was executed, even if it failed, 0 otherwise | | indexed | INTEGER | 1 if this transaction's jig states were calculated successfully, 0 otherwise |

spends

Stores spend information about transaction outputs.

| Column | Type | Description | | ------ | ---- | ----------- | | location | TEXT | <txid>_o<output-index> string describing an output | spend_txid| TXID | Hex txid that spent this output, or NULL if unspent

deps

Stores the transaction needed to load a RUN transaction.

| Column | Type | Description | | ------ | ---- | ----------- | | up | TEXT | A transaction ID in hex | | down | TEXT | Hex txid for a transaction that depends on up |

berry

Stores berry states for third-party protocol data.

| Column | Type | Description | | ------ | ---- | ----------- | | location | TEXT | Berry location without the &hash query param | | state | TEXT | JSON string describing the object state |

trust

Stores the transactions which have been trusted and whose code will be executed.

| Column | Type | Description | | ------ | ---- | ----------- | | txid | TEXT | Hex string txid | | value | INTEGER | 1 if trusted, 0 if untrusted |

ban

Stores the transactions which have been blacklisted.

| Column | Type | Description | | ------ | ---- | ----------- | | txid | TEXT | Hex string txid | | value | INTEGER | 1 if blacklisted, 0 otherwise |

crawl

Stores the crawled block tip height and hash for data in the database.

| Column | Type | Description | | ------ | ---- | ----------- | | key | TEXT | 'height' or 'hash' | value | TEXT | String value for the key |