derek

Tools for converting data into schema.

(Still very much pre-alpha!)

Implemented in multiple languages.

| | Index | Coverage | Supported versions | Downloads | | --------------------- | ------------------------------------------- | ---------------------------------------------------- | ----------------------------------------------------------- | ---------------------------------------------------- | | Python | | | | - | | JavaScript (node.js) | | | | | | Rust (coming soon!) | - | - | - | - | | Nim (coming soon!) | - | - | - | - |

1. Installation
2. What is Derek?
   1. Document data structures
   2. Extract schemas from APIs
   3. Really lightweight
   4. Extensible
   5. KISS
3. Documentation
   1. Features
   2. Specification
   3. API

Installation

Python

You can install this from the pypi index. It's available as the derek-py package.

Simple example with pip (poetry is recommended):

pip install derek-py

Complete set of supported installation methods:

| Package manager | pypi | git | | --------------- | ---------------------- | ------------------------------------------------------------- | | pip | pip install derek-py | pip install git+https://github.com/benjaminwoods/derek@main | | poetry | poetry add derek-py | poetry add git+https://github.com/benjaminwoods/derek#main |

Javascript (Node.js)

You can install this from the npm index. It's available as the derek-ts package.

Simple example with yarn:

yarn add derek-ts

Complete set of supported installation methods:

| Package manager | npm | git | | --------------- | ------------------- | ---------------------------------------------------------- | | npm | npm i derek-ts | npm i git+https://github.com/benjaminwoods/derek#main | | yarn | yarn add derek-ts | yarn add git+https://github.com/benjaminwoods/derek#main |

What is Derek?

Here's a quick guide showing what you can do with derek. These examples are for a Python implementation.

Derek documents data structures.

Load some data into a tree of nodes:

# Import the main class
from derek import Derek

# Suppose that you have some JSON-compatible data
obj = [
  {
    'some': [1.0, 3, "4.5"],
    'data': [3.4, 4.5]
  },
  {
    'some': [2, "4.0", 1.5],
    'data': [1.4]
  }
]

# Feed this data into Derek.tree
root_node = Derek.tree(obj, name='MyDataStructure')

You can use .example() to see a simple example item of data:

>>> root_node.example()
[{'some': [1.0], 'data': [3.4]}]

You can produce an OAS2/OAS3 JSON schema from this data, too:

j = root_node.parse(format='oas3')
import json
print(json.dumps(j, indent=2))

{
  "MyDataStructure": {
    "type": "array",
    "items": {
      "type": "object",
      "additionalProperties": {
        "oneOf": [
          {
            "type": "array",
            "items": {
              "oneOf": [
                {
                  "type": "string"
                },
                {
                  "type": "integer"
                },
                {
                  "type": "number"
                }
              ]
            }
          },
          {
            "type": "array",
            "items": {
              "type": "number"
            }
          }
        ]
      }
    },
    "example": [
      {
        "some": [1.0],
        "data": [3.4]
      }
    ]
  }
}

Install and use the yaml package to convert this structure to an OAS3-compliant data schema.

import yaml
print(yaml.dump(j))

MyDataStructure:
  example:
    - data:
        - 3.4
      some:
        - 1.0
  items:
    additionalProperties:
      oneOf:
        - items:
            type: number
          type: array
        - items:
            oneOf:
              - type: number
              - type: integer
              - type: string
          type: array
    type: object
  type: array

Derek extracts schemas from APIs.

Quickly extract schemas from APIs, by feeding the returned JSON into Derek.

from derek import Derek

from pycoingecko import CoinGeckoAPI
cg = CoinGeckoAPI()

# Get all coins from CoinGecko
root_node = Derek.tree(cg.get_coins_list(), name='GetCoins')

Parse to get your schema:

j = root_node.parse(format='oas3')
import json
print(json.dumps(j, indent=2))

{
  "GetCoins": {
    "type": "array",
    "items": {
      "type": "object",
      "additionalProperties": {
        "type": "string"
      }
    },
    "example": [
      {
        "id": "01coin",
        "symbol": "zoc",
        "name": "01coin"
      }
    ]
  }
}

Derek is really lightweight.

No required dependencies. Always.

Derek is extensible.

Use libraries like pywhat and yaml to quickly extend Derek:

import json, yaml

from derek import Derek, Parser

from pywhat import Identifier

class PywhatDerek(Derek):
    @property
    def parser(self):
        return PywhatParser()

    def get_oas3_yaml(self):
        return yaml.dump(
            self.parse(format="oas3")
        )

class PywhatParser(Parser):
    @classmethod
    def oas2(cls, node):
        # Call the superclass parser for the current node:
        #   _sup = cls.__mro__[PywhatParser.__mro__.index(int):]
        #   j = _sup.oas2(cls, node)
        # All calls to the oas2 method in the superclass therefore re-route
        # back to this class method, automatically handling all recursive calls
        # here.
        j = super(PywhatParser, cls).oas2(node)

        # The rest of this function simply patches in results from a call
        # to the pywhat API.
        identifier = Identifier()

        if all(map(lambda t: not isinstance(node.value, t), [list, dict])):
            result = identifier.identify(str(node.value))

            if result['Regexes'] is not None:
                matches = [entry for entry in result['Regexes']['text']]

                # Select the match as the longest string
                map_func = lambda d: (d['Matched'], d['Regex Pattern']['Name'])
                max_func = lambda tup: len(tup[0])
                _, match = max(
                    map(map_func, matches),
                    key=max_func
                )

                j = {
                    **j,
                    'description': match
                }

        return j

Allowing for functionality like:

root_node = PywhatDerek.tree(
    {'data': ['17VZNX1SN5NtKa8UQFxwQbFeFc3iqRYhem']},
    name='Addresses'
)
root_node.get_oas3_yaml()

returning:

Addresses:
  additionalProperties:
    items:
      description: "Bitcoin (\u20BF) Wallet Address"
      type: string
    type: array
  example:
    data:
      - 17VZNX1SN5NtKa8UQFxwQbFeFc3iqRYhem
  type: object

Derek is straightforward.

Derek is designed for ease of use. If you're trying to use Derek functionality in a workflow and it feels like it should be easier to get your desired result, please make an issue.