restify-git-json

v0.0.10

Published

4 years ago

Restful interface to a git repo

0High
0Medium
0Low

bewest

restify-git-json

An API to git repos exposed as a RESTful interface.

Uses libraries to package up js-git as a set of streaming interfaces. Uses git streams to provide a RESTful API ala github to inspect, create, and manipulate git repos.

Includes an endpoint to upload files, streaming them into a uniquely identified git reference per user per namespace per upload.

Install

$ git clone [email protected]:tidepool-org/restify-git-json.git
# or
$ git clone git://github.com/tidepool-org/restify-git-json.git
$ cd restify-git-json
$ npm install

Run

$ node server.js

Environment variables

PORT Expects to be assigned a tcp port to serve http requests on via PORT environment variable. Default is 6776.
BASE the directory housing bare git repos if using fs-db backend. Default value is ./out.
GIT_BACKEND fs-db (default) or memdb.

Choosing a git backend

Bare file database `fs-db`

Each git database is stored on disk using the "bare" repo layout (no working directory, no files are left "checked out"). See git repository layout for more information.

The current advice is to use this one if you aren't sure.

Memory based database `memdb`

The git database is stored in memory, which is cool, but you can only get the contents via the http endpoints supported by the server. (Cloning is post-mvp.) See lib/handlers/experiments for WIP.

API

HTTP REST Endpoints

Basic idea is to wrap a configurable git repo (backed by s3, memory, or bare fs) using js-git, providing a RESTful API identical to the github json API.

Basic pattern is:

METHOD	ENDPOINT	Description
GET	/repos/:owner/:repo/git/:type
GET	/repos/:owner/:repo/git/refs
GET	/repos/:owner/:repo/git/refs/:ref
GET	/repos/:owner/:repo/git/refs/:ref
GET	/repos/:owner/:repo/git/commits/:sha
GET	/repos/:owner/:repo/git/trees/:sha
GET	/repos/:owner/:repo/git/blobs/:sha

`Uploads`

As a bonus, we expose an additional endpoint

METHOD	ENDPOINT	Description
POST	/repos/:owner/:repo/upload

This pins uploads to a uniquely identified branch. The result can then be cloned for replication.

The semantics for working with this endpoint are the exact same as normal browser file uploads. As a bonus feature, if a header with content-md5 is sent, the checksum of the file is validated before being saved. For all cases, sha1 sums are provided for each upload.

Caveats cloning is currently only possible when using fs bare option (default).

Example

+ curl -ivs -F web=@./History.md -F [email protected] localhost:6776/repos/me/test/upload
+ json
* About to connect() to localhost port 6776 (#0)
*   Trying 127.0.0.1... connected
> POST /repos/me/test/upload HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: localhost:6776
> Accept: */*
> Content-Length: 2911
> Expect: 100-continue
> Content-Type: multipart/form-data; boundary=----------------------------124e1b80ed69
> 
< HTTP/1.1 100 Continue
} [data not shown]
< HTTP/1.1 201 Created
< Connection: close
< Content-Type: application/json
< Content-Length: 866
< Date: Wed, 15 Jan 2014 05:58:03 GMT
< 
{ [data not shown]
* Closing connection #0
{
  "err": null,
  "body": [
    {
      "ref": "upload/incoming/2014-01-15-79083446/489841",
      "sha": "4898416679ef377c29690b3e4fe0ed2c7aa7ef23",
      "head": {
        "commit": "4898416679ef377c29690b3e4fe0ed2c7aa7ef23",
        "tree": {
          "tree": "f9526358da6b35eb90d365309dfea451f91e096c",
          "author": {
            "name": "MY AUTHOR",
            "email": "git@js"
          },
          "committer": {
            "name": "MY COMMITTER",
            "email": "git@js"
          },
          "message": "MY JUSTIFICATION",
          "url": "http://localhost:6776/repos/me/test/git/trees/f9526358da6b35eb90d365309dfea451f91e096c"
        },
        "url": "http://localhost:6776/repos/me/test/git/commits/4898416679ef377c29690b3e4fe0ed2c7aa7ef23"
      },
      "content": [
        "http://localhost:6776/repos/me/test/raw/upload/incoming/2014-01-15-79083446/489841/History.md",
        "http://localhost:6776/repos/me/test/raw/upload/incoming/2014-01-15-79083446/489841/env.js"
      ],
      "url": "http://localhost:6776/repos/me/test/git/refs/heads/upload/incoming/2014-01-15-79083446/489841"
    }
  ]
}

So each request like this creates a new set of blobs on a new uniquely identified branch, per upload per namespace (repo) per user (per server).

Installed routes

+ curl -ivs localhost:6776/help
+ json
* About to connect() to localhost port 6776 (#0)
*   Trying 127.0.0.1... connected
> GET /help HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: localhost:6776
> Accept: */*
> 
< HTTP/1.1 200 OK
< Connection: close
< Content-Type: application/json
< Content-Length: 998
< Date: Wed, 15 Jan 2014 06:01:29 GMT
< 
{ [data not shown]
* Closing connection #0
[
  {
    "path": "/status",
    "version": "0.0.3",
    "middleware": 0,
    "method": "GET"
  },
  {
    "path": "/repos/:owner/:repo",
    "version": "0.0.3",
    "middleware": 4,
    "method": "GET"
  },
  {
    "path": "/repos/:owner/",
    "version": "0.0.3",
    "middleware": 0,
    "method": "GET"
  },
  {
    "path": "/repo/test",
    "version": "0.0.3",
    "middleware": 2,
    "method": "POST"
  },
  {
    "path": "/repos/:owner/:repo/upload",
    "version": "0.0.3",
    "middleware": 5,
    "method": "POST"
  },
  {
    "path": "/repos/:owner/:repo/git/refs",
    "version": "0.0.3",
    "middleware": 4,
    "method": "GET"
  },
  {
    "path": "/repos/:owner/:repo/git/refs/(.*)",
    "version": "0.0.3",
    "middleware": 4,
    "method": "GET"
  },
  {
    "path": "/repos/:owner/:repo/git/commits/:sha",
    "version": "0.0.3",
    "middleware": 4,
    "method": "GET"
  },
  {
    "path": "/repos/:owner/:repo/git/trees/:sha",
    "version": "0.0.3",
    "middleware": 4,
    "method": "GET"
  },
  {
    "path": "/repos/:owner/:repo/git/blobs/:sha",
    "version": "0.0.3",
    "middleware": 4,
    "method": "GET"
  },
  {
    "path": "/repos/:owner/:repo/raw/(.*)",
    "version": "0.0.3",
    "middleware": 4,
    "method": "GET"
  },
  {
    "path": "/help/",
    "version": "0.0.3",
    "middleware": 1,
    "method": "GET"
  }
]

Use API to fetch data

+ curl -ivs http://localhost:6776/repos/me/proof/git/refs
+ json -H
* About to connect() to localhost port 6776 (#0)
*   Trying 127.0.0.1... connected
> GET /repos/me/proof/git/refs HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: localhost:6776
> Accept: */*
> 
< HTTP/1.1 200 OK
< Connection: close
< Content-Type: application/json
< Content-Length: 2452
< Date: Thu, 26 Dec 2013 00:05:36 GMT
< 
{ [data not shown]
* Closing connection #0
HTTP/1.1 200 OK
Connection: close
Content-Type: application/json
Content-Length: 2452
Date: Thu, 26 Dec 2013 00:05:36 GMT

{
  "err": null,
  "body": [
    {
      "ref": "refs/heads/upload/incoming/2013-12-25-51951039/722e28",
      "sha": "722e28bf2d3aa24cb3eae6b34c63a1bef0f2311a",
      "url": "http://localhost:6776/repos/me/proof/git/refs/heads/upload/incoming/2013-12-25-51951039/722e28"
    },
    {
      "ref": "refs/heads/upload/incoming/2013-12-25-51956598/cb3955",
      "sha": "cb3955e12f4b81c25ed307a6d196c0b5decc7883",
      "url": "http://localhost:6776/repos/me/proof/git/refs/heads/upload/incoming/2013-12-25-51956598/cb3955"
    },
    {
      "ref": "refs/heads/upload/incoming/2013-12-25-51959099/167675",
      "sha": "167675a1605d03d9eaa5448b432e250a86fccb39",
      "url": "http://localhost:6776/repos/me/proof/git/refs/heads/upload/incoming/2013-12-25-51959099/167675"
    },
    {
      "ref": "refs/heads/upload/incoming/2013-12-25-51961695/bf9c8e",
      "sha": "bf9c8ec01ef9f8926109f12e64c3b0a965c7e845",
      "url": "http://localhost:6776/repos/me/proof/git/refs/heads/upload/incoming/2013-12-25-51961695/bf9c8e"
    },
    {
      "ref": "refs/heads/upload/incoming/2013-12-25-51962813/4f9f5c",
      "sha": "4f9f5c47326c677fb91718c953b7d1dec21e7921",
      "url": "http://localhost:6776/repos/me/proof/git/refs/heads/upload/incoming/2013-12-25-51962813/4f9f5c"
    },
    {
      "ref": "refs/heads/upload/incoming/2013-12-25-54381462/494606",
      "sha": "494606f961d5b519df094662365f52c539a3a142",
      "url": "http://localhost:6776/repos/me/proof/git/refs/heads/upload/incoming/2013-12-25-54381462/494606"
    },
    {
      "ref": "refs/heads/upload/incoming/2013-12-25-57444767/09ae06",
      "sha": "09ae0668f6792d11a55fc050af494d04e6616e14",
      "url": "http://localhost:6776/repos/me/proof/git/refs/heads/upload/incoming/2013-12-25-57444767/09ae06"
    },
    {
      "ref": "refs/heads/upload/incoming/2013-12-25-57479601/9d8cd2",
      "sha": "9d8cd26166485fcbf63bd0667456a1ef94255a35",
      "url": "http://localhost:6776/repos/me/proof/git/refs/heads/upload/incoming/2013-12-25-57479601/9d8cd2"
    },
    {
      "ref": "refs/heads/upload/incoming/2013-12-25-57500649/127d5a",
      "sha": "127d5a329803597987a912788c260812ead21af7",
      "url": "http://localhost:6776/repos/me/proof/git/refs/heads/upload/incoming/2013-12-25-57500649/127d5a"
    },
    {
      "ref": "refs/heads/upload/incoming/2013-12-25-57521748/49a3c6",
      "sha": "49a3c6019a37aa4d6762236c58bf5a8cb78fda0a",
      "url": "http://localhost:6776/repos/me/proof/git/refs/heads/upload/incoming/2013-12-25-57521748/49a3c6"
    },
    {
      "ref": "refs/heads/upload/incoming/2013-12-25-57540359/36ca88",
      "sha": "36ca88c51023c38b135b243fe054da98d3eb6ac5",
      "url": "http://localhost:6776/repos/me/proof/git/refs/heads/upload/incoming/2013-12-25-57540359/36ca88"
    }
  ],
  "url": "http://localhost:6776/repos/me/proof/git/refs/"
}

Get ref

+ curl -ivs http://localhost:6776/repos/me/proof/git/refs/heads/upload/incoming/2013-12-25-57540359/36ca88
+ json -H
* About to connect() to localhost port 6776 (#0)
*   Trying 127.0.0.1... connected
> GET /repos/me/proof/git/refs/heads/upload/incoming/2013-12-25-57540359/36ca88 HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: localhost:6776
> Accept: */*
> 
< HTTP/1.1 200 OK
< Connection: close
< Content-Type: application/json
< Content-Length: 184
< Date: Thu, 26 Dec 2013 00:00:56 GMT
< 
{ [data not shown]
* Closing connection #0
HTTP/1.1 200 OK
Connection: close
Content-Type: application/json
Content-Length: 184
Date: Thu, 26 Dec 2013 00:00:56 GMT

{
  "err": null,
  "body": {
    "type": "commit",
    "sha": "1daf69f11ee1f5ff253e1a3300839cf6c7be710f",
    "url": "http://localhost:6776/repos/me/proof/git/commits/36ca88c51023c38b135b243fe054da98d3eb6ac5"
  }
}

Get commit

+ curl -ivs http://localhost:6776/repos/me/proof/git/commits/36ca88c51023c38b135b243fe054da98d3eb6ac5
+ json -H
* About to connect() to localhost port 6776 (#0)
*   Trying 127.0.0.1... connected
> GET /repos/me/proof/git/commits/36ca88c51023c38b135b243fe054da98d3eb6ac5 HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: localhost:6776
> Accept: */*
> 
< HTTP/1.1 200 OK
< Connection: close
< Content-Type: application/json
< Content-Length: 402
< Date: Thu, 26 Dec 2013 00:01:44 GMT
< 
{ [data not shown]
* Closing connection #0
HTTP/1.1 200 OK
Connection: close
Content-Type: application/json
Content-Length: 402
Date: Thu, 26 Dec 2013 00:01:44 GMT

{
  "err": null,
  "body": {
    "type": "commit",
    "body": {
      "tree": "1daf69f11ee1f5ff253e1a3300839cf6c7be710f",
      "parents": [],
      "author": {
        "name": "MY AUTHOR",
        "email": "git@js",
        "date": "2013-12-25T23:59:00.000Z"
      },
      "committer": {
        "name": "MY COMMITTER",
        "email": "git@js",
        "date": "2013-12-25T23:59:00.000Z"
      },
      "message": "MY JUSTIFICATION"
    },
    "url": "http://localhost:6776/repos/me/proof/git/trees/1daf69f11ee1f5ff253e1a3300839cf6c7be710f"
  }
}

Get tree

+ json -H
+ curl -ivs http://localhost:6776/repos/me/proof/git/trees/1daf69f11ee1f5ff253e1a3300839cf6c7be710f
* About to connect() to localhost port 6776 (#0)
*   Trying 127.0.0.1... connected
> GET /repos/me/proof/git/trees/1daf69f11ee1f5ff253e1a3300839cf6c7be710f HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: localhost:6776
> Accept: */*
> 
< HTTP/1.1 200 OK
< Connection: close
< Content-Type: application/json
< Content-Length: 291
< Date: Thu, 26 Dec 2013 00:02:02 GMT
< 
{ [data not shown]
* Closing connection #0
HTTP/1.1 200 OK
Connection: close
Content-Type: application/json
Content-Length: 291
Date: Thu, 26 Dec 2013 00:02:02 GMT

{
  "err": null,
  "body": {
    "input.txt": {
      "mode": 33188,
      "hash": "ca32e189ce9fc464583de225ae5170bc7b8bf776",
      "url": "http://localhost:6776/repos/me/proof/git/blobs/ca32e189ce9fc464583de225ae5170bc7b8bf776"
    }
  },
  "url": "http://localhost:6776/repos/me/proof/git/trees/1daf69f11ee1f5ff253e1a3300839cf6c7be710f/"
}

Get blob

+ curl -ivs http://localhost:6776/repos/me/proof/git/blobs/ca32e189ce9fc464583de225ae5170bc7b8bf776
+ json -H -C content
* About to connect() to localhost port 6776 (#0)
*   Trying 127.0.0.1... connected
> GET /repos/me/proof/git/blobs/ca32e189ce9fc464583de225ae5170bc7b8bf776 HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: localhost:6776
> Accept: */*
> 
< HTTP/1.1 200 OK
< Connection: close
< Content-Type: application/json
< Content-Length: 691
< Date: Thu, 26 Dec 2013 00:02:34 GMT
< 
{ [data not shown]
* Closing connection #0
HTTP/1.1 200 OK
Connection: close
Content-Type: application/json
Content-Length: 691
Date: Thu, 26 Dec 2013 00:02:34 GMT

2011-03-09T08:38:01	242
2013-05-08T14:43:02	89
2013-06-08T12:22:34	100
2013-06-08T11:28:39	143
2013-07-14T13:14:37	98
2011-03-08T21:34:27	126
2011-04-08T19:30:37	94
2011-04-08T18:29:12	91
2011-11-08T16:55:07	100
2011-11-08T16:21:07	84
2012-01-14T11:58:00	66
2012-01-14T09:44:39	94
2012-03-14T02:42:41	144
2012-05-14T01:54:29	204
2013-03-08T16:20:21	57
2013-03-08T14:43:02	89
2013-03-08T12:22:34	100
2013-03-08T11:28:39	143
2013-04-14T13:14:37	98
2013-05-14T11:58:00	66
2013-05-14T09:44:39	94
2013-05-14T02:42:41	144
2013-05-14T01:54:29	204
2013-07-14T11:58:00	66
2013-08-14T09:44:39	94

Inspired by

https://github.com/creationix/js-git/blob/master/examples/create.js
https://gist.github.com/bewest/8008603
https://github.com/substack/stream-handbook#transform
https://github.com/dominictarr/event-stream#eventstream

TODO

the rest

update repo metadata
update user metadata

ROADMAP

get POST json api endpoints working
- doublecheck API compatibility with github
test with memdb
- develop module for memcache/redis
- develop module for s3 (knox, request, use proxy to s3)
test git http support XXX: so far I have only tested using file based git bare repo... would prefer http support for mem and s3 modules
develop export module
- bundle with encrypt?
develop import module
develop remotes/push module

fetch content

+ curl -ivs http://localhost:6776/repos/me/test/raw/upload/incoming/2014-01-15-79083446/489841/History.md
* About to connect() to localhost port 6776 (#0)
*   Trying 127.0.0.1... connected
> GET /repos/me/test/raw/upload/incoming/2014-01-15-79083446/489841/History.md HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: localhost:6776
> Accept: */*
> 
< HTTP/1.1 200 OK
< Connection: close
< Content-Type: text/plain
< Content-Length: 2192
< Date: Wed, 15 Jan 2014 06:06:16 GMT
< 
{ [data not shown]
* Closing connection #0
HTTP/1.1 200 OK
Connection: close
Content-Type: text/plain
Content-Length: 2192
Date: Wed, 15 Jan 2014 06:06:16 GMT



v0.0.3 / 2013-12-25
==================

 * 0.0.2

... commits since last month

  * Ben West - 0.0.2
  * Ben West - 0.0.1
  * Ben West - final move cleanup
  * Ben West - move rest over to single install method
  * Ben West - record WIP, final move
  * Ben West - tweak comment
  * Ben West - delete code moved to handlers
  * Ben West - move get-blob to own handler
  * Ben West - move get tree to own handler
  * Ben West - migrate get-commit handler
  * Ben West - move upload branch handler
  * Ben West - get ready to move rest of handlers
  * Ben West - remove experiment from main area
  * Ben West - update post-upload-pack
  * Ben West - move rest of experiments out of way
  * Ben West - remove previous moved code
  * Ben West - moving one experiment out of the way
  * Ben West - start stubbing out experiment
  * Ben West - remove old version of handler
  * Ben West - tweak whitespace
  * Ben West - get moved handler working
  * Ben West - stub out moving another handler
  * Ben West - move another handler to urlize
  * Ben West - update TODO
  * Ben West - moved to handlers
  * Ben West - move one handler over
  * Ben West - get urlize middleware working
  * Ben West - successfully re-organized this code into ./lib/middleware/
  * Ben West - depend on new upload middleware
  * Ben West - initialize upload middleware
  * Ben West - stub out uploads middleware
  * Ben West - move more middleware
  * Ben West - stub out separating more middleware
  * Ben West - Example re-organize single middleware
  * Ben West - markup TODOs
  * Ben West - make ls-remote work and get start on clone
  * Ben West - ls-remote works now
  * Ben West - ...
  * Ben West - add working and non working examples
  * Ben West - A bit closer?
  * Ben West - basic fixes
  * Ben West - update docs
  * Ben West - make link to blobs work
  * Ben West - working links
  * Ben West - remove spurious README
  * Ben West - Merge branch 'master' of github.com:bewest/restify-git-json
  * Ben West - Initial commit
  * Ben West - stub out bunch more stuff
  * Ben West - Add stubbed out experiments
  * Ben West - add packages
  * Ben West - init
n.n.n / 2013-12-25
==================

 * 0.0.2

Customize profile creation

+ json
+ curl -isv -XPOST 'http://localhost:6776/users/crash/[email protected]&name=Tessy+Tidepool&handle=secret&user\[email\][email protected]&user\[name\]=Tessy+Tidepool&author\[name\]=Foo+Bar&author\[email\][email protected]&committer\[name\]=blip+on+behalf+of+secret&committer\[email\]=blip%4Atideool.io&message=my+create+message'
* About to connect() to localhost port 6776 (#0)
*   Trying 127.0.0.1... connected
> POST /users/crash/[email protected]&name=Tessy+Tidepool&handle=secret&user[email][email protected]&user[name]=Tessy+Tidepool&author[name]=Foo+Bar&author[email][email protected]&committer[name]=blip+on+behalf+of+secret&committer[email]=blip%4Atideool.io&message=my+create+message HTTP/1.1

> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3

> Host: localhost:6776

> Accept: */*

> 

< HTTP/1.1 201 Created

< Connection: close

< Content-Type: application/json

< Content-Length: 656

< Date: Sat, 18 Jan 2014 18:43:31 GMT

< 

{ [data not shown]
* Closing connection #0

{
  "name": "Tessy Tidepool",
  "handle": "secret",
  "email": "[email protected]",
  "committer": {
    "name": "blip on behalf of secret",
    "email": "blipJtideool.io"
  },
  "author": {
    "name": "Foo Bar",
    "email": "[email protected]"
  },
  "user": "crash",
  "url": "http://localhost:6776/users/crash",
  "updated": {
    "ref": "upload/incoming/2014-01-18-38611838/538497",
    "sha": "538497ec4b9db2eca4c7eb63e9acd0ec234a2c9c",
    "head": {
      "commit": "538497ec4b9db2eca4c7eb63e9acd0ec234a2c9c",
      "tree": {
        "tree": "728aee4bbb551213b1cc829357da3ce118d2b55a",
        "author": {
          "name": "Foo Bar",
          "email": "[email protected]"
        },
        "committer": {
          "name": "blip on behalf of secret",
          "email": "blipJtideool.io"
        },
        "message": "my create message"
      }
    }
  }
}

Published

Vulnerabilities

Links

Maintainers

Keywords

Readme

restify-git-json

Install

Run

Environment variables

Choosing a git backend

Bare file database fs-db

Memory based database memdb

API

HTTP REST Endpoints

Uploads

Example

Installed routes

Use API to fetch data

Get ref

Get commit

Get tree

Get blob

Inspired by

TODO

the rest

ROADMAP

fetch content

Customize profile creation

Bare file database `fs-db`

Memory based database `memdb`

`Uploads`