git-off handles large files in git repos.

This is a prototype inspired by git lfs and git fat.

By default the files are stored in the home directory.

Supported transports are:

• copy - store locally
• rsync
• ssh/scp
• http (read-only)
• s3

Install

npm install -g git-off

Then

git off

for help.

https://www.npmjs.com/package/git-off

C Version

There is a git-off written in C in the c directory. To use it do the following step:

In Debian:

apt-get install tcc
cd /usr/local/bin
ln -s $PATH_TO_GITOFF_GIT/c/git-off.c git-off

In MacOS:

cd /usr/local/bin
clang -o git-off $PATH_TO_GITOFF_GIT/c/gitoff.c

Quick Start

Setup:

git off track '*.bin'
git add .
git commit
git push
git checkout master

Other

git off install
git off mode scp
git off scp localhost:/tmp/offStore
git off scpuser username
git off cc
git off ca
git off env
git off defaults

Configuration

The configuration is saved in 3 locations:

• $GIT_OFF_CONFIG
• repo config: .git-off file at the root of repo
• global git config

For each key:

• git off looks for the value in $GIT_OFF_CONFIG
• if the key is not found, git off looks for the value in repo config
• if the key is not found, git off looks for the value in the global git config

Run:

git off env

to see the current configuration.

The log file location is always configured in the global git config.

SSH Setup

Create an SSH key:

ssh-keygen -t rsa

Copy id_rsa.pub file to SSH server and id_rsa to ~/.ssh

Configure your private ssh key:

git off pem ~/.ssh/id_rsa

Use git off scp mode:

git off mode scp

Input user, ssh host and path for store:

git off scp $USER@localhost:offStore

Alternatively, setup SSH with ssh config, edit ~/.ssh/config and add your config:

host localhost
  HostName localhost
  IdentityFile ~/.ssh/id_rsa
  User username

Quick ssh and http server setup with docker

Start http and ssh servers:

docker run --name gitoffHttp -p 8080:80 -v ~/.git-off/offStore:/usr/share/nginx/html:ro -d nginx:alpine
docker run --name gitoffSSH -p 2222:22 -v ~/.git-off/offStore:/root/offStore -d sickp/alpine-sshd

Setup SSH public key:

ssh root@localhost -p 2222 "mkdir .ssh"

The password is root

Setup git off:

git off mode scp
git off scp root@localhost:2222/root/offStore

Change root password in SSH server to something more difficult to guess:

ssh root@localhost -p 2222
passwd

Amazon S3

In AWS S3, create a new bucket for your git off store. Enter the name of the bucket for the bucket option in your S3 store options.

Select the bucket, and then select Properties. Note the region, and enter the correct region in your S3 store options in git off.

In AWS IAM, create a new user. Copy the generated key and secret to git off config

Select your newly created user. In the bottom area, select Permissions > Inline Policy.

Select Custom Policy.

To create the custom policy, give it any name you want, and then copy and paste the example policy below. Replace "mybucketname" with your actual bucket name.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListAllMyBuckets",
        "s3:ListBucket"
      ],
      "Resource": "arn:aws:s3:::*"
    },
    {
      "Action": [
        "s3:PutObject",
        "s3:PutObjectAcl",
        "s3:GetObject",
        "s3:GetObjectAcl",
        "s3:DeleteObject",
        "s3:DeleteObjectAcl"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::mybucketname/*"
      ]
    }
  ]
}

Create a credentials file at ~/.aws/credentials:

[default]
aws_access_key_id = your_access_key
aws_secret_access_key = your_secret_key

Setup s3 in git off:

git off mode s3
git off s3region us-west-2
git off s3bucket mybucketname

COMMANDS

git off install [thisrepo]
  setup git config (default global)
  thisrepo sets up config in current repo
git off mode [thisrepo] [copy|rsync|scp|http|s3]
  set/show git off mode
git off store [thisrepo] [path]
  set/show git off store path for copy mode
git off scp [thisrepo] [host]
  setup scp config
  host has format host:path, user@host:path, user@host:port/path
  Example: localhost:/tmp/offStore
git off http [thisrepo] [host]
  setup http config
  host has format http://host/path
git off curl [thisrepo] [options]
  setup curl config
git off integrity [thisrepo] [enable|disable]
  set/show git off integrity.
  when enabled, the SHA of the file received from the store is
  checked again the SHA of the original file
git off pem [thisrepo] [pathToPrivateKey]
  set/show git off pem.
  off.pem is the private key for ssh and scp
  set 'offNoValue' to set an empty value (useful when there are multiple configs)
git off sshoptions [thisrepo] [options]
  set/show git off sshoptions
git off scpoptions [thisrepo] [options]
  set/show git off scpoptions
git off rsyncoptions [thisrepo] [options]
  set/show git off rsyncoptions
git off scpuser [thisrepo] [username]
  setup scp username config
git off track
  setup gitattribute filters
  example: git off track '*.bin'
  without parameter, list git off attributes
  calls git off install
git off configAlways [''|GIT_OFF_CONFIG|repo|global]
  '' disable configAlways
  GIT_OFF_CONFIG load all configurations from $GIT_OFF_CONFIG
  repo load all configurations from current git repo
  global load all configurations from global git config
  set 'offNoValue' to set an empty value
git off s3region [thisrepo] [region]
  setup amazon s3 region for the bucket
git off s3bucket [thisrepo] [bucket]
  setup amazon s3 bucket
git off transform [thisrepo] [enable|disable]
  enable transform in clean and smudge filters
git off transformTo [thisrepo] ['cmd _1 _2']
  setup transform command for clear filter
  When the command is empty the regular transport is performed
git off transformFrom [thisrepo] ['cmd _1 _2']
  setup transform command for smudge filter
  When the command is empty the regular transport is performed
git off clean
  internal filter
  dont use directly
git off pre-push
  internal filter
  dont use directly
git off smudge
  internal filter
  dont use directly
git off copyTo [copy|rsync|scp|s3]
  copy cache to store for specified mode
git off push
  copy cache to store for selected mode
git off clearAll
  delete store, cache and log
git off ca
  delete store, cache and log
git off clearCache
  delete cache in current git
git off cc
  delete cache in current git
git off clearStore
  delete store
git off cs
  delete store
git off clearTmp
  delete tmp in git off cache
  Useful when transform is enabled
git off ct
  delete tmp in git off cache
  Useful when transform is enabled
git off defaults
  shows first time config
git off env
  shows config
git off help [cmd]
  git off help. Run git off help command to get help for a specific command.

Dependencies

• nodejs
• git
• rsync for rsync mode
• ssh and scp for scp mode
• curl for http mode

Platforms

• Linux
• MacOS

Hacking

Edit src/gitoff.coffee and then run ./gen.sh

Alternatively, edit the generated javascript file directly (src/git-off)

Contributions

I'm looking for someone who can contribute to the windows version.