signatures (@data-leakage-protection/signatures)

Identify confidential and sensitive info in source code repositories by data-loss "signatures".

@data-leakage-protection/signatures is a Node.js module for storing and accessing to data-leakage detection definitions. We call the data structure that represents a data-leakage detection defintion a "signature." We store a community-tested list of signatures in a file called signatures.json.

Table of Contents

• 1. Security
• 2. Install
• 3. Usage
• 4. API
  • 4.1. @data-leakage-protection/signatures.Signature
  • 4.2. @data-leakage-protection/signatures.Signature.prototype.match
• 5. Accessing signatures with other tools and programming languages
• 6. Maintainers
• 7. Contributions
  • 7.1. Adding a Signature
  • 7.2. Editing a Signature
  • 7.3. Removing a Signature
• 8. License
• 9. References and Attributions

1. Security

Data leakage is the unauthorized transmission of data from within an organization to an external destination or recipient.[^1]

One of the most common forms of data-loss (aka, "data leakage") happens when developers (inadvertently) commit and push passwords, access-tokens, and sensitive data to a source-control management system (like Git). Consequently, confidential information "leaks" into search results and commit history.

The signatures.json contains a growing list of definitions to help you detect secrets in your source code repositories.

| | Signature | Detected in | |----:|-----------|-------------| | 1 | .asc file extensionPotential cryptographic key bundle | extension | | 2 | .p12 file extensionPKCS#12 (.p12): potential cryptographic key bundle | extension | | 3 | .pem file extensionPotential cryptographic private key | extension | | 4 | .pfx file extensionPKCS#12 (.pfx): Potential cryptographic key bundle | extension | | 5 | .pkcs12 file extensionPKCS#12 (.pkcs12): Potential cryptographic key bundle | extension | | 6 | 1Password password manager database fileFeed it to Hashcat and see if you're lucky | extension | | 7 | AWS API Key | contents | | 8 | AWS CLI credentials file | path | | 9 | Apache htpasswd file | filename | | 10 | Apple Keychain database file | extension | | 11 | Azure service configuration schema file | extension | | 12 | Carrierwave configuration fileCan contain credentials for cloud storage systems such as Amazon S3 and Google Storage | filename | | 13 | Chef Knife configuration fileCan contain references to Chef servers | filename | | 14 | Chef private keyCan be used to authenticate against Chef servers | path | | 15 | Configuration file for auto-login processCan contain username and password | filename | | 16 | Contains word: credential | path | | 17 | Contains word: password | path | | 18 | DBeaver SQL database manager configuration file | filename | | 19 | Day One journal fileNow it's getting creepy... | extension | | 20 | DigitalOcean doctl command-line client configuration fileContains DigitalOcean API key and other information | path | | 21 | Django configuration fileCan contain database credentials, cloud storage system credentials, and other secrets | filename | | 22 | Docker configuration fileCan contain credentials for public or private Docker registries | filename | | 23 | Environment configuration file | filename | | 24 | Facebook Oauth | contents | | 25 | FileZilla FTP configuration fileCan contain credentials for FTP servers | filename | | 26 | FileZilla FTP recent servers fileCan contain credentials for FTP servers | filename | | 27 | GNOME Keyring database file | extension | | 28 | Generic API Key | contents | | 29 | Generic Secret | contents | | 30 | Git configuration file | filename | | 31 | GitHub | contents | | 32 | GitHub Hub command-line client configuration fileCan contain GitHub API access token | path | | 33 | GnuCash database file | extension | | 34 | Google (GCP) Service-account | contents | | 35 | Google Oauth | contents | | 36 | Heroku API Key | contents | | 37 | Hexchat/XChat IRC client server list configuration file | path | | 38 | Irssi IRC client configuration file | path | | 39 | Java keystore file | extension | | 40 | Jenkins publish over SSH plugin file | filename | | 41 | KDE Wallet Manager database file | extension | | 42 | KeePass password manager database fileFeed it to Hashcat and see if you're lucky | extension | | 43 | Little Snitch firewall configuration fileContains traffic rules for applications | filename | | 44 | Log fileLog files can contain secret HTTP endpoints, session IDs, API keys and other goodies | extension | | 45 | Microsoft BitLocker Trusted Platform Module password file | extension | | 46 | Microsoft BitLocker recovery key file | extension | | 47 | Microsoft SQL database file | extension | | 48 | Microsoft SQL server compact database file | extension | | 49 | Mutt e-mail client configuration file | filename | | 50 | MySQL client command history file | filename | | 51 | NPM configuration fileCan contain credentials for NPM registries | filename | | 52 | Network traffic capture file | extension | | 53 | OmniAuth configuration fileThe OmniAuth configuration file can contain client application secrets | filename | | 54 | OpenVPN client configuration file | extension | | 55 | PGP private key block | contents | | 56 | PHP configuration file | filename | | 57 | Password Safe database file | extension | | 58 | Password in URL | contents | | 59 | Pidgin OTR private key | filename | | 60 | Pidgin chat client account configuration file | path | | 61 | PostgreSQL client command history file | filename | | 62 | PostgreSQL password file | filename | | 63 | Potential Jenkins credentials file | filename | | 64 | Potential Linux passwd fileContains system user information | path | | 65 | Potential Linux shadow fileContains hashed passwords for system users | path | | 66 | Potential MediaWiki configuration file | filename | | 67 | Potential Ruby On Rails database configuration fileCan contain database credentials | filename | | 68 | Potential cryptographic private key | extension | | 69 | Potential jrnl journal fileNow it's getting creepy... | filename | | 70 | Private SSH key_rsa | filename | | 71 | Private SSH key_dsa | filename | | 72 | Private SSH key_ed25519 | filename | | 73 | Private SSH key_ecdsa | filename | | 74 | RSA private key | contents | | 75 | Recon-ng web reconnaissance framework API key database | path | | 76 | Remote Desktop connection file | extension | | 77 | Robomongo MongoDB manager configuration fileCan contain credentials for MongoDB databases | filename | | 78 | Ruby IRB console history file | filename | | 79 | Ruby On Rails secret token configuration fileIf the Rails secret token is known, it can allow for remote code execution (http://www.exploit-db.com/exploits/27527/) | filename | | 80 | Rubygems credentials fileCan contain API key for a rubygems.org account | path | | 81 | S3cmd configuration file | filename | | 82 | SFTP connection configuration file | filename | | 83 | SQL dump file | extension | | 84 | SQLite database file | extension | | 85 | SSH (DSA) private key | contents | | 86 | SSH (EC) private key | contents | | 87 | SSH (OPENSSH) private key | contents | | 88 | SSH configuration file | path | | 89 | Sequel Pro MySQL database manager bookmark file | filename | | 90 | Shell command alias configuration fileShell configuration files can contain passwords, API keys, hostnames and other goodies | filename | | 91 | Shell command history file | filename | | 92 | Shell configuration file(.exports): Shell configuration files can contain passwords, API keys, hostnames and other goodies | filename | | 93 | Shell configuration file(.functions): Shell configuration files can contain passwords, API keys, hostnames and other goodies | filename | | 94 | Shell configuration file(.extra): Shell configuration files can contain passwords, API keys, hostnames and other goodies | filename | | 95 | Shell configuration file(bash, zsh, csh): Shell configuration files can contain passwords, API keys, hostnames and other goodies | filename | | 96 | Shell profile configuration file(profile): Shell configuration files can contain passwords, API keys, hostnames and other goodies | filename | | 97 | Slack Token | contents | | 98 | Slack Webhook | contents | | 99 | T command-line Twitter client configuration file | filename | | 100 | Terraform variable config fileCan contain credentials for terraform providers | filename | | 101 | Tugboat DigitalOcean management tool configuration | filename | | 102 | Tunnelblick VPN configuration file | extension | | 103 | Twilio API Key | contents | | 104 | Twitter Oauth | contents | | 105 | Ventrilo server configuration fileCan contain passwords | filename | | 106 | Windows BitLocker full volume encrypted data file | extension | | 107 | cPanel backup ProFTPd credentials fileContains usernames and password hashes for FTP accounts | filename | | 108 | git-credential-store helper credentials file | filename | | 109 | gitrob configuration file__ | filename |

2. Install

Before you begin, you'll need to have these

• Programming languages:

  • Node.js >= 10.x

  • NPM >= 4.0.0

• Skills:

  You'll need to know how to access the command line (aka, "Terminal") on your machine.

Open a Terminal and enter the following command:

# As a dependency in your Node.js app
npm i @data-leakage-protection/signatures --save-prod

3. Usage

Use @data-leakage-protection/signatures.signatures to find file extensions, names, and paths that commonly leak secrets.

const { signatures } = require('@data-leakage-protection/signatures')
// ⚠️ Note: the 'recursive-readdir' module is not bundled with
//    @data-leakage-protection/signatures. 'recursive-readdir' is referenced
//    only as an example.
const recursiveReaddir = require('recursive-readdir')

const potentialLeaks = recursiveReaddir('/path/to/local/repo')
  .then(files => files
    .map(file => signatures
    .map(signature => signature.match(file)))
  )
  .catch(err => err)

4. API

The @data-leakage-protection/signatures module provides a Signatures class, which validates @data-leakage-protection/signatures and converts regular expression strings to RE2 (whenever possible).

The @data-leakage-protection/signatures module's public API provides:

1. factory method: a convenience function that creates a signature object.
2. nullSignature: implements a default object literal with all signatures properties set to null.
3. Signature: a class that constructs a signature object.
4. signatures: an array of Signature instances.
5. toArray(data: {String|Array.<Object>}): generates an Array.<Signature> from a JSON string or object literal array.
6. validParts: a constants enum of valid Signature.prototype.part values.
7. validTypes: a constants enum of valid Signature.prototype.type values.

4.1. @data-leakage-protection/signatures.Signature

A class that constructs Signature objects.

const { Signature, validParts, validTypes } = require('@data-leakage-protection/signatures')

const signature = new Signature({
  caption: 'Potential cryptographic private key',
  description: '',
  part: validParts.EXTENSION,
  pattern: '.pem',
  type: validTypes.MATCH
})

4.2. @data-leakage-protection/signatures.Signature.prototype.match

Discover possible data leaks by matching a Signature pattern against file extensions, names, and paths.

const rsaTokenSignature = new Signature({
  'caption': 'Private SSH key',
  'description': '',
  'part': 'filename',
  'pattern': '^.*_rsa$',
  'type': 'regex'
})

const suspiciousFilePath = '/hmm/what/might/this/be/id_rsa'
rsaTokenSignature.match(suspiciousFilePath)
// => ['/hmm/what/might/this/be/id_rsa']

const fileThatIsJustBeingCoolBruh = 'file/that/is/just/being/cool/bruh'
rsaTokenSignature.match(suspiciousFilePath)
// => null

Review the source code for signature.

5. Accessing signatures with other tools and programming languages

You can access signatures.json without the @data-leakage-protection/signatures Node module. Select a tool or programming language below to view examples.

You can access data-loss rules using HTTPS. You can GET all signatures directly from Gitlab with cURL.

curl -X GET \
  'https://gitlab.com/data-leakage-protection/signatures/raw/master/signatures.json'

package main

import (
	"fmt"
	"net/http"
	"io/ioutil"
)

func main() {

	url := "https://gitlab.com/data-leakage-protection/signatures/raw/master/signatures.json"

	req, _ := http.NewRequest("GET", url, nil)

	req.Header.Add("Private-Token", "<your-personal-token>")
	req.Header.Add("cache-control", "no-cache")

	res, _ := http.DefaultClient.Do(req)

	defer res.Body.Close()
	body, _ := ioutil.ReadAll(res.Body)

	fmt.Println(res)
	fmt.Println(string(body))

}

OkHttpClient client = new OkHttpClient();

String signaturesJson = "https://gitlab.com/data-leakage-protection/signatures/raw/master/signatures.json";

Request request = new Request.Builder()
  .url(signaturesJson)
  .get()
  .addHeader("Accept", "*/*")
  .addHeader("Cache-Control", "no-cache")
  .addHeader("Host", "gitlab.com")
  .addHeader("accept-encoding", "gzip, deflate")
  .addHeader("Connection", "keep-alive")
  .addHeader("cache-control", "no-cache")
  .build();

Response response = client.newCall(request).execute();

const http = require('https')

const options = {
  method: 'GET',
  hostname: ['gitlab', 'com'],
  path: ['api', 'v4', 'projects'],
  headers: {
    'Private-Token': '<your-access-token>',
    'cache-control': 'no-cache'
  }
}

const req = http.request(options, res => {
  const chunks = []

  res.on('data', chunk => {
    chunks.push(chunk)
  })

  res.on('end', () => {
    var body = Buffer.concat(chunks)
    console.log(body.toString())
  })
})

req.end()

Python3

import http.client

conn = http.client.HTTPConnection("gitlab,com")

payload = ""

headers = {
  'Accept': "application/json",
  'cache-control': "no-cache"
}

conn.request("GET", "data-leakage-protection/signatures,raw,master,signatures.json", payload, headers)

res = conn.getresponse()
data = res.read()

print(data.decode("utf-8"))

Python2

import requests

url = "https://gitlab.com/data-leakage-protection/signatures/raw/master/signatures.json"

payload = ""
headers = {
  'Accept': "application/json",
  'cache-control': "no-cache"
}

response = requests.request("GET", url, data=payload, headers=headers)

print(response.text)

require 'uri'
require 'net/http'

url = URI("'https://gitlab.com/data-leakage-protection/signatures/raw/master/signatures.json")

http = Net::HTTP.new(url.host, url.port)

request = Net::HTTP::Get.new(url)
request["Private-Token"] = '<your-personal-token>'
request["cache-control"] = 'no-cache'

response = http.request(request)
puts response.read_body

6. Maintainers

@gregswindle

The Maintainer Guide has useful information for Maintainers and Trusted Committers.

7. Contributions

We gratefully accept Merge Requests! Here's what you need to know to get started.

Before submitting a Merge Request, please read our:

• Code of Conduct
• Contributing Aggreement
• Developer Guide
• Maintainer/Trusted Committer Guide

Thanks goes to our awesome contributors (emoji key):

This project follows the all-contributors specification. Contributions of any kind welcome!

7.1. Adding a Signature

Before adding a new Signature, please review all current definitions: the Signature might already exist.

If the Signature does not exist, please be sure to add your Signature with the following properties:

1. caption: A succinct summary for the Signature. Think of caption as a well-written email subject.

2. description: Provide more details about the Signature if necessary. description is especially useful for differentiating similar Signatures.

3. hash: A hexidecimal SHA256 representation of a Signature (with ordered properties).

4. name: The Signature's caption, converted to kebab-case.

5. part: An enumeration that defines what the Signature is evaluating. Valid values are:

   • contents: The string(s) within a file.
   • extension: A file extension (which defines the Content-Type or mime-type).
   • filename: The unique name of the file.
   • path: The directory path relative to the repo and without the filename.

6. pattern: The string or regular expression to look for.

7. type: An enumeration that defines how to evaluate for secrets. Valid values are:

   • match: A strict string equivalency evaluation.
   • regex: A regular expression "search" or "test".

7.2. Editing a Signature

Edits are welcome! Just be sure to unit test.

7.3. Removing a Signature

Please provide a testable justification for any Signature removal.

8. License

© 2019 Greg Swindle.

View detailed legal NOTICEs .

9. References and Attributions

[^1]: What is Data Leakage? Defined, Explained, and Explored | Forcepoint. (2019) Retrieved January 27, 2019, from https://www.forcepoint.com/cyber-edu/data-leakage