IoC extractor

This is a fork of https://github.com/ninoseki/ioc-extractor with the following changes

• sorting is removed
• IPv4 and IPv6 address also accepts optional CIDR mask (can be disable by setting enableOptionalMask: false)
• new only option for extractIOC to only extract certain types of IoCs

I'm too lazy to make this configurable and send PR, so here we are.

Contents from original readme:

IoC extractor is an npm package for extracting common IoC (Indicator of Compromise) from a block of text.

Note: the package is highly influenced by cacador.

Installation

npm install -g ioc-extractor
# or if you want to use ioc-extractor as a library in your JS/TS project
npm install ioc-extractor

Usage

As a CLI

$ ioc-extractor --help
Usage: ioc-extractor [options]

Options:
  -s2, --stix2          output in STIX2 format (default: false)
  -t, --threads         use threads (default: false)
  --disable-idn         disable IDN extraction (default: false)
  --disable-strict-tld  disable strict TLD validation (default: false)
  --disable-refang      disable refang (default: false)
  -h, --help            display help for command

$ echo "1.1.1.1 8.8.8.8 example.com" | ioc-extractor
{"asns":[],"btcs":[],"cves":[],"domains":["example.com"],"emails":[],"eths":[],"gaPubIDs":[],"gaTrackIDs":[],"ipv4s":["1.1.1.1","8.8.8.8"],"ipv6s":[],"macAddresses":[],"md5s":[],"sha1s":[],"sha256s":[],"sha512s":[],"ssdeeps":[],"urls":[],"xmrs":[]}

# Using with jq
$ echo "1.1.1.1 8.8.8.8 example.com " | ioc-extractor | jq
{
  "asns": [],
  "btcs": [],
  "cves": [],
  "domains": [
    "example.com"
  ],
  "emails": [],
  "eths": [],
  "gaPubIDs": [],
  "gaTrackIDs": [],
  "ipv4s": [
    "1.1.1.1",
    "8.8.8.8"
  ],
  "ipv6s": [],
  "macAddresses": [],
  "md5s": [],
  "sha1s": [],
  "sha256s": [],
  "sha512s": [],
  "ssdeeps": [],
  "urls": [],
  "xmrs": []
}

# Using -t(--threads) option makes sense if you want to process a big chunk of text
$ cat big.txt | ioc-extractor -t

As a library

import { extractIOC } from "ioc-extractor";

const input = '1.1.1[.]1 google(.)com f6f8179ac71eaabff12b8c024342109b';
const ioc = extractIOC(input);
console.log(ioc.md5s);
// => ['f6f8179ac71eaabff12b8c024342109b']
console.log(ioc.ipv4s);
// => ['1.1.1.1']
console.log(ioc.domains);
// => ['google.com']

console.log(JSON.stringify(ioc))
// => {"asns":[],"btcs":[],"cves":[],"domains":["google.com"],"emails":[],"gaPubIDs":[],"gaTrackIDs":[],"ipv4s":["1.1.1.1"],"ipv6s":[],"macAddresses":[],"md5s":["f6f8179ac71eaabff12b8c024342109b"],"sha1s":[],"sha256s":[],"sha512s":[],"ssdeeps":[],"urls":[],"xmrs":[]}

If you want to extract a specific type of IOC, you can use extractXXX function.

import { refang, extractDomains, extractIPv4s, extractMD5s } from "ioc-extractor";

const input = "1.1.1[.]1 google(.)com f6f8179ac71eaabff12b8c024342109b";
const refanged = refang(input);
// => 1.1.1.1 google.com f6f8179ac71eaabff12b8c024342109b

const ipv4s = extractIPv4s(refanged);
// => ['1.1.1.1']

const domains = extractDomains(refanged);
// => ['google.com']

const md5s = extractMD5s(refanged);
// => ['f6f8179ac71eaabff12b8c024342109b']

See docs for more details.

Details

This package supports the following IOCs:

• Hashes: md5, sha1, sha256, sha512, ssdeep
• Networks: domain, email, ipv4, ipv6, url, asn
• Hardwares: mac_address
• Utilities: cve(CVE ID)
• Cryptocurrencies: btc (BTC address), eth (ETH address), xmr (XMR address)
• Trackers: gaTrackID (Google Analytics tracking ID), gaPubID (Google Adsense Publisher ID)

For Networks IOCs, the following defang/refang techniques are supported:

| Techniques | Defanged | Refanged | |------------------|----------------------------------------|---------------------------------| | . => . | 1.1.1 . 1 | 1.1.1.1 | | [.] => . | 1.1.1[.]1 | 1.1.1.1 | | (.) => . | 1.1.1(.)1 | 1.1.1.1 | | {.} => . | 1.1.1{.}1 | 1.1.1.1 | | \. => . | example\.com | example.com | | [/] => / | http://example.com[/]path | http://example.com/path | | [:] => : | http[:]//example.com | http://example.com | | [://] => :// | http[://]example.com | http://example.com | | hxxp => http | hxxps://google.com | https://google.com | | [at] => @ | test[at]example.com | [email protected] | | [@] => @ | test[@]example.com | [email protected] | | (@) => @ | test(@)example.com | [email protected] | | {@} => @ | test{@}example.com | [email protected] | | [dot] => . | test@example[dot]com | [email protected] | | (dot) => . | test@example(dot)com | [email protected] | | {dot} => . | test@example{dot}com | [email protected] | | Partial | 1.1.1[.1 | 1.1.1.1 | | Any combination | hxxps[:]//test\.example[.)com[/]path | https://test.example.com/path |

Known limitations

A domain with an IDN TLD (e.g. みんな) is not supported. Please convert an input into Punycode beforehand. Then it will work.

# OK
xn--p8j9a0d9c9a.xn--q9jyb4c
はじめよう.com

# NG
はじめよう.みんな
example.みんな

SITX2 support

This package provides a partial support of the STIX2 format.

$ echo "1.1.1.1 8.8.8.8 example.com" | ioc-extractor --sitx2 | jq
{
  "spec_version": "2.0",
  "type": "bundle",
  "objects": [
    {
      "type": "indicator",
      "id": "indicator--e0dc210b-fc7e-4dcc-8a5e-a220b32bd070",
      "created": "2019-09-07T12:40:13.104Z",
      "modified": "2019-09-07T12:40:13.104Z",
      "labels": [
        "malicious-activity"
      ],
      "pattern": "[ipv4-addr:value = '1.1.1.1']",
      "valid_from": "2019-09-07T12:40:13.104Z"
    },
    {
      "type": "indicator",
      "id": "indicator--f77971ea-37de-4ddb-a147-613fec3401b3",
      "created": "2019-09-07T12:40:13.104Z",
      "modified": "2019-09-07T12:40:13.104Z",
      "labels": [
        "malicious-activity"
      ],
      "pattern": "[domain-name:value = 'google.com']",
      "valid_from": "2019-09-07T12:40:13.104Z"
    },
    {
      "type": "indicator",
      "id": "indicator--0461539a-dc75-4cd1-ab74-24d964c8609c",
      "created": "2019-09-07T12:40:13.104Z",
      "modified": "2019-09-07T12:40:13.104Z",
      "labels": [
        "malicious-activity"
      ],
      "pattern": "[file:hashes.md5 = 'f6f8179ac71eaabff12b8c024342109b']",
      "valid_from": "2019-09-07T12:40:13.104Z"
    }
  ]
}

The following indicator patterns are supported.

• ipv4-addr
• ipv6-addr
• domain-name
• url
• email-addr
• file:hashes.{md5|sha1|sha256|sha512}

Alternatives

• Python:
  • InQuest/python-iocextract
  • cmu-sei/cyobstract
  • fhightower/ioc-finder
• Golang:
  • sroberts/cacador