Introduction

This tool is designed to efficiently capture network flows and compute a range of statistics and features for each flow. These features include packet count, byte count, inter-arrival time (IAT), protocol type, TCP flags, and packet length.

Requirements

All of these prerequisites are usually installed togehter with nodeJS as the suggested required tools, bu we advice going over the following steps to make sure everything needed is present:

On Linux:

• libpcap-dev: Library crucial for network traffic capture

  sudo apt-get install libpcap-dev

• Network Permissions: ensure these privileges are enabled in your program execution environment
  • NET_ADMIN: For manipulating network interfaces.
  • NET_RAW: For raw socket access, essential for flow capture.

  sudo setcap 'cap_net_admin,cap_net_raw=eip' $(which node)

  However, this command only sets these permission once while you actually need to programmatically incorporate the cap_net_admin and cap_net_raw capabilities into your deployment (this will vary depending on your environment and should be always fairly easy to set up; it is not explained here due to the impossibility to cover the countless number of different setup possible). As an example, below is the configuration for a contenerized environment setup:

  docker run --cap-add=NET_ADMIN --cap-add=NET_RAW -d your_image_name

  or with docket-compose with:

  version: '3.8'
  services:
    your_service:
      image: your_image_name
      ***cap_add:**
        **- NET_ADMIN**
        **- NET_RAW***

On Windows:

In a terminal with Administrative Privileges, install:

• Python ^3.x: Releases for Windows
• Visual Studio Build Tools This is used to compile binary specific dependencies of pcap

  curl -o vs_buildtools.exe https://aka.ms/vs/17/release/vs_buildtools.exe
  .\vs_buildtools.exe --add Microsoft.VisualStudio.Workload.VCTools --includeRecommended --add Microsoft.VisualStudio.Component.VC.Tools.x86.x64 --add Microsoft.VisualStudio.Component.Windows10SDK.20348

  Adjust version of Tools (debending of your OS, you might use x32 instead of x64, etc …) and WindowsSDK(switch to windows11SDK instead of 10 if needed) accordingly depending on your setup. if you are running this on a Windows server you can use the —quiet or —passive flags to automatically do any required interaction without needing a GUI. Remember the installation might take a while and usually requires a restart of the machine after for the changes to be applied. These tools are used to compile the node pcap binding for windows.
• Npcap

  curl -o npcap-1.78.exe https://[npcap.com/dist/npcap-1.78.exe](https://npcap.com/dist/npcap-1.78.exe)
  npcap-1.78.exe

  On the npcap GUI installer select "Install Npcap in WinPcap API-compatible Mode" option.
• Network Permissions: ensure these privileges are enabled in your program execution environment
  • NET_ADMIN: For manipulating network interfaces.
  • NET_RAW: For raw socket access, essential for flow capture. Make sure to run your application with a user that has the right Network Administrative permissions and ensure your firewall settings allow for packet capturing.

Installation

Instructions for installing the application and its dependencies.

npm i --save @innerworks/flow-extractor-node

Usage

Integrating the Network Flow Extractor in Node.js:

1. Import the Module: At the beginning of your service file, import the Network Flow Extractor: jsx import NetworkFlowExtractor from "@innerworks/flow-extractor-node";

2. Initialize Once:

   To minimize performance overhead, instantiate the Network Flow Extractor only once where needed. Ideally, do this as a class field within a service constructor or similar scope where it can be easily accessed later:

   const iwFlowExtractor = new NetworkFlowExtractor();

3. Usage During User Login: When processing a user login request, retrieve the specific network flow features for that user using the flow extractor instance initialised earlier passin the srcIp and srcport for the current request: jsx const flowFeatures = await this.flowExtractor.getFlowFeaturesWhenReady(req.socket.remoteAddress, req.socket.remotePort);

   As outlined above, ensure the request object, or just the values **`req.socket.remoteAddress`** and **`req.socket.remotePort`** , is passed from its initial access point (usually the endpoint controller) through to the flow extractor function call.
   
   When triggered, **`getFlowFeaturesWhenReady()`** immediately retrieves a flow from the cache for that id if available; otherwise, it awaits a 'flowDataUpdated' event signaling a new flow's arrival. This asynchronous function times out after a default of 1.5 seconds if no flow is found, timeout is adjustable via the function third optional parameter.

Optional Parameters Tuning

• Constructor

  new NetworkFlowExtractor(
  	networkInterface?: string,
  	flowLengthThreshold: number = 3,
  	flowsCacheCleanupInterval: number = 60000;
  )

  • networkInterface?: string: The flow extractor auto-detects by default the network interface used by your backend. If this does not work as expected, you can manually specify the interface to listen to in this parameter.
  • flowLengthThreshold: number = 3: Defines the minimum packet count for a valid flow. It's advised to keep this default value, as a valid flow usually comprises at least 3 packets. Increase this threshold only if you're certain that your endpointd requests will consistently involve more packets, which can enhance flow feature accuracy.
  • flowsCacheCleanupInterval: number = 60000: Determines the interval (in milliseconds) for purging flows older than 1 minute from memory.

Testing Network Flow Extractor Locally

To test the flow extractor locally using your laptops's network interface follow below steps

• generate openssl sertificates server.cert and server.key:

  • openssl genpkey -algorithm RSA -out server.key -pkeyopt rsa_keygen_bits:2048
  • openssl req -new -x509 -key server.key -out server.cert -days 365

• make your own copy of index.js from scripts/index.js.example

  • adjust network interface
  • adjust correct flow id

• run sudo node index.js

Offline Pcap file processing

• this is needed for debugging & feature engineering purposes
• in the moment, we need this to analyze feature importance of our extractor on https://www.unb.ca/cic/datasets/iotdataset-2023.html dataset

run the example Pcap=>CSV feature extraction

• sudo node scripts/pcap2csv.js

output: flow_features.csv

trouble shooting

there could be issue with pcap package on Mac M1

• nvm install 20.5.0
• nvm use 20.5.0
• npm rebuild pcap --update-binary