[TDD v1]: Sensitive column detection Engine

Description:

This is a piece of software that detects sensitive columns in a dataset Through a combination of heuristic name searches and data classifications. A correlation approach through a multi view fusion function translates to a column being sensitive or not.

Problem Defintion & Constraints:

A Column in this design is described sensitive if it contains data that can introduce bias into a models output or breach complicance and regulatory standards when exposed and can pose a threat of any-form legally defined to a Human.

-- Design Constraints:

1. Scope of Search is bounded by dataset columns. No cross referencing
2. Human Factor Dominance. The Algorithm is primarily designed to catch Human Centric Sensitive Columns.

Usage:

1. npm install @yankey/sensitivecolumndetection
2. Import DetectionEngine from 'DetectionEngine'
3. result= DetectionEngine(<"CsVFileDataPath">)

Implementation Overview

Given a dataset the algorithm peforms a heursitic search on column names. Further column data sampling is applied to column data and a regular expression match is performed. Regular expressions are vectorized into features and using KNN style classification, we classify based on features.

Sytantic Defintion of a Sensitive Column

Column Name Heuristic Guide:

 [
  'name', 'fullname', 'first_name', 'last_name', 'surname',
  'email', 'mail','age','sex',
  'phone', 'mobile', 'tel', 'telephone',
  'address', 'addr', 'location','latitude,'longitude',
  'ssn', 'nin', 'national_id', 'passport', 'id_number',
  'credit_card', 'card_number', 'cc_number', 'iban', 'account_number',
  'dob', 'birthdate', 'date_of_birth',
  'salary', 'income', 'tax',
  'medical', 'diagnosis', 'disease', 'condition',
  'religion', 'ethnicity', 'race',
  'sexual_orientation', 'orientation',
  'political', 'party','password'
 ]

Statically defined syntax:

1. Email ----> Regex[Email]
2. PhoneNumber ----> Regex[Phonenumber]
3. Credit/Debit_Card_Number ----> Regex[credit/debitcard]
4. DateOfBirth ----> Regex[DateofBirth]
5. BankAccount/IBAN ----> Regex[BankAccount]
6. Gps_Coordiantes ----> Regex[LatLongGPs]
7. Gps_CoordiantesFul ----> Regex[FullGPs]

Dynamically defined Syntax:

1. Identity --> [Anything like an ID]
2. Address --> Regex[Address&Postcodes]

Process Flow:

Describing the process flow for the matching, classification and Detection Phases.

graph TD
    subgraph ColumnNameSearch [ColNameSearch]
    direction LR
    H*Search --> score["{0, 0.5}"]
    end

    subgraph DataClassiifer [DataSearch]
    direction TB
    DataX --> x["RegEX Matching Static"]
    DataX -->|IF Static fails| y["RegEX Matching Dynamic"]
    y --> s["{0 - 0.5}"]
    x --> s
    end

    Fusion --> dep["$$2.max(s1,s2)$$"]
    DataClassiifer --> Fusion 
    ColumnNameSearch --> Fusion
    dep --> Result["{0 - 1}"]

Detection Engine Design:

Heuristic COL Name Search Design:

Semantic Directed Acyclic Word Graph (S-DAWG):

This is a custom designed algorithm that implements Directed Acyclic Word Graph style datastructure to infer semantic meaning from a given column name against a heuristics search guide. The Datastructure stores a compressed database of sensitive column names and has algorthms build and search build the Directed Acyclic word Graph and search against to infer semantic meaning respectively

Column Data Search Using Regular Expressions.

This approach uses regular expressions to capture column data signature and classifies with a score indiciating percentage rating of likelhood of the data been a sensitive data. Sampled data in a column is matched against static and dynamic regular expressions. staitcs matches are assigned a score {0,0.5} and dynamic matches classify how of the data pattern match the regualar expression. This is computed percentage match and scaled down to 0.5

Fusion Technique

To correctly infer that a column is a sensitive column, we implement a multi view fusion technique to fuse the column name heuristic score and the data regular expression match score to give a more informed predictabiltiy score as described in the logic flow . The fusion regualarizes both scores dependence. This creates a more grounded truth for prediction. Each score carries Equal weight in the event of unpropery named colummn names hence the data detected as senstive has enough weight ot classify the column as sensitive.