🔍 Find Duplicate JS

A powerful and intelligent tool to detect duplicate and similar code in JavaScript and TypeScript projects. Find Duplicate JS helps you maintain cleaner codebases by automatically identifying redundant functions and code patterns across your project.

📋 Table of Contents

• Why Use Find Duplicate JS?
• Features
• Installation
• Usage
  • CLI Mode
  • Web UI Mode
• How It Works
• Configuration Options
• Examples
• API
• Contributing
• Links
• License

🎯 Why Use Find Duplicate JS?

Duplicate code is a common problem in software development that leads to:

• Maintenance Headaches: Fixing bugs requires updating code in multiple places
• Increased File Size: Unnecessary code bloat
• Inconsistencies: Changes in one place might not be reflected elsewhere
• Technical Debt: Harder to refactor and improve code quality

Find Duplicate JS helps you identify these issues automatically, saving time and improving code quality.

✨ Features

• 🎯 Smart Function Detection: Recognizes multiple function types

  • Arrow functions (const func = () => {})
  • Function declarations (function func() {})
  • Class methods and object methods
  • Async functions
  • TypeScript functions with type annotations
  • Generic functions (<T>)

• 🧠 Intelligent Code Analysis:

  • Normalizes code to ignore irrelevant differences (whitespace, comments, variable names)
  • Automatically removes TypeScript type annotations for semantic comparison
  • Uses Levenshtein distance algorithm for accurate similarity scoring
  • Configurable similarity threshold (default 70%)

• 🎨 Two Usage Modes:

  • CLI Mode: Quick terminal-based analysis with detailed text output
  • Web UI Mode: Beautiful, interactive web interface with visual comparisons

• ⚡ Performance:

  • Recursively scans entire project directories
  • Automatically skips node_modules, .git, dist, and build folders
  • Handles .js, .jsx, .ts, and .tsx files

• 🔧 Zero Configuration: Works out of the box with sensible defaults

📦 Installation

Global Installation (Recommended)

npm install -g find-duplicate-js

Local Installation

npm install --save-dev find-duplicate-js

Then add to your package.json scripts:

{
  "scripts": {
    "find-duplicate": "find-duplicate",
    "find-duplicate:ui": "find-duplicate --ui"
  }
}

Or use the built-in npm scripts:

npm start        # Run CLI mode
npm run ui       # Run UI mode
npm test         # Run tests

🚀 Usage

CLI Mode

Run a quick analysis from the command line and get results in your terminal:

# Analyze current directory with default threshold (70%)
find-duplicate

# Analyze specific directory
find-duplicate ./src

# Custom similarity threshold (80%)
find-duplicate ./src 80

# Analyze entire project
find-duplicate . 75

CLI Output Example:

🚀 Searching for duplicate code in: ./src
📏 Similarity threshold: 70%

🔍 Scanning 15 JavaScript/TypeScript files...

📊 Found 42 functions total

⚠️  Found 3 pairs of similar functions:

═══════════════════════════════════════════════════════════════════════════════════════

📋 Match #1 - Similarity: 95.5%

   File 1: src/utils/math.js
   Function: calculateTotal()
   Code: const sum = items.reduce((acc, item) => acc + item.price, 0)...

   File 2: src/components/cart.js
   Function: getTotalPrice()
   Code: const total = products.reduce((acc, prod) => acc + prod.pri...

─────────────────────────────────────────────────────────────────────────────────────

💡 Summary: Found 3 duplicate function pairs

Web UI Mode

Launch an interactive web interface for a better visual experience using the --ui flag:

# Start web UI server (opens browser automatically)
find-duplicate --ui

# Analyze specific directory
find-duplicate --ui ./src

# Custom threshold
find-duplicate --ui ./src 80

Alternative: You can also use the dedicated UI command (backwards compatibility):

find-duplicate-ui ./src 80

The web interface will:

1. Start a local server on http://localhost:2712
2. Automatically open your default browser
3. Display an interactive dashboard with:
   • Project statistics
   • Color-coded duplicate pairs
   • Side-by-side code comparison
   • Similarity percentages
   • File paths and function names

Features in Web UI:

• 📊 Statistics Dashboard: Overview of scanned files, functions found, and duplicates
• 🎨 Beautiful Design: Modern, responsive interface
• 🔄 Live Refresh: Re-analyze your code with a single click
• 📱 Mobile Friendly: Works on all devices
• 🎯 Easy Navigation: Jump directly to problematic code

🔧 How It Works

1. File Discovery

Recursively scans your project directory and finds all .js, .jsx, .ts, and .tsx files, while intelligently skipping:

• node_modules
• .git
• dist
• build

2. Function Extraction

Uses sophisticated regex patterns to identify and extract:

• Arrow functions with const, let, or var
• Traditional function declarations
• Class and object methods
• Async functions
• TypeScript functions with type annotations
• Generic functions with type parameters

3. Code Normalization

Before comparison, the code is normalized to focus on logic rather than style:

• Removes all whitespace and line breaks
• Strips comments (single-line and multi-line)
• Removes TypeScript type annotations and generic parameters
• Replaces variable names with generic placeholders
• Replaces string literals with generic strings
• Removes template literals

Example:

// Original JavaScript Code
function calculateSum(num1, num2) {
  // Calculate sum of two numbers
  const result = num1 + num2;
  return result;
}

// Original TypeScript Code
function calculateSum(num1: number, num2: number): number {
  // Calculate sum of two numbers
  const result: number = num1 + num2;
  return result;
}

// Both Normalized to
V(){V=V+V;V;}

This allows the tool to recognize that TypeScript and JavaScript versions of the same function are duplicates!

4. Similarity Calculation

Uses the Levenshtein Distance algorithm to calculate how similar two functions are:

• Measures the minimum number of edits needed to transform one string into another
• Converts to a percentage (0-100%)
• Compares against your configured threshold

5. Results Presentation

Presents findings in an easy-to-understand format (CLI or Web UI) showing:

• Which functions are similar
• Their similarity percentage
• File locations
• Code previews

⚙️ Configuration Options

Command Line Arguments

find-duplicate [directory] [threshold]
find-duplicate-ui [directory] [threshold]

Parameters:

• directory (optional): Path to analyze. Default: current directory (.)
• threshold (optional): Similarity percentage (0-100). Default: 70

Examples:

# Very strict (only near-identical code)
find-duplicate ./src 95

# Moderate (recommended)
find-duplicate ./src 70

# Lenient (catches more potential duplicates)
find-duplicate ./src 50

Threshold Guidelines:

• 90-100%: Nearly identical functions (different variable names only)
• 70-89%: Very similar logic with minor variations
• 50-69%: Similar patterns but with notable differences
• Below 50%: May produce many false positives

🎯 TypeScript Support

Find Duplicate JS now fully supports TypeScript! The tool intelligently handles TypeScript-specific syntax:

Supported TypeScript Features

• ✅ Type Annotations: Function parameters, return types, and variable types
• ✅ Interfaces and Type Aliases: Automatically filtered during normalization
• ✅ Generics: Generic type parameters (<T>, <T extends U>)
• ✅ Type Assertions: as keyword syntax
• ✅ Access Modifiers: public, private, protected
• ✅ Optional Parameters: param?: type
• ✅ Union and Intersection Types: type1 | type2, type1 & type2

How It Works

The tool normalizes TypeScript code by removing all type information, allowing it to detect duplicates regardless of whether they're written in JavaScript or TypeScript:

// TypeScript version
function fetchUser(id: string): Promise<User> {
  return api.get<User>(`/users/${id}`);
}

// JavaScript version  
function getUserData(userId) {
  return api.get(`/users/${userId}`);
}

// Both will be detected as similar! ✅

Mixed Projects

Works seamlessly in projects that contain both JavaScript and TypeScript:

# Analyze a mixed JS/TS project
find-duplicate ./src

# Files analyzed:
# ✅ .js files
# ✅ .jsx files  
# ✅ .ts files
# ✅ .tsx files

📚 Examples

Example 1: Finding Exact Duplicates

File 1: auth.js

function validateUser(username, password) {
  if (!username || !password) {
    return false;
  }
  return true;
}

File 2: login.js

function checkCredentials(user, pass) {
  if (!user || !pass) {
    return false;
  }
  return true;
}

Result: 100% similarity - Same logic, different names

Example 2: Similar Functions

File 1: cart.js

const calculateTotal = (items) => {
  return items.reduce((sum, item) => sum + item.price, 0);
}

File 2: checkout.js

const getTotalPrice = (products) => {
  let total = 0;
  products.forEach(product => {
    total += product.price;
  });
  return total;
}

Result: ~75% similarity - Same logic, different implementation

🔌 API

You can also use Find Duplicate JS programmatically in your Node.js projects:

import { findDuplicates, findJsFiles } from 'find-duplicate-js';

// Find all JavaScript files
const files = findJsFiles('./src');
console.log(`Found ${files.length} files`);

// Find duplicates with 70% threshold
const result = findDuplicates('./src', 70);

console.log(`Total functions: ${result.totalFunctions}`);
console.log(`Duplicate pairs: ${result.duplicates.length}`);

// Iterate through duplicates
result.duplicates.forEach((dup, index) => {
  console.log(`\nMatch #${index + 1}:`);
  console.log(`Similarity: ${dup.similarity}%`);
  console.log(`Function 1: ${dup.func1.name} in ${dup.func1.filePath}`);
  console.log(`Function 2: ${dup.func2.name} in ${dup.func2.filePath}`);
});

Available Functions

findJsFiles(directory)

Returns an array of all .js and .jsx file paths in the directory.

findDuplicates(directory, threshold = 70)

Analyzes the directory and returns:

{
  duplicates: [
    {
      func1: { name, body, originalBody, filePath, startIndex },
      func2: { name, body, originalBody, filePath, startIndex },
      similarity: "95.50"
    }
  ],
  totalFunctions: 42
}

calculateSimilarity(code1, code2)

Calculates similarity percentage between two code strings.

normalizeCode(code)

Normalizes JavaScript code for comparison.

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.

Development Setup

# Clone the repository
git clone https://github.com/benshabbat/find-duplicate-js.git

# Install dependencies
cd find-duplicate-js
npm install

# Run locally
node find-duplicates.js ./src
node find-duplicates-ui.js ./src

🔗 Links

• npm Package
• GitHub Repository
• Issues & Bug Reports

📄 License

MIT © benshabbat

Made with ❤️ to help developers write better code

If you find this tool helpful, please consider giving it a ⭐ on GitHub!