@repo/deduper

AST-based code duplication detection for the crossplatform.ai monorepo. Detects duplicate code patterns across packages using TypeScript Abstract Syntax Tree analysis.

Overview

The deduper package scans your monorepo for duplicate code patterns such as types, functions, classes, components, and more. It uses TypeScript's AST parser to perform semantic analysis, detecting duplicates based on code structure rather than just text matching.

Installation

This package is already available in the monorepo. Run it using:

pnpm check-duplicates

Available Detectors

The package includes 13 specialized detectors:

| Detector | Description | |----------|-------------| | types | TypeScript type aliases and interfaces | | functions | Function declarations and expressions | | constants | Constant variable declarations | | classes | ES6 class declarations | | abstract-classes | Abstract class declarations | | components | React/JSX components | | enums | TypeScript enum declarations | | hooks | React hooks (functions starting with use) | | type-guards | TypeScript type guard functions | | zod-schemas | Zod schema definitions | | routes | Route definitions (API/web routes) | | context-providers | React context provider components | | error-classes | Custom error class declarations |

Usage

Run All Detectors

By default, running without arguments checks for all duplicate patterns:

pnpm check-duplicates

Or explicitly:

pnpm check-duplicates --all

Run Specific Detector

Check for duplicates of a specific type:

# Check for duplicate type definitions
pnpm check-duplicates types

# Check for duplicate functions
pnpm check-duplicates functions

# Check for duplicate React components
pnpm check-duplicates components

# Check for duplicate React hooks
pnpm check-duplicates hooks

Get Help

pnpm check-duplicates --help

Output Format

When duplicates are found, the output shows:

Reported Duplicates:

  Group 1
    Name: User
    Kind: interface
    Files (2):
      packages/auth/src/types.ts (@repo/auth)
      packages/api/src/types.ts (@repo/api)

✗ Found 1 duplicate group(s) to address

When no duplicates are found:

✓ No duplicate code to report

Config Suggestions

Use --suggest to get copy-paste config snippets:

pnpm check-duplicates types --suggest

This adds a ready-to-use JSON config for each duplicate:

  To allow this duplicate, add to deduper.config.js:
  {
    "name": "User",
    "kind": "interface",
    "files": [
      "packages/auth/src/types.ts",
      "packages/api/src/types.ts"
    ],
    "reason": "TODO: explain why this duplication is intentional"
  }

Exit Codes

• 0: No duplicates found
• 1: Duplicates found or error occurred

This makes the tool useful in CI/CD pipelines:

# Fail the build if duplicates are detected
pnpm check-duplicates || exit 1

Common Use Cases

Pre-commit Hook

Add to your pre-commit workflow to catch duplicates before they're committed:

{
  "pre-commit": "pnpm check-duplicates"
}

CI/CD Pipeline

Run in your CI pipeline to enforce code quality:

- name: Check for code duplicates
  run: pnpm check-duplicates

Code Review

Use specific detectors during code review to focus on particular patterns:

# When reviewing API changes
pnpm check-duplicates types

# When reviewing component changes
pnpm check-duplicates components

# When reviewing error handling
pnpm check-duplicates error-classes

Refactoring Analysis

Identify candidates for consolidation:

# Find all duplicate utilities
pnpm check-duplicates functions

# Find duplicate constants that could be centralized
pnpm check-duplicates constants

Architecture

Plugin System

Each detector implements the DetectorPlugin interface:

interface DetectorPlugin {
  name: string;
  detect(files: string[], config: DetectionConfig): Promise<DuplicateGroup[]>;
}

Detection Process

1. File Scanning: Glob all .ts and .tsx files (excluding node_modules, dist, .turbo)
2. AST Parsing: Parse files using TypeScript compiler API
3. Pattern Extraction: Each detector extracts its specific pattern type
4. Signature Normalization: Generate canonical signatures for comparison
5. Duplicate Grouping: Group items with identical names and signatures
6. Package Resolution: Extract package names from file paths
7. Result Formatting: Format and display duplicate groups

Adding New Detectors

To add a new detector:

1. Create a new file in src/detectors/
2. Implement the DetectorPlugin interface
3. Export from src/detectors/index.ts
4. Register in src/detect.ts DETECTORS map

Configuration

Deduper looks for configuration in deduper.config.ts or deduper.config.js in your project root.

TypeScript Configuration

Create deduper.config.ts:

import type { DeduperConfig } from '@repo/deduper';

export default {
  allowRules: [
    {
      name: 'AppType',
      kind: 'type',
      files: [
        'apps/api-origin/src/index.ts',
        'apps/api-edge/src/index.ts',
      ],
      reason: 'Each API exports its own Hono instance type for RPC clients',
    },
  ],
} satisfies DeduperConfig;

JavaScript Configuration

Create deduper.config.js:

/** @type {import('@repo/deduper').DeduperConfig} */
export default {
  allowRules: [
    {
      name: 'AppType',
      kind: 'type',
      files: [
        'apps/api-origin/src/index.ts',
        'apps/api-edge/src/index.ts',
      ],
      reason: 'Each API exports its own Hono instance type for RPC clients',
    },
  ],
};

Programmatic Usage

For custom detection logic:

import { detect } from '@repo/deduper';

const results = await detect('types', {
  ignore: ['**/*.test.ts'],
  threshold: 2
});

Development

Running Tests

pnpm test

Type Checking

pnpm check-types

Linting

pnpm lint

Testing

This package uses a hybrid testing pattern:

• Unit tests (src/**/*.test.ts): Co-located with source code, testing individual detectors and utilities
• Integration tests (src/__tests__/**/*.test.ts): Separated tests for end-to-end detection pipeline scenarios

Run all tests:

pnpm test

The Vitest configuration explicitly includes both patterns to ensure all 262+ tests are discovered.

Technical Details

• AST Analysis: Uses TypeScript Compiler API for accurate parsing
• Monorepo-Aware: Automatically detects package boundaries
• Performance: Processes files in parallel where possible
• Error Handling: Gracefully skips unparseable files
• Output Formatting: Uses picocolors for readable terminal output

Limitations

• Only scans .ts and .tsx files
• Requires valid TypeScript syntax
• Signature comparison is structure-based (may not catch semantically identical but syntactically different code)
• Does not analyze runtime behavior or values

See Configuration Guide for detailed documentation.