🔍 Searchify-DB

A powerful, production-ready Node.js (TypeScript) search engine library designed for product databases. Features inverted index, stemming, synonyms, DB-agnostic indexing, and smart NLP query understanding.

✨ Features

🚀 Core Features

• ⚡ Inverted Index (Super Fast Search)

  • Builds a token → product ID map for instant keyword search
  • Handles 100k–1M+ products efficiently
  • Lightning-fast retrieval

• 🔤 Stemming

  • Understands word variations automatically
  • pant, pants, panting → pant
  • shoe, shoes → shoe
  • Ensures better matches even with different word forms

• 🔗 Synonym Matching

  • Manual Synonyms: Define custom synonym mappings
  • Auto-Generated Synonyms: Zero manual work - automatically detects words that appear together frequently
  • Example: "kurti" and "tunic" appear often together → automatically linked as synonyms

• 🗄️ Works With Any Database

  • MySQL
  • PostgreSQL
  • MongoDB
  • SQLite
  • REST API

• 🧠 Rule-Based NLP Layer (Coming Soon)

  • Understands numeric filters: "shoes under 300", "bags below 500"
  • Range filters: "phone 10k–15k", "between 1000 and 2000"
  • Unit normalization: "10k" → 10000, "₹300" → 300

• 📊 Smart Ranking & Scoring

  • Relevance-based scoring
  • Field-weighted matching
  • Title boost for exact matches

📦 Installation

Install the package using npm:

npm i searchify-db

or using yarn:

yarn add searchify-db

or using pnpm:

pnpm add searchify-db

🚀 Quick Start

import { SearchEngine } from 'searchify-db';

// Initialize
const engine = new SearchEngine({
  db: {
    type: 'mysql',
    host: 'localhost',
    user: 'root',
    password: 'password',
    database: 'products_db',
  },
  enableStemming: true,
  enableSynonyms: true,
  autoGenerateSynonyms: true,
  enableNLP: false, // Will be enabled later
});

// Add manual synonyms
engine.addManualSynonyms({
  pants: ['trousers', 'jeans'],
  tee: ['t-shirt', 'tshirt'],
});

// Build index
await engine.buildIndex({
  table: 'products',
  fields: ['title', 'description', 'category', 'price'],
});

// Search
const results = engine.search('blue jeans', 10);

// Disconnect
await engine.disconnect();

📚 API Reference

SearchEngine

Constructor

new SearchEngine(config: SearchEngineConfig)

Config Options:

| Option | Type | Default | Description | |--------|------|---------|-------------| | db | DatabaseConfig | Required | Database configuration object | | enableStemming | boolean | true | Enable word stemming | | enableSynonyms | boolean | true | Enable synonym matching | | autoGenerateSynonyms | boolean | false | Auto-generate synonyms from catalog | | enableNLP | boolean | false | Enable NLP query understanding | | minSynonymFrequency | number | 3 | Minimum frequency for auto-synonyms | | synonymThreshold | number | 0.6 | Similarity threshold for auto-synonyms |

Methods

buildIndex(options: BuildIndexOptions): Promise<void>

Builds the search index from database.

await engine.buildIndex({
  table: 'products',           // Table/collection name (for SQL/NoSQL)
  collection: 'products',      // Alternative for MongoDB
  fields: ['title', 'description'], // Fields to index
  idField: 'id',               // ID field name (optional, defaults to 'id')
});

Parameters:

• table (string, optional): Table name for SQL databases
• collection (string, optional): Collection name for MongoDB
• fields (string[] | IndexField[]): Fields to index. Can be array of strings or objects with name and weight
• idField (string, optional): ID field name (default: 'id')

Example with field weights:

await engine.buildIndex({
  table: 'products',
  fields: [
    { name: 'title', weight: 2.0 },      // Title matches are 2x more important
    { name: 'description', weight: 1.0 },
    { name: 'category', weight: 1.5 },
  ],
});

search(query: string, limit?: number): SearchResult[]

Searches the index and returns ranked results.

const results = engine.search('blue jeans', 10);

Parameters:

• query (string): Search query
• limit (number, optional): Maximum number of results (default: 10)

Returns: Array of SearchResult objects with:

• id: Product ID
• score: Relevance score
• product: Full product object

addManualSynonyms(synonyms: { [key: string]: string[] }): void

Adds manual synonym mappings.

engine.addManualSynonyms({
  pants: ['trousers', 'jeans'],
  tee: ['t-shirt', 'tshirt'],
  kurti: ['tunic'],
});

addManualSynonym(word1: string, word2: string): void

Adds a single synonym pair.

engine.addManualSynonym('pants', 'trousers');

getStats(): { indexed: boolean, productCount: number, tokenCount: number }

Returns index statistics.

const stats = engine.getStats();
console.log(stats);
// {
//   indexed: true,
//   productCount: 15000,
//   tokenCount: 5420
// }

disconnect(): Promise<void>

Disconnects from database.

await engine.disconnect();

🗄️ Database Support

MySQL

{
  type: 'mysql',
  host: 'localhost',
  port: 3306,
  user: 'root',
  password: 'password',
  database: 'products_db',
}

PostgreSQL

{
  type: 'postgresql',
  host: 'localhost',
  port: 5432,
  user: 'postgres',
  password: 'password',
  database: 'ecommerce',
}

MongoDB

{
  type: 'mongodb',
  host: 'localhost',
  port: 27017,
  user: 'admin',
  password: 'password',
  database: 'products',
}

Note: For MongoDB, use collection instead of table in buildIndex:

await engine.buildIndex({
  collection: 'products',
  fields: ['title', 'description'],
});

SQLite

{
  type: 'sqlite',
  database: './products.db',
}

REST API

{
  type: 'rest',
  apiUrl: 'https://api.example.com',
  apiKey: 'your-api-key',
}

Note: For REST API, you may need to provide a custom query function in buildIndex to fetch products.

🔍 Search Flow

1. Tokenize query text into individual words
2. Stem tokens (if enabled) - converts words to their root form
3. Expand with synonyms (if enabled) - adds related terms
4. Retrieve product IDs from inverted index
5. Filter by NLP constraints (when NLP enabled)
6. Rank by relevance score
7. Return top N results

💡 Examples

Basic Usage

import { SearchEngine } from 'searchify-db';

const engine = new SearchEngine({
  db: {
    type: 'mysql',
    host: 'localhost',
    user: 'root',
    password: 'password',
    database: 'products_db',
  },
  enableStemming: true,
  enableSynonyms: true,
});

await engine.buildIndex({
  table: 'products',
  fields: ['title', 'description', 'category'],
});

const results = engine.search('blue jeans', 10);
results.forEach(result => {
  console.log(`${result.product.title} - Score: ${result.score}`);
});

await engine.disconnect();

With Manual Synonyms

const engine = new SearchEngine({
  db: { type: 'postgresql', /* ... */ },
  enableSynonyms: true,
});

// Add custom synonyms
engine.addManualSynonyms({
  pants: ['trousers', 'jeans', 'slacks'],
  tee: ['t-shirt', 'tshirt', 'shirt'],
  kurti: ['tunic', 'top'],
});

await engine.buildIndex({
  table: 'products',
  fields: ['title', 'description'],
});

// Now "pants" will also match "trousers", "jeans", etc.
const results = engine.search('pants', 10);

With Auto-Generated Synonyms

const engine = new SearchEngine({
  db: { type: 'mongodb', /* ... */ },
  enableSynonyms: true,
  autoGenerateSynonyms: true,
  minSynonymFrequency: 5,  // Words must appear together 5+ times
  synonymThreshold: 0.7,   // 70% similarity threshold
});

await engine.buildIndex({
  collection: 'products',
  fields: ['title', 'description'],
});

// Synonyms are automatically generated from your catalog!

Field Weighting

await engine.buildIndex({
  table: 'products',
  fields: [
    { name: 'title', weight: 3.0 },        // Title matches are 3x more important
    { name: 'description', weight: 1.0 },
    { name: 'category', weight: 2.0 },
    { name: 'tags', weight: 1.5 },
  ],
});

🎯 Use Cases

• E-commerce Product Search - Fast, relevant product search
• Catalog Search - Search through product catalogs
• Content Search - Search articles, blogs, documents
• Document Search - Full-text document search
• Any Text-Based Search Application - Flexible and adaptable

🛠️ Development

Prerequisites

• Node.js >= 16.0.0
• npm, yarn, or pnpm

Setup

# Clone the repository
git clone https://github.com/VijaiMegala/Searchify-DB.git
cd Searchify-DB

# Install dependencies
npm install
# or
yarn install
# or
pnpm install

# Build the project
npm run build

Running Examples

# Run basic usage example
ts-node examples/basic-usage.ts

# Run MongoDB example
ts-node examples/mongodb-example.ts

# Run PostgreSQL example
ts-node examples/postgresql-example.ts

# Run test search
npm run test:search

Project Structure

Searchify-DB/
├── src/
│   ├── adapters/          # Database adapters
│   ├── core/              # Core search engine logic
│   ├── plugins/           # Plugins (NLP, etc.)
│   ├── utils/             # Utilities (tokenizer, stemmer, etc.)
│   ├── types.ts          # TypeScript type definitions
│   └── index.ts          # Main export file
├── examples/             # Example usage files
├── dist/                 # Compiled JavaScript (generated)
└── README.md

📝 License

MIT

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/AmazingFeature)
3. Commit your changes (git commit -m 'Add some AmazingFeature')
4. Push to the branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

📧 Support

If you encounter any issues or have questions, please open an issue on GitHub.

🙏 Acknowledgments

Built with ❤️ using TypeScript and Node.js