NoSQLite

A high-performance, disk-based NoSQL database library for Node.js with LSM-Tree architecture and MongoDB-compatible API.

Features

• LSM-Tree Storage Architecture: Write-optimized performance with efficient compaction
• MongoDB-Compatible API: Familiar query interface with support for CRUD operations
• Secondary Indexes: B-Tree implementation for fast queries
• Write-Ahead Log (WAL): Durability guarantees with configurable sync intervals
• Background Compaction: Automatic space optimization and performance maintenance
• TypeScript Support: Full type safety and excellent developer experience
• Skip List Memtable: Efficient in-memory sorted storage
• Bloom Filters: Reduced disk I/O for non-existent keys

Installation

npm install qd-nosqlite

Quick Start

import { NoSQLite } from 'qd-nosqlite';

// Initialize database
const db = new NoSQLite('./data');
await db.initialize();

// Get a collection
const users = await db.collection('users');

// Insert documents
await users.insertOne({ 
  name: 'Alice', 
  age: 30, 
  email: '[email protected]' 
});

// Create indexes for better query performance
await users.createIndex({ age: 1 });

// Query documents
const olderUsers = await users.find({ age: { $gt: 25 } });

// Close database
await db.close();

Architecture

NoSQLite implements a Log-Structured Merge-Tree (LSM-Tree) architecture:

1. Write Path: Data flows through WAL → Memtable → SSTables
2. Read Path: Query checks Memtable first, then SSTables with Bloom filters
3. Compaction: Background process merges SSTables to optimize storage
4. Indexing: B-Tree secondary indexes for fast lookups

Complete API Documentation

Table of Contents

• Getting Started
• Database API
• Collection API
• Query Filters
• Update Operations
• Index Management
• Error Handling
• TypeScript Types

Getting Started

Basic Usage

// Create a new database instance
const db = new NoSQLite({
  dataDir: './data',
  memtableSize: 64 * 1024 * 1024, // 64MB
  maxLevel0Tables: 4,
  compactionThreshold: 4,
  walSyncInterval: 1000,
  bloomFilterBitsPerKey: 10,
  maxOpenFiles: 1000
});

// Initialize the database
await db.initialize();

// Get a collection
const users = db.collection('users');

// Insert a document
const result = await users.insertOne({
  name: 'John Doe',
  email: '[email protected]',
  age: 30
});

console.log(result.insertedId);

Database API

The Database class (exported as NoSQLite) provides database-level operations.

Constructor

Creates a new database instance.

new NoSQLite(config: NoSQLiteConfig)

Parameters:

• config: Database configuration object

Configuration Options:

• dataDir: Directory for storing database files
• memtableSize: Size of in-memory table in bytes (default: 64MB)
• maxLevel0Tables: Maximum number of Level 0 SSTables before compaction
• compactionThreshold: Threshold for triggering compaction
• walSyncInterval: Write-ahead log sync interval in milliseconds
• bloomFilterBitsPerKey: Bits per key for bloom filters
• maxOpenFiles: Maximum number of open file handles

Example:

const db = new NoSQLite({
  dataDir: './my-database',
  memtableSize: 128 * 1024 * 1024, // 128MB
  maxLevel0Tables: 6,
  compactionThreshold: 8,
  walSyncInterval: 500,
  bloomFilterBitsPerKey: 12,
  maxOpenFiles: 1500
});

initialize()

Initializes the database and prepares it for operations.

async initialize(): Promise<void>

Returns: Promise that resolves when initialization is complete

Example:

await db.initialize();

collection()

Gets or creates a collection.

collection(name: string): Collection

Parameters:

• name: Name of the collection

Returns: Collection instance

Example:

const users = db.collection('users');
const products = db.collection('products');

dropCollection()

Drops (deletes) a collection and all its data.

async dropCollection(name: string): Promise<boolean>

Parameters:

• name: Name of the collection to drop

Returns: Promise that resolves to true if collection was dropped, false if it didn't exist

Example:

const dropped = await db.dropCollection('old_data');
if (dropped) {
  console.log('Collection dropped successfully');
}

listCollections()

Lists all collections in the database.

listCollections(): string[]

Returns: Array of collection names

Example:

const collections = db.listCollections();
console.log('Available collections:', collections);

getStats()

Gets database statistics including storage usage and performance metrics.

getStats(): Promise<any>

Returns: Promise that resolves to database statistics object

Example:

const stats = await db.getStats();
console.log('Database size:', stats.totalSize);
console.log('Number of collections:', stats.collectionCount);

compact()

Manually triggers database compaction to optimize storage.

async compact(): Promise<void>

Returns: Promise that resolves when compaction is complete

Example:

await db.compact();
console.log('Database compaction completed');

close()

Closes the database and releases all resources.

async close(): Promise<void>

Returns: Promise that resolves when database is closed

Example:

await db.close();
console.log('Database closed');

getConfig()

Gets the current database configuration.

getConfig(): NoSQLiteConfig

Returns: Copy of the current configuration object

Example:

const config = db.getConfig();
console.log('Data directory:', config.dataDir);

updateConfig()

Updates the database configuration.

updateConfig(newConfig: Partial<NoSQLiteConfig>): void

Parameters:

• newConfig: Partial configuration object with fields to update

Example:

db.updateConfig({
  memtableSize: 256 * 1024 * 1024, // Increase to 256MB
  maxLevel0Tables: 8
});

Collection API

The Collection class provides MongoDB-compatible document operations.

insertOne()

Inserts a single document into the collection.

async insertOne(document: Omit<Document, '_id'> & { _id?: DocumentId }): Promise<{
  insertedId: DocumentId;
  acknowledged: boolean;
}>

Parameters:

• document: Document to insert (can include optional _id)

Returns: Object containing the inserted document ID and acknowledgment status

Example:

const result = await users.insertOne({
  name: 'Alice Smith',
  email: '[email protected]',
  age: 28,
  active: true
});

console.log('Inserted with ID:', result.insertedId);

// With custom ID
const result2 = await users.insertOne({
  _id: 'user_123',
  name: 'Bob Johnson',
  email: '[email protected]'
});

insertMany()

Inserts multiple documents into the collection.

async insertMany(documents: (Omit<Document, '_id'> & { _id?: DocumentId })[]): Promise<{
  insertedIds: DocumentId[];
  insertedCount: number;
  acknowledged: boolean;
}>

Parameters:

• documents: Array of documents to insert

Returns: Object containing inserted IDs, count, and acknowledgment status

Example:

const result = await users.insertMany([
  { name: 'User 1', email: '[email protected]' },
  { name: 'User 2', email: '[email protected]' },
  { name: 'User 3', email: '[email protected]' }
]);

console.log(`Inserted ${result.insertedCount} documents`);
console.log('IDs:', result.insertedIds);

find()

Finds documents matching a filter.

async find(filter?: QueryFilter, options?: QueryOptions): Promise<Document[]>

Parameters:

• filter: Query filter object (optional, defaults to {} - find all)
• options: Query options (optional)

Options:

• limit: Maximum number of documents to return
• skip: Number of documents to skip
• sort: Sort order object { field: 1 | -1 }
• projection: Fields to include/exclude { field: 1 | 0 }

Returns: Array of matching documents

Example:

// Find all users
const allUsers = await users.find();

// Find with filter
const activeUsers = await users.find({ active: true });

// Find with options
const youngUsers = await users.find(
  { age: { $lt: 30 } },
  { 
    limit: 10, 
    sort: { age: 1 },
    projection: { name: 1, age: 1 }
  }
);

// Complex queries
const result = await users.find({
  age: { $gte: 25, $lte: 35 },
  active: true,
  email: { $regex: '@company.com$' }
});

findOne()

Finds the first document matching a filter.

async findOne(filter?: QueryFilter, options?: QueryOptions): Promise<Document | null>

Parameters:

• filter: Query filter object (optional)
• options: Query options (optional)

Returns: First matching document or null if none found

Example:

// Find first user
const firstUser = await users.findOne();

// Find specific user
const user = await users.findOne({ email: '[email protected]' });

if (user) {
  console.log('Found user:', user.name);
} else {
  console.log('User not found');
}

findById()

Finds a document by its ID.

async findById(id: DocumentId): Promise<Document | null>

Parameters:

• id: Document ID to search for

Returns: Document with the specified ID or null if not found

Example:

const user = await users.findById('user_123');
if (user) {
  console.log('User found:', user.name);
}

updateOne()

Updates the first document matching a filter.

async updateOne(
  filter: QueryFilter,
  update: Record<string, any>,
  options?: { upsert?: boolean }
): Promise<{
  matchedCount: number;
  modifiedCount: number;
  upsertedId?: DocumentId;
  acknowledged: boolean;
}>

Parameters:

• filter: Query filter to match documents
• update: Update operations object
• options: Update options (optional)
  • upsert: If true, creates a new document if none match

Returns: Object containing update statistics

Example:

// Simple update
const result = await users.updateOne(
  { email: '[email protected]' },
  { $set: { age: 29, lastLogin: new Date() } }
);

// Increment operation
await users.updateOne(
  { _id: 'user_123' },
  { $inc: { loginCount: 1 } }
);

// Upsert (insert if not found)
const upsertResult = await users.updateOne(
  { email: '[email protected]' },
  { $set: { name: 'New User', active: true } },
  { upsert: true }
);

if (upsertResult.upsertedId) {
  console.log('Created new user:', upsertResult.upsertedId);
}

updateMany()

Updates all documents matching a filter.

async updateMany(
  filter: QueryFilter,
  update: Record<string, any>
): Promise<{
  matchedCount: number;
  modifiedCount: number;
  acknowledged: boolean;
}>

Parameters:

• filter: Query filter to match documents
• update: Update operations object

Returns: Object containing update statistics

Example:

// Update multiple documents
const result = await users.updateMany(
  { active: false },
  { $set: { status: 'inactive', lastModified: new Date() } }
);

console.log(`Updated ${result.modifiedCount} documents`);

// Bulk increment
await users.updateMany(
  { age: { $gte: 18 } },
  { $inc: { credits: 10 } }
);

replaceOne()

Replaces the first document matching a filter.

async replaceOne(
  filter: QueryFilter,
  replacement: Omit<Document, '_id'>,
  options?: { upsert?: boolean }
): Promise<{
  matchedCount: number;
  modifiedCount: number;
  upsertedId?: DocumentId;
  acknowledged: boolean;
}>

Parameters:

• filter: Query filter to match documents
• replacement: New document data (without _id)
• options: Replace options (optional)

Returns: Object containing replacement statistics

Example:

const result = await users.replaceOne(
  { _id: 'user_123' },
  {
    name: 'Updated Name',
    email: '[email protected]',
    age: 35,
    active: true,
    profile: {
      bio: 'Updated bio',
      interests: ['coding', 'reading']
    }
  }
);

deleteOne()

Deletes the first document matching a filter.

async deleteOne(filter: QueryFilter): Promise<{
  deletedCount: number;
  acknowledged: boolean;
}>

Parameters:

• filter: Query filter to match documents

Returns: Object containing deletion statistics

Example:

const result = await users.deleteOne({ 
  email: '[email protected]' 
});

if (result.deletedCount > 0) {
  console.log('User deleted successfully');
}

deleteMany()

Deletes all documents matching a filter.

async deleteMany(filter: QueryFilter): Promise<{
  deletedCount: number;
  acknowledged: boolean;
}>

Parameters:

• filter: Query filter to match documents

Returns: Object containing deletion statistics

Example:

// Delete inactive users
const result = await users.deleteMany({ 
  active: false,
  lastLogin: { $lt: new Date('2023-01-01') }
});

console.log(`Deleted ${result.deletedCount} inactive users`);

// Delete all documents (use with caution!)
await users.deleteMany({});

countDocuments()

Counts documents matching a filter.

async countDocuments(filter?: QueryFilter): Promise<number>

Parameters:

• filter: Query filter object (optional, defaults to {} - count all)

Returns: Number of matching documents

Example:

// Count all users
const totalUsers = await users.countDocuments();

// Count active users
const activeCount = await users.countDocuments({ active: true });

// Count users in age range
const youngAdults = await users.countDocuments({
  age: { $gte: 18, $lt: 30 }
});

console.log(`Total: ${totalUsers}, Active: ${activeCount}, Young adults: ${youngAdults}`);

getName()

Gets the collection name.

getName(): string

Returns: Name of the collection

Example:

const name = users.getName();
console.log('Collection name:', name);

close()

Closes the collection and releases resources.

async close(): Promise<void>

Returns: Promise that resolves when collection is closed

Example:

await users.close();

Query Filters

NoSQLite supports MongoDB-style query operators for filtering documents.

Comparison Operators

// Equality
await users.find({ age: 30 });

// Not equal
await users.find({ age: { $ne: 30 } });

// Greater than
await users.find({ age: { $gt: 25 } });

// Greater than or equal
await users.find({ age: { $gte: 25 } });

// Less than
await users.find({ age: { $lt: 50 } });

// Less than or equal
await users.find({ age: { $lte: 50 } });

// In array
await users.find({ status: { $in: ['active', 'pending'] } });

// Not in array
await users.find({ status: { $nin: ['inactive', 'banned'] } });

Logical Operators

// AND (implicit)
await users.find({ 
  age: { $gte: 18 }, 
  active: true 
});

// OR
await users.find({
  $or: [
    { age: { $lt: 18 } },
    { age: { $gt: 65 } }
  ]
});

// AND explicit
await users.find({
  $and: [
    { age: { $gte: 18 } },
    { active: true }
  ]
});

// NOT
await users.find({
  age: { $not: { $lt: 18 } }
});

Element Operators

// Field exists
await users.find({ phone: { $exists: true } });

// Field doesn't exist
await users.find({ phone: { $exists: false } });

// Type checking
await users.find({ age: { $type: 'number' } });

Array Operators

// Element in array
await users.find({ 
  interests: { $elemMatch: { $eq: 'coding' } }
});

// Array size
await users.find({ 
  interests: { $size: 3 }
});

// All elements match
await users.find({
  tags: { $all: ['verified', 'premium'] }
});

String Operators

// Regular expression
await users.find({ 
  email: { $regex: '^admin@' }
});

// Case-insensitive regex
await users.find({
  name: { $regex: 'john', $options: 'i' }
});

Nested Field Queries

// Dot notation for nested objects
await users.find({ 
  'profile.age': { $gte: 21 } 
});

// Nested object matching
await users.find({
  address: {
    city: 'New York',
    country: 'USA'
  }
});

Update Operations

NoSQLite supports MongoDB-style update operators for modifying documents.

Field Update Operators

// Set field values
await users.updateOne(
  { _id: 'user_123' },
  {
    $set: {
      name: 'New Name',
      'profile.bio': 'Updated bio',
      lastModified: new Date()
    }
  }
);

// Remove fields
await users.updateOne(
  { _id: 'user_123' },
  {
    $unset: {
      temporaryField: '',
      'profile.oldData': ''
    }
  }
);

// Increment numeric values
await users.updateOne(
  { _id: 'user_123' },
  {
    $inc: {
      loginCount: 1,
      credits: -10,
      'stats.points': 5
    }
  }
);

Array Update Operators

// Add element to array
await users.updateOne(
  { _id: 'user_123' },
  {
    $push: {
      interests: 'photography',
      'history.actions': {
        type: 'login',
        timestamp: new Date()
      }
    }
  }
);

// Remove elements from array
await users.updateOne(
  { _id: 'user_123' },
  {
    $pull: {
      interests: 'outdated_interest',
      tags: 'remove_this_tag'
    }
  }
);

// Add to set (only if not already present)
await users.updateOne(
  { _id: 'user_123' },
  {
    $addToSet: {
      skills: 'JavaScript',
      certifications: 'AWS Certified'
    }
  }
);

Update without Operators

// Direct field assignment (equivalent to $set)
await users.updateOne(
  { _id: 'user_123' },
  {
    name: 'Direct Update',
    active: true
  }
);

Index Management

NoSQLite provides B-Tree indexes for efficient querying.

createIndex()

Creates an index on one or more fields.

async createIndex(
  definition: IndexDefinition,
  options?: IndexOptions
): Promise<string>

Parameters:

• definition: Index field definition { field: 1 | -1 }
• options: Index options (optional)
  • unique: If true, creates a unique index
  • sparse: If true, only indexes documents with the field
  • name: Custom index name

Returns: Promise that resolves to the index name

Example:

// Simple ascending index
const indexName1 = await users.createIndex({ email: 1 });

// Descending index
const indexName2 = await users.createIndex({ createdAt: -1 });

// Compound index
const indexName3 = await users.createIndex({ 
  category: 1, 
  priority: -1 
});

// Unique index
const indexName4 = await users.createIndex(
  { email: 1 },
  { unique: true, name: 'unique_email' }
);

// Sparse index (only indexes documents that have the field)
const indexName5 = await users.createIndex(
  { phone: 1 },
  { sparse: true, name: 'sparse_phone' }
);

console.log('Created indexes:', [indexName1, indexName2, indexName3]);

dropIndex()

Drops an index.

async dropIndex(indexName: string): Promise<boolean>

Parameters:

• indexName: Name of the index to drop

Returns: Promise that resolves to true if index was dropped

Example:

const dropped = await users.dropIndex('unique_email');
if (dropped) {
  console.log('Index dropped successfully');
}

listIndexes()

Lists all indexes on the collection.

listIndexes(): string[]

Returns: Array of index names

Example:

const indexes = users.listIndexes();
console.log('Available indexes:', indexes);

getIndexInfo()

Gets information about a specific index.

getIndexInfo(indexName: string): any

Parameters:

• indexName: Name of the index

Returns: Index information object

Example:

const info = users.getIndexInfo('unique_email');
console.log('Index info:', info);

Error Handling

NoSQLite provides specific error types for different failure scenarios.

Error Types

import { ValidationError, StorageError, IndexError } from 'qd-nosqlite';

try {
  await users.insertOne({ /* invalid document */ });
} catch (error) {
  if (error instanceof ValidationError) {
    console.error('Validation failed:', error.message);
  } else if (error instanceof StorageError) {
    console.error('Storage error:', error.message);
  } else if (error instanceof IndexError) {
    console.error('Index error:', error.message);
  } else {
    console.error('Unknown error:', error);
  }
}

Common Error Scenarios

// Validation errors
try {
  await users.insertOne({
    // Missing _id or invalid data
  });
} catch (error) {
  console.error('Insert failed:', error.message);
}

// Unique constraint violations
try {
  await users.createIndex({ email: 1 }, { unique: true });
  await users.insertOne({ email: '[email protected]' });
  await users.insertOne({ email: '[email protected]' }); // Will fail
} catch (error) {
  console.error('Unique constraint violation:', error.message);
}

// Storage errors
try {
  await db.initialize();
} catch (error) {
  console.error('Database initialization failed:', error.message);
}

TypeScript Types

NoSQLite is fully typed with TypeScript interfaces.

Core Types

import { 
  Document, 
  DocumentId, 
  QueryFilter, 
  QueryOptions,
  IndexDefinition,
  IndexOptions,
  NoSQLiteConfig 
} from 'qd-nosqlite';

// Document type
interface Document {
  [key: string]: any;
}

// Document ID type
type DocumentId = string;

// Query filter for find operations
interface QueryFilter {
  [key: string]: any;
}

// Options for queries
interface QueryOptions {
  limit?: number;
  skip?: number;
  sort?: { [key: string]: 1 | -1 };
  projection?: { [key: string]: 1 | 0 };
}

// Index definition
interface IndexDefinition {
  [field: string]: 1 | -1; // 1 for ascending, -1 for descending
}

// Index creation options
interface IndexOptions {
  unique?: boolean;
  sparse?: boolean;
  name?: string;
}

Type-Safe Usage

// Define your document interface
interface User extends Document {
  name: string;
  email: string;
  age: number;
  active: boolean;
  profile?: {
    bio: string;
    interests: string[];
  };
}

// Type-safe operations
const users = db.collection('users');

const result = await users.insertOne({
  name: 'John Doe',
  email: '[email protected]',
  age: 30,
  active: true
} as User);

const user = await users.findOne({ email: '[email protected]' }) as User | null;

if (user) {
  console.log(`User ${user.name} is ${user.age} years old`);
}

Generic Collection Operations

// You can create type-safe collection wrappers
class UserCollection {
  constructor(private collection: Collection) {}

  async insertUser(userData: Omit<User, '_id'>): Promise<User> {
    const result = await this.collection.insertOne(userData);
    return await this.collection.findById(result.insertedId) as User;
  }

  async findUserByEmail(email: string): Promise<User | null> {
    return await this.collection.findOne({ email }) as User | null;
  }

  async updateUserProfile(userId: DocumentId, profile: User['profile']): Promise<void> {
    await this.collection.updateOne(
      { _id: userId },
      { $set: { profile } }
    );
  }
}

// Usage
const userCollection = new UserCollection(db.collection('users'));
const user = await userCollection.insertUser({
  name: 'Jane Smith',
  email: '[email protected]',
  age: 25,
  active: true
});

Complete Example

Here's a comprehensive example demonstrating various NoSQLite features:

import { NoSQLite } from 'qd-nosqlite';

async function example() {
  // Initialize database
  const db = new NoSQLite({
    dataDir: './example-db',
    memtableSize: 64 * 1024 * 1024,
    maxLevel0Tables: 4,
    compactionThreshold: 4,
    walSyncInterval: 1000,
    bloomFilterBitsPerKey: 10,
    maxOpenFiles: 1000
  });

  await db.initialize();

  // Get collections
  const users = db.collection('users');
  const products = db.collection('products');

  // Create indexes
  await users.createIndex({ email: 1 }, { unique: true, name: 'unique_email' });
  await users.createIndex({ age: 1, active: 1 });
  await products.createIndex({ category: 1, price: -1 });

  // Insert data
  const userResult = await users.insertMany([
    {
      name: 'Alice Johnson',
      email: '[email protected]',
      age: 28,
      active: true,
      profile: {
        bio: 'Software developer',
        interests: ['coding', 'reading', 'hiking']
      }
    },
    {
      name: 'Bob Smith',
      email: '[email protected]',
      age: 35,
      active: true,
      profile: {
        bio: 'Product manager',
        interests: ['technology', 'business', 'travel']
      }
    },
    {
      name: 'Carol Davis',
      email: '[email protected]',
      age: 42,
      active: false,
      profile: {
        bio: 'Designer',
        interests: ['art', 'design', 'photography']
      }
    }
  ]);

  console.log(`Inserted ${userResult.insertedCount} users`);

  // Query data
  const activeUsers = await users.find(
    { active: true },
    { 
      sort: { age: 1 },
      projection: { name: 1, email: 1, age: 1 }
    }
  );

  console.log('Active users:', activeUsers);

  // Complex query
  const youngActiveUsers = await users.find({
    $and: [
      { age: { $lt: 40 } },
      { active: true },
      { 'profile.interests': { $elemMatch: { $eq: 'coding' } } }
    ]
  });

  console.log('Young active developers:', youngActiveUsers);

  // Update operations
  await users.updateOne(
    { email: '[email protected]' },
    {
      $set: { lastLogin: new Date() },
      $inc: { loginCount: 1 },
      $push: { 'profile.interests': 'machine learning' }
    }
  );

  // Count documents
  const totalUsers = await users.countDocuments();
  const activeUserCount = await users.countDocuments({ active: true });

  console.log(`Total users: ${totalUsers}, Active: ${activeUserCount}`);

  // Database statistics
  const stats = await db.getStats();
  console.log('Database stats:', stats);

  // Clean up
  await db.close();
}

// Run the example
example().catch(console.error);

This documentation covers all the major API calls and features of NoSQLite. The library provides a MongoDB-compatible interface with efficient LSM-Tree storage and B-Tree indexing for high-performance document operations.

License

MIT License - see LICENSE file for details.