@push.rocks/smartdata

v7.2.0

Published

5 days ago

An advanced library for NoSQL data organization and manipulation using TypeScript with support for MongoDB, data validation, collections, and custom data types.

0High
0Medium
0Low

lossless

data manipulation NoSQL MongoDB TypeScript data validation collections custom data types ODM

@push.rocks/smartdata 🚀

The ultimate TypeScript-first MongoDB ODM — type-safe decorators, real-time change streams, Lucene-powered search, distributed leader election, and cursor streaming. Built for modern applications that demand performance, correctness, and developer experience.

Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit community.foss.global/. This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a code.foss.global/ account to submit Pull Requests directly.

🌟 Why SmartData?

🔒 100% Type-Safe — TC39 Stage 3 decorators, generic filters, and compile-time query validation
⚡ High Performance — Connection pooling, cursor streaming, and automatic indexing
🔄 Real-time Ready — MongoDB Change Streams with RxJS for reactive applications
🌍 Distributed Systems — Built-in leader election and task coordination via @push.rocks/taskbuffer
🛡️ Security First — $where injection prevention, operator allow-listing, and input sanitization
🔎 Lucene Search — Full-text, wildcard, boolean, and range queries out of the box
🎯 Great DX — Intuitive API, IntelliSense that just works, and lifecycle hooks

📦 Installation

pnpm add @push.rocks/smartdata

🚦 Requirements

Node.js >= 20.x
Deno >= 2.0 (for Deno projects)
MongoDB >= 5.0
TypeScript >= 5.2 (for TC39 decorator support)

Note: SmartData uses TC39 Stage 3 decorators (the standard). Make sure experimentalDecorators is not set in your tsconfig.json. Bun is not currently supported as it doesn't implement TC39 decorators yet.

🎯 Quick Start

1️⃣ Connect to Your Database

import { SmartdataDb } from '@push.rocks/smartdata';

const db = new SmartdataDb({
  mongoDbUrl: 'mongodb://localhost:27017/myapp',
  mongoDbName: 'myapp',
  mongoDbUser: 'username',
  mongoDbPass: 'password',
});

await db.init();
console.log(db.status); // 'connected'

2️⃣ Define Your Data Models

import {
  SmartDataDbDoc,
  Collection,
  unI,
  svDb,
  index,
  searchable,
} from '@push.rocks/smartdata';

@Collection(() => db)
class User extends SmartDataDbDoc<User, User> {
  @unI()
  public id!: string;

  @svDb()
  @searchable()
  public username!: string;

  @svDb()
  @searchable()
  @index({ unique: false })
  public email!: string;

  @svDb()
  public status!: 'active' | 'inactive' | 'pending';

  @svDb()
  public tags!: string[];

  @svDb()
  public createdAt: Date = new Date();

  constructor(username: string, email: string) {
    super();
    this.username = username;
    this.email = email;
  }
}

3️⃣ CRUD Operations

// ✨ Create
const user = new User('johndoe', '[email protected]');
user.status = 'active';
user.tags = ['developer', 'typescript'];
await user.save();

// 🔍 Read — fully type-safe filters
const foundUser = await User.getInstance({ username: 'johndoe' });
const activeUsers = await User.getInstances({ status: 'active' });

// ✏️ Update
foundUser.email = '[email protected]';
await foundUser.save();

// 🗑️ Delete
await foundUser.delete();

🔥 Features

🎯 Type-Safe Query Filters

SmartData provides a rich, type-safe filtering system supporting all MongoDB operators with full IntelliSense:

// Comparison operators
const adults = await User.getInstances({
  age: { $gte: 18, $lt: 65 },
});

// Array operators
const experts = await User.getInstances({
  tags: { $all: ['typescript', 'mongodb'] },
  skills: { $size: 5 },
});

// Logical operators
const complex = await Order.getInstances({
  $and: [
    { status: 'active' },
    { $or: [{ priority: 'high' }, { value: { $gte: 1000 } }] },
  ],
});

// Deep nested object queries
const users = await User.getInstances({
  profile: {
    settings: {
      notifications: { email: true },
    },
  },
});

// Dot notation
const sameUsers = await User.getInstances({
  'profile.settings.notifications.email': true,
});

// Regex patterns
const gmailUsers = await User.getInstances({
  email: { $regex: '@gmail\\.com$', $options: 'i' },
});

Security: The $where operator is automatically blocked to prevent NoSQL injection. Unknown operators trigger warnings.

🔎 Lucene-Powered Search

Mark fields with @searchable() to enable a built-in search engine with automatic compound text indexing:

@Collection(() => db)
class Product extends SmartDataDbDoc<Product, Product> {
  @unI() public id!: string;
  @svDb() @searchable() public name!: string;
  @svDb() @searchable() public description!: string;
  @svDb() @searchable() public category!: string;
  @svDb() public price!: number;
}

// Simple text search across all @searchable fields
const results = await Product.search('laptop');

// Field-specific search
const electronics = await Product.search('category:Electronics');

// Wildcard
const matches = await Product.search('Mac*');

// Boolean operators (AND, OR, NOT)
const query = await Product.search('laptop AND NOT gaming');

// Phrase search
const exact = await Product.search('"MacBook Pro"');

// Range queries
const midRange = await Product.search('price:[100 TO 500]');

// Combined with MongoDB filters and post-fetch validation
const affordable = await Product.search('laptop', {
  filter: { price: { $lte: 1500 } },
  validate: async (p) => p.price > 0,
});

📡 Real-Time Change Streams

Watch for database changes with RxJS subjects and EventEmitter support:

const watcher = await User.watch(
  { status: 'active' },
  {
    fullDocument: 'updateLookup',
    bufferTimeMs: 100, // optional: buffer changes via RxJS
  },
);

// RxJS subscription
watcher.changeSubject.subscribe((user) => {
  console.log('User changed:', user);
});

// Or EventEmitter style
watcher.on('change', (user) => {
  console.log('User changed:', user);
});

// Clean up
await watcher.close();

🔄 Cursor Streaming

Process large datasets without memory pressure:

const cursor = await User.getCursor(
  { status: 'active' },
  {
    modifier: (c) => c.sort({ createdAt: -1 }).limit(10000),
  },
);

// Iterate one-by-one
await cursor.forEach(async (user) => {
  await processUser(user);
});

// Or collect into an array
const users = await cursor.toArray();

// Always close when done
await cursor.close();

🔐 Transactions

Ensure atomic consistency across multiple operations:

const session = db.startSession();

try {
  await session.withTransaction(async () => {
    const sender = await User.getInstance({ id: 'user-1' }, { session });
    sender.balance -= 100;
    await sender.save({ session });

    const receiver = await User.getInstance({ id: 'user-2' }, { session });
    receiver.balance += 100;
    await receiver.save({ session });
  });
} finally {
  await session.endSession();
}

💾 EasyStore — Type-Safe Key-Value Storage

Built on top of SmartData collections, EasyStore provides simple key-value persistence:

interface AppConfig {
  apiKey: string;
  features: { darkMode: boolean; notifications: boolean };
  limits: { maxUsers: number };
}

const config = await db.createEasyStore<AppConfig>('app-config');

// Write
await config.writeKey('features', { darkMode: true, notifications: false });

// Read
const features = await config.readKey('features');
if (features) {
  // TypeScript knows: features.darkMode is boolean ✅
  console.log(features.darkMode);
}

// Read all
const all = await config.readAll();

// Write multiple keys
await config.writeAll({ apiKey: 'new-key', limits: { maxUsers: 500 } });

// Delete a key
await config.deleteKey('features');

// Wipe the store
await config.wipe();

🌐 Distributed Coordination

Built-in leader election using MongoDB for coordination, integrating with @push.rocks/taskbuffer:

import { SmartdataDistributedCoordinator } from '@push.rocks/smartdata';

const coordinator = new SmartdataDistributedCoordinator(db);

// Start coordination — automatic heartbeat and leader election
await coordinator.start();

// Fire distributed task requests
const result = await coordinator.fireDistributedTaskRequest({
  submitterId: 'instance-1',
  requestResponseId: 'unique-id',
  taskName: 'process-payments',
  taskVersion: '1.0.0',
  taskExecutionTime: Date.now(),
  taskExecutionTimeout: 30000,
  taskExecutionParallel: 1,
  status: 'requesting',
});

// Graceful shutdown with leadership handoff
await coordinator.stop();

🎨 Custom Serialization

Transform data on its way in and out of MongoDB:

@Collection(() => db)
class Doc extends SmartDataDbDoc<Doc, Doc> {
  @svDb({
    serialize: (set) => Array.from(set),
    deserialize: (arr) => new Set(arr),
  })
  public tags!: Set<string>;

  @svDb({
    serialize: (date) => date?.toISOString(),
    deserialize: (str) => (str ? new Date(str) : null),
  })
  public scheduledAt!: Date | null;
}

🧹 Cached Documents With TTL

Use SmartdataCachedDocument for cache-like documents that need creation, last-access, and absolute-expiration timestamps:

import {
  Collection,
  SmartdataCachedDocument,
  smartdataTtlValues,
  svDb,
  unI,
} from '@push.rocks/smartdata';

@Collection(() => db)
class CachedLookup extends SmartdataCachedDocument<CachedLookup> {
  @unI()
  public cacheKey!: string;

  @svDb()
  public payload!: string;
}

const lookup = new CachedLookup();
lookup.cacheKey = 'example';
lookup.payload = 'cached value';
lookup.setTTL(smartdataTtlValues.HOURS_24);
await lookup.save();

expiresAt is indexed with MongoDB TTL semantics using expireAfterSeconds: 0, so the document expires after the absolute date stored in expiresAt. MongoDB TTL cleanup is asynchronous and may lag behind the exact expiration time.

🎣 Lifecycle Hooks

Add custom logic before and after save/delete:

@Collection(() => db)
class Order extends SmartDataDbDoc<Order, Order> {
  @unI() public id!: string;
  @svDb() public items!: Array<{ product: string; quantity: number; price: number }>;
  @svDb() public totalAmount!: number;

  async beforeSave() {
    this.totalAmount = this.items.reduce((s, i) => s + i.price * i.quantity, 0);
  }

  async afterSave() {
    await notificationService.orderUpdated(this.id);
  }

  async beforeDelete() {
    if (this.totalAmount > 0) throw new Error('Cannot delete non-zero orders');
  }

  async afterDelete() {
    await cache.delete(`order:${this.id}`);
  }
}

🏗️ Indexing

@Collection(() => db)
class HighPerformanceDoc extends SmartDataDbDoc<HighPerformanceDoc, HighPerformanceDoc> {
  @unI()
  public id!: string; // Unique index

  @index()
  public userId!: string; // Regular index

  @index({ sparse: true })
  public deletedAt?: Date; // Sparse index — only indexes docs where field exists

  @index({ expireAfterSeconds: 86400 })
  public expiresAt!: Date; // TTL index — auto-expires 24h after this Date
}

🔧 Connection Options

const db = new SmartdataDb({
  mongoDbUrl: 'mongodb://localhost:27017',
  mongoDbName: 'myapp',
  mongoDbUser: 'admin',
  mongoDbPass: 's3cret',

  // Connection pool tuning (all optional)
  maxPoolSize: 100,           // Max connections (default: 100)
  maxIdleTimeMS: 300000,      // Close idle connections after 5min (default)
  serverSelectionTimeoutMS: 30000, // Timeout for server selection
  socketTimeoutMS: 30000,     // Socket timeout to prevent hung operations
});

📚 Decorators Reference

| Decorator | Target | Description | |-----------|--------|-------------| | @Collection(dbGetter) | Class | Binds a document class to a MongoDB collection | | @managed(managerGetter?) | Class | Like @Collection but controlled by a manager instance | | @unI() | Field | Marks as unique index + saveable | | @svDb(options?) | Field | Marks field as saveable, with optional serialize/deserialize | | @index(options?) | Field | Creates a regular MongoDB index | | @searchable() | Field | Enables Lucene-style text search on this field | | @globalSvDb() | Field | Marks field as globally saveable across all doc types |

📚 API Reference

Core Classes

| Class | Description | |-------|-------------| | SmartdataDb | Database connection, session management, EasyStore factory | | SmartDataDbDoc<T, TImpl> | Base class for all document models | | SmartdataCollection<T> | Underlying collection manager (usually accessed indirectly) | | SmartdataDbCursor<T> | Cursor for streaming large result sets | | SmartdataDbWatcher<T> | Change stream watcher with RxJS + EventEmitter | | SmartdataDistributedCoordinator | Leader election and distributed task coordination | | SmartdataCachedDocument<T> | Base class for cache documents with creation, access, and expiration timestamps | | EasyStore<T> | Type-safe key-value store backed by a collection |

Utilities

| Utility | Description | |---------|-------------| | smartdataTtlValues | Common TTL durations in milliseconds for cache documents |

Key Static Methods on `SmartDataDbDoc`

| Method | Description | |--------|-------------| | getInstances(filter, opts?) | Find multiple documents | | getInstance(filter, opts?) | Find a single document (or null) | | getCursor(filter, opts?) | Get a streaming cursor | | getCount(filter?) | Count matching documents | | watch(filter, opts?) | Watch for real-time changes | | search(query, opts?) | Lucene-style full-text search | | forEach(filter, fn) | Iterate all matches with a callback | | getNewId(length?) | Generate a class-prefixed unique ID | | createSearchFilter(luceneQuery) | Convert Lucene query to MongoDB filter | | getSearchableFields() | List all @searchable() fields |

Key Instance Methods on `SmartDataDbDoc`

| Method | Description | |--------|-------------| | save(opts?) | Insert or update the document | | delete(opts?) | Delete the document | | updateFromDb() | Refresh fields from the database | | saveDeep(savedMap?) | Recursively save referenced documents | | createSavableObject() | Serialize to a plain object for persistence | | createIdentifiableObject() | Extract unique index fields for filtering |

License and Legal Information

This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the LICENSE file.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.

Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.