@objectstack/metadata

Metadata Loading, Persistence & Customization Layer for ObjectStack.

@objectstack/metadata is the central service responsible for loading, validating, persisting and watching all metadata definitions (Objects, Views, Flows, Apps, Agents, etc.) in the ObjectStack platform.

It implements the IMetadataService contract from @objectstack/spec and acts as the single source of truth that all other packages depend on.

Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                     IMetadataService                        │
│              (Contract: @objectstack/spec)                   │
├─────────────────────────────────────────────────────────────┤
│                     MetadataManager                         │
│            (Orchestrator: this package)                      │
│                                                             │
│  ┌─────────────┐  ┌──────────────┐  ┌───────────────────┐  │
│  │  In-Memory   │  │   Overlay    │  │  Type Registry    │  │
│  │  Registry    │  │   System     │  │  & Dependencies   │  │
│  └─────────────┘  └──────────────┘  └───────────────────┘  │
├─────────────────────────────────────────────────────────────┤
│                      Loader Layer                           │
│  ┌─────────────┐  ┌──────────────┐  ┌───────────────────┐  │
│  │ Filesystem   │  │   Remote     │  │   Memory          │  │
│  │ Loader       │  │   Loader     │  │   Loader          │  │
│  │ (files)      │  │   (HTTP)     │  │   (test)          │  │
│  └─────────────┘  └──────────────┘  └───────────────────┘  │
│  ┌──────────────────────────────────────────────────────┐   │
│  │ DatabaseLoader (planned — datasource-backed storage) │   │
│  └──────────────────────────────────────────────────────┘   │
├─────────────────────────────────────────────────────────────┤
│                    Serializer Layer                          │
│  ┌──────────┐  ┌──────────┐  ┌──────────────────────────┐  │
│  │   JSON   │  │   YAML   │  │   TypeScript/JavaScript  │  │
│  └──────────┘  └──────────┘  └──────────────────────────┘  │
└─────────────────────────────────────────────────────────────┘

Core Concepts

1. Metadata Sources (Three-Scope Model)

ObjectStack adopts a three-scope layered model for metadata:

| Scope | Storage | Mutability | Description | |:-----------|:-------------|:-------------|:-------------------------------------------| | system | Filesystem | Read-only | Defined in code, shipped with packages | | platform | Database | Admin-editable | Created/modified by admins via UI | | user | Database | User-editable | Personal customizations per user |

Resolution order: system ← merge(platform) ← merge(user).

2. Loaders

Loaders are pluggable data sources that know how to read/write metadata from different backends. Each loader declares a MetadataLoaderContract with name, protocol, and capabilities:

| Loader | Protocol | Read | Write | Watch | Status | |:--------------------|:---------------|:-----|:------|:------|:-------------| | FilesystemLoader | file: | ✅ | ✅ | ✅ | Implemented | | MemoryLoader | memory: | ✅ | ✅ | ❌ | Implemented | | RemoteLoader | http: | ✅ | ✅ | ❌ | Implemented | | DatabaseLoader | datasource: | ✅ | ✅ | ✅ | Planned |

3. Serializers

Serializers convert metadata objects to/from different file formats:

• JSONSerializer — .json files with optional key sorting
• YAMLSerializer — .yaml/.yml files (JSON_SCHEMA for security)
• TypeScriptSerializer — .ts/.js module exports (for defineObject(), defineView(), etc.)

4. Overlay / Customization System

The overlay system enables non-destructive customizations on top of package-delivered (system) metadata, following a delta-based approach (JSON Merge Patch):

• getOverlay / saveOverlay / removeOverlay — manage customization deltas
• getEffective — returns the merged result of base + platform overlay + user overlay
• Overlays never modify the base definition — they are additive patches

5. MetadataManager (IMetadataService Implementation)

The MetadataManager is the main orchestrator. It provides:

• Core CRUD: register, get, list, unregister, exists, listNames
• Convenience: getObject, listObjects
• Package Management: unregisterPackage — unload all metadata from a package
• Query / Search: query with filtering, pagination, sorting by type/scope/state/tags
• Bulk Operations: bulkRegister, bulkUnregister with error handling
• Import / Export: exportMetadata, importMetadata with conflict resolution (skip/overwrite/merge)
• Validation: validate — structural validation of metadata items
• Type Registry: getRegisteredTypes, getTypeInfo — discover available metadata types
• Dependency Tracking: getDependencies, getDependents — cross-reference analysis
• Watch / Subscribe: watchService — observe metadata changes in real-time
• Loader Delegation: load, loadMany, save — delegate I/O to registered loaders

6. NodeMetadataManager

Extends MetadataManager with Node.js-specific capabilities:

• Auto-configures FilesystemLoader for local development
• File watching via chokidar for hot-reload during development
• Detects file add/change/delete events and notifies subscribers

7. MetadataPlugin

Integrates with the ObjectStack kernel plugin system:

• Registers as the primary IMetadataService provider
• Auto-loads all metadata types from the filesystem on startup (sorted by loadOrder)
• Supports YAML, JSON, TypeScript, and JavaScript metadata formats

Metadata Types

The platform supports 26 built-in metadata types across 6 protocol domains:

| Domain | Types | |:-------------|:----------------------------------------------------------------------------| | Data | object, field, datasource, validation | | UI | view, app, dashboard, report, action, theme | | Automation | flow, workflow, trigger, schedule | | System | manifest, translation, api, permission_set, role, profile | | Security | permission_set, role | | AI | agent, rag_pipeline, model, prompt, tool |

Each type has a defined loadOrder (dependencies load before dependents), file patterns (e.g. **/*.object.{ts,json,yaml}), and overlay support flag.

Spec Protocol References

This package depends on schemas and contracts defined in @objectstack/spec:

| Spec Module | What It Defines | |:---------------------------------|:----------------------------------------------------| | spec/contracts/metadata-service | IMetadataService — the async service interface | | spec/kernel/metadata-loader | Loader contract, load/save/watch schemas, MetadataManagerConfig | | spec/kernel/metadata-plugin | Type registry, plugin manifest, capabilities | | spec/kernel/metadata-customization | Overlay, merge strategy, customization policy | | spec/system/metadata-persistence | MetadataRecord — DB persistence envelope | | spec/data/datasource | DatasourceSchema, DriverDefinition, capabilities | | spec/contracts/data-driver | IDataDriver — database driver interface |

Installation

pnpm add @objectstack/metadata

Usage

Basic (Browser-Compatible)

import { MetadataManager, MemoryLoader } from '@objectstack/metadata';

const manager = new MetadataManager({
  formats: ['json'],
  loaders: [new MemoryLoader()],
});

// Register metadata
await manager.register('object', 'account', { name: 'account', label: 'Account', fields: {} });

// Retrieve
const obj = await manager.get('object', 'account');

// Query
const result = await manager.query({ types: ['object'], search: 'account' });

Node.js (with Filesystem)

import { NodeMetadataManager, MetadataPlugin } from '@objectstack/metadata/node';

const manager = new NodeMetadataManager({
  rootDir: './src',
  formats: ['typescript', 'json', 'yaml'],
  watch: true,
});

// Load all objects from filesystem
const objects = await manager.loadMany('object');

// Watch for changes
manager.watchService('object', (event) => {
  console.log(`Object ${event.name} was ${event.type}`);
});

With Kernel Plugin

import { MetadataPlugin } from '@objectstack/metadata/node';

const plugin = MetadataPlugin({
  rootDir: './src',
  watch: process.env.NODE_ENV === 'development',
});
// Register with ObjectStack kernel
kernel.use(plugin);

Package Structure

packages/metadata/
├── src/
│   ├── index.ts                     # Main exports (browser-compatible)
│   ├── node.ts                      # Node.js exports (filesystem, watching)
│   ├── metadata-manager.ts          # MetadataManager (IMetadataService impl)
│   ├── node-metadata-manager.ts     # NodeMetadataManager (+ file watching)
│   ├── plugin.ts                    # MetadataPlugin (kernel integration)
│   ├── loaders/
│   │   ├── loader-interface.ts      # MetadataLoader contract
│   │   ├── filesystem-loader.ts     # File I/O with glob, cache, ETag
│   │   ├── memory-loader.ts         # In-memory store (tests/overrides)
│   │   └── remote-loader.ts         # HTTP API loader with auth
│   ├── serializers/
│   │   ├── serializer-interface.ts  # MetadataSerializer contract
│   │   ├── json-serializer.ts       # JSON format
│   │   ├── yaml-serializer.ts       # YAML format
│   │   └── typescript-serializer.ts # TS/JS module format
│   └── migration/
│       ├── index.ts                 # Barrel export
│       └── executor.ts              # ChangeSet executor (DDL operations)
├── package.json
├── tsconfig.json
├── vitest.config.ts
├── README.md                        # This file
└── ROADMAP.md                       # Development roadmap

Related Packages

| Package | Relationship | |:------------------------|:-------------------------------------------------| | @objectstack/spec | Protocol definitions (schemas, contracts, types) | | @objectstack/core | Logger, service registry, kernel utilities | | @objectstack/runtime | Uses this package to bootstrap metadata | | apps/studio | Visual metadata editor (consumes IMetadataService)|

License

Apache-2.0 — see LICENSE for details.