RTZ-IO

TypeScript library for RTZ file format serialization and persistence. RTZ-IO provides a comprehensive solution for creating, packing, unpacking, and managing RTZ files with support for encryption, assets, and metadata validation.

Features

• Working State Management: Initialize and validate RTZ working states
• Serialization: Pack working states into compressed RTZ files
• Deserialization: Unpack RTZ files back to working states
• Encryption: Optional AES-256-GCM encryption with key management
• Assets: Support for binary assets and file attachments
• Validation: Comprehensive data validation and integrity checking
• CLI Tools: Command-line interface for common operations
• TypeScript: Full TypeScript support with type definitions

Installation

npm install rtz-io

Quick Start

Basic Usage

import { 
  initWorkingState, 
  workingToPersistence, 
  persistenceToWorking 
} from 'rtz-io';

// Create a new working state
const workingState = initWorkingState();
workingState.header.title = 'My RTZ Document';
workingState.content.data = { message: 'Hello, RTZ!' };

// Pack to RTZ file
const buffer = await workingToPersistence(workingState);

// Unpack from RTZ file
const restored = await persistenceToWorking(buffer);

With Encryption

import { workingToPersistence, persistenceToWorking, generateKey } from 'rtz-io';

// Generate encryption key
const key = generateKey(32);
const keyProvider = async (keyId: string) => key;

// Pack with encryption
const encrypted = await workingToPersistence(workingState, {
  encrypt: true,
  keyProvider,
  keyId: 'my-key'
});

// Unpack with decryption
const restored = await persistenceToWorking(encrypted, { keyProvider });

With Assets

import fs from 'fs';

const workingState = initWorkingState();

// Add assets
workingState.content.assets = [
  {
    id: 'image-1',
    name: 'photo.jpg',
    type: 'image',
    size: imageData.length,
    checksum: calculateHash(imageData),
    data: imageData
  }
];

const buffer = await workingToPersistence(workingState, { includeAssets: true });

API Reference

Core Functions

initWorkingState(): WorkingState

Creates a new WorkingState object with default values.

workingToPersistence(state: WorkingState, options?: PersistOptions): Promise<Buffer>

Converts a WorkingState to RTZ file format.

Options:

• encrypt?: boolean - Enable AES-256-GCM encryption
• keyProvider?: (keyId: string) => Promise<Buffer> - Key provider function
• keyId?: string - Key identifier for encryption
• compression?: 'none' | 'gzip' | 'deflate' - Compression method
• includeAssets?: boolean - Include asset data in the archive

persistenceToWorking(buffer: Buffer, options?: LoadOptions): Promise<WorkingState>

Converts RTZ file format back to WorkingState.

Options:

• keyProvider?: (keyId: string) => Promise<Buffer> - Key provider for decryption
• validateSignature?: boolean - Verify manifest signatures
• extractAssets?: boolean - Extract asset data

validateWorkingState(state: any): asserts state is WorkingState

Validates that an object conforms to the WorkingState interface.

Utility Functions

canonicalizeJson(obj: any): string

Creates a canonical JSON representation for consistent serialization.

calculateHash(data: Buffer | string): string

Calculates SHA-256 hash of data.

generateKey(length?: number): Buffer

Generates a random encryption key (default: 32 bytes for AES-256).

generateId(): string

Generates a random UUID v4.

Types

interface WorkingState {
  header: Header;
  content: Content;
  history: HistoryEntry[];
  usagePolicy: UsagePolicy;
}

interface Header {
  version: string;
  created: string;
  modified: string;
  id: string;
  title?: string;
  description?: string;
  tags?: string[];
  metadata?: Record<string, any>;
}

interface Content {
  type: string;
  data: any;
  encoding?: string;
  checksum?: string;
  assets?: Asset[];
}

interface Asset {
  id: string;
  name: string;
  type: string;
  size: number;
  checksum: string;
  path?: string;
  data?: Buffer;
}

CLI Usage

The package includes a command-line tool for common RTZ operations:

# Initialize a new RTZ working state
rtz init ./my-project

# Pack a directory into an RTZ file
rtz pack ./my-project project.rtz

# Unpack an RTZ file
rtz unpack project.rtz ./extracted

# Show RTZ file information
rtz info project.rtz

# Generate an encryption key
rtz --generate-key ./my-key.bin

# Pack with encryption
rtz pack ./my-project project.rtz --encrypt --key-file ./my-key.bin

# Unpack encrypted file
rtz unpack project.rtz ./extracted --key-file ./my-key.bin

JSON Splitting and Partitioning

RTZ-IO includes powerful JSON splitting capabilities that allow you to partition any JSON object into two files:

1. A "plain" file with sensitive fields replaced by UUID references
2. An "encrypted" file containing the original sensitive values keyed by UUIDs

This is useful for scenarios where you need to separate sensitive data for security, storage, or transmission purposes.

Basic JSON Splitting

import { splitJsonToFiles, mergeJsonFromFiles, CommonPredicates } from 'rtz-io';

const sensitiveData = {
  name: 'John Doe',
  email: '[email protected]',
  password_enc: 'secret123',
  profile: {
    ssn_enc: '123-45-6789',
    credit_card_enc: '4111-1111-1111-1111',
    public_info: 'This is public'
  }
};

// Split into two files
const result = await splitJsonToFiles(sensitiveData, {
  plainFile: './output/plain.json',
  encryptedFile: './output/encrypted.json',
  predicate: CommonPredicates.encryptedFields // Matches fields ending with '_enc'
});

console.log(`Encrypted ${result.stats.totalEncryptedFields} fields`);
console.log(`Total size: ${result.sizes.totalBytes} bytes`);

// Later, reconstruct the original data
const reconstructed = await mergeJsonFromFiles({
  plainFile: './output/plain.json',
  encryptedFile: './output/encrypted.json'
});

// reconstructed === sensitiveData (deep equality)

Custom Predicates

You can define custom logic for determining which fields should be encrypted:

import { partitionJson, PartitionPredicates } from 'rtz-io';

// Custom predicate: encrypt fields containing 'secret' or 'password'
const customPredicate = (path: string, value: any) => {
  const fieldName = path.split('.').pop() || '';
  return /(?:secret|password)/i.test(fieldName);
};

// Built-in predicate utilities
const predicates = {
  // Fields ending with specific suffixes
  encryptedFields: PartitionPredicates.fieldEndsWith('_enc'),
  
  // Fields matching regex patterns
  sensitiveFields: PartitionPredicates.fieldMatches(/(?:password|secret|key|token)/i),
  
  // Specific field paths
  specificPaths: PartitionPredicates.specificPaths(['user.ssn', 'payment.card_number']),
  
  // Values matching patterns (e.g., credit card numbers)
  creditCards: PartitionPredicates.valueMatches([/^\d{4}[-\s]?\d{4}[-\s]?\d{4}[-\s]?\d{4}$/])
};

Common Predicate Types

The library includes pre-built predicates for common sensitive data patterns:

import { CommonPredicates } from 'rtz-io';

const predicates = {
  // Fields ending with '_enc', '_encrypted', '_secret'
  encryptedFields: CommonPredicates.encryptedFields,
  
  // Fields containing 'password', 'secret', 'key', 'token'
  sensitiveFields: CommonPredicates.sensitiveFields,
  
  // Email addresses
  emails: CommonPredicates.emails,
  
  // Credit card numbers
  creditCards: CommonPredicates.creditCards,
  
  // Social Security Numbers (US format)
  ssn: CommonPredicates.ssn,
  
  // Phone numbers (various formats)
  phoneNumbers: CommonPredicates.phoneNumbers,
  
  // Combine multiple predicates
  combined: CommonPredicates.anyOf(
    CommonPredicates.encryptedFields,
    CommonPredicates.creditCards,
    CommonPredicates.ssn
  )
};

Advanced Features

File Validation

import { validateJsonSplitFiles } from 'rtz-io';

const validation = await validateJsonSplitFiles({
  plainFile: './plain.json',
  encryptedFile: './encrypted.json'
});

if (validation.valid) {
  console.log('Files are valid and can be merged');
} else {
  console.log('Validation errors:', validation.errors);
}

Backup Creation

import { backupJsonSplitFiles } from 'rtz-io';

const backupResult = await backupJsonSplitFiles({
  plainFile: './plain.json',
  encryptedFile: './encrypted.json'
}, './backups');

console.log('Backups created:', backupResult);

Low-level Partitioning

For more control, use the low-level partitioning API:

import { partitionJson, mergeJson, validatePartition, getPartitionStats } from 'rtz-io';

// Partition in memory (no file I/O)
const { plain, encryptedMap } = partitionJson(data, predicate);

// Get statistics
const stats = getPartitionStats({ plain, encryptedMap });
console.log(`Encrypted ${stats.totalEncryptedFields} fields`);

// Validate partition integrity
const isValid = validatePartition(originalData, { plain, encryptedMap });

// Merge back to original
const reconstructed = mergeJson(plain, encryptedMap);

CLI Commands for JSON Splitting

The CLI includes commands for JSON splitting operations:

# Split a JSON file using encrypted fields predicate
rtz split-json data.json plain.json encrypted.json encrypted-fields

# Split using different predicates
rtz split-json data.json plain.json encrypted.json sensitive-fields
rtz split-json data.json plain.json encrypted.json emails
rtz split-json data.json plain.json encrypted.json credit-cards

# Merge split files back together
rtz merge-json plain.json encrypted.json reconstructed.json

Available predicate types for CLI:

• encrypted-fields - Fields ending with _enc, _encrypted, _secret
• sensitive-fields - Fields containing password, secret, key, token
• emails - Email address values
• credit-cards - Credit card number values
• ssn - Social Security Number values
• phone-numbers - Phone number values

JSON Splitting API Reference

splitJsonToFiles(data, options): Promise<JsonSplitResult>

Splits a JSON object into two files.

Parameters:

• data: any - The JSON object to split
• options: JsonSplitOptions - Configuration options

Options:

• plainFile: string - Path for the plain JSON file (with UUID references)
• encryptedFile: string - Path for the encrypted JSON file (sensitive values)
• predicate: PartitionPredicate - Function to determine which fields to encrypt
• createDirs?: boolean - Create directories if they don't exist (default: true)
• prettyPrint?: boolean - Pretty print JSON with indentation (default: true)
• validate?: boolean - Validate the split operation (default: true)

Returns:

• JsonSplitResult - Object containing statistics and file information

mergeJsonFromFiles(options): Promise<any>

Merges JSON data from two files to reconstruct the original object.

Parameters:

• options: JsonMergeOptions - Configuration options

Options:

• plainFile: string - Path to the plain JSON file
• encryptedFile: string - Path to the encrypted JSON file
• validate?: boolean - Validate the merge operation (default: true)

Returns:

• any - The reconstructed original object