fs-data

A robust, promise-based file management utility for Node.js, designed to handle structured project files with built-in caching, concurrency control, and file system monitoring. This library simplifies safe reading/writing of JSON files, enforces project directory structure, and provides tools for bulk operations.

Table of Contents

• Features
• Installation
• Core Concepts
• Quick Start
• API Reference
• Configuration Options
• Project Structure
• Error Handling
• Best Practices
• License

Features

• Safe JSON I/O: Atomic writes (via temporary files) and multi-process file locking to prevent data corruption.
• Concurrency Control: Per-file write queues to handle concurrent modification requests safely.
• Dual Caching Layers: Separate LRU caches for core project files and request-specific data (configurable TTL/size).
• File System Monitoring: Automatic cache invalidation when files are modified/deleted externally.
• Path Security: Built-in path validation to prevent directory traversal attacks.
• Bulk Operations: Efficient bulk reading/writing of multiple files.
• Structured Projects: Enforces a standardized project directory structure (configurable).
• Graceful Error Handling: Custom error events, JSON syntax error tolerance (via strictMode), and missing file defaults.

Installation

Install via npm or yarn:

# npm
npm install fs-data

# yarn
yarn add fs-data

This library depends on the following peer dependencies (automatically installed):

• lodash: For deep cloning and object manipulation
• fast-glob: For efficient file system globbing
• lru-cache: For in-memory caching
• chokidar: For file system monitoring
• proper-lockfile: For multi-process file locking

Core Concepts

1. Base Directory: The root folder where all projects and system files are stored.
2. Project ID: A unique identifier for each project (maps to a subdirectory under baseDir/projects/).
3. Dual Caches:
   • coreCache: For frequently accessed project/system configuration files.
   • requestCache: For temporary request-specific data (e.g., individual API requests).
4. Atomic Writes: Files are first written to a temporary file, then renamed to the target path to avoid partial writes.
5. File Locking: Prevents race conditions across multiple processes using proper-lockfile.

Quick Start

1. Initialize the File Manager

const { OfflineFileManager } = require('fs-data');

// Initialize with base directory and optional configuration
const fileManager = new OfflineFileManager('./my-data-root', {
  strictMode: false,          // Tolerate JSON syntax errors (log + return default)
  coreCacheSize: 200,         // Max entries in core cache
  coreCacheTTL: 300000,       // 5-minute TTL for core cache (ms)
  requestCacheSize: 300,      // Max entries in request cache
  requestCacheTTL: 60000      // 1-minute TTL for request cache (ms)
});

// Listen for errors (recommended)
fileManager.on('error', (error) => {
  console.error(`fs-data Error: ${error.file} - ${error.error.message}`);
});

2. Basic File Operations

// 1. Create a new project
const projectId = 'my-first-project';
await fileManager.writeProject(projectId, {
  name: 'My First Project',
  description: 'A test project for fs-data',
  createdAt: new Date().toISOString()
});

// 2. Read project metadata
const project = await fileManager.readProject(projectId);
console.log('Project:', project);

// 3. Write a request-specific file
const requestUuid = 'req-1234-5678-90ab-cdef';
await fileManager.writeRequest(projectId, requestUuid, {
  method: 'GET',
  url: 'https://api.example.com/data',
  headers: { 'Content-Type': 'application/json' }
});

// 4. Read the request file
const request = await fileManager.readRequest(projectId, requestUuid);
console.log('Request:', request);

// 5. List all projects
const allProjects = await fileManager.listProjects();
console.log('All Projects:', allProjects);

3. Bulk Load a Project

Load all files (configs, mocks, resources, requests) for a project in one call:

const fullProject = await fileManager.loadProjectAll(projectId, {
  includeMocks: true,         // Include mock files (default: true)
  includeResources: true,     // Include resource files (default: true)
  includeRequests: true       // Include request files (default: true)
});

console.log('Full Project Config:', fullProject.project);
console.log('Project Mocks:', fullProject.mock);
console.log('Project Requests:', fullProject.requests);

4. Complete write example

A complete write-in example, including project initialization, etc:

(async () => {
    try {
        // 1. Create an instance of OfflineFileManager, specify base directory, enable strict mode
        const baseDir = path.resolve(__dirname, "apipost-offline");
        const manager = new OfflineFileManager(baseDir, {
            strictMode: true,          // Throw error on invalid JSON
            coreCacheSize: 300,        // Core files LRU cache size
            requestCacheSize: 500      // Requests files LRU cache size
        });

        // 2. Global files (manifest.json, system/setting.json) are automatically initialized
        console.log("Global files are initialized automatically at class instantiation.");

        // 3. Initialize a new project (create missing files if not exist)
        const projectId = "my-new-project"; // Can be UUID or any unique string
        await manager.initProject(projectId);
        console.log(`Project '${projectId}' initialized.`);

        // 4. Bulk write some initial data for project-level files
        await manager.bulkWrite({
            [manager._resolveProjectFile(projectId, "project.json")]: { name: "My New Project" },
            [manager._resolveProjectFile(projectId, "globals.json")]: { headers: [] },
            [manager._resolveProjectFile(projectId, "variables.json")]: { token: "123456" },
            [manager._resolveProjectFile(projectId, "environments.json")]: [
                { name: "dev", baseUrl: "https://dev.example.com" },
                { name: "prod", baseUrl: "https://api.example.com" }
            ],
            // 5. Add some sample request data
            [manager._resolveProjectFile(projectId, "requests", "req-001.json")]: {
                name: "Get User Info",
                method: "GET",
                url: "/users/{userId}",
                headers: [{ key: "Authorization", value: "Bearer ${token}" }],
                query: [],
                body: null
            },
            [manager._resolveProjectFile(projectId, "requests", "req-002.json")]: {
                name: "Create New User",
                method: "POST",
                url: "/users",
                headers: [{ key: "Content-Type", value: "application/json" }],
                query: [],
                body: { username: "newuser", email: "[email protected]" }
            },
            [manager._resolveProjectFile(projectId, "mock", "setting.json")]: { enabled: true },
            [manager._resolveProjectFile(projectId, "resources", "functions.json")]: {}
        });
        console.log("Initial project files and sample requests written.");

        // 6. Load the full project data (core files + mocks + resources + requests)
        const projectData = await manager.loadProjectAll(projectId, {
            includeMocks: true,
            includeResources: true,
            includeRequests: true
        });

        console.log("Project full data loaded:");
        console.log(JSON.stringify(projectData, null, 2));

        // 7. Close watchers when finished
        manager.close();

    } catch (err) {
        console.error("Initialization failed:", err);
    }
})();

5. Cleanup

Always close the file manager to stop watchers and free resources:

// Close watchers when done (e.g., on application shutdown)
process.on('SIGINT', async () => {
  await fileManager.close();
  process.exit(0);
});

API Reference

Core Initialization

| Method | Description | |--------|-------------| | new OfflineFileManager(baseDir, [options]) | Creates a new file manager instance. baseDir is required (root path for all files). | | fileManager.close() | Stops all file system watchers and clears resources. Call on shutdown. |

Manifest & System Settings

| Method | Description | |--------|-------------| | readManifest([defaultValue]) | Reads baseDir/manifest.json (returns defaultValue if missing; default: {}). | | writeManifest(data) | Writes data to baseDir/manifest.json (automatically creates parent directories). | | readSystemSetting([defaultValue]) | Reads baseDir/system/setting.json (returns defaultValue if missing; default: {}). | | writeSystemSetting(data) | Writes data to baseDir/system/setting.json. |

Project Operations

| Method | Description | |--------|-------------| | readProject(projectId, [defaultValue]) | Reads baseDir/projects/{projectId}/project.json. | | writeProject(projectId, data) | Writes data to baseDir/projects/{projectId}/project.json. | | listProjects() | Returns an array of all project IDs (directories under baseDir/projects/). | | readArbitrary(projectId, relativePath, [defaultValue]) | Reads a custom file at baseDir/projects/{projectId}/{relativePath}. | | writeArbitrary(projectId, relativePath, data) | Writes to a custom file at baseDir/projects/{projectId}/{relativePath}. |

Request Operations

| Method | Description | |--------|-------------| | readRequest(projectId, uuid, [defaultValue]) | Reads baseDir/projects/{projectId}/requests/{uuid}.json (uses requestCache). | | writeRequest(projectId, uuid, data) | Writes to baseDir/projects/{projectId}/requests/{uuid}.json. |

Bulk Operations

| Method | Description | |--------|-------------| | bulkWrite(fileMap) | Writes multiple files in parallel. fileMap is an object where keys are absolute file paths and values are the data to write. | | loadProjectAll(projectId, [options]) | Bulk loads all standard project files (base configs, mocks, resources, requests). Returns a nested object with paths mapped to dot notation (e.g., mock.setting for mock/setting.json). |

Configuration Options

When initializing OfflineFileManager, pass an options object to customize behavior:

| Option | Type | Default | Description | |--------|------|---------|-------------| | strictMode | boolean | false | If true, throws errors for invalid JSON; if false, logs errors and returns defaultValue. | | coreCacheSize | number | 200 | Maximum number of entries in the coreCache (for project/system files). | | coreCacheTTL | number | 0 | TTL (in milliseconds) for coreCache entries. 0 = no expiration. | | requestCacheSize | number | 300 | Maximum number of entries in the requestCache (for request-specific files). | | requestCacheTTL | number | 0 | TTL (in milliseconds) for requestCache entries. 0 = no expiration. |

Project Structure

The library enforces a standardized project structure (defined in PROJECT_STRUCTURE):

baseDir/
├── manifest.json          # Global manifest
├── system/
│   └── setting.json       # System-wide settings
└── projects/
    └── {projectId}/       # Project-specific directory
        ├── project.json   # Project metadata
        ├── globals.json   # Global variables
        ├── cookies.json   # Cookie configurations
        ├── variables.json # Project variables
        ├── environments.json # Environment configurations
        ├── databases.json # Database connections
        ├── datadicts.json # Data dictionaries
        ├── models.json    # Data models
        ├── import/
        │   └── cron.json  # Cron job configurations
        ├── mock/
        │   ├── setting.json # Mock settings
        │   └── rules.json # Mock rules
        ├── resources/
        │   ├── functions.json # Custom functions
        │   ├── statusdict.json # Status code dictionaries
        │   ├── attribute.json # Resource attributes
        │   └── prostatus.json # Project statuses
        └── requests/
            └── {uuid}.json # Individual request configurations

Error Handling

1. Event-Based Errors

All errors (e.g., write failures, lock timeouts) are emitted via the error event. Always listen for this event to avoid unhandled promise rejections:

fileManager.on('error', (error) => {
  console.error(`[${new Date().toISOString()}] fs-data Error:`, {
    file: error.file,
    message: error.error.message,
    stack: error.error.stack
  });
});

2. Common Error Scenarios

| Error Type | Cause | Handling | |------------|-------|----------| | ENOENT | File/directory not found. | Returns defaultValue (if provided) for reads; creates parent directories for writes. | | SyntaxError (JSON) | Invalid JSON in a file. | Logs error and returns defaultValue (if strictMode: false); throws error (if strictMode: true). | | LockError | Failed to acquire file lock (multi-process conflict). | Logs warning and proceeds (write may still succeed if lock is released). | | PathError | Attempt to access a file outside baseDir. | Throws error (prevents directory traversal). |

Best Practices

1. Always Call close(): Stop watchers on application shutdown to avoid memory leaks.
2. Use strictMode in Development: Catch invalid JSON early by setting strictMode: true during development.
3. Limit Cache Size: Adjust coreCacheSize and requestCacheSize based on available memory (avoid caching large files).
4. Handle ENOENT Gracefully: Provide meaningful defaultValue for reads to avoid null checks.
5. Batch Operations: Use bulkWrite and loadProjectAll for multiple file operations to reduce I/O overhead.
6. Monitor Cache Hit Ratio: Add logging to track cache effectiveness (e.g., log when a file is read from cache vs. disk).

License

MIT License

Copyright (c) 2024 [Your Name/Organization]

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.