@memberjunction/doc-utils

A TypeScript library for dynamically retrieving, parsing, and caching MemberJunction library documentation from the official documentation website. Designed to support AI models, code generation tools, and other documentation-driven features that need structured access to MJ library metadata.

Overview

The @memberjunction/doc-utils package provides a singleton DocumentationEngine that loads library and library item metadata from the MemberJunction entity system, then fetches the corresponding HTML documentation pages from the official MJ documentation site (https://memberjunction.github.io/MJ/). Parsed content is cached in memory for efficient repeated access.

This package is particularly useful for:

• Feeding structured library documentation to AI models for context-aware code generation
• Building developer tools that reference MJ API documentation
• Providing searchable documentation indexes for MJ packages and their exported items

Architecture

graph TB
    subgraph DocUtils["DocUtils Package"]
        DE["DocumentationEngine<br/>(Singleton)"]
        LEE["MJLibraryEntityExtended"]
        LIEE["MJLibraryItemEntityExtended"]
    end

    subgraph MJCore["@memberjunction/core"]
        BE["BaseEngine"]
        BEnt["BaseEntity"]
    end

    subgraph CoreEntities["@memberjunction/core-entities"]
        LE["LibraryEntity"]
        LIE["LibraryItemEntity"]
    end

    subgraph External["External Services"]
        DB[("MJ Database<br/>Libraries &<br/>Library Items")]
        DOCS["MJ Docs Site<br/>memberjunction.github.io"]
    end

    DE -->|extends| BE
    LEE -->|extends| LE
    LIEE -->|extends| LIE
    LE -->|extends| BEnt
    LIE -->|extends| BEnt

    DE -->|loads metadata| DB
    DE -->|fetches HTML| DOCS
    DE -->|manages| LEE
    DE -->|manages| LIEE
    LEE -->|contains| LIEE

    style DE fill:#2d6a9f,stroke:#1a4971,color:#fff
    style LEE fill:#2d8659,stroke:#1a5c3a,color:#fff
    style LIEE fill:#2d8659,stroke:#1a5c3a,color:#fff
    style BE fill:#7c5295,stroke:#563a6b,color:#fff
    style BEnt fill:#7c5295,stroke:#563a6b,color:#fff
    style LE fill:#7c5295,stroke:#563a6b,color:#fff
    style LIE fill:#7c5295,stroke:#563a6b,color:#fff
    style DB fill:#b8762f,stroke:#8a5722,color:#fff
    style DOCS fill:#b8762f,stroke:#8a5722,color:#fff

Data Flow

The engine follows a two-phase loading process: first it loads entity metadata from the database, then it fetches and parses HTML documentation for each library item.

sequenceDiagram
    participant App as Application
    participant DE as DocumentationEngine
    participant DB as MJ Database
    participant Site as Docs Website

    App->>DE: Config(forceRefresh, contextUser)
    DE->>DB: Load Libraries (entity metadata)
    DB-->>DE: MJLibraryEntityExtended[]
    DE->>DB: Load Library Items (entity metadata)
    DB-->>DE: MJLibraryItemEntityExtended[]

    Note over DE: AdditionalLoading phase

    loop For each Library
        loop For each Item in Library
            DE->>DE: Build URL from library name + item type
            DE->>Site: GET /{type}/{sanitized_name}.{ItemName}.html
            Site-->>DE: HTML response
            DE->>DE: Parse HTML via JSDOM (extract div.col-content)
            DE->>DE: Cache HTMLContent and URL on item
        end
    end

    DE-->>App: Config complete, data cached
    App->>DE: Access Libraries / LibraryItems
    DE-->>App: Cached entity data with HTML content

Class Hierarchy

classDiagram
    class BaseEngine {
        <<abstract>>
        +Load(configs, provider, forceRefresh, contextUser)
        #AdditionalLoading(contextUser)*
    }

    class DocumentationEngine {
        +Instance$ DocumentationEngine
        +Libraries MJLibraryEntityExtended[]
        +LibraryItems MJLibraryItemEntityExtended[]
        +Config(forceRefresh, contextUser, provider)
        #AdditionalLoading(contextUser)
        #GetContent(url, rootSelector) string
        #fetchDocumentation(url) string
        #parseDocumentation(html, rootSelector) string
    }

    class BaseEntity {
        <<abstract>>
        +Get(fieldName)
        +Set(fieldName, value)
        +Save()
    }

    class LibraryEntity {
        +ID string
        +Name string
        +Description string?
        +Status "Active" | "Disabled" | "Pending"
        +TypeDefinitions string?
        +SampleCode string?
    }

    class MJLibraryEntityExtended {
        +Items MJLibraryItemEntityExtended[]
    }

    class LibraryItemEntity {
        +ID string
        +Name string
        +LibraryID string
        +Type "Class"|"Function"|"Interface"|"Module"|"Type"|"Variable"
        +Library string
    }

    class MJLibraryItemEntityExtended {
        +URL string
        +HTMLContent string
        +TypeURLSegment string
    }

    BaseEngine <|-- DocumentationEngine
    BaseEntity <|-- LibraryEntity
    BaseEntity <|-- LibraryItemEntity
    LibraryEntity <|-- MJLibraryEntityExtended
    LibraryItemEntity <|-- MJLibraryItemEntityExtended
    DocumentationEngine o-- MJLibraryEntityExtended
    DocumentationEngine o-- MJLibraryItemEntityExtended
    MJLibraryEntityExtended o-- MJLibraryItemEntityExtended

    style DocumentationEngine fill:#2d6a9f,stroke:#1a4971,color:#fff
    style MJLibraryEntityExtended fill:#2d8659,stroke:#1a5c3a,color:#fff
    style MJLibraryItemEntityExtended fill:#2d8659,stroke:#1a5c3a,color:#fff
    style BaseEngine fill:#7c5295,stroke:#563a6b,color:#fff
    style BaseEntity fill:#7c5295,stroke:#563a6b,color:#fff
    style LibraryEntity fill:#7c5295,stroke:#563a6b,color:#fff
    style LibraryItemEntity fill:#7c5295,stroke:#563a6b,color:#fff

Installation

npm install @memberjunction/doc-utils

Dependencies

MemberJunction packages:

| Package | Purpose | |---------|---------| | @memberjunction/core | Base engine class, metadata provider, user context | | @memberjunction/core-entities | LibraryEntity and LibraryItemEntity base classes | | @memberjunction/global | @RegisterClass decorator for entity registration |

External packages:

| Package | Purpose | |---------|---------| | axios | HTTP requests to the documentation website | | jsdom | HTML parsing and DOM manipulation for content extraction |

Usage

Initializing the Engine

The DocumentationEngine uses the singleton pattern. Call Config() once to load metadata and documentation content before accessing data.

import { DocumentationEngine } from '@memberjunction/doc-utils';
import { UserInfo } from '@memberjunction/core';

// Get the singleton instance
const docEngine = DocumentationEngine.Instance;

// Server-side: pass contextUser for proper data isolation
const contextUser: UserInfo = /* from your authentication context */;
await docEngine.Config(false, contextUser);

// Client-side: contextUser is optional
await docEngine.Config();

Accessing Libraries and Items

After configuration, libraries and their items are available through cached properties.

const docEngine = DocumentationEngine.Instance;

// Get all libraries
const libraries = docEngine.Libraries;
console.log(`Loaded ${libraries.length} libraries`);

// Find a specific library
const coreLib = libraries.find(lib => lib.Name === '@memberjunction/core');
if (coreLib) {
    console.log(`${coreLib.Name} (${coreLib.Status})`);
    console.log(`Items: ${coreLib.Items.length}`);

    // Iterate through items in the library
    for (const item of coreLib.Items) {
        console.log(`  ${item.Type}: ${item.Name}`);
        console.log(`  URL: ${item.URL}`);
        console.log(`  Content length: ${item.HTMLContent.length} chars`);
    }
}

Accessing All Library Items

The flat LibraryItems array provides access to all items across all libraries.

const docEngine = DocumentationEngine.Instance;

// Get all library items regardless of library
const allItems = docEngine.LibraryItems;

// Filter by type
const allClasses = allItems.filter(item => item.Type === 'Class');
const allInterfaces = allItems.filter(item => item.Type === 'Interface');

// Find a specific item by name and library
const baseEntity = allItems.find(
    item => item.Name === 'BaseEntity' && item.Library === '@memberjunction/core'
);

if (baseEntity) {
    console.log(`Documentation URL: ${baseEntity.URL}`);
    console.log(`HTML Content: ${baseEntity.HTMLContent}`);
}

Force Refreshing Documentation

Pass true to Config() to reload all metadata from the database and re-fetch documentation content.

const docEngine = DocumentationEngine.Instance;

// Force a complete reload
await docEngine.Config(true);

API Reference

DocumentationEngine

The primary class providing access to MJ library documentation. Extends BaseEngine and uses the singleton pattern.

| Member | Type | Description | |--------|------|-------------| | Instance (static) | DocumentationEngine | Returns the singleton instance | | Libraries | MJLibraryEntityExtended[] | All loaded libraries with their items | | LibraryItems | MJLibraryItemEntityExtended[] | Flat list of all library items across all libraries | | Config(forceRefresh?, contextUser?, provider?) | Promise<void> | Initializes the engine: loads metadata, fetches and parses documentation |

MJLibraryEntityExtended

Extended entity class for libraries. Registered via @RegisterClass(BaseEntity, "Libraries").

| Member | Type | Description | |--------|------|-------------| | ID | string | Unique identifier | | Name | string | Library package name (e.g., @memberjunction/core) | | Description | string \| null | Human-readable description | | Status | 'Active' \| 'Disabled' \| 'Pending' | Library availability status | | TypeDefinitions | string \| null | Type/function definitions for reference by humans and AI | | SampleCode | string \| null | Usage examples for the library | | Items | MJLibraryItemEntityExtended[] | All items belonging to this library |

MJLibraryItemEntityExtended

Extended entity class for individual library items (classes, interfaces, functions, etc.). Registered via @RegisterClass(BaseEntity, "Library Items").

| Member | Type | Description | |--------|------|-------------| | ID | string | Unique identifier | | Name | string | Item name (e.g., BaseEntity, RunView) | | LibraryID | string | Foreign key to the parent library | | Library | string | Denormalized library name from the view | | Type | 'Class' \| 'Function' \| 'Interface' \| 'Module' \| 'Type' \| 'Variable' | The kind of exported item | | URL | string | Generated documentation URL for this item | | HTMLContent | string | Parsed HTML content from the documentation page | | TypeURLSegment | string (getter) | URL path segment derived from Type (e.g., classes, interfaces) |

URL Generation

Documentation URLs are constructed automatically using the pattern:

https://memberjunction.github.io/MJ/{typeSegment}/{sanitizedLibraryName}.{ItemName}.html

Where:

• {typeSegment} is the pluralized, lowercase item type (classes, interfaces, functions, modules, types, variables)
• {sanitizedLibraryName} is the library name with @, ., /, and \ characters replaced by _
• {ItemName} is the exact name of the exported item

For example, BaseEntity (a Class) in @memberjunction/core resolves to:

https://memberjunction.github.io/MJ/classes/_memberjunction_core.BaseEntity.html

URL Routing by Item Type

graph LR
    Item["Library Item"]

    Item -->|Class| C["/classes/"]
    Item -->|Interface| I["/interfaces/"]
    Item -->|Function| F["/functions/"]
    Item -->|Module| M["/modules/"]
    Item -->|Type| T["/types/"]
    Item -->|Variable| V["/variables/"]

    style Item fill:#2d6a9f,stroke:#1a4971,color:#fff
    style C fill:#2d8659,stroke:#1a5c3a,color:#fff
    style I fill:#2d8659,stroke:#1a5c3a,color:#fff
    style F fill:#2d8659,stroke:#1a5c3a,color:#fff
    style M fill:#b8762f,stroke:#8a5722,color:#fff
    style T fill:#b8762f,stroke:#8a5722,color:#fff
    style V fill:#b8762f,stroke:#8a5722,color:#fff

Integration with MemberJunction

This package integrates with the broader MemberJunction ecosystem through several mechanisms:

• Entity System: Uses @RegisterClass to register MJLibraryEntityExtended and MJLibraryItemEntityExtended as entity subclasses, ensuring the MJ class factory returns the extended types when loading Libraries and Library Items.
• BaseEngine Pattern: Extends BaseEngine with the Config() / AdditionalLoading() lifecycle, allowing consistent initialization and caching across all MJ engine classes.
• User Context: Supports contextUser for proper data isolation and security when running on the server side.
• Metadata Provider: Accepts an optional IMetadataProvider for environments with custom metadata access patterns.

Build

# Build the package
cd packages/DocUtils
npm run build

# Development mode with file watching
npm start

Source Files

| File | Description | |------|-------------| | src/index.ts | Public API surface -- re-exports from Engine.ts | | src/Engine.ts | Core implementation: DocumentationEngine, MJLibraryEntityExtended, MJLibraryItemEntityExtended |

Notes

• Documentation is fetched from https://memberjunction.github.io/MJ/ using axios HTTP GET requests.
• HTML content is parsed with jsdom, extracting the inner HTML of the div.col-content element from each page.
• All fetched content is cached in memory after the initial Config() call. Subsequent calls to Config() without forceRefresh=true return immediately.
• Library names containing special characters (@, ., /, \) are sanitized by replacing them with underscores for URL compatibility.
• If a documentation page cannot be fetched (network error, 404, etc.), the HTMLContent for that item is set to an error or placeholder string rather than throwing an exception.