🚀 SchemaShift: Dynamic Code-Drift & Autonomous Self-Healing Plugin System

SchemaShift is a state-of-the-art, framework-agnostic plugin framework designed to protect third-party application platforms against payload drift, environment fragmentation, and code integrity degradation.

By utilizing a Hybrid Agency Architecture, SchemaShift combines local, deterministic symbolic agents (for sub-second diagnostics and safe filesystem recovery) with generative connectionist models (for explaining schema anomalies and synthesizing database migrations).

🏗️ Core Architecture & Design Philosophy

Traditional plugins assume a single host environment (e.g. tightly coupling to Vite, React, or specific server architectures). This makes them brittle and impossible to scale across modern microservice environments.

SchemaShift breaks this dependency with a strict Core vs. Adapter Strategy:

graph TD
    subgraph Core Layer [SchemaShift Core]
        A["schemashift-core (Agnostic JS/TS Engine)"]
        B["schema-engine (AST, diffing, patch math)"]
    end

    subgraph Adapter Layer [SchemaShift Adapters]
        C["schemashift-activepieces-adapter (Microservice sandbox integrations)"]
        D["Next.js / Serverless Adapter"]
        E["Express / Node.js Middleware Adapter"]
        F["Vite / Pure Client Adapter"]
        G["CLI / Sidecar Adapter"]
    end

    CoreLayer --> B
    B --> AdapterLayer
    C -.-> AP["Activepieces Host (Bun/PGLite)"]
    D -.-> NEXT["Next.js Backend (Vercel)"]
    E -.-> EXP["Express API (Node.js)"]
    F -.-> VIT["Browser SPA (Vite Client)"]
    G -.-> SHELL["Terminal Shell"]

Key Pillars:

1. Zero Host Assumptions: The core calculation engines have no dependencies on Vite, React, or browser-specific objects (window, document).
2. Compile-Once, Adapt-Anywhere: The core processes the logical drifts and git hunks, while adapters translate those outputs into the specific host's UI components, storage layers, and security frameworks.
3. Decoupled Telemetry: Installation and environment health checks execute on independent diagnostic backplanes.

📁 Monorepo Workspace Directory Layout

SchemaShift is organized as a high-velocity pnpm monorepo. Below is a map of the internal package ecosystem:

| Package / Folder | Tier | Core Responsibility | | :--- | :--- | :--- | | packages/schemashift-core | Core | Framework-agnostic calculus of schema mismatches, diff generation, and code-drift calculations. | | packages/schema-engine | Core | Low-level AST parsing, diff hunk tracking, and logical execution math. | | packages/schemashift-activepieces-adapter | Adapter | Integrates the core engine with Activepieces. Maps flows, registers actions, and controls local telemetry. | | packages/schemashift-installer | Tooling | A developer-focused CLI utility used to inspect environments and trigger force uploads to remote API hosts. | | packages/schemashift-agent | AI | The cognitive agent layer housing prompts, prompt injection sanitizers, and LLM orchestration schemas. | | packages/schemashift-agentify | AI | An automated decorator utility that adds agentic introspection to existing third-party JavaScript plugins. | | packages/design-system | Core UI | Shared design tokens, Tailwind configurations, and primitive HSL variables for glassmorphism panels. |

🧠 The Hybrid Agency Engine (Symbolic vs. Generative)

SchemaShift separates Agency into two distinct tiers to ensure high security, offline capability, and zero latency during mission-critical tasks:

flowchart LR
    A[Perception: Error Log Stream] --> B{Heuristic Match?}
    B -- YES (Cache/File Anomaly) --> C[Local Symbolic Agent]
    B -- NO (Text Analysis Request) --> D[Generative AI Agent]
    
    C --> E[Execute fs.rm & Re-Link]
    D --> F[Ask for connection credentials]
    F --> G[Secure Direct API request to LLM]

Tier 1: The Local Symbolic Agent (Deterministic & Safe)

During installation, local diagnostics, and self-healing, SchemaShift deliberately avoids LLM API calls.

• Why? Avoids the paradox of needing an LLM connection to install the LLM piece, maintains zero execution latency, guarantees 100% path accuracy, and eliminates cost/hallucination risk.
• Mechanism: RegEx patterns analyze stdout/stderr strings dynamically on the client, mapping matching conflicts to localized recovery steps. System repairs are carried out securely using targeted filesystem functions.

Tier 2: The Generative AI Agent (Cognitive & Adaptive)

Once successfully installed, SchemaShift leverages generative AI models to handle cognitive, high-context tasks:

• Capabilities: Explaining complex schema changes in human-readable terms, predicting drifted payloads, and generating database migration scripts on the fly.
• LLM Strategy: The adapter uses the host's native credentials model to secure requests:

| Host Runtime | LLM Credentials Strategy | Security Isolation | | :--- | :--- | :--- | | Activepieces | Decrypted DB Connections: Decrypts connections (Property.CustomAuth) in-memory and injects them into the sandboxed worker context.auth. | Sandboxed runner isolation | | Next.js | Server Environment Variables: Uses secure Next.js server route handlers to query process.env.OPENAI_API_KEY. | API keys never hit the browser | | Express | System Environment: Reads keys from system variables or local config files directly in the Express Node process. | Protected by host operating environment | | Vite SPA | Dev Proxy / Local Session: Uses Vite proxies during development, and temporary in-memory browser sessionStorage in production. | Cleaned up when browser tab is closed |

🧹 Stale Cache & Hoisting Resolution (The Self-Healing Wizard)

One of SchemaShift's most advanced autonomous capabilities is Self-Healing Installation Diagnostics, which resolves the common monorepo hoisting and caching conflicts:

🔍 The Root Cause of Stale Installs

When a user uninstalls a piece in Activepieces:

1. The database metadata is deleted, making the platform believe the piece is gone.
2. However, the filesystem cache directories (activepieces/cache/v11/common/pieces/...) and the ready check files are left behind.
3. When the user attempts to re-install, Activepieces' piece-installer.ts detects the old folder/files, assumes the piece is already fully installed, and skips the fresh download and linking phase (bun install).
4. The system is left in a "split-brain" state, throwing PieceNotFoundError or ENGINE_OPERATION_FAILURE when running flows.

🛡️ The Self-Healing Diagnostic Overlay

SchemaShift resolves this cleanly inside InstallPieceDialog.tsx:

1. Telemetry Intercept: When installation fails, the dialog parses logs for indicators like "PieceNotFoundError", "ENGINE_OPERATION_FAILURE", or "cache/v11/common".
2. Glassmorphic Diagnostics Overlay: Instead of showing raw error blocks, it mounts a premium, Rocket Orange (#ff6b35) accented SchemaShift Autonomous Diagnostics overlay:

[!IMPORTANT] State Mismatch Detected The platform database indicates this piece is not installed, but stale caches exist under the worker path, which blocks standard dependency linking.

3. Auto-Heal & Clean Install: Clicking "Auto-Heal Cache & Retry Clean Install" triggers the installer with the replaceExisting: true parameter:
   • It resets the telemetry feed, reporting its autonomous cleanup operations.
   • On the backend, pieceInstallService.ts intercepts replaceExisting to execute fs.rm recursively on the stale disk directories (cache/v11/common/pieces/ and cache/v11/custom_pieces/...).
   • It then forces a completely clean dependency-linking sequence, successfully compiling the package.

🚀 Quickstart & Developer Runbook

Prerequisites

• Bun Runtime (v1.0 or higher)
• pnpm (v8.0 or higher)
• Node.js (v18 or higher)

Installation

Clone the workspace and initialize monorepo dependencies:

# In project root
pnpm install

Developing the Activepieces Adapter

To launch the complete dev stack (backend API, worker process, and frontend Vite server):

cd activepieces
npm run dev

Verifying Type-Safety (Zero TypeScript Errors Policy)

All developers must ensure the codebase compiles cleanly with zero TypeScript errors. Run the following check before submitting commits:

# In activepieces subdirectory
npx tsc --noEmit -p packages/web/tsconfig.json

Running Smoke Tests

Run end-to-end integration tests using the pre-configured Playwright suite:

cd activepieces
AP_HOST_URL=http://localhost:4200 AP_EMAIL=admin@local AP_PASSWORD=adminpw npx playwright test tests/smoke/schemashift-ui-install.playwright.spec.ts

💻 Running the CLI Tools

SchemaShift provides two developer-facing CLI packages for repository auditing, self-healing installation planning, and dependency wrapping.

🛠️ Option 1: Workspace pnpm Scripts (Recommended)

The easiest way is to run the pre-configured TypeScript execution scripts from the repository root:

• SchemaShift Installer (@schemashift/installer):

  • Scan Environment:

    pnpm schemashift:scan [-r <root_dir>] [-o <output_json>]

  • Build Integration Plan:

    pnpm schemashift:plan [-r <root_dir>] [-d <output_dir>]

  • Perform Self-Healing Install:

    pnpm schemashift:install --host-url <url> --token <token> [--enable-plugin-support]

  • Verify Host Status:

    pnpm schemashift:verify --host-url <url> --token <token>

• SchemaShift Agentify Wrapper Tool (@schemashift/agentify):

  • Scan Workspace Dependencies:

    pnpm agentify:scan

  • Inspect Specific Package:

    pnpm agentify:inspect <package_name>

  • Generate Safe Agent Wrappers:

    pnpm agentify:generate <package_name>

  • Validate Generated Wrapper:

    pnpm agentify:validate <package_name>

📦 Option 2: Compiling and Running Compiled Binaries

Alternatively, compile the monorepo first and execute the build distribution files or global command lines:

# 1. Compile TS workspace files
pnpm build

# 2. Invoke installer or agentify tools directly
node packages/schemashift-installer/dist/index.js --help
node packages/schemashift-agentify/dist/cli.js --help

🔒 System Integrity Principles

1. User Gating: Self-healing recovery runs autonomously but requires explicit user consent via the UI before executing destructive cache purges.
2. Visual Consistency: Diagnostic cards must feature a uniform border design (no left-border color accents) and adhere to the Rocket Orange theme (#ff6b35).
3. No Placeholders: Telemetry logs and error indicators must display live system statistics, avoiding mock descriptions or simulated values.