Flow-Frame

Flow-Frame is an advanced automation platform designed to orchestrate complex workflows involving AI, media processing, browser interactions, and email management.

It acts as a central nervous system for automation, connecting LLMs (OpenAI) with real-world interfaces (Gmail, Chrome, File System) and heavy computation tasks (Image/Audio processing).

🚀 Quick Start

1. Install Dependencies:

   npm install

2. Setup Environment: Create a .env file with your API keys.
3. Start Server:

   npm start

🧪 Testing

This project uses Jest for unit and integration testing.

Running Tests

To run the automated test suite:

npm test

Writing Tests

• Place unit tests alongside source files (e.g., src/utils.test.ts).
• Tests should use import syntax (ESM).
• Existing manual testing scripts (e.g. *.manual-test.ts) are excluded from the automated runner.

📚 Documentation

Detailed documentation is available in the docs/ folder:

• Setup & Installation: Configuration including src/config.json and GCP.
• Architecture: System design, services, and resiliency.
• API Reference: Endpoints for workflows, config files, media.
• Browser Automation: robotjs integration and extension details.
• Email Automation: Gmail polling and filtering.
• Workflows: How to build and run workflows.

🔧 Tool Reference (for Agent Integration)

flow-frame-core is designed to be used as a tool library by AI agents. The complete tool glossary is in TOOLS.md, covering every function and REST endpoint organized by capability:

| Category | Tools | Integration | |----------|-------|-------------| | LLM / AI | runPrompt, runPromptGrok, runImagePromptGrok | Import or POST /api/prompts/chat | | Prompt Chains | executeChain — serial/parallel steps with auto-repair | Import | | Workflow Engine | executeFlow — 25+ node types in a DAG | Import or REST | | JSON Utilities | extractJSON, cleanJSONString, containsJSON | Import | | Schema & Code Gen | inferSchema, generateJsTransformFromPrompt | Import | | Prompt Optimization | runAutoPromptSearch — iterative prompt refinement | Import | | Image Analysis | classifyImageQuery, findImageInImage | Import or REST | | Audio / TTS | processAudio — ElevenLabs TTS | Import or POST /generateAudio | | PDF Processing | extractPdf — text, images, page renders | Import or GET /pdf/extract | | Email | extractGmailBodyText, polling endpoints | Import or REST | | Web Scraping | crawl — full-site depth-first crawl | Import or POST /scrape | | Storage | getItem, setItem, profile management | Import or REST | | Queue | queueManager — serial job queue with events | Import or REST | | File System | 30+ file utilities (read, write, download, zip) | Import or REST | | Templates | generateTextFromTemplate — LLM variable generation | Import or REST | | Browser / Desktop | Chrome control, mouse, keyboard, app management | Import or REST | | Screenshots | captureScreenshotBase64, captureFullScreenshot | Import or REST | | Self-Learning Vision | Ingest, recognize, discover screens, transition graphs | Import or REST | | UI Planning | UiPlanner, StepByStepAiPlanner — NL → automation steps | Import | | Workflows CRUD | Create, load, save, validate .workflow files | REST | | Extensions | Plugin discovery and execution via api.json | Import or REST | | Media | Video/audio dimensions, YOLO training data | Import or REST | | Config | File-based config CRUD with backup/validation | REST |

See TOOLS.md for full function signatures, parameters, return types, and REST endpoint details.

🌟 Key Features

• Agentic UI Planning: Deeply integrated AI that maps web pages and uses robotjs to simulate human clicks and typing.
• Dynamic Workflows: Can materialize file structures from JSON and execute them (build-workflow.js).
• Production Resilience: Global error handlers prevent server crashes (server.js).
• Media Pipeline: Native support for processing Audio, Images, and PDFs.

📂 Project Structure

• src/server.js: Main entry point with error handling.
• src/operations.js: Low-level OS automation (mouse/keyboard).
• src/configEndpoints.js: API for managing file-based configs.
• src/services/: Business logic and AI services.
• src/extension/: Chrome extension for browser integration.
• src/gmail-poller/: Email automation tools.

License

Private / Proprietary.