crewlyze

v1.0.7

Published

5 days ago

Autonomous Multi-Agent Business Intelligence and Data Engineering Platform

0High
0Medium
0Low

sowmiyan-s

ai data-analysis crewai fastapi python

📑 Table of Contents

🌍 Executive Vision & Enterprise Value

In the era of Big Data, the primary bottleneck is no longer data collection or storage—it is data interpretation. Traditional data analysis is heavily gated by technical requirements: writing complex Python scripts (Pandas, Numpy), manually cleaning messy datasets, identifying statistical correlations, and spending hours formatting PowerPoint presentations for stakeholders.

Crewlyze completely shatters this paradigm by introducing an Autonomous Multi-Agent Swarm. Powered by the orchestration logic of CrewAI and the universal model compatibility of LiteLLM, Crewlyze mimics an entire physical data department. It assigns distinct, hyper-specialized AI personas—such as the Data Quality Engineer, the Statistical Pattern Spotter, and the Senior Business Strategist—to independently audit, clean, analyze, and visualize your data in minutes.

💡 Why Crewlyze is a "Million-Dollar" Solution

For enterprises, data science teams, and C-Suite executives, Crewlyze offers compounding ROI:

Exponential Time Savings: What normally takes a data scientist 15-20 hours of Exploratory Data Analysis (EDA) and reporting is accomplished autonomously in under 3 minutes.
Democratization of Data: Non-technical executives can upload raw CSV/Excel files and instantly receive actionable business strategies—without needing to write a single line of SQL or Python.
Privacy-First Architecture: Unlike ChatGPT or Claude web interfaces, Crewlyze can run completely air-gapped on your local hardware using Ollama, ensuring highly confidential corporate data never leaves your internal network.

✨ Core Capabilities & Feature Deep-Dive

Crewlyze is not a thin wrapper around a language model; it is a complex orchestration engine equipped with secure, sandboxed code-execution tools.

🕵️ Profiling Agent: Scans the dataset for missing values, extreme outliers, cardinality issues, and data type inconsistencies.
🧹 Cleaning Agent: Autonomously executes Python code in a secure environment to drop highly-null columns, impute missing values (using mean/median/mode depending on distribution), and sanitize string formatting.
📊 Relational Analysis Agent: Uses statistical libraries (like scipy and statsmodels) to hunt for non-linear correlations, feature importances, and hidden multi-variable relationships.
💼 Senior Strategic Agent: Translates raw statistics ("Feature X has a 0.8 Pearson correlation with Target Y") into boardroom-ready language ("Increasing Feature X by 10% is projected to increase Revenue Y, suggesting immediate marketing budget reallocation").

Natural Language Queries: "What was our highest selling month in 2023?"
On-the-Fly Transformations: "Drop all rows where the 'Status' column is 'Pending'."
Dynamic Visuals: "Plot a scatter graph of Age vs Salary, colored by Department."

🏗️ Technical Architecture & Design

Crewlyze operates on a dual-engine architecture designed for maximum speed, security, and scalability.

⚙️ Backend: FastAPI & Python

Asynchronous Execution: FastAPI handles heavy LLM network requests concurrently, ensuring the UI remains snappy even during massive dataset ingestion.
Thread-Isolated Sandboxes: When the AI writes Python code to clean your data, Crewlyze executes it in isolated subprocesses. This prevents malicious code execution (e.g., os.system('rm -rf /')) and protects the host system.
Session Management: Project files, intermediate CSVs, JSON logs, and SQLite metadata are stored securely in ~/.crewlyze/data, separated by unique Session UUIDs.

🎨 Frontend: Vanilla JS & CSS Glassmorphism

We specifically avoided heavy frameworks like React or Angular to keep the application blisteringly fast and dependency-light.

State Management: Handled entirely via localized DOM manipulation and JavaScript Proxy objects.
Aesthetics: Utilizes state-of-the-art Glassmorphism CSS, smooth CSS variables for real-time dark/light mode toggling, and micro-animations to create a premium, "living" application feel.
Streaming UI: Real-time log streaming using Server-Sent Events (SSE) allows the user to watch the AI agents "think" and execute in real-time.

🛡️ Security & Privacy (Local Execution)

Data privacy is the number one concern for enterprise adoption of AI. Crewlyze tackles this head-on.

No Cloud Storage: Crewlyze does not upload your CSV files to any external database. All files remain strictly on the machine running the FastAPI server.
Local LLM Integration (Air-Gapped): By integrating Ollama, you can download open-source models (like Llama 3 8B or Mistral) directly to your machine. When configured in the Crewlyze Settings, your data is processed 100% offline. Zero bytes of data leave your network.
Secure API Key Storage: If using Cloud LLMs (OpenAI, Anthropic), API keys are stored exclusively in your browser's local IndexedDB/localStorage. They are transmitted securely only when actively making a request.

🚀 Installation & Local Deployment

Whether you are a developer, a data analyst, or an IT administrator, deploying Crewlyze is incredibly straightforward.

⚡ Option 1: NPM Install (Recommended for End-Users)

The fastest, most seamless way to get Crewlyze running on Windows, macOS, or Linux.

Prerequisites:

Node.js (v16+)
Python (3.10+)

# 1. Install Crewlyze globally via NPM
npm install crewlyze

# 2. Launch the application from anywhere in your terminal
crewlyze

🎉 Success: The backend server will automatically initialize, and your default web browser will open to http://localhost:8000.

🐳 Option 2: Docker (Enterprise & Cloud Ready)

Ideal for deploying on AWS, GCP, Azure, or keeping your local machine clean.

# Clone the repository
git clone https://github.com/sowmiyan-s/Multi-Agent-Data-Analysis-System-with-CrewAI.git
cd Multi-Agent-Data-Analysis-System-with-CrewAI

# Build and start the container in detached mode
docker-compose up --build -d

The application is now running securely inside a containerized Linux environment at http://localhost:8000.

💻 Option 3: Developer Source Setup

For contributors who want to modify the source code, adjust CSS, or engineer new CrewAI prompts.

# 1. Clone the repository
git clone https://github.com/sowmiyan-s/Multi-Agent-Data-Analysis-System-with-CrewAI.git
cd Multi-Agent-Data-Analysis-System-with-CrewAI

# 2. Create a virtual environment
python -m venv venv

# 3. Activate the environment
# For macOS/Linux:
source venv/bin/activate
# For Windows:
venv\Scripts\activate

# 4. Install required dependencies
pip install -r requirements.txt

# 5. Start the FastAPI development server
python main.py

🔑 API Guidelines & Model Configuration

Crewlyze's integration with LiteLLM means you have access to over 100+ language models globally. Configuration is handled entirely within the UI.

📸 Step-by-Step Platform Walkthrough

Our frontend is meticulously designed for high readability, minimal friction, and cognitive ease.

1️⃣ The Global Dashboard (Home Page)

Features: Instantly view your currently active AI model, track token usage, manage all historical projects, and effortlessly create new analysis sessions or import shared .zip project files.

2️⃣ Comprehensive Settings & Integrations

Features: A unified hub for API keys. If LiteLLM supports it, Crewlyze supports it. Dynamically search for niche providers (like Groq for ultra-fast inference) and bind them instantly.

3️⃣ The Project Workspace Hub

Features: Once a CSV is uploaded, you face a strategic choice. Will you interrogate the data manually via the Chat AI, or will you deploy the Data Analyst Crew for an exhaustive, autonomous audit?

4️⃣ Interactive Chat Assistant (Copilot)

Features: A ChatGPT-like interface injected directly with your data's context. Ask the AI to perform complex filtering, calculate standard deviations, or explain anomalies in plain English.

5️⃣ Multi-Agent Configuration Panel

Features: The cockpit for your AI team. Define the overarching business goal (e.g., "Maximize user retention"), select the target feature column, and hit run.

6️⃣ Live Agentic Processing (Terminal View)

Features: Total transparency. Watch the agents converse, write Python code, debug errors, and synthesize data in real-time. This ensures trust and verifiability in the AI's methodology.

7️⃣ Executive Business Insights Dashboard

Features: The magnum opus of the platform. The UI splits findings into three hyper-readable columns: Observation (The stat), Business Implication (Why it matters), and Actionable Strategy (What you should do today).

8️⃣ Natively Embedded Visualizations

Features: Interactive, dark-mode optimized Plotly charts generated to back up the strategic claims made by the AI. Hover over data points for exact values.

📊 Deliverables: Reports & Visualizations

Crewlyze ensures that the final mile of data analysis—the presentation—is handled with utmost professionalism.

📥 The Executive PDF Report

At the click of a button, Crewlyze aggregates the insights, renders the interactive Plotly graphs into high-resolution PNGs, and packages them into a stunning PDF document.

Dynamic Conclusions: The conclusion is not boiler-plate; it dynamically references the most critical business risk discovered during the run.
Branded Styling: Includes headers, footers, pagination, and color-coded risk alerts.

🎯 Click Here to View the Example Executive Report PDF

💾 Data Artifacts

cleaned_dataset.csv: Download the mathematically imputed and sanitized version of your dataset.
execution_trace.json: Download the complete LLM token trace for compliance and auditing purposes.

🏢 Industry Use Cases & ROI

Crewlyze is designed to be highly versatile across sectors.

| Industry Sector | Example Dataset | How Crewlyze Transforms the Workflow | Projected ROI | | :--- | :--- | :--- | :--- | | 🏥 Healthcare & Pharma | Patient clinical trials, vitals logs. | Secure, local analysis of patient outcomes. Agents identify correlations between drug dosage and recovery times without exposing PII to the cloud. | Saves months of manual biostatistical coding. | | 📈 Finance & FinTech | Ledger exports, stock price histories, transaction logs. | The Strategic Agent flags anomalies in transaction frequency, identifying potential fraud or optimizing portfolio rebalancing rules. | Drastic reduction in risk exposure and manual audit hours. | | 🛒 E-Commerce & Retail | Shopify/Amazon sales exports, customer churn data. | Instantly maps which demographic segments are driving the highest Customer Lifetime Value (CLV) and suggests targeted ad spend shifts. | Direct increase in ROAS (Return on Ad Spend). | | 🏭 Manufacturing & Supply Chain | Sensor IoT data, logistics timetables. | Profiles massive datasets to find failure thresholds in machinery, enabling preventative maintenance strategies. | Prevents costly factory downtime. |

📂 Project Structure & API Routes

For developers looking to understand the inner workings:

Crewlyze/
├── main.py                 # FastAPI core, route definitions, SSE streaming logic
├── crew.py                 # CrewAI orchestration, agent definitions, and task routing
├── requirements.txt        # Python dependency manifest
├── package.json            # NPM configuration for global installation
├── README.md               # You are here
├── config/                 # YAML configurations for prompts, roles, and settings
├── tools/                  # Python sandboxing tools (dataset_tools.py)
├── ui/                     # PDF Generation scripts (export.py) using ReportLab
├── web/                    # 100% Vanilla Frontend (HTML, CSS, JS)
│   ├── index.html          # Main SPA Entrypoint
│   ├── style.css           # Glassmorphism design system
│   └── app.js              # State management and DOM routing
└── assets/                 # Branding and Documentation Screenshots

🔌 Core API Routes

POST /api/upload: Handles multipart form data for initial CSV ingestion.
GET /api/stream: Opens a Server-Sent Events (SSE) connection for real-time terminal logs.
POST /api/analyze: Triggers the async crew.py background task.
POST /api/chat: Handles conversational queries against the dataset context.
GET /api/download_pdf: Assembles and returns the binary PDF stream.

🛣️ Roadmap & Future Scope

Crewlyze is actively maintained. Here is our vision for the next 12-18 months:

🛠️ Comprehensive Troubleshooting

Encountering an issue? Check our detailed solutions below:

🤝 Contributing & Development

Crewlyze thrives on open-source collaboration! Whether you are fixing typos, designing better CSS, or writing advanced CrewAI prompt templates, your help is welcome.

Fork the Repository on GitHub.
Clone your Fork locally.
Create a Feature Branch: git checkout -b feature/AmazingNewFeature
Commit your Changes: git commit -m 'Add AmazingNewFeature'
Push to the Branch: git push origin feature/AmazingNewFeature
Open a Pull Request against the main Multi-Agent-Data-Analysis-System-with-CrewAI repository.

Note: Please ensure all Python code passes standard flake8 linting and that no sensitive API keys are accidentally committed!

👥 Meet the Team & Contributors

A massive thank you to the brilliant minds building the future of autonomous data analysis.

✨ Core Contributors

💖 Sponsorship & Support

If Crewlyze has saved you or your enterprise countless hours of manual data labor, please consider supporting the project. Open-source development requires massive amounts of coffee, API testing credits, and server hosting!

Your sponsorship directly funds the development of v2.0 (Database Integrations) and ensures the project remains free and open forever.

Follow the Creator:

🌐 GitHub: @sowmiyan-s
💼 LinkedIn: Sowmiyan S

📜 License & Legal

Crewlyze is proudly open-sourced software licensed under the MIT License. You are free to use, modify, and distribute this software in both commercial and non-commercial settings.

Disclaimer: Always ensure you comply with your organization's internal data privacy policies and GDPR/HIPAA regulations when uploading sensitive datasets to cloud LLM providers (OpenAI, Google, Anthropic). For maximum security with confidential data, strictly utilize local Ollama instances.