Data Science Agent

Data Science Agent is an AI-assisted data science platform with a command-line interface and a FastAPI backend. It supports dataset profiling, exploratory analysis, model training, forecasting, NLP workflows, business intelligence summaries, and iterative follow-up analysis using session memory.

This repository is configured for two primary usage patterns:

• Local development from source
• NPM-based CLI execution via the ds-agent-cli launcher (published package)

Table of Contents

• Overview
• Architecture
• Prerequisites
• Installation and Run Options
• CLI Command Reference
• API Reference
• Configuration
• Frontend Serving Behavior
• Dependency Strategy
• Troubleshooting
• Development and Deployment

Overview

Core capabilities include:

• Dataset profiling and quality checks
• End-to-end analysis pipelines from natural language tasks
• Model training and tuning workflows
• Time-series forecasting
• NLP workflows
• Executive brief and judge-mode workflows
• Session-based follow-up analysis
• REST API with streaming progress endpoints

Architecture

High-level components:

• Python CLI: src/cli.py
• Workflow helpers: src/cli_workflows.py
• API server: src/api/app.py
• Tool modules: src/tools/*
• NPM bootstrap launcher: bin/ds-agent.js

The NPM launcher resolves Python, creates or reuses an environment, installs dependency groups on demand, and then executes the Python CLI.

Prerequisites

Required:

• Node.js 18+ (for NPM launcher usage)
• Python 3.11 to 3.13 for managed venv mode
• Python 3.11 specifically for local311 mode

Recommended:

• pip up to date
• Virtual environment for source development

API key requirements:

• At least one supported LLM provider key:
  • MISTRAL_API_KEY
  • GROQ_API_KEY
  • GOOGLE_API_KEY (or GEMINI_API_KEY)

Installation and Run Options

Option A: Global NPM Install

Install once globally:

npm install -g ds-agent-cli

Run:

ds-agent quickstart
ds-agent auth --provider mistral
ds-agent analyze data.csv --target price --task "Predict price"

Option B: No Global Install (Recommended for locked-down machines)

Run with npx:

npx ds-agent-cli quickstart
npx ds-agent-cli auth --provider mistral
npx ds-agent-cli analyze data.csv --target price --task "Predict price"

Run with npm exec:

npm exec --package ds-agent-cli -- ds-agent quickstart

Environment Modes in NPM Launcher

The launcher supports two modes:

1. Managed virtual environment mode (default)
2. Local Python 3.11 mode

Managed venv mode:

DS_AGENT_ENV_MODE=venv npx ds-agent-cli quickstart

Local Python 3.11 mode:

DS_AGENT_ENV_MODE=local311 DS_AGENT_PYTHON=python3.11 npx ds-agent-cli quickstart

Windows launcher shorthand (auto-detect Python via py):

$env:DS_AGENT_ENV_MODE="local311"
$env:DS_AGENT_PYTHON="py"
npx ds-agent-cli quickstart

Pin exact Python version through launcher args:

$env:DS_AGENT_ENV_MODE="local311"
$env:DS_AGENT_PYTHON="py -3.11"
npx ds-agent-cli quickstart

Windows PowerShell equivalent:

$env:DS_AGENT_ENV_MODE="local311"
$env:DS_AGENT_PYTHON="C:\\Python311\\python.exe"
npx ds-agent-cli quickstart

Windows cmd.exe equivalent:

set DS_AGENT_ENV_MODE=local311 && set DS_AGENT_PYTHON=C:\Python311\python.exe && npx ds-agent-cli quickstart

Option C: Run from Source

1. Create and activate a virtual environment

macOS/Linux:

python3 -m venv .venv
source .venv/bin/activate

Windows PowerShell:

py -3.11 -m venv .venv
.\.venv\Scripts\Activate.ps1

2. Install dependencies

pip install --upgrade pip
pip install -r requirements.txt

3. Configure environment

cp .env.example .env

4. Run CLI

python src/cli.py quickstart

5. Configure auth in CLI

python src/cli.py auth --provider mistral

CLI Command Reference

Top-level commands currently available:

• quickstart
• auth
• analyze
• executive-brief
• doctor
• judge-mode
• profile
• eda
• report
• compare
• bi
• chat
• plot
• forecast
• nlp
• pipeline
• tune
• clean
• train
• cache-stats
• clear-cache

Session commands:

• sessions list
• sessions resume <session_id>
• sessions delete <session_id>
• sessions clear

Get command help:

ds-agent --help
ds-agent <command> --help

API Reference

Run API server from source:

python src/api/app.py

Alternative run command:

uvicorn src.api.app:app --host 0.0.0.0 --port 8080

Default local URL:

• http://localhost:8080

Interactive docs:

• http://localhost:8080/docs

Key endpoints:

• GET /api/health
• GET /health
• POST /run
• POST /run-async
• GET /api/progress/{session_id}
• GET /api/progress/stream/{session_id}
• POST /profile
• POST /chat
• GET /tools
• GET /api/files
• GET /api/files/{file_id}
• DELETE /api/files/{file_id}
• GET /api/files/stats/{user_id}
• POST /api/files/extend/{file_id}
• POST /api/export/huggingface
• GET /outputs/{file_path:path}

Simple run example:

curl -X POST http://localhost:8080/run \
  -F "[email protected]" \
  -F "task_description=Analyze this dataset and predict the target column" \
  -F "target_col=target"

Configuration

Core Environment Variables

From .env.example:

• GOOGLE_API_KEY
• LLM_PROVIDER
• REASONING_EFFORT
• CACHE_DB_PATH
• CACHE_TTL_SECONDS
• OUTPUT_DIR
• DATA_DIR
• MAX_PARALLEL_TOOLS
• MAX_RETRIES
• TIMEOUT_SECONDS

Additional provider keys used by auth/provider logic:

• MISTRAL_API_KEY
• GROQ_API_KEY
• GEMINI_API_KEY

NPM Launcher Environment Variables

• DS_AGENT_HOME: custom home directory for launcher state and managed venv
• DS_AGENT_ENV_MODE: venv or local311
• DS_AGENT_PYTHON: override Python command used by launcher (supports args, e.g. py -3.11)
• DS_AGENT_INSTALL_ALL: force full requirements.txt sync when set to true-like value
• DS_AGENT_VERBOSE_PROGRESS: enables verbose progress behavior

Frontend Serving Behavior

The API can serve a frontend build when available.

• Expected build directory: FRRONTEEEND/dist
• If present, GET / and client routes serve frontend assets
• If absent, GET / returns an API-first JSON response with guidance to use /docs

If the web interface does not appear, verify that frontend build artifacts exist at FRRONTEEEND/dist.

Dependency Strategy

This project is intentionally configured to reduce startup failures and initial install cost:

• Minimal core dependencies install first
• Command-specific dependency groups install on demand in the NPM launcher
• Several advanced Python libraries are optional in source dependencies
• Multiple modules include graceful fallback paths when optional packages are missing

Practical effect:

• Lightweight commands start faster and with fewer hard failures
• Heavy workflows install required extras only when invoked

Troubleshooting

1) ds-agent command not found

Cause:

• Global package not installed, or global npm bin path not on PATH

Fix:

npm install -g ds-agent-cli

or avoid global install:

npx ds-agent-cli quickstart

2) PowerShell environment variable syntax errors

Use PowerShell syntax (not bash inline env syntax):

$env:DS_AGENT_ENV_MODE="local311"
$env:DS_AGENT_PYTHON="py"
npx ds-agent-cli quickstart

3) Python detection failure in local311 mode

Cause:

• Launcher requires Python 3.11.x in local311

Fix:

• Install Python 3.11
• Set explicit interpreter:

DS_AGENT_ENV_MODE=local311 DS_AGENT_PYTHON=python3.11 npx ds-agent-cli quickstart

Windows options:

$env:DS_AGENT_ENV_MODE="local311"
$env:DS_AGENT_PYTHON="py"
npx ds-agent-cli quickstart

or pin exact launcher target:

$env:DS_AGENT_ENV_MODE="local311"
$env:DS_AGENT_PYTHON="py -3.11"
npx ds-agent-cli quickstart

4) Frontend interface is not showing

Cause:

• Frontend build artifacts missing from FRRONTEEEND/dist

Fix:

• Build and place frontend dist assets in FRRONTEEEND/dist
• Or use API docs at /docs

5) Optional package errors in advanced workflows

Cause:

• Some advanced features rely on optional libraries

Fix:

• Re-run command through launcher to trigger on-demand installs
• For source mode, install required extras manually

Development and Deployment

Development helpers in repo root:

• start.sh
• start.ps1
• build-and-deploy.sh
• build-and-deploy.ps1
• deploy.sh
• Dockerfile
• Dockerfile.spaces

Example local API start scripts:

./start.sh

.\start.ps1

Cloud and container deployment scripts are provided as operational examples and may require environment-specific adjustments.

Versioning

Current package version in this repository:

• ds-agent-cli: 0.1.4
• Python package metadata: 0.1.4

License

MIT