duoops

v0.3.0

Published

a month ago

Toolset for Explainable and Sustainable CI on Gitlab.

0High
0Medium
0Low

youneslaaroussi

oclif

DuoOps

Sustainable and Explainable CI/CD for GitLab — powered by AI.

DuoOps helps developers understand, debug, and reduce the environmental impact of their GitLab CI/CD pipelines. It combines AI-powered troubleshooting with carbon footprint measurement, giving teams actionable insights to build greener software.

| Section | Link | |---------|------| | Key Features | Key Features | | Quick Start | Quick Start | | Usage | Usage | | Duo Agent Platform | Duo Agent Platform | | Heal | Heal | | MCP Server | MCP Server | | CI Components | CI Components | | Carbon Calculation Methodology | Carbon Calculation Methodology | | Architecture | Architecture | | Project Structure | Project Structure | | Development | Development | | Deliverables | Deliverables | | License | License |

Key Features

AI-Powered Troubleshooting — Ask questions about your pipelines in plain English. The AI agent (GitLab Duo or Vertex AI) analyzes logs, identifies root causes, and suggests fixes.
Automated Pipeline Healing — @duoops on a failing MR triggers a 5-step flow: fetch errors, diagnose root cause, validate CI config, apply fix commit, post report.
Carbon Measurement — Calculate CO2 emissions of CI jobs based on real CPU/RAM usage timeseries. Track your environmental impact over time.
MCP Integration — Connect DuoOps to GitLab Duo Chat so your AI assistant can query emissions data directly.
Web Portal — A local React dashboard (built with Palantir Blueprint.js) to visualize metrics, manage budgets, and chat with the AI assistant.
Deep Pipeline Inspection — View pipeline status, job details, and logs from your terminal.
Intelligent Summarization — Automatically summarizes long job logs using sub-agents for concise failure analysis.

Quick Start

Prerequisites

Node.js >= 18.0.0
GitLab Account with a Personal Access Token (api scope)
(Optional) Google Cloud SDK — for BigQuery metrics and MCP server deployment

Installation

Install from npm:

npm install -g duoops

Alternative: Build from Source

git clone https://gitlab.com/youneslaaroussi/duoops.git
cd duoops
pnpm install
pnpm build
npm link

Configuration

Run the interactive setup:

duoops init

This prompts you for your GitLab URL, Personal Access Token, and optional BigQuery settings.

Or use environment variables:

Usage

Inspect Pipelines

duoops pipelines list <project-id-or-path>

Analyze Jobs & Logs

duoops job show <project-id> <job-id>

Ask the AI Agent

duoops ask "Why did the last pipeline fail?"
duoops ask "How do I optimize my .gitlab-ci.yml for faster builds?"

Measure Carbon Footprint

duoops measure calculate \
  --provider gcp \
  --machine e2-standard-4 \
  --region us-central1 \
  --cpu-timeseries ./data/cpu.json \
  --ram-used-timeseries ./data/ram.json \
  --out-md report.md

Get AI Recommendations on Emissions

duoops act [project-id] [--limit N]

Fetches recent carbon metrics from BigQuery (per-job CO2e, energy breakdown, scope 3) and sends them to the AI agent. The agent analyzes trends and returns concrete recommendations: cache optimization, image size reduction, runner downsizing, stage parallelization, and region migration for lower carbon intensity.

Launch the Web Portal

duoops portal

Opens a local dashboard at http://localhost:58327 for visualizing metrics and chatting with the AI assistant.

Duo Agent Platform

DuoOps is built on the GitLab Duo Agent Platform, providing a Custom Agent, a Custom Flow, and an External Agent Trigger.

DuoOps Advisor (Custom Agent)

Available in Duo Chat for interactive troubleshooting with 17 platform tools — pipeline analysis, code review, log inspection, and sustainability recommendations.

# templates/agents/agent.yml
name: "DuoOps Advisor"
tools:
  - get_job_logs
  - get_pipeline_errors
  - get_pipeline_failing_jobs
  - get_merge_request
  - list_merge_request_diffs
  - create_merge_request_note
  - get_repository_file
  - read_file / read_files / find_files / grep / list_dir
  - get_commit_diff / list_commits
  - get_project / get_issue / create_issue_note

Pipeline Doctor (Custom Flow)

A 5-step automated pipeline healing flow using all 3 Duo component types:

| Step | Component Type | What It Does | |------|---------------|--------------| | fetch_failures | DeterministicStep | Fetches pipeline errors without an LLM call | | diagnose | Agent | Reads logs, CI config, source files, MR diffs to find root cause | | validate_ci | OneOff | Runs ci_linter on .gitlab-ci.yml with up to 3 retries | | apply_fix | Agent | Edits files and creates a fix commit on the MR branch | | report | Agent | Posts structured MR note: root cause, fix, CI validation, sustainability insights |

Conditional routing: if the diagnosis says FIXABLE: no, the flow skips validate_ci and apply_fix and goes straight to report.

External Agent Trigger

Drop .gitlab/duo/flows/duoops.yaml into any project. Mentioning @duoops on an MR or Issue triggers the agent:

# Routes by intent:
#   "fix/fail/debug"     -> duoops heal-ci (analyzes failure, commits fix, posts sustainability report to MR)
#   "carbon/emission"    -> duoops act (sustainability analysis from BigQuery)
#   else                 -> duoops ask (general Q&A)

Set up everything automatically with duoops init — it creates the agent, flow, trigger config, and CI variables in one command.

Heal

Interactive (CLI + Portal)

duoops heal

Launches the web portal, switches to the console tab, and preloads a request asking the agent to inspect the latest pipeline failure and suggest sustainability improvements. The agent reads logs, diagnoses the issue, proposes a fix, and recommends ways to reduce CI carbon footprint.

Headless (CI / MR)

duoops heal-ci <project-id> [--pipeline <id>] [--mr <iid>]

Runs the full diagnosis-and-fix cycle without user interaction. When --mr is provided, the agent posts a structured report directly to the merge request with:

Root cause analysis (job names, error messages, line numbers)
Applied fix (commit SHA and files changed, or manual remediation steps)
CI config validation (lint status)
Sustainability improvements (caching, image optimization, job consolidation, runner right-sizing, carbon impact estimates)

MCP Server

The MCP server exposes DuoOps emissions data to AI agents via the Model Context Protocol.

Tools

Deployment

# Using the CLI (recommended)
duoops mcp deploy --gcp-project <PROJECT_ID> --region us-central1

# Or manually with gcloud
gcloud run deploy duoops-mcp \
  --image registry.gitlab.com/youneslaaroussi/duoops/mcp-server:latest \
  --set-env-vars "GCP_PROJECT_ID=<ID>,BQ_DATASET=<DATASET>,BQ_TABLE=<TABLE>" \
  --allow-unauthenticated \
  --region us-central1

GitLab Duo Chat Integration

Add to your project's .gitlab/duo/mcp.json:

{
  "mcpServers": {
    "duoops-carbon": {
      "type": "http",
      "url": "https://<your-cloud-run-url>/mcp",
      "approvedTools": true
    }
  }
}

CI Components

Carbon Measurement (runs after every pipeline)

include:
  - component: gitlab.com/youneslaaroussi/duoops/duoops-measure-component@main
    inputs:
      provider: gcp
      machine: e2-standard-4
      region: us-central1

Runs in the .post stage. Fetches real CPU/RAM timeseries from GCP Monitoring, calculates emissions, and persists results to BigQuery.

Heal (runs on failure)

include:
  - component: gitlab.com/youneslaaroussi/duoops/duoops-heal-component@main

Runs in the .post stage with when: on_failure. Analyzes the failing pipeline, diagnoses root cause, posts a fix report with sustainability recommendations to the MR.

Both Together

include:
  - component: gitlab.com/youneslaaroussi/duoops/duoops-measure-component@main
    inputs:
      provider: gcp
      machine: e2-standard-4
      region: us-central1
  - component: gitlab.com/youneslaaroussi/duoops/duoops-heal-component@main

CI/CD Variables

Carbon Calculation Methodology

DuoOps builds on the methodology of re:cinq and Teads, integrating real cloud monitoring data, real-time grid carbon intensity, and region-specific PUE for high-accuracy emissions estimates.

Step-by-Step Calculation

1. Calculate CPU Energy from Utilization Timeseries

For each data point in the CPU utilization timeseries, we compute the instantaneous power draw and integrate over time:

\text{vCPUHours} = \left(\frac{\text{interval (s)}}{3600}\right) \times \text{vCPUs}

2. Interpolate CPU Power at Utilization

We use 4 data points (0%, 10%, 50%, 100% utilization) from real CPU power profiles and apply cubic spline interpolation:

P_{\text{CPU}}(u) = \text{CubicSpline}(\{0, 10, 50, 100\}, \{W_0, W_{10}, W_{50}, W_{100}\})(u)

A physical vCPU correction is applied to avoid overestimating power:

P_{\text{corrected}} = P_{\text{base}} \times \frac{1}{\text{physical\_threads}}

3. Calculate CPU Energy

E_{\text{CPU}} \text{ (kWh)} = \sum_{i} P_{\text{CPU}}(u_i) \times \frac{\Delta t_i}{3600 \times 1000}

4. Calculate RAM Energy

RAM power is estimated using a conservative flat rate of 0.5 W/GB (based on used memory, not allocated):

E_{\text{RAM}} \text{ (kWh)} = \sum_{i} \left( \text{RAM}_{\text{used,GB}} \times 0.5 \right) \times \frac{\Delta t_i}{3600 \times 1000}

5. Apply PUE and Carbon Intensity

\text{CO}_2\text{e}_{\text{operational}} = (E_{\text{CPU}} + E_{\text{RAM}}) \times \text{PUE} \times \text{Carbon Intensity (gCO}_2\text{/kWh)}

6. Add Scope 3 (Embodied Emissions)

\text{CO}_2\text{e}_{\text{embodied}} = \text{scope3\_hourly} \times \text{runtime\_hours}

Final Formula

E_{\text{total}} = \underbrace{(E_{\text{CPU}} + E_{\text{RAM}}) \times PUE \times CI}_{\text{operational}} + \underbrace{E_{\text{mfg}} \times t}_{\text{embodied}}

Key Constants & Data Sources

| Parameter | Value | Source | |---|---|---| | RAM power | 0.5 W/GB | Conservative flat estimate (CCF uses ~0.392 W/GB) | | Default PUE | 1.2 | Industry standard fallback | | Fallback carbon intensity | 475 gCO₂/kWh | Global average | | GCP region PUEs | 1.06–1.22 | Google Data Center Efficiency | | AWS region PUEs | 1.14–1.22 | Published estimates per region | | CPU power profiles | Per-model curves | re:cinq emissions-data | | Carbon intensity | Real-time | Electricity Maps API | | Machine specs | GCP & AWS | data/gcp_machine_power_profiles.json, data/aws_machine_power_profiles.json |

What Makes DuoOps Different

Real Monitoring Data — Actual CPU/RAM utilization timeseries from GCP Monitoring or AWS CloudWatch, not static estimates.
Live Grid Carbon Intensity — Real-time gCO₂/kWh from Electricity Maps API for the specific cloud region.
Region-Specific PUE — Published datacenter efficiency values, not a one-size-fits-all multiplier.
Physical vCPU Correction — Corrects for shared-core architectures to avoid inflating power estimates.
Scope 3 Included — Amortized embodied emissions from hardware manufacturing are added to operational emissions.

Architecture

See the architecture diagram at the top of this README.

Project Structure

duoops/
├── bin/                  # CLI entry point
├── src/                  # CLI commands and core logic
│   ├── commands/         # Oclif command implementations
│   └── lib/              # AI agents, GitLab client, measurement
├── mcp-server/           # MCP server (standalone, deployable)
├── portal/               # React web dashboard (Blueprint.js)
├── templates/            # GitLab CI component templates
├── data/                 # Sample data and fixtures
└── test/                 # Test suite

Development

# Run tests
pnpm test

# Build everything
pnpm build

# Run the portal in dev mode
cd portal && pnpm dev

Deliverables

| Deliverable | Description | | :--- | :--- | | CI Component | Drop-in GitLab CI component that measures carbon emissions per job using CPU/RAM timeseries, then persists results to BigQuery. | | MCP Server | Model Context Protocol server that exposes emissions data to AI agents (GitLab Duo Chat, VS Code, etc.) via standardized tools. | | CLI Tool | Local command-line interface for pipeline inspection, AI-powered troubleshooting, carbon measurement, and launching the web portal. |

License

MIT