M365 Copilot Agent Evaluations

This tool is Generally Available. Full documentation on Microsoft Learn

A CLI for evaluating M365 Copilot agents. Send prompts to your agent, get responses, and automatically score them with Azure AI Evaluation metrics.

| Evaluator | Type | Scale | Default | Description| |-----------|------|-------|---------|------------| | Relevance ⭐ | LLM-based | 1-5 | Yes | Assesses how well the agent's response addresses the user's query.| | Coherence ⭐ | LLM-based | 1-5 | Yes | Measures the logical and orderly presentation of ideas in the agent's response.| | Groundedness | LLM-based | 1-5 | No | Assesses whether the agent's response is consistent with and supported by the provided grounding context.| | Similarity | LLM-based | 1-5 | No | Measures the degree of semantic similarity between the agent's response and a provided expected_response| | Citations | Count-based | ≥ 0 | No | Counts the number of citation references. | | RetrievalQuery | Non-LLM | pass/fail | No | Assesses if Copilot correctly translated user intent into retrieval queries.| | RetrievalResult | Non-LLM | pass/fail | No | Validates that expected resources actually appear in the documents, messages, and items returned by retrieval executions.| | ExactMatch | String match | boolean | No | Measures the degree of textual overlap between the agent's response and the expected_response. | | PartialMatch | String match | 0.0–1.0 | No | Performs a direct string comparison between the agent's response and the expected_response. |

• Multiple input modes: command‑line list, JSON file, interactive.
• Multiple output formats: console (colorized), JSON, CSV, HTML (auto‑opens report).

📋 Prerequisites

• M365 Copilot License for your tenant
• M365 Copilot Agent deployed to your tenant (can be created with M365 Agents Toolkit or any other method)
• Node.js 24.12.0+ (check: node --version)
• Python 3.13.x is downloaded automatically. If the download fails (e.g., network restrictions), set PYTHON_PATH to a local Python 3.13.x installation (see Troubleshooting)
• Environment file with your credentials and agent ID (see Environment Setup below)
• Your Tenant ID - get your tenant id using the instructions here
• Admin approval to run WORKIQ Client App for your tenant here
• Azure OpenAI endpoint, and API key (see Getting Variables below)

Platform authentication support:

• Windows — Windows Account Manager (WAM) broker, built-in.
• macOS — Company Portal broker. Install Microsoft Company Portal before running.
  • Known limitation (Intel Macs): Sign-in via the broker is currently failing on Intel-based Macs. Apple Silicon (M-series) Macs are not affected. The MSAL team is investigating; progress is tracked in AzureAD/microsoft-authentication-library-for-python#908.
• Linux / WSL — Intune broker. Install the required system libraries first:

  sudo apt install libwebkit2gtk-4.1-0 libdbus-1-dev python3-gi gir1.2-secret-1 libubsan1

  If the required libraries are missing, the authentication library raises an ImportError instead of falling back to browser-based authentication — install the packages above before running.

🔧 Environment Setup

Install the Tool

# 1. Install
npm install -g @microsoft/m365-copilot-eval

# 2. Run from your agent project directory
cd /path/to/your-agent-project
runevals --version

Setup Steps

Now, set the following environment variables wherever you are running your evals

TEAMS_APP_TENANT_ID="your-tenant-id" 
AZURE_AI_OPENAI_ENDPOINT="<your-azure-openai-endpoint>"
AZURE_AI_API_KEY="<your-api-key-from-azure-portal>"
AZURE_AI_API_VERSION="2024-12-01-preview"  
AZURE_AI_MODEL_NAME="gpt-4o-mini"      

Or store environment variables in a .env file for the tool to pick up:

Option 1: For M365 Agents Toolkit (ATK) Projects

ATK projects already check in .env.local with M365_TITLE_ID. Do not put secrets in .env.local — use .env.local.user instead, which is loaded automatically and should be added to your .gitignore.

# .env.local (checked in — no secrets!)
# Already present from ATK:
M365_TITLE_ID="T_your-title-id-here"  # Auto-generated by ATK
TEAMS_APP_TENANT_ID="your-tenant-id"  # Auto-generated by ATK

# .env.local.user (NOT checked in — secrets go here)
AZURE_AI_OPENAI_ENDPOINT="<your-azure-openai-endpoint>"
AZURE_AI_API_KEY="<your-api-key-from-azure-portal>"
AZURE_AI_API_VERSION="2024-12-01-preview"  # default
AZURE_AI_MODEL_NAME="gpt-4o-mini"           # recommended

Add .env.local.user to your .gitignore:

# User-specific secrets — never commit
.env.local.user
env/.env.local.user

Option 2: For Non-ATK Projects

Create env/.env.dev in your project directory:

# env/.env.dev (new file you create)
# Your agent ID (Optional):
M365_AGENT_ID="your-agent-id"  # e.g., U_0dc4a8a2-b95f-edac-91c8-d802023ec2d4

# You'll add these (see Getting Variables section below):
AZURE_AI_OPENAI_ENDPOINT="<your-azure-openai-endpoint>"
AZURE_AI_API_KEY="<your-api-key-from-azure-portal>"
AZURE_AI_API_VERSION="2024-12-01-preview"  # default
AZURE_AI_MODEL_NAME="gpt-4o-mini"           # recommended
TENANT_ID="<your-tenant-id>"

• If you are storing your environment variables in .env.dev or .env.local files, you can run runevals --env dev or runevals --env local
• You can also override the agent ID at runtime: runevals --m365-agent-id "custom-id"

🔑 Getting Variables

📖 How to get these values →

New feature alert! You can now use DefaultAzureCredential instead of AZURE_AI_API_KEY for authenticating your Azure LLM models!

Azure OpenAI Authentication Mode

By default, the CLI authenticates to Azure OpenAI using an API key (AZURE_AI_API_KEY). If your organization disables key-based access or you prefer keyless authentication, you can use DefaultAzureCredential (Microsoft Entra ID) instead.

Option A: API Key (default)

Set AZURE_AI_API_KEY in your env file — the CLI uses it automatically.

Option B: DefaultAzureCredential (keyless)

1. Sign in to Azure CLI: az login --tenant <your-tenant-id>
2. Assign the Cognitive Services OpenAI User role to your identity on the Azure OpenAI resource
3. Remove or leave AZURE_AI_API_KEY empty in your env file
4. Run the CLI — it auto-detects the missing key and uses DefaultAzureCredential

You can also explicitly select the auth mode:

# Explicit keyless authentication
runevals --azure-ai-auth-mode default-credential

# Explicit API key authentication
runevals --azure-ai-auth-mode key

Fallback behavior (auto-detect):

| AZURE_AI_API_KEY set? | --azure-ai-auth-mode flag | Auth used | |---|---|---| | ✅ Yes | (not provided) | API key | | ❌ No | (not provided) | DefaultAzureCredential | | — | key | API key (fails if key missing) | | — | default-credential | DefaultAzureCredential |

Note: DefaultAzureCredential tries multiple credential sources in order: Azure CLI, Azure PowerShell, environment variables, managed identity, and more. See Azure Identity docs for details.

Advanced: Request Timeout & Retries (Optional)

Calls to the Work IQ A2A agent use sensible defaults that work for most workloads. For long-running agents (multi-step reasoning, large tool calls, slow downstream services) you can tune the HTTP request behavior with these optional environment variables:

| Variable | Default | Description | |---|---|---| | WORKIQ_REQUEST_TIMEOUT_SECS | 300 | HTTP request timeout, in seconds, applied to each prompt/response request sent to the agent. Increase it if agent responses routinely exceed five minutes. Invalid or non-positive values fall back to the default. (Agent discovery and agent-card resolution use a fixed 300s timeout and are not affected by this setting.) | | WORKIQ_REQUEST_MAX_ATTEMPTS | 4 | Maximum number of attempts (initial try + retries) for an agent request. Retries cover transient failures: retryable HTTP statuses (429, 503, 504) and socket timeouts. Values below 1 or non-integers fall back to the default. |

# Example: allow a bit more time per request and one extra retry
WORKIQ_REQUEST_TIMEOUT_SECS="420"
WORKIQ_REQUEST_MAX_ATTEMPTS="5"

Note: Transient failures (retryable HTTP statuses and socket timeouts) are retried for both single-turn prompts and individual multi-turn turns. Because a timed-out turn may have already been processed server-side, a multi-turn retry can occasionally duplicate that turn in the conversation; the agent is still expected to respond with the correct content. HTTP 401 responses are handled separately by a single automatic token refresh.

🚀 Quick Start

Now that you have your environment variables set up, you're ready to run evaluations!

Important: Run this tool FROM your M365 agent project directory (where your agent code lives), not from this repository. You don't need to clone or download this repo.

# Navigate to YOUR agent project directory
cd /path/to/your-agent-project

# Run evaluations (auto-discovers .env.local for ATK projects)
runevals

# Or specify an environment file
runevals --env dev

No prompts file? If you don't have a prompts file yet, the tool will offer to create a starter file with example prompts for you.

Environment file lookup:

• Checks .env.local first (ATK projects)
• Then checks env/.env.{name} if --env {name} is specified
• Prompts file auto-discovery works the same for all projects

📝 Eval Document Format

The eval document schema is versioned independently from the CLI, following Semantic Versioning.

• Schema location: schema/v1/eval-document.schema.json
• Schema changelog: schema/CHANGELOG.md

New in Schema v1.2.0: Multi-turn conversation threads — test context persistence across multiple turns within a shared conversation session. Each thread supports 1-20 turns.

New in Schema v1.1.0: Per-prompt evaluator overrides with evaluators_mode (extend/replace), file-level default_evaluators, and ExactMatch/PartialMatch evaluators.

Getting Started

The CLI auto-discovers prompts files in your project. When you run runevals, it searches:

1. Current directory: prompts.json, evals.json, tests.json
2. ./evals/ subdirectory: prompts.json, evals.json, tests.json

No prompts file? The CLI will offer to create a starter file with example prompts for you.

A minimal eval document:

{
  "schemaVersion": "1.6.0",
  "items": [
    {
      "prompt": "What is Microsoft 365?",
      "expected_response": "Microsoft 365 is a cloud-based productivity suite..."
    }
  ]
}

Evaluator Configuration

Use default_evaluators to set file-level defaults, and per-item evaluators with evaluators_mode to customize:

{
  "schemaVersion": "1.6.0",
  "default_evaluators": {
    "Relevance": {},
    "Coherence": {}
  },
  "items": [
    {
      "prompt": "What is Microsoft Graph?",
      "expected_response": "A unified API endpoint for Microsoft services.",
      "evaluators": {
        "Groundedness": { "threshold": 4 }
      },
      "evaluators_mode": "extend"
    },
    {
      "name": "Expense policy flow",
      "turns": [
        {
          "prompt": "I spent $250 on dinner. Is that okay?",
          "expected_response": "The per-diem meal allowance is $200.",
          "evaluators": {
            "Citations": { "citation_format": "mixed" },
            "RetrievalQuery": {
              "capability": "OneDriveAndSharePoint",
              "selector": "dinner",
              "includes": [
                "allowance"
              ],
              "excludes": [
                "restaurant"
              ]
            },
            "RetrievalResult": {
              "capability": "OneDriveAndSharePoint",
              "max_rank": 5,
              "expected_items": [
                {
                  "retrievalExtract_contains": "$200"
                },
                {
                  "retrievalExtract_contains": "allowance"
                }
              ]
            }
          },
          "evaluators_mode": "extend"
        },
        {
          "prompt": "What should I do about the overage?",
          "expected_response": "Request manager approval.",
          "evaluators": {
            "ExactMatch": { "case_sensitive": false }
          },
          "evaluators_mode": "replace"
        }
      ]
    }
  ]
}

How evaluator modes work in this example:

| Item | evaluators_mode | Active Evaluators | Why | |------|-------------------|-------------------|-----| | Single-turn (Graph) | extend | Relevance, Coherence, Groundedness | Per-prompt Groundedness merged with defaults | | Multi-turn turn 1 (dinner) | extend | Relevance, Coherence, Citations, RetrievalQuery, RetrievalResult | Per-turn evaluators merged with defaults | | Multi-turn turn 2 (overage) | replace | ExactMatch | Per-turn ExactMatch replaces defaults entirely |

Evaluator Modes

| Mode | Behavior | |------|----------| | "extend" (default) | Per-item evaluators merge with defaults. Both run. | | "replace" | Per-item evaluators replace defaults entirely. Only per-item evaluators run. | | (none) | Inherits file-level default_evaluators, or system defaults (Relevance, Coherence) if not set. |

See schema/v1/examples/ in the package for more examples including per-turn evaluator overrides, mixed single/multi-turn files, and output format.

Custom Evaluators (New in Schema v1.6.0)

In addition to the 10 built-in evaluators, you can define your own custom LLM-judge evaluators for domain-specific scoring (regulatory compliance, brand tone, custom relevance rubrics, etc.). Drop a .prompty file and a .py wrapper into <your_project>/custom-evaluators/<name>/ and reference it from any eval document:

"evaluators": {
  "Relevance": {},
  "professional_tone": { "threshold": 4 }
}

Each custom evaluator pairs a .prompty file (the LLM judge prompt) with a .py wrapper class that invokes it and parses the result. This supports everything from simple single-prompt scoring to multi-step LLM calls, custom output parsing, and score aggregation.

See docs/custom-evaluators/README.md for the full authoring guide and reference examples (professional_tone, consistency_check, answer_accuracy).

Auto-Upgrade Behavior

When the CLI loads an eval document:

• Legacy documents (missing schemaVersion): Automatically upgraded with a timestamped backup (e.g., file.json.bak.20260205143052)
• Older versions (same major version): schemaVersion field updated without backup
• Invalid documents: CLI exits with an error message and guidance to review the schema changelog
• Future versions: CLI rejects with a message suggesting a CLI update

Version Compatibility

Within a major version (e.g., 1.x.x), we aim to maintain backward compatibility for documents that conform to the published schema for their version. Compatibility does not extend to undeclared or ad-hoc fields outside the schema definition; review the schema changelog when upgrading between minor versions.

🎯 Usage Examples

Remember: All commands below assume you're running them FROM your agent project directory, not from this repository.

What to Expect

When you run an evaluation from your agent project directory, you'll see:

🚀 M365 Copilot Agent Evaluations CLI

📂 Loading environment: dev
🤖 Agent ID: T_my-agent.declarativeAgent
📄 Using prompts file: ./evals/evals.json

📊 Running evaluations...

─────────────────────────────────────────────────────────────

✓ Evals completed successfully!

Results saved to: ./evals/2025-12-03_14-30-45.html

Commands to run from your project root:

# Use .env.local (checked in current dir, then env/ folder)
runevals

# Use env/.env.dev configuration
runevals --env dev

# Use specific prompts file in your project
runevals --prompts-file ./evals/my-tests.json

# Inline prompts (no file needed, useful for quick tests)
runevals --prompts "What is Microsoft Graph?" --expected "Gateway to M365 data"

# Interactive mode (enter prompts interactively)
runevals --interactive

# Canonical logging verbosity
runevals --log-level debug
runevals --log-level info
runevals --log-level warning
runevals --log-level error

# Parallel prompt execution control
runevals --concurrency 5 --prompts-file ./evals/evals.json
runevals --concurrency 1000 --prompts-file ./evals/evals.json   # Python CLI clamps to 5

# Multi-account sign-in: pick which cached account to use
runevals --account [email protected] --prompts-file ./evals/evals.json

# Custom output location in your project
runevals --output ./reports/results.html

Sample Scorecard

⚠️ Debug log safety notice: The --log-level debug option is opt-in and may include raw API payloads and response data in console output. Redaction is pattern-based (API keys, tokens, passwords, long mixed-case strings) and will not catch arbitrary PII or custom credentials embedded in prompts or responses. Do not share debug-level output publicly without manual review.

Auth and SDK errors: Warnings and errors from the Microsoft sign-in flow (MSAL) and Azure AI Evaluation SDK appear alongside the CLI's own diagnostics — useful when a run fails to authenticate or an evaluator can't reach Azure. Routine SDK chatter (token cache hits, HTTP retries) is hidden by default. If you're troubleshooting an auth or evaluator issue and want to see everything those libraries report, add --log-level debug.

Optional: Add Shortcuts to package.json

You can add shortcuts (npm scripts) to your agent project's package.json:

{
  "scripts": {
    "eval": "runevals",
    "eval:local": "runevals --env local",
    "eval:dev": "runevals --env dev"
  }
}

Then use shorter commands:

# Uses .env.local (ATK default)
npm run eval

# Uses env/.env.local
npm run eval:local

# Uses env/.env.dev
npm run eval:dev

Production note: For production environments, use CI/CD pipelines instead of local npm run commands. See CICD_CACHE_GUIDE.md for examples.

📊 Output Formats

Results are automatically saved to ./evals/YYYY-MM-DD_HH-MM-SS.html with:

• Per-prompt and per-turn evaluation scores from configured evaluators
• Aggregate statistics across all evaluated items
• Multi-turn thread summaries (turns passed/failed, overall status)

Other formats:

# JSON output
runevals --output results.json

# CSV output
runevals --output results.csv

🔧 Command Reference

Options:
  -V, --version                 output version number
  --log-level [level]           log level: debug|info|warning|error (bare flag -> info)
  --prompts <prompts...>        inline prompts to evaluate
  --expected <responses...>     expected responses (with --prompts)
  --prompts-file <file>         JSON file with prompts
  -o, --output <file>           output file (JSON, CSV, or HTML)
  -i, --interactive             interactive prompt entry mode
  --m365-agent-id <id>          override agent ID
  --account <account>           user account (email/UPN) to sign in with when multiple are cached
  --env <environment>           environment name (default: dev)
  --concurrency <number>        parallel workers for prompt processing (1-5)
  --azure-ai-auth-mode <mode>   Azure AI auth: key | default-credential
  --judge-backend <backend>     LLM judge backend: azure (default) | copilot
  --no-judge-auto-fallback      disable auto-retry with model="auto" on rate limit (copilot)
  --init-only                   just setup, don't run evals
  -h, --help                    display help

Cache Commands:
  cache-info                    show cache statistics
  cache-clear                   remove cached Python runtime
  cache-dir                     print cache directory path

🧑‍⚖️ LLM Judge Backend

LLM-based evaluators (Relevance, Coherence, Groundedness, Similarity) are scored by a "judge" model. Two backends are available via --judge-backend:

| Backend | Flag | Model is configured by | Notes | |---------|------|------------------------|-------| | Azure OpenAI (default) | --judge-backend azure | AZURE_AI_MODEL_NAME env var | Requires the AZURE_AI_* variables. | | GitHub Copilot | --judge-backend github-copilot | GITHUB_COPILOT_JUDGE_MODEL env var | No Azure OpenAI keys needed; authenticates with GitHub (gh auth login or GITHUB_TOKEN). |

# Use GitHub Copilot as the judge (no Azure OpenAI configuration required)
runevals --judge-backend github-copilot --prompts-file ./evals/evals.json

Choosing the Copilot judge model

There is no --judge-model flag — the model is read from the GITHUB_COPILOT_JUDGE_MODEL environment variable, mirroring how the Azure backend reads AZURE_AI_MODEL_NAME.

GITHUB_COPILOT_JUDGE_MODEL="gpt-4.1"   # pin a specific model
# (unset)                       # defaults to "auto"

• Default is "auto" — Copilot selects a model per request. The actual model used is reported in the run output (e.g. Judge: GitHub Copilot (model: auto → resolved: gpt-4.1-mini)).
• If you pin a model that your account can't access, the run fails fast at startup with the list of available models (instead of erroring on every prompt).

Rate limits and auto-fallback

By default, if a pinned model hits its rate limit, the judge automatically retries that evaluation with model="auto" so the run can finish. Pass --no-judge-auto-fallback to disable this and surface the rate-limit error instead.

GPT‑5.x and o‑series judge models (Microsoft Foundry cloud evaluation)

GPT‑5.x and o‑series models can't be used with the default local evaluators. To use one of these models as the LLM judge, point the CLI at a Microsoft Foundry project — the LLM evaluators (Relevance, Coherence, Groundedness, Similarity) then run through Microsoft Foundry cloud evaluation instead.

This is automatic and independent of --judge-backend:

• Set AZURE_AI_PROJECT_ENDPOINT to your Foundry project endpoint (https://<account>.services.ai.azure.com/api/projects/<project>).
• When AZURE_AI_PROJECT_ENDPOINT and AZURE_AI_MODEL_NAME are set, the LLM evaluators run in Foundry. The results and report format are unchanged.
• Leave AZURE_AI_PROJECT_ENDPOINT unset to use the local evaluator path for gpt‑4x models — no other configuration change needed.

| AZURE_AI_PROJECT_ENDPOINT set? | Judge models supported | |:---:|-----------------| | ✅ yes | gpt‑5x / o‑series and gpt‑4x (via Microsoft Foundry) | | ❌ no | gpt‑4x only (local evaluators) |

Note: Microsoft Foundry has deprecated the gpt‑4x / gpt‑4o judge models, with retirement dates through 2026. Plan to move your judge model to gpt‑5.x (which requires the Foundry cloud evaluation path above). See the Foundry model retirement schedule.

Requirements: a Foundry project with a chat‑capable model deployment, the Azure AI Developer role on the project, and Entra sign‑in (az login / DefaultAzureCredential). AZURE_AI_API_KEY is not used for this path. If you're not signed in or lack the required role, the run reports an authentication or permission error. See Cloud evaluation with the Microsoft Foundry SDK and RAG evaluators.

# Point at a Foundry project to evaluate with a gpt‑5x / o‑series judge model
AZURE_AI_PROJECT_ENDPOINT="https://myacct.services.ai.azure.com/api/projects/myproj"
AZURE_AI_MODEL_NAME="gpt-5-mini"
runevals --prompts-file ./evals/evals.json

❓ Troubleshooting

Pre-cache Python Environment (Optional)

If you want to set up the Python environment ahead of time without running evaluations:

runevals --init-only

This is useful for:

• Pre-warming the cache in CI/CD pipelines
• Testing the setup without running evaluations
• Troubleshooting installation issues

Cache Issues

# View cache info
runevals cache-info

# Clear and rebuild
runevals cache-clear
runevals --init-only --log-level debug

Network/Proxy Issues

# Set proxy
export HTTPS_PROXY=http://proxy:8080

# Retry with verbose output
runevals --init-only --log-level debug

Permission Issues

# Check cache directory
runevals cache-dir

# Fix permissions (Unix/macOS)
chmod -R u+w $(runevals cache-dir)

Custom Python Runtime (PYTHON_PATH)

If the automatic Python download fails (e.g., network restrictions, unsupported platform), provide your own Python installation:

# Windows
set PYTHON_PATH=C:\Python313\python.exe

# macOS/Linux
export PYTHON_PATH=/usr/local/bin/python3.13

Python 3.13.x is the tested version. If a different version is found, you'll be prompted to confirm before proceeding. In CI/CD, a version mismatch fails automatically.

📚 Advanced Documentation

• CI/CD Integration - GitHub Actions, Azure DevOps caching
• Testing Guide - Cross-platform testing procedures
• Python CLI Guide - Direct Python usage (without Node.js)
• Local Development Setup - Setting up the repo for local development

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit Contributor License Agreements.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

Terms of Use

By using this tool, you agree to the Microsoft Software License Terms.

See LICENSE for the full license text.