playground-cli

A programmatic simulation library for Microsoft 365 bots/agents. Drive your bot's responses without a browser or manual interaction — ideal for automated testing, CI pipelines, and LLM-based evaluation.

Installation

The package is part of the monorepo. Link it in your project:

{
  "dependencies": {
    "@microsoft/m365agentsplayground-cli": "file:../../packages/playground-cli"
  }
}

Build the package before use:

npm run build --workspace=packages/playground-cli

Quick Start

import { TestClient } from "@microsoft/m365agentsplayground-cli";

const client = new TestClient({
  botEndpoint: "http://localhost:3978/api/messages",
});

await client.start();
const [response] = await client.sendMessage("Hello!");
console.log(response.text);
await client.stop();

API

TestClient

The main class for interacting with your bot.

new TestClient(config: TestClientConfig)

TestClientConfig

| Property | Type | Default | Description | | -------------- | ------------------------------ | ---------- | -------------------------------- | | botEndpoint | string | required | Your bot's endpoint URL | | timeout | number | 5000 | Response timeout in milliseconds | | bot | BotConfig | - | Bot identity configuration | | deliveryMode | "expectReplies" \| "default" | - | How the bot delivers responses |

BotConfig

| Property | Type | Description | | --------------- | ---------------------------------- | ------------------------------------------------------ | | id | string | Bot's unique identifier | | name | string | Display name (max 42 chars) | | role | "user" \| "bot" \| "agenticUser" | Bot role | | tenantId | string | Tenant ID (also sets activity.conversation.tenantId) | | agenticUserId | string | User ID for agentic bots | | agenticAppId | string | App ID for agentic bots |

Methods

| Method | Returns | Description | | --------------------- | ------------------------ | ------------------------------------ | | start() | Promise<void> | Initialize the client | | stop() | Promise<void> | Shut down and clean up | | sendMessage(text) | Promise<BotResponse[]> | Send a message, wait for response(s) | | getMessages() | Message[] | All messages in the conversation | | getLastBotMessage() | Message \| undefined | Most recent bot message | | getConversationId() | string | Current conversation ID | | newConversation() | void | Reset conversation state |

WebSocket Events

TestClient extends EventEmitter and emits real-time events:

| Event | Payload | Description | | ----------------- | ---------------------- | ------------------------------------- | | message:created | ICreateMessageAction | New message from bot or user | | message:updated | IUpdateMessageAction | Message was updated (streaming, edit) | | typing | ITypingAction | Bot typing indicator | | websocket:error | Error | WebSocket connection error | | websocket:close | - | WebSocket connection closed |

BotResponse

Returned by sendMessage().

| Property | Type | Description | | ------------- | --------------- | ------------------------------------------ | | messageId | string | Unique message ID | | text | string? | Text content | | attachments | Attachment[]? | Attachments (Adaptive Cards, etc.) | | timestamp | number | Unix timestamp | | raw | Message | Raw Message object for detailed inspection |

Conversation Server

An HTTP wrapper around TestClient so non-Node languages (Python, curl, etc.) can drive bot conversations by POSTing JSON.

# From the repo root (after install + build):
npm run server --workspace=packages/agents-simulator -- --port 9000

Or programmatically:

import { createConversationServer } from "agents-simulator";
const server = await createConversationServer({ port: 9000 });

POST /run-conversation

Run a multi-turn conversation:

{
  "config": {
    "botEndpoint": "http://localhost:3978/api/messages",
    "timeout": 120000,
    "deliveryMode": "expectReplies",
    "bot": { "id": "[email protected]", "name": "My Bot" },
    "personas": {
      "alice": { "id": "alice-id", "name": "Alice", "email": "[email protected]" }
    }
  },
  "scenario": "greeting-test",
  "input": {
    "turns": [
      { "test_id": "t1", "prompt": "Hello!" },
      { "test_id": "t2", "prompt": "What can you do?" }
    ]
  }
}

Response:

{
  "type": "conversation_result",
  "scenario": "greeting-test",
  "status": "Completed",
  "duration_seconds": 4.2,
  "turns": [
    {
      "test_id": "t1",
      "prompt": "Hello!",
      "actual_response": "Hi! I'd be happy to help. What do you need?",
      "status": "Completed",
      "duration_seconds": 1.8
    },
    {
      "test_id": "t2",
      "prompt": "What can you do?",
      "actual_response": "I can help you with...",
      "status": "Completed",
      "duration_seconds": 2.4
    }
  ]
}

Configuration

The config object in the request body:

| Field | Type | Default | Description | | -------------- | ------------------------------- | ----------------- | ------------------------------------- | | botEndpoint | string | required | Bot's messaging endpoint URL | | timeout | number | 120000 | Per-turn response timeout (ms) | | deliveryMode | "expectReplies" \| "default" | "expectReplies" | How the bot delivers responses | | chatType | "personal" \| "group" \| "channel" | "personal" | Default chat context for all turns | | streamingSettleDelayMs | number | 800 | Quiet-period (ms) for streaming bots — see below | | bot | BotConfig | - | Bot identity (see above) | | personas | Record<string, PersonaConfig> | - | Named personas for notification turns |

Streaming bots (teams-ai / teams.ts stream: true): When the simulator receives the placeholder "Loading stream results...", it automatically switches into streaming mode. It first waits for a streamType:"final" WebSocket event (the precise end-of-stream signal from teams-ai and teams.ts SDK) — this is instant and accurate. If the bot doesn't send streamType, it falls back to a quiet-period: waits until no new updateActivity events arrive for streamingSettleDelayMs ms (default: 800 ms). Increase streamingSettleDelayMs only if you have a slow LLM that pauses mid-stream for over 800 ms. For bots with stream: false this option has no effect.

Turn Types

Each turn has an optional turn_type that controls what kind of Bot Framework activity is sent:

| Turn type | Description | | ------------------- | ------------------------------------------------ | | "chat" | Standard chat message (default) | | "sendEmail" | Email notification activity | | "mentionInWord" | Word document mention notification activity | | "meetingStart" | Meeting started event | | "meetingEnd" | Meeting ended event | | "participantJoin" | Meeting participant joined event | | "participantLeave"| Meeting participant left event | | "messageReaction" | Reaction added to a message (like, heart, etc.) | | "install" | Bot installation update | | "userAdded" | User added to conversation | | "botAdded" | Bot added to conversation | | "channelCreated" | Channel created (team context required) | | "teamRenamed" | Team renamed (team context required) |

All values from CustomActivityTemplateType are supported. You can mix turn types within a single conversation:

{
  "input": {
    "turns": [
      { "test_id": "t1", "prompt": "Hello", "turn_type": "chat" },
      {
        "test_id": "t2",
        "prompt": "<html><body>Customer complaint about order #789...</body></html>",
        "turn_type": "sendEmail",
        "persona": "customer1"
      },
      { "test_id": "t3", "prompt": "Summarize the email I just received", "turn_type": "chat" },
      { "test_id": "t4", "prompt": "", "turn_type": "meetingStart" },
      { "test_id": "t5", "prompt": "", "turn_type": "messageReaction", "reaction_type": "like" }
    ]
  }
}

Turn-specific fields

| Field | Applies to | Description | | ----------------- | ------------------------------- | -------------------------------------------------------- | | reaction_type | messageReaction | Reaction emoji: like (default), heart, laugh, surprised, sad, angry | | reply_to_id | messageReaction | Message ID to react to. Defaults to last bot message. | | prompt_metadata.meetingTitle | meetingStart, meetingEnd | Override the meeting title in the event. | | prompt_metadata.documentUrl | mentionInWord | Override the document URL. | | prompt_metadata.documentName| mentionInWord | Override the document name. |

Chat Types

The chat_type field on a turn (or chatType in config) controls the conversation context:

| Chat type | Description | | ------------ | ---------------------------------------------- | | "personal" | 1:1 personal chat with the bot (default) | | "group" | Group chat context | | "channel" | Team channel context (uses general channel) |

Set the default for the whole conversation in config.chatType, or override per turn with turn.chat_type:

{
  "config": {
    "botEndpoint": "http://localhost:3978/api/messages",
    "chatType": "group"
  },
  "scenario": "group-chat-test",
  "input": {
    "turns": [
      { "test_id": "t1", "prompt": "@bot Hello everyone!", "turn_type": "chat" },
      { "test_id": "t2", "prompt": "Personal follow-up", "turn_type": "chat", "chat_type": "personal" }
    ]
  }
}

Parallel Testing

Multiple TestClient instances in the same process each get a unique conversationId, so concurrent tests don't interfere with each other's messages.

To test multiple bot endpoints in parallel, run separate conversationServer processes on different ports:

# Terminal 1 — bot-a on port 9001
npm run server --workspace=packages/agents-simulator -- --port 9001

# Terminal 2 — bot-b on port 9002
npm run server --workspace=packages/agents-simulator -- --port 9002

Then point each test suite at its own server URL.

Personas

Personas set the from identity on notification activities. Define them in config.personas and reference by name:

• Conversation-level default: Set input.persona to apply to all turns
• Per-turn override: Set turn.persona to override for a specific turn
• Persona fields: id (required), name (required), email (optional — used as from.id when present)

Turn Result Statuses

| Status | Description | | ----------- | ----------------------------------------------- | | Completed | Bot responded successfully | | TimedOut | No response within the configured timeout | | Errored | An error occurred during execution | | Skipped | Turn was skipped because a previous turn failed |

Example: curl

curl http://localhost:9000/health

curl -X POST http://localhost:9000/run-conversation \
  -H "Content-Type: application/json" \
  -d '{
    "config": { "botEndpoint": "http://localhost:3978/api/messages" },
    "scenario": "smoke-test",
    "input": { "turns": [{ "test_id": "t1", "prompt": "Hello" }] }
  }'

Example: Python

import requests

resp = requests.post("http://localhost:9000/run-conversation", json={
    "config": {"botEndpoint": "http://localhost:3978/api/messages"},
    "scenario": "my-test",
    "input": {
        "turns": [
            {"test_id": "t1", "prompt": "Hello"},
            {"test_id": "t2", "prompt": "What can you do?"},
        ]
    },
})

for turn in resp.json()["turns"]:
    print(f"[{turn['status']}] {turn['actual_response'][:80]}")

GET /health

Returns { "status": "ok" }.

Example Project

See samples/agents-simulator-example for a complete example with Mocha tests and a sanity-test script.