@yavda/powerbi-testing-kit

A reusable, production-grade testing framework for Power BI dashboards and reports. Enables visual regression testing, API-based data validation, and Power BI monitoring across any Power BI project.

Features

• Visual Regression Testing: Pixel-to-pixel comparison between native Power BI (baseline) and embedded reports
• Power BI REST API Testing: DAX query execution, row-level security (RLS) validation, dataset refresh monitoring
• Flexible Authentication: Power BI service principal (default), Azure CIAM, or custom OAuth flows
• Render Detection: Two strategies — custom-flag (for instrumented embeds) or native-visual-settle (for native Power BI or third-party embeds)
• Included Skills/Utilities: BPA health checks, DAX validation, RLS testing, refresh monitoring, workspace inventory
• Framework-Agnostic: Works with any Power BI embedding approach or native Power BI Service dashboards

Installation

npm install @yavda/powerbi-testing-kit

Or for local development:

npm link ../powerbi-testing-kit

Quick Start

1. Create a Config File

Create .pbi-test-config.json at your project root:

{
  "appName": "my-pbi-project",
  "baseUrl": "$BASE_URL",
  "workspaceId": "$POWERBI_WORKSPACE_ID",
  "reportId": "$POWERBI_REPORT_ID",
  "datasetId": "$POWERBI_DATASET_ID",
  "auth": {
    "type": "powerbi-service-principal",
    "config": {}
  },
  "dataEndpoint": {
    "type": "powerbi-rest-api"
  },
  "render": {
    "mode": "native-visual-settle",
    "timeout": 60000
  },
  "visualRegression": {
    "enabled": true,
    "tolerance": 8
  }
}

Environment variables (prefixed with $) are resolved at runtime:

export BASE_URL="https://app.powerbi.com"
export POWERBI_WORKSPACE_ID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
export POWERBI_REPORT_ID="yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy"
export POWERBI_DATASET_ID="zzzzzzzz-zzzz-zzzz-zzzz-zzzzzzzzzzzz"
export POWERBI_CLIENT_ID="aaa..."
export POWERBI_CLIENT_SECRET="bbb..."
export POWERBI_TENANT_ID="ccc..."

2. Run Tests

# Visual regression (pixel-diff)
npx powerbi-testing-kit run visual-regression

# DAX validation
npx powerbi-testing-kit run dax-query-validation

# RLS validation
npx powerbi-testing-kit run rls-validation

# Dataset refresh monitoring
npx powerbi-testing-kit run dataset-refresh-monitor

# Workspace inventory check
npx powerbi-testing-kit run workspace-inventory

Configuration

Auth Types

powerbi-service-principal (Default)

Uses Azure AD client credentials flow. Recommended for CI/CD and API-based testing.

Required Environment Variables:

• POWERBI_CLIENT_ID: Azure AD app registration client ID
• POWERBI_CLIENT_SECRET: Azure AD app registration client secret
• POWERBI_TENANT_ID: Azure AD tenant ID

Use Cases:

• Power BI REST API queries (DAX, RLS, refresh monitoring)
• Unattended/CI automation
• No browser interaction needed

ciam-azure

Azure Consumer Identity and Access Management (CIAM). Example for tenant-specific login flows.

Required Environment Variables:

• CIAM_TEST_USER_EMAIL: test user email
• CIAM_TEST_PASSWORD: test user password

Use Cases:

• Testing embedded dashboards with user authentication
• Verifying login flows and tenant isolation
• Browser-based end-to-end tests

Render Modes

custom-flag

Requires your embedding app to signal when rendering is complete.

How to implement (example: React + Power BI Embed SDK):

// In your PowerBIEmbed component
report.on('rendered', () => {
  window._pbiRenderComplete = true;
});

report.on('error', (e) => {
  window._pbiRenderError = e.message;
});

Advantages: Fast, precise, no polling.
Disadvantages: Requires app instrumentation.

native-visual-settle

Polls Power BI's DOM for stable visual count + network idle. Works on native powerbi.com and any embed.

Advantages: Works everywhere, no app changes needed.
Disadvantages: Slightly slower (polling-based).

Data Endpoints

powerbi-rest-api

Uses Power BI REST API to fetch/validate data. Requires service principal auth.

Features:

• Execute DAX queries for value assertions
• Test row-level security with impersonatedUserName
• Trigger and monitor dataset refreshes
• List workspaces, datasets, reports

custom-endpoint

Example for app-specific data APIs (e.g., /api/v1/reports in Yavda).

Use Cases:

• Testing app-specific data aggregation layers
• Validating custom RLS implementations
• Non-Power BI backend data sources

API Usage

Playwright Utils

import {
  waitForPowerBiRender,
  captureReportScreenshot,
  RenderConfig,
} from '@yavda/powerbi-testing-kit/playwright-utils';
import { test } from '@playwright/test';

test('capture report screenshot', async ({ page }) => {
  // Navigate to report
  await page.goto('https://app.powerbi.com/...');

  // Wait for render using native-visual-settle mode
  const renderConfig: RenderConfig = {
    mode: 'native-visual-settle',
    timeout: 60000,
  };
  await waitForPowerBiRender(page, renderConfig);

  // Capture screenshot
  await captureReportScreenshot(page, './screenshot.png', renderConfig);
});

Power BI REST API

import { PowerBiRestApi } from '@yavda/powerbi-testing-kit/adapters/data/rest-api';
import { ServicePrincipalAuth } from '@yavda/powerbi-testing-kit/adapters/auth/service-principal';

const auth = new ServicePrincipalAuth();
const api = new PowerBiRestApi({
  getToken: () => auth.getAccessToken(),
});

// Execute a DAX query
const result = await api.executeDaxQuery(
  'workspace-id',
  'dataset-id',
  'EVALUATE SUMMARIZECOLUMNS(DimDate[Date], "Total Sales", SUM(FactSales[Amount]))'
);

// Test RLS
const rlsResult = await api.executeDaxQueryWithRls(
  'workspace-id',
  'dataset-id',
  'EVALUATE VALUES(DimRegion[Region])',
  '[email protected]'
);

// Refresh dataset
await api.refreshDataset('workspace-id', 'dataset-id');
await api.waitForRefreshCompletion('workspace-id', 'dataset-id', 300000);

// List workspace contents
const workspaces = await api.listWorkspaces();
const datasets = await api.listDatasets(workspaces[0].id);
const reports = await api.listReports(workspaces[0].id);

Service Principal Auth

import { ServicePrincipalAuth } from '@yavda/powerbi-testing-kit/adapters/auth/service-principal';

const auth = new ServicePrincipalAuth();
const token = await auth.getAccessToken(); // auto-cached, refreshed as needed
const headers = await auth.getApiHeaders(); // ready for axios/fetch

Examples

See the examples/ directory for reference configs:

• powerbi-only-config.json — Native Power BI Service dashboards with REST API testing
• yavda-config.json — Embedded dashboards with CIAM login
• other-pbi-app-config.template.json — Template for your own Power BI embedding app

Environment Variables

Common variables used by the kit:

# Power BI Service Principal (default auth)
POWERBI_CLIENT_ID=...
POWERBI_CLIENT_SECRET=...
POWERBI_TENANT_ID=...

# Test app URLs and IDs
BASE_URL=https://staging.my-app.com
POWERBI_WORKSPACE_ID=...
POWERBI_REPORT_ID=...
POWERBI_DATASET_ID=...

# Azure CIAM (optional, for user-based testing)
[email protected]
CIAM_TEST_PASSWORD=...

License

MIT

Contributing

Contributions welcome! Please open an issue or PR on yavdaanalytics/powerbi-testing-kit.