Playwright Prometheus Reporter

This is a custom reporter for Playwright.

Use it for collecting metrics when running a suite of tests written in Playwright, and push those metrics to a Prometheus Pushgateway.

The metrics can then be scrapped by Prometheus and used as you want, for example, in Grafana visualizations.

Getting Started

Assuming that you already have one or more Playwright test suites ready in your system, all you need to do is:

• Install this package in your project.
• Add this package name to the list of reporters in your Playwright configuration file.

Installation

Install this package in your project which includes the Playwright tests:

$ npm install --save-dev @pumpo5.dev/playwright-prometheus-reporter

Or if you prefer Yarn:

$ yarn add --dev @pumpo5.dev/playwright-prometheus-reporter

Add this package to the list of reporters in your Playwright configuration (usually in a playwright.config.ts file):

...

reporter: "@pumpo5.dev/playwright-prometheus-reporter",

...

For more information on Playwright reporters, and other ways to add a new reporter to your test suite, please read the official documentation.

Usage

In order to enable the reporter and start collecting metrics, you need to create a configuration file. This file must be named config.conf.

Before any tests are executed, the reporter will load and look for this file in three ways:

• First, it will check if an environment variable named CONFIG_PATH is set.

  If it is, it will then check its value and verify if it is a path that points to a file named config.conf.

  In case positive, it will then try to load the configuration by reading this file.

• Second, if no CONFIG_PATH environment variable is set, the reporter will check if a command line argument was passed to NodeJS contaning a path to any file named config.conf. In case positive, it will then try to load the configuration by reading the file pointed by this command line argument.

• Third and last, in case the other methods failed, the reporter will look for a config.conf file in the project's root folder and directories above.

  In case it finds a config.conf in any of those directories it will proceed to load it and use it to set its internal configuration.

The keys that could or should be included in this configuration file are as follows:

Key | Type | Mandatory | Description ---|---------|---|--- pn5.reporting.enabled | boolean | Yes | Enables the Prometheus reporter. pn5.reporting.prometheus.enabled | boolean | Yes | Enables the collection and pushing of metrics during tests. pn5.reporting.prometheus.endpoint | String | Yes | URL for the Pushgateway endpoint to which metrics should be pushed. pn5.reporting.prometheus.suite | String | Yes | Name to be used for the suite_name label in metrics. In case this is empty, some class name will be used as default, which can be confusing. pn5.reporting.prometheus.testcases.track | boolean | No| Enables metrics for test cases. If disabled, only suite metrics will be collected. pn5.reporting.prometheus.suite_url | String | No | URL to be used as a value for the suite_url label in metrics. pn5.reporting.prometheus.branch | String | No | The current development branch in the versioning system. Used as a value for the branch label in metrics. pn5.reporting.prometheus.job | String | No | A name to be given for the job when pushing metrics, and for the job label. If empty, the suite name will be used as default. Warning: the slash '/' and backslash '' characters is not allowed in the job name. In case it is present, the Pushgateway will not accept the metrics. pn5.reporting.prometheus.project | String | No | The project name, to be prepended to the jobname pn5.reporting.prometheus.labels | String | No| Custom labels and values to be pushed in metrics, together with the base labels listed in the tables above. Should be in format label1=value1,label2=value2,.... pn5.reporting.prometheus.username | String | No | The user name, in case the endpoint requires basic authentication pn5.reporting.prometheus.password | String | No | The password, in case the endpoint requires basic authentication

The metrics currently reported are:

Suite metrics

Metric Name | Description | Base Labels ---|--------------------------------------------------------------------------------------------------------------------------------|--- suite_execution_total | Total number of tests in the test suite. | suite_name: a specific identifier for the test suitesuite_url: any URL the user wants the test suite to point to, e.g., a CI/CD pipelinebranch: the branch in the version control system from where the tests were run suite_execution_failed | Total number of failing tests in the test suite. This includes tests with status failed, timedOut and interrupted. | Same as above. suite_execution_finish_time | Timestamp from when the last execution finished (in seconds). | Same as above. suite_execution_duration_seconds | How long the suite took to finish, in seconds. | Same as above.

Test case metrics

Metric Name | Description | Base Labels ---|---------------------------------------------------------------------------------------------------------------|--- test_execution_duration_seconds | How long the test case took to finish, in seconds. | Same as above, with the addition of:test_case_name: the name of the test case. test_execution_finish_time | Timestamp from when the last test case execution finished (in seconds). | Same as above. test_execution_total | Displays a test case execution result, in a label. The metric value is the order in which the tests finished. | Same as above, with the addition of:result: what was the outcome from test case. Possible values are either passed, failed, timedOut or interrupted.

Examples

Please, check our repository to see example Playwright test suites using the Playwright Prometheus Reporter here.

Other versions

If you like this project, please check Pumpo#5 in Java as well, including another Prometheus reporter, here and here :)

Authors

• Alvaro Silva - Initial work - Linkedin

License

Apache License 2.0 © PumpITup