testrail-reporter-playwright

The testrail-reporter-playwright package can be used to integrate to the cloud version of testrail (testrail.com or testrail.net). This package is built as a reporter to be incorporated in the playwright config file. This integrates with TestRail and automatically creates TestRail Runs and add test results on matching testrail test case IDs.

Installation

npm install testrail-reporter-playwright

Features

The package supports the below features:

• Integrates to Playwright with provisions to be addded as a reporter.
• Creates a New test run in testrail or use an existing testrail run.
• Update the test cases in a test run after a test is complete.
• Update the Error log too to the testrail test case on failures.
• Attach the screenshot too to the Test run level on failures.
• Print the test run on the test completion.
• Support for a single test run for multiple playwright spec files run.

Usage

1. Install the package.

2. Add 'testrail-reporter-playwright' as a reporter to the playwright config file. This should be added under "module.exports = defineConfig({" block. Sample code below:
   reporter: ['line', 'html', 'testrail-reporter-playwright'],

3. Create a 'testrail.config.js' file in your project's root directory. Below should be the data under it:

{
    "TESTRAIL_HOST": "https://<<your domain>>.testrail.com",
    "TESTRAIL_USERNAME": "your testrail login email",
    "TESTRAIL_PASSWORD": {
        "encrypted": "<your base64 encrypted testrail API key using encrypt.spec.js>",
        "iv": "<your base64 IV value when encrypting the testrail API key using encrypt.spec.js",
        "testrailPassword": process.env.TESTRAIL_PWD
    },
    "TESTRAIL_PROJECT_ID": "<testrail project id>",
    "TESTRAIL_SUITE_ID": {
        "<project short name 1>": "test suite id in testrail",
        "<project short name 2>": "test suite id in testrail"
    },
    "TESTRAIL_API_ATTACHMENT_PATH": "/index.php?/api/v2/add_attachment_to_run/",
    "TESTRAIL_RUN_NAME": "<Test run name that you want the test run to be created as, will always be followed by the date time>",
    "TESTRAIL_RUN_ID": "<Leave empty if you want a new test run created everytime, else add the test run id of an existing run>"
}

• TESTRAIL_HOST - TestRail instance domain name e.g https://<company name>.testrail.com
• TESTRAIL_USERNAME - This should be a valid testrail login email id; can be a service account email id created for the team or a personal email based on security policies
• TESTRAIL_PASSWORD - Here, the combination of 'encrypted' and 'iv' OR the 'testrailPassword' will be used. Recommended to encrypt the testrail plain text password using https://www.npmjs.com/package/encrypt-decrypt-crypto, the same package that is used for decrypting to a plain text password. In case the encrypted form is not used, then the user can send it from command line to the process.env.TESTRAIL_PWD environment variable.
• TESTRAIL_PROJECT_ID - Numeric number/Id of the project
• TESTRAIL_SUITE_ID - Multiple line items, each line item contains a short name followed by its test suites id.
• TESTRAIL_API_ATTACHMENT_PATH - Does not change. Should stay the same.
• TESTRAIL_RUN_NAME - Sample name can be ' Regression tests' and then it will be followed by the date and time when a new test run is created.
• TESTRAIL_RUN_ID - Should stay blank - "" in most cases to create a new test run for every test cycle. In case you need to use an existing test run id, then update this field with the numerical testrail test run id.

4. Create/Update the test cases in your playwright suites including the TestRail test IDs in the test names or descriptions. For example:

test('<testrail test case id - say C9708595> <Test case name say - 'Validate the home page is loaded'>', async () => {
    // Test steps here
});

Note: Please have the testrailid followed by a space and then the test case name/title

5. Run your tests with the same commands as before.

• The current reporter will create a new run if needed, update the test results and publish the test run url at the end of the tests. In case of failures, it will log the console error to the test case and the screenshot to the test run.
• Note: Mandatory** Please add a process.env.ENV to the command line run command with a value matching the testrail.config.js -> TESTRAIL_SUITE_ID -> 'project short name 1'. This is used to match the test case suite id and create a test run accordingly

TestRail Configuration

Navigate to Testrail UI -> Administration -> Site Settings -> Check both the checkboxes for 'Enable API' and 'Enable session authentication for API'. These settings needs to be enabled for the reporter to upload results to testrail.

Sample testrail config js file

{
    "TESTRAIL_HOST": "https://test.testrail.com",
    "TESTRAIL_USERNAME": "[email protected]",
    "TESTRAIL_PASSWORD": {
        "encrypted": "<encrypted password>",
        "iv": "<IV value>"
    },
    "TESTRAIL_PROJECT_ID": 159,
    "TESTRAIL_SUITE_ID": {
        "AR": 13104,
        "RAPI": 13314,
        "ER": 13044
    },
    "TESTRAIL_API_ATTACHMENT_PATH": "/index.php?/api/v2/add_attachment_to_run/",
    "TESTRAIL_RUN_NAME": "Front End_Regression Automation_Playwright - Created on ",
    "TESTRAIL_RUN_ID": ""
}