@camunda/e2e-test-suite

v0.0.22

Published

a day ago

End-to-end test helpers for Camunda 8

C8 Cross-Component End-to-End Test Suite

Overview

The C8 Cross-Component End-to-End Test Suite represents an extensive collection of end-to-end tests meticulously tailored for Camunda C8, encompassing the entire product stack. This specialized suite is dedicated to test scenarios that traverse multiple C8 components, harnessing the power of Playwright with TypeScript. It adheres to the Page Object Model design pattern, ensuring effortless test creation and seamless maintenance. The focus of the test suite is to align with the C8 persona profiles and their user flows as well as the most important cross-component user flows for the product stack determined by each product team. The persona profiles that will be focused on can be observed in this link

Supported Testing Platforms

C8 SaaS, C8 Self-Managed (SM) with Helm, and C8 Run

This test suite supports testing across three primary platforms:

C8 SaaS
C8 Self-Managed with Helm Chart
C8 Run

Supported Cluster Versions for SaaS

For the Camunda SaaS offering, the test suite supports following cluster versions:

8.5.X
8.6.X
8.7.X
8.8.X
8.9.X

By default, the test suite is configured to run tests on the latest snapshot versions. If you wish to run tests on a different generation, you can adjust the configurations in the .github/c8_versions.json file.

Supported Versions for SM with Helm

For the Camunda SM Helm offering, the test suite supports following versions:

8.5.X
8.6.X
8.7.X
8.8.X
8.9.X

The test suite primarily uses Elasticsearch as the default search engine. However, it also supports OpenSearch, with dedicated workflows explicitly named to indicate OpenSearch as the selected engine.

Supported Versions for C8Run

For the Camunda C8Run offering, the test suite supports following versions:

8.6.X
8.7.X
8.8.X
8.9.X

GitHub Actions for Testing

GitHub Actions can be used to facilitate testing across various cluster versions and components:

SaaS

For SaaS, a test run can be manually triggered for specific minor versions and their associated cluster generation. This can be done through the SaaS On-Demand GitHub Action.

SM with Helm

For SM using Helm, a test run can manually trigger by specifying any component version as an input. TThis can be done through the following workflows:

C8Run

For C8Run, a test run can be manually triggered for specific minor versions, for each supported Operating System. This can be done through the following workflows:

Prerequisites

Before running the test suite, make sure you have the following prerequisites installed:

Node.js (with npm, >= v14)

Getting Started

Clone the repository:

git clone https://github.com/camunda/c8-cross-component-e2e-tests.git

Install the dependencies:

npm install
npx playwright install

Add Enviornmental Variables:

Create a .env file in the root of the project. Add the credentials found in Keeper

Keeper location:
Shared → Engineering All → C8 Cross Component E2E Test Suite Credentials

Important:
Keep the .env file private and do not commit it to version control.
The credentials stored in Keeper are shared across all of Engineering.

If you prefer to use your own credentials, follow the steps below.

Setting up your own organization (optional)

Request the creation of an organisation identical to Automation QA
Org ID: 3626cf46-8f97-4711-81b3-ecfc341348c2
Environment: INT

Requirements:

You must be set as the Owner
Name the organisation: Automation QA - Your_Name

Inviting test users

Within the Automation QA - Your_Name organisation, invite the following users:

Alias of C8_USERNAME
Alias of C8_USERNAME_TEST
Alias of C8_USERNAME_TEST_2
Alias of C8_USERNAME_TEST_3
Alias of C8_USERNAME_TEST_4 (v8.8+)
Alias of C8_USERNAME_TEST_5 (v8.8+)
Alias of C8_USERNAME_TEST_6 (v8.8+)
Alias of C8_USERNAME_TEST_7 (v8.8+)
Alias of C8_USERNAME_TEST_8 (v8.8+)
Alias of C8_USERNAME_TEST_9 (v8.8+)

Passwords & invitation acceptance

Retrieve the Gmail passwords from Keeper:
Shared → Engineering All → C8 Cross Component E2E Test Suite Credentials
Use these emails to sign up and accept the invitations.
Align the passwords with:
- C8_PASSWORD
- C8_PASSWORD_TEST

Running Tests Locally

Run Tests in Headless Mode

To run the test suite in headless mode (without a graphical user interface), use the following command:

npm run test:local

To run one specific test file in headless mode (without a graphical user interface), use the following command:

npx playwright test --project=chromium tests/your-test-folder/your-test-spec

Run Tests with UI (Interactive Mode)

To run the test suite with the UI (interactive mode), append --ui at the end of the npx playwright test command (note the test 'Create Cluster' must be run first):

npx playwright test --ui

Additional Notes for Local Testing

C8 SaaS

For a full test run, no additional configuration is required. However, if you are running a single test, ensure you have a healthy and active cluster.

If you want to run tests against the production environment, add the following to your local .env file:

IS_PROD=true

C8 Self-Managed (SM) with Helm

To test locally on the SM Helm environment, you need an active SM instance. You can create one through any of the following GitHub Actions workflows:

Once the SM instance is created, copy the generated test URL and input it into your .env file under the PLAYWRIGHT_BASE_URL variable.

Note: For workflows ≥ 8.8.x, if you only need to retrieve the test URL without running tests, simply uncheck the Enable tests checkbox.

If you want to run tests with Resource-Based Access (RBA) enabled, add the following to your .env file, otherwise SM tests default to RBA disabled:

IS_RBA=true

If you want to run tests with Multi-Tenancy (MT) enabled, add the following to your .env file, otherwise SM tests default to MT disabled:

IS_MT=true

C8 Run

To test locally against C8 Run, you must set up a local instance of C8 Run. Please follow the instructions provided in the C8 Run README.md. By default, the BASE_URL is localhost:8080, however if this is configured as something differently, tests can be run against this by inputting the new BASE_URL into your .env file under the PLAYWRIGHT_BASE_URL variable.

Automated Nightly Test Runs

The test suite is configured to run automatically every night. The results of these nightly test runs are published on TestRail. If any test fails during these automated runs, the Designated Responsible Individual (DRI) of the project is notified via email for immediate attention and resolution. On a nightly basis, the test suite is configured to run the following tests:

C8 SaaS: Tests are run against a SNAPSHOT generation cluster.
C8 SM with Helm: Tests are run against the alpha branch of the camunda-platform-helm repository, using SNAPSHOT images for each component. This is peformed with both Elasticsearch and OpenSearch.
C8 Run: Tests are run against the main branch of the camunda/camunda repository.

Test Suite Structure

The test suite is organized using the Page Object Model design pattern, where each test file contains test scenarios related to a specific component or feature of the Camunda C8 product stack. Additionally, it leverages separate page objects for each web page or component being tested.

You can find the page objects in the pages directory, and the test files in the tests directory.

For example:

Page files:

LoginPage.ts - Page object for the login functionality.

Test files:

Feel free to add more test files or page objects to expand the test coverage based on your specific requirements.

GitHub Actions Workflow Naming Convention

To maintain consistency and clarity in workflow names displayed in the GitHub Actions UI, we use the following naming pattern:

Naming Pattern:

<Distribution> <Trigger> <Scope (if needed)>

Distribution:
- SaaS → Cloud-based environment
- SM → Self-Managed using the Helm Chart
- C8Run → C8Run environment
Trigger:
- Nightly → Runs automatically every night
- PR → Runs when a pull request is opened/updated
- On-Demand → Runs when triggered via workflow dispatch
Scope (optional):
- Linux, Connectors, OpenSearch, No Tests etc.

Examples

| Workflow Name | Meaning | |--------------|---------| | SaaS Nightly 8.8 | Runs SaaS tests nightly against the latest 8.8 SNAPSHOT | | SM Nightly Chrome 8.7 OpenSearch | Runs SM tests nightly against the latest 8.7 SNAPSHOT in Chrome with OpenSearch | | SaaS/SM/C8Run PR | Runs SaaS or Self-Manged or C8Run tests on PR updates | | SM On-Demand MT No Tests | Creates a Self-Managed environment with Multi-Tenancy Enabled via workflow dispatch with no test run|

By following this convention, we ensure clarity, consistency, and alignment with GitHub terminology.

SM Possible Directory Names

| Directory Name | Meaning | |---------------------------------|--------------------------------------------------------------------------------------------------------------| | camunda-platform-8.9-latest | Runs SM tests against the latest 8.9 SNAPSHOT | | camunda-platform-8.8-latest | Runs SM tests against the latest 8.8 SNAPSHOT | | camunda-platform-8.7-latest | Runs SM tests against the latest 8.7 SNAPSHOT | | camunda-platform-8.6-latest | Runs SM tests against the latest 8.6 SNAPSHOT | | camunda-platform-8.9 | Runs SM tests against the 8.9 directory. This is where the release-candidate-8.9 branch is located | | camunda-platform-8.8 | Runs SM tests against the 8.8 directory. This is where the release-candidate-8.8 branch is located | | camunda-platform-8.7 | Runs SM tests against the 8.7 directory. This is where the release-candidate-8.7 branch is located | | camunda-platform-8.6 | Runs SM tests against the 8.6 directory. This is where the release-candidate-8.6 branch is located | | all-latest | Runs SM tests against all latest versions |

Viewing Test Results

Locally:

The test results are presented as a comprehensive list in the terminal when all tests pass. Additionally, within the project's test-results folder, you can find the test results accompanied by images. In the event of a test failure, a video capturing the issue is provided. Moreover, if any test encounters issues or instability, an HTML report automatically opens, facilitating in-depth analysis.

On the CI:

Within the CI environment, test results are accessible through TestRail, located in the C8 Project folder under Test Runs. These reports not only include detailed test results but also feature screenshots. In case of a test failure, you will find the stacktrace and an accompanying test video attached. Furthermore, the CI environment offers extensive tools for in-depth analysis of test results and reports.

Contributing

Contributions to the C8 Cross-Component End-to-End Test Suite are welcome! If you find any issues or want to add new test cases, please submit a pull request or open an issue on GitHub.

To contribute, please follow these guidelines:

Anyone can contribute.
The reviewer of the pull request must be someone from the Test Automation Team
Test cases must adhere to the Page Object Model design pattern.
Test cases must involve multiple C8 components.

TestRail Integration

To ensure our test case documentation on TestRail remains up-to-date, if a PR contains updates to any test file or page file, the associated TestRail test case must be linked in the PR description.

If you do not have access to TestRail, please contact the Test Automation Team to request access.

Thank you for using the C8 Cross-Component End-to-End Test Suite for Camunda C8 full product stack e2e testing. Happy testing! If you have any questions or need assistance, feel free to reach out to the maintainers for support.