e2e-core

UI test automation core framework which is based on Playwright and created specifically for SPS Commerce internal UI projects.

Connect e2e-core subproject to an exisitng UI project

1. Create a new folder for an e2e subproject (e.g. tests-e2e/) with a separate package.json file there.
2. Add e2e-core and all other packages that you need for the e2e testing into package.json.
3. Go into the tests-e2e/ directory and install node modules from package.json.
4. Create .credentials.sh file in the tests-e2e/ directory with the external SPS user credentials.

export USER_EMAIL="EXAMPLE_USER"
export USER_PASSWORD="EXAMPLE_PASSWORD"
# Note: if USER_PASSWORD contains \ or " symbols, please add \ symbol before it.

Attention: Internal SPS user login flow requires MFA step. Accounts with MFA cannot be fully automated, and need manual intervention. Follow the discussion: https://github.com/SPSCommerce/sps-atlas/discussions/111

5. Create .env file in the tests-e2e/ directory with an appropriate process.env variables listed below in the table. E.g.:

TEST_TIMEOUT=75000
EXPECT_TIMEOUT=10000
ACTION_TIMEOUT=45000
WORKERS=6
REPO_NAME=fulfillment-ui-v2
BASEURL=https://test.commerce.spscommerce.com/fulfillment
VIEWPORT={"width": 1920, "height": 1080}

| Variable | description | default | possible values | | ------------------: | -------------------------------------------------- | ------------------------------ | ------------------------------------ | | HEADLESS | launch browser in headless/headful mode | true | true, false | | BROWSER | use + as a separator, e.g edge+firefox | chromium | chromium, firefox, webkit, edge | | TEST_TIMEOUT | global timeout for each test | 30000 | [number_in_ms] | | EXPECT_TIMEOUT | global timeout for each expect | 5000 | [number_in_ms] | | ACTION_TIMEOUT | global timeout for each PW action, like click() | 30000 | [number_in_ms] | | SLOWMO | slows down the actions by the specified ms | 0 | [number_in_ms] | | WORKERS | number of workers (test threads) | 1 | [number] | | TRACER | create PW actions trace file per each failed test | false | true, false | | VIEWPORT | emulates consistent viewport for each page | {"width": 1536, "height": 960} | [string] | | CI | used in CI: forbidOnly and maxFailures are applied | false | true, false | | SUITE | used in CI: report file name is created per suite | TEST-results.xml | [string] | | ENV | invoked auth state environment flag* | test | test, prod | | BASEURL | auth state is invoked for the provided url* | - (required) | [string] | | TEST_DIR | used in package.json: tests directory path | - (required) | [string] | | CONFIG_PATH | used in package.json: .env file path | - | [string] | | SETUP_PATH | used in package.json: set sm-th before test start | - | [string] | | REPO_NAME | creates GH repo link on test in log files | - | [string] |

* If you have several urls to run your tests for, consider the following possible set up in the SETUP_PATH file instead of providing BASEURL in the .env file:

const domain = 'commerce.spscommerce.com';

process.env.BASEURL = `https://test.${domain}/fulfillment`;

if (process.env.ENV) {
  switch (process.env.ENV.toLowerCase()) {
    case 'prod':
      process.env.BASEURL = `https://${domain}/fulfillment`;
      break;
    case 'prodrc':
      process.env.BASEURL = `https://${domain}/fulfillment-rc`;
      break;
    case 'test':
      process.env.BASEURL = `https://test.${domain}/fulfillment`;
      break;
    case 'testrc':
      process.env.BASEURL = `https://test.${domain}/fulfillment-rc`;
      break;
    default:
      process.env.BASEURL = `https://test.${domain}/fulfillment-${process.env.ENV.toLowerCase()}`;
      break;
  }
}

It allows to set url for testing via CLI instead of setting it up in .env file. Plus, auth state is invoked only once for test/prod url groups due to ENV logic in the globalSetup.ts file.

6. Setup the following "scripts" commands in the package.json according to your paths:

"test": ". .credentials.sh && CONFIG_PATH=$(pwd)/.env SETUP_PATH=$(pwd)/src/helpers/urlSetup.js TEST_DIR=$(pwd) HEADLESS=false playwright test --config=node_modules/@spscommerce/e2e-core/playwright.config.js",
"test:ci": "CONFIG_PATH=$(pwd)/.env SETUP_PATH=$(pwd)/src/helpers/urlSetup.js TEST_DIR=$(pwd) CI=true playwright test --config=node_modules/@spscommerce/e2e-core/playwright.config.js"

7. Add your first test file (see Example.test.ts file).

Attention: Start all the decribe block names with SPS_ prefix. If not, created screenshots and log files won't be attached to an appropriate test.

Additional options

If CI is set to true, the forbidOnly and maxFailures options are applied.

• forbidOnly: Fail the build on CI if you accidentally left test.only in the source code
• maxFailures: Stop test suite execution after reaching 8 failed tests and skip any tests that were not executed yet.

Authentication

Authentication flow was implemented in the e2e-core library. When tests are launnhed for the first time, it takes some time for the authentication state to be invoked. If ENV variable is not set, the storageState is created for the test environment, otherwise set ENV to 'prod'. After auth state is invoked, it appears in the results-e2e/auth folder. All the next test runs will use an already created storageState.json file till its TTL. After TTL is reached, the auth state is invoked again.

Global setup

To set something up once before running all tests, create a JS file and link it via SETUP_PATH variable in the package.json. Global setup file must export a single function that runs once before all the tests.

Results

After tests finishes, you can observe test results in the results-e2e/ folder:

• screenshot folder with screen and trace file (if TRACER=true) for each failed test
• log folder with log file for each test regardless its status
• reports folder with compatible junit xml files (Azure test results friendly format)
• auth folder with an invoked auth state per ENV

Tracer

You can observe the PW trace file in 3 ways:

• open a trace file via the terminal CLI: npx playwright show-trace [trace_file_path].zip
• use PW Trace Viewer web app: https://trace.playwright.dev/
• install "Playwright Trace Viewer for VSCode" extension: https://marketplace.visualstudio.com/items?itemName=ryanrosello-og.playwright-vscode-trace-viewer

Dockerfile

PW provides an official Docker image. These image includes all the dependencies needed to run browsers in a Docker container, and also include the browsers themselves. Here is an example of the Dockerfile:

FROM mcr.microsoft.com/playwright:v1.28.0-focal

ARG BROWSER

RUN mkdir e2e
WORKDIR /e2e
COPY . .
RUN yarn
RUN if [ "$BROWSER" = "edge" ]; then npx playwright install msedge; fi
RUN if [ "$BROWSER" = "chrome" ]; then npx playwright install chrome; fi

Attention: Installed Playwright package version must match the Docker image version.

Run tests in CI (Azure Pipelines)

In order to add an Azure Pipeline for your framework, head over to https://dev.azure.com/spscommerce/ and find your team. You can create a new pipeline by selecting your Git repository. For more information on getting started with Azure, visit https://github.com/SPSCommerce/azure-pipelines-config.

Attention: Do not forget to set up USER_EMAIL and USER_PASSWORD private variables in Azure UI. Here is an example of a job in the yaml file:

jobs:	
  - job: TestRun	
    pool: BUILD-LINUX	
    timeoutInMinutes: 20
    steps:	
    - task: DockerCmd@7	
      displayName: 'Test execution'	
      continueOnError: true	
      inputs:	
        dockerfile: 'Dockerfile'	
        dockerBuildArgs: '--build-arg BROWSER=${{parameters.browser}}'	
        dockerImage: 'ffui/e2e'	
        imageSource: 'Dockerfile'	
        dockerRunArgs: --ipc=host --memory=6g --shm-size=1g	
        awsAssumeRole: $(BuildRole)	
        dockerEnvVars: |	
          USER_PASSWORD=$(USER_PASSWORD)
          USER_EMAIL=$(USER_EMAIL)
          BROWSER=${{parameters.browser}}
          ENV=${{parameters.url}}
          SUITE=$(suite)
          CI=true
        dockerCmd: |
          npx playwright install
          xvfb-run --auto-servernum npm run test:ci $(suite)
        mountCustomVolume: true	
        externalMountLocation: '$(Build.ArtifactStagingDirectory)/results'	
        internalMountLocation: '/e2e/results-e2e'
    - task: PublishTestResults@2 
      displayName: 'Publish test results'
      inputs:
        testResultsFiles: '**/*.xml'
        searchFolder: '$(Build.ArtifactStagingDirectory)/results/reports/'
    - task: TestingInspector@1
      displayName: 'Publish log files and failed test screenshots'
      inputs:
        screenshotsFolder: '$(Build.ArtifactStagingDirectory)/results/screenshots'
        logsFolder: '$(Build.ArtifactStagingDirectory)/results/logs'
    - task: PublishBuildArtifacts@1
      displayName: 'Upload test results artifacts'
      condition: always()
      inputs:
        pathtoPublish: '$(Build.ArtifactStagingDirectory)/results'
        artifactName: 'results'
        publishLocation: 'Container'
    - task: SlackNotification@1
      displayName: Slack notification (qa channel)
      condition: in(variables['Build.Reason'],'Manual')
      inputs:
        channel: 'azure-autotests-private'
        defaultMessage: true

Notes:

• --ipc=host is used since Chrome can run out of memory without this flag Docker doc
• --shm-size=1g is used since Docker runs a container with a /dev/shm shared memory space 64MB which is typically too small for Chromium and will cause Chromium to crash when rendering large pages
• --auto-servernum usage is recommended in order to run command with a random display number
• xvfb-run is used on Linux agents for headed execution since it requires Xvfb to be installed. Playwright official Docker image has Xvfb pre-installed

TestingInspector@1 - Azure DevOps extension used in CI pipeline to upload screenshot and log files into the test results report.

Q&A

If you have any questions or difficulties with this package, reach out to us in #ffqa on Slack. Additional info over this framework could be found on Confluence. Put in a PR against this repo if you fix a bug or come up with an improvement!

Used tools

• @playwright/test - browser automation and test runner tool
• @typescript-eslint/eslint-plugin - ESLint plugin which provides lint rules for TypeScript codebases
• @typescript-eslint/parser - ESLint parser which leverages TypeScript ESTree to allow for ESLint to lint TypeScript source code
• dotenv - loads environment variables from a .env file into process.env
• eslint - static code analysis tool
• jest-expect-message - gives ability to add a custom error message to the assertions
• jest-extended - adds additional matchers to default ones making it easy to test everything 🙌
• typescript - language for application-scale JavaScript that adds optional types