FUNCTIONAL TEST RUNNER (eol-ft-runner)

This framework uses Selenium-webdriver, wd, Appium & CucumberJS to run UI tests written in BDD format. This project is a work in progress. As we build towards v1.0, you can checkout the "Projects" section of GitHub to keep track of our progress.

How to install

npm install eol-ft-runner -D

Browser Support status

| Browser | macOS/Linux | Windows | | ------------- | ------------- | ------------- | | Chrome | ✅ | ✅ | | Chrome Headless | ✅ | ✅ | | Firefox | ✅ | ✅ | | Firefox Headless | ✅ | ✅ | | Safari | ✅ | N/A | | Edge | ✅ | ✅ | | Edge Headless | 🛠 WIP | 🛠 WIP | | Android Chrome | ✅ | ✅ | | iOS Safari | ✅ | N/A | | Internet Explorer | N/A | 🛠 WIP | | Opera | ❌ | ❌ | | Brave | ❌ | ❌ |

Mobile App status

| Platform | macOS/Linux | Windows | | -------- | ----------- | ---------- | | Android App | ✅ | ✅ | | iOS App | ✅ | N/A |

Note on browser drivers

Browser drivers are not bundled with this package. You will need to install the browser drivers (Chromedriver or Gecko Driver or Edge Driver) - in your project using the below command:

npm install chromedriver --save-dev

(or)

npm install geckodriver --save-dev

(or)

npm install @sitespeed.io/edgedriver --save-dev

This will allow you to control your browser driver version, based on the browser version you are testing against.

Run GUI test(s) on your local environment:

Run your test with a configuration file:

./node_modules/eol-ft-runner/bin/ft-runner --config config.json

Run your test with CLI arguments:

./node_modules/eol-ft-runner/bin/ft-runner --config config.json --browser chrome --tags @sanity --cores 2

Run headless using --headless (only supported on chrome/firefox):

./node_modules/eol-ft-runner/bin/ft-runner --config config.json --browser chrome --tags @sanity --cores 2 --headless

Run using npm scripts:

npm run test -- --config config.json --browser chrome --tags @sanity --cores 2

In your package.json, the npm script test should point to the ft-runner executable: ./node_modules/eol-ft-runner/bin/ft-runner.

browser can be 'chrome', 'firefox', 'safari', 'edge', 'android', 'ios'.

tags are cucumber tags found on the top of a scenario inside a feature file. Tags are optional, and will execute all scenarios if not provided. To run multiple tags, use --tags "@sanity @smoke @etc".

cores are the number of parallel threads of execution specified in the format "--cores 3" or "--cores 10". Cores are optional, and will run on 2 cores if not specified. For Safari & Edge browsers, cores will be defaulted to 1 due to their respective browser instance restrictions.

Configuration file attributes

| Attribute name | Mandatory? | Type | Example | | -------------- | ---------- | ---- | ------- | | featurePath | Yes | String | "./path/to/feature/dir" | | stepDefinitionPath | Yes | String | "./path/to/step_def/dir" | | supportFolderPath | Yes | String | "./path/to/support/dir" | | reportFolderPath | Yes | String | "./path/to/report/dir" | | browser | No | Array of Strings | [ "chrome", "firefox" ] | | tags | No | String | "@sanity" "@sanity @smoke" | | parallelType | No | String | "features" | | rerun | No | Boolean | true | | remoteAppiumHub | No | Object | {"address": "https://www.example-cloud.com/wd/hub","port": 8081} port is optional| | desiredCapabilities | No | Object | View below sample |

{
  "configurations": {
    "browser": [ "chrome" ],
    "tags": "@sanity",
    "featurePath": [ "test/features/" ],
    "stepDefinitionPath": [ "test/step_definitions/" ],
    "supportFolderPath": [ "test/support" ],
    "reportFolderPath": "test/reports",
    "rerun": true,
    "remoteAppiumHub": {
      "address": "https://www.example-cloud.com/wd/hub",
      "port": 8081
    },
    "desiredCapabilities": {
      "chrome": {
        "browserName": "chrome",
        "unhandledPromptBehavior": "accept"
      },
      "ios": {
        "browserName": "Safari",
        "platformName": "iOS",
        "platformVersion": "14.1",
        "deviceName": "iPhone 12",
        "automationName": "XCUITest",
        "startIWDP": true,
        "newCommandTimeout": 30,
        "safariAllowPopups": true
      }
    }
  }
}

Create the config.json file anywhere in your project, and provide its relative path as a command-line argument: --config <relative_path_of_config.json>. For Windows, replace all instances of forward-slashes(/) each with 2 backslashes(\).

CLI arguments

|CLI argument|Description|Expected value type|Example| |----------|--------|----------|----| |--config /path/to/config.json|Mandatory configuration file that defines the location of feature, step-definition and hook files|String| --config ./config.json | |--browser [browsers]|Specify the browser name for the session (can be 'chrome', 'firefox', 'safari', 'edge', 'android', 'ios')|String|--browser chrome --browser chrome firefox (for parallel execution)| |--headless|Attempt to run a headless session (applicable for Chrome, Firefox)| Boolean (optional)|
|--tags <tags>|Provide select cucumber tags to be executed|String|e.g. --tags @sanity e.g. --tags "@smoke @sanity" (for multiple tags)| |--parallelType features|Provide this flag to run scenarios from a single feature within the same thread|String|e.g. --parallelType features| |--remoteAppiumHub <url>|Provide this flag to override the appium url even if one is provided in the config file|String|e.g. --remoteAppiumHub https://appium.samplehuburl.com/| |--addDesiredCaps <desiredCapObject>|Add an entirely new desired capability object or add new attributes to an existing desired capability object|Stringified JSON|e.g. --addDesiredCap "{\"ios\":{\"secretKey\":\"shhd0ntte11any1\"}}"| |--rerun|Re-execute all failing tests one more time|Boolean (optional)| |--cores <n>|Number of threads in execution during parallel run|Number (default: 2)| --cores 4| |--retry <n>|Cucumber-js native retry mechanism to run any failed test scenario(s)|Number| --retry 3| |--webhookUrl <url>|Slack webhook url for Slack notifications|String|--webhookUrl https://webhookurl.slack.com/blah/| |--ciLinkTitle <title>|Set title of the Slack message forwarded using Slack webhook URL|String|--ciLinkTitle "Build 14"| |--ciLinkUrl <url>|Set hyperlink URL for the slack title|String|--ciLinkUrl https://jenkins/url|

Important: key-value pairs passed via CLI arguments will override the same key-value pairs provided in the configuration json file. For example, --browser chrome CLI argument will override any browser attribute setting in the configuration file.

Read more:

• Driver class methods
• Differentiating desktop browser and mobile browser tests
• Running tests on Android chrome browser
• Running tests on iOS Safari browser
• Desired capabilities for Android & iOS using Appium
• Debugging tests in VSCode

Sample test project

Sample node project using eol-ft-runner: https://github.com/moneesh2011/test-eol-ft-runner