test-run

1. Introduction

@empiricalrun/test-run is a CLI tool (and library) designed to facilitate running Playwright (or Appwright) tests in a project. It allows you to:

• Run all tests or a specific test.
• Filter tests by name, file, or suites.
• Filter Playwright projects to run.
• Skip teardown test blocks.
• Integrate with an external dashboard for environment and build management.
• Forward additional arguments directly to the underlying Playwright/Appwright test runner.

2. Installation

Install from npm (usually as part of an Empirical-run monorepo, but can be done standalone):

npm install -D @empiricalrun/test-run

You can also use the CLI directly with npx.

3. Basic Usage

The CLI entry point is defined in package.json under "bin". You can invoke it using:

npx @empiricalrun/test-run [flags] [Playwright/test arguments]

Example (running a test named "foo"):

npx @empiricalrun/test-run -n "foo" --retries 0

Here, --retries 0 (and any unrecognized options) get passed on to the underlying Playwright command.

4. CLI Flags

Below are the main CLI options implemented in src/bin/index.ts: | Flag/Option | Description | |-----------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------| | -n, --name <test-name> | The scenario name (title) of the specific test to run. If provided, only that particular test/scenario is run instead of all tests. | | -d, --dir <test-dir> | Directory containing tests. Defaults to tests. | | -f, --file <test-file-path> | Specific test file path. If provided along with --name, it targets that file for the named test. | | -p, --project <project-name...> | One or more (space-delimited) project names/patterns used to filter the test run. Examples: --project web or --project ios desktop. | | -s, --suites <suites> | A comma-separated list of suite or describe-block names. Example: --suites "suiteA,suiteB". | | --skip-teardown | Skips teardown tests by marking them as “skipped.” Reverts changes when the process exits. | | --token <token> | A base64-encoded token that, when decoded, can define test configuration (such as a test list). | | --forbid-only | Fails the run if there are any .only blocks in the code. |

Other unrecognized flags on the command line will be appended as Playwright arguments.

5. Environment Variables

The following environment variables can further customize behavior:

• PROJECT_NAME
  Overrides project name detection from package.json.

• TEST_RUN_ENVIRONMENT
  This helps fetch playwright projects, environment details and potential build artifacts before tests start.

• BUILD_URL
  Overrides the build URL for downloading any build artifacts.

6. Running group of tests vs. All Tests

• Group Test:
  Use --name to identify one tests with the specfied name, optionally adding --file or --suites if needed.

  npx @empiricalrun/test-run --name "my test" --file tests/example.spec.ts

• All Tests:
  Omit --name. This runs all tests.

  npx @empiricalrun/test-run --reporter=list

7. Skipping Teardown Tests

If you pass --skip-teardown, the CLI will:

1. Temporarily annotate *.teardown.ts files so that they are skipped (for example, converting teardown calls to teardown.skip).
2. After the run, it automatically reverts the code changes, triggered by process exit signals.

This is useful for omitting teardown steps during certain runs or local development.

9. Environment & Build (Optional)

If TEST_RUN_ENVIRONMENT is defined:

1. The CLI fetches environment details (and optionally a build artifact) via the remote “dashboard” (see dashboard.ts).
2. If no custom BUILD_URL is set, it uses the environment’s latest build.
3. It runs an npm script named download to grab the build.

This flow makes sure tests run against the relevant environment or a fresh build artifact.

10. Examples

1. Run a single named test

   npx @empiricalrun/test-run -n "login test"

2. Use specific test file and name

   npx @empiricalrun/test-run -n "logout test" -f tests/logout.spec.ts

3. Run multiple projects and skip teardown

   npx @empiricalrun/test-run -n "mobile login" --project android --project ios --skip-teardown

4. Run all tests with Playwright arguments

   npx @empiricalrun/test-run -- --worker=2 --reporter=line

11. Troubleshooting & Notes

• The CLI automatically appends unrecognized flags to the underlying playwright test (or appwright test) command.
• Errors like “No test block found” often mean the CLI did not locate your test name or file. Double-check --name, --file, etc.
• The teardown skip logic modifies files but reverts them on exit. If a process is forcibly killed, you may need to manually restore.
• If environment fetching or build downloading fails, confirm your environment slug and project name match those recognized by your dashboard.

14. Conclusion

@test-run simplifies orchestrating Playwright tests, especially for:

• Filtering or focusing on specific test scenarios.
• Managing environment-based test runs.
• Skipping teardown blocks.
• Working with multiple or specialized Playwright projects.

Explore the source (particularly src/bin/index.ts and src/utils) for deeper technical insights and advanced customization.