squally-reporter

Playwright reporter that sends test results to your Squally dashboard.

Learn more at app.squally.dev.

Requirements

• Node.js >= 20.18.1
• @playwright/test >= 1.30.0

Install

npm install -D squally-reporter

Config

Set one env var (e.g. as a CI secret):

SQUALLY_INGEST_KEY=your-project-ingest-key

Find your ingest key in the Squally dashboard under Settings › Project › Ingest key.

Since 0.12.0 the URL defaults to https://app.squally.dev. Set SQUALLY_INGEST_URL only to report to another Squally. Without a key the reporter sends nothing and says so once.

Renamed in 0.9.0. These were SQUALLY_API_URL and SQUALLY_API_KEY. The old names still work for now, with a deprecation warning in the CI log - but they belong to Squally's read API now, so rename them. SQUALLY_API_URL is only used together with SQUALLY_API_KEY, never on its own - and when it is set but not used, the reporter sends nothing rather than falling back to the default URL, since the variable says you meant another target.

A read key is never sent. A key starting with sqly_ro_ is a read API key, for scripts and agents; reporting needs the ingest key. Since 0.12.0 the reporter refuses one in any of SQUALLY_INGEST_KEY, SQUALLY_API_KEY or the apiKey option before sending anything, and says which one to fix.

Then add the reporter to playwright.config.ts:

import { defineConfig } from "@playwright/test";

export default defineConfig({
  reporter: [
    ["html"],
    ["squally-reporter"],
  ],
});

Replace your reporter line, don't add a second one. npm init playwright writes reporter: 'html',, and with two reporter: keys in the config the later one wins without a warning - either the HTML report or Squally silently goes. Using other reporters? Keep them in the list and add ["squally-reporter"].

Alternatively, pass config explicitly instead of using env vars - but prefer the env vars above for apiKey, and not only because a literal in a committed file is obviously wrong. Playwright's json and blob reporters serialize your resolved config into their output file, options and all: an apiKey passed here ends up in that file in plain text, even when its value came from an environment variable at runtime. What gets written is the option, not where it came from. The reporter warns, naming the file, when it sees this combination. Measured against a real report, 20.09.2026.

reporter: [
  ["squally-reporter", { apiUrl: "https://app.squally.dev", apiKey: process.env.SQUALLY_INGEST_KEY }],
],

That's it - run your tests as usual (npx playwright test); results stream to Squally while the suite runs (see How results reach Squally below).

Artifact upload (screenshot, trace, browser console)

If your playwright.config.ts captures screenshots or traces, the reporter uploads them to Squally so they show up next to the failing test in the dashboard - no separate setup beyond the two env vars above.

Three kinds are uploaded:

| Kind | Where it comes from | | --- | --- | | Screenshot | the file Playwright writes for a failed attempt | | Trace | the trace zip Playwright writes | | Browser console | read out of that trace zip by this reporter |

The browser console needs a trace. It is not something Playwright hands a reporter: result.stdout/stderr are the output of the Node process - what your test file printed - while what the page printed only exists inside the trace. So the reporter opens the trace and reads it out, which means an attempt with no trace has no browser console, and nothing can change that but turning tracing on. Squally's attempt page says which of the two it is rather than showing an empty section.

Supported trace schema versions: 5 to 10, which covers Playwright 1.30 through 1.62 (1.62 writes version 8). A trace outside that range is left alone and the reporter prints one warning naming the version - it never uploads an empty console file, because "no messages" and "this reporter could not read it" must not look the same in the dashboard.

Only from attempts worth looking at: artifacts are uploaded only for attempts that failed, or that needed a retry (even if that retry then passed) - never for an attempt that passed on its first try, regardless of how screenshot/trace are configured. Nobody looks at a screenshot of a test that just passed, and uploading one anyway would only eat into your organization's storage for no benefit.

You need to actually enable capture in playwright.config.ts - without it, Playwright doesn't produce any attachments, so there's nothing for the reporter to find or upload (no error either way, it's a silent no-op):

export default defineConfig({
  use: {
    screenshot: "only-on-failure",
    trace: "retain-on-failure",
  },
  // ...
});

Limits. None of these are enforced by the reporter - Squally applies them when the upload is requested and answers with a reason, which is what the reporter prints. The numbers below are therefore Squally's current ones rather than anything this package guarantees, and can change server-side without a new release of it:

• 50 MB per screenshot or trace, 4 MB per browser console, the same on every plan. A larger file is rejected before any bytes are uploaded; the reporter logs a console.warn naming the file and its limit and moves on. The test result itself is reported as usual, and the run's outcome and exit code are untouched. In the dashboard that artifact is shown as skipped, with the reason, rather than silently missing.
• At most 250 artifacts per shard. Squally keeps the first 250 and skips the rest with the reason limit, which the reporter explains in one line.
• A total storage cap per organization, depending on your Squally plan. Once it is reached, further uploads are rejected the same way until you free up space or upgrade. Note that a fully instrumented suite now produces up to three artifacts per failed attempt rather than two.
• The browser console is a text file of a few KB - it is extracted from the trace you are already uploading, so it adds no capture cost to your run.
• Your test's own stdout/stderr is cut at 16 KB before it is sent. That is this reporter's own limit, not Squally's, and it applies whether or not you capture artifacts at all.
• Two minutes per upload, and 30 seconds for the calls that surround it. An unreachable or very slow server means a skipped upload and a console.warn, not a CI job waiting indefinitely.

Upload is best-effort and fully non-blocking, same principle as the rest of this reporter - a failed or skipped upload never fails your test run or changes its exit code, it only logs a console.warn explaining what happened.

How results reach Squally

Results stream to Squally as your suite runs, not all at once at the end: each test's result is sent in a small batch, so the "running" view in the dashboard stays close to real time even for a long suite. A batch goes out on whichever of these comes first:

• 25 results. Fixed, and the same whether you shard or not.
• A timer: 2 seconds, or 0.4 seconds × the number of shards when that is longer, capped at 30 seconds (0.12.0). Up to five shards every shard flushes every 2 seconds; eight shards every 3.2; sixteen every 6.4; from 75 shards the cap holds it at 30.

Only the timer scales. All shards of a run share one rate budget on Squally's side (300 requests a minute per key), and the timer keeps all of them together at about half of it. The 25-result threshold is left alone deliberately: it bounds how long a fast suite waits, and stretching it would delay results without bounding the burst any better.

A batch also stays under 3 MB (0.12.0): results with long error texts or many steps simply go out in more, smaller requests.

There is no longer a single-request size limit that could discard a whole run - a very large suite simply takes more, smaller requests. As a courtesy the reporter still warns if a run's result count climbs well past 5,000, since that usually means retries are configured higher than intended rather than a suite that's actually that large.

If Squally is unreachable partway through, results already sent stay in the dashboard; only what could not be delivered is lost, and your test run's exit code is never affected either way.

Steps (0.12.0)

For an attempt that failed, or needed a retry, the reporter also sends the steps Playwright recorded - hooks, fixtures, actions, assertions and your own test.steps, with their durations and file:line. Squally shows them on the attempt page, the failing step marked. An attempt that passed on its first try sends none.

• Typed values never leave your runner. The value of a fill, type or pressSequentially is replaced: Fill "•••" getByLabel('Password'). Every title also goes through the same secret masking as error texts.
• At most 300 steps per attempt, 200 characters per title. A longer list keeps the failing step and the steps it sits in, then the rest in order, and tells Squally how many were left out.
• Nothing to configure: steps come from Playwright's own report, with or without tracing.

Only actual test runs appear. A run shows up in Squally once your first test starts, so commands that never run a test create nothing: --list, a globalSetup or webServer that fails to start, a --grep that matches nothing. A run that does execute and legitimately contains no tests — an empty shard, --only-changed with no changed test file, --pass-with-no-tests — is still reported, as a run with zero results.

Timeouts and interrupted tests. A test that runs out of time is reported as failed, and Squally is told that it timed out and what its timeout was - including a test.setTimeout() or test.slow() from inside the test. A test that Playwright cuts off because the whole run is stopping (maxFailures reached, Ctrl+C) is reported as skipped: it did not fail, and it does not count against the test.

Cancelled runs

Cancelled in Squally. An admin can cancel a running run from its page in Squally. That marks the run there; it does not stop your CI job. What the job sends afterwards is discarded, and from 0.12.0 the reporter says so in one line - "This run was cancelled in Squally" - instead of advice about job keys.

A run stopped with SIGINT - Ctrl+C, or a cancelled CI job - shows in Squally as Cancelled: the reporter sends what finished and tells Squally the run was interrupted. Your exit code stays Playwright's own (130). This needs Playwright itself to receive the signal.

On GitHub Actions a cancelled workflow sends the signal to the step's shell, not to Playwright, so the reporter never reports the end. Add one step after the tests (0.14.0) that marks the run cancelled in Squally:

jobs:
  test:
    runs-on: ubuntu-latest
    env:
      SQUALLY_INGEST_KEY: ${{ secrets.SQUALLY_INGEST_KEY }}
    steps:
      # ...checkout, setup-node, npm ci, playwright install
      - name: Run Playwright tests
        run: npx playwright test
      - name: Mark Squally run cancelled
        if: ${{ cancelled() }}
        run: npx squally-reporter cancel

npx squally-reporter cancel finds the run by the same CI run id the reporter sent from this job, marks every shard of it that is still running as cancelled, and prints one line: "Marked run #N as cancelled in Squally (X shards)", "Nothing to cancel: no run recorded for " when the job was cancelled before its first test, or "Run already finished". It never fails the step: a missing key, an unreachable Squally, a 401 or a 5xx is one warning and exit code 0. It needs the key in the step's environment - set it at job level, as above, or on the step too. A Squally without the cancel endpoint answers 404, which is a warning as well.

Two things to know:

• Set SQUALLY_CI_JOB_KEY at job level, not only on the test step. The cancel step builds the id the same way the reporter did; without the key it names another run - or none. npx squally-reporter cancel --job-key <key> passes it explicitly.
• A fail-fast matrix marks the whole group cancelled. With GitHub's default fail-fast: true, one failing shard cancels the others, their cancel steps run, and the whole run shows as Cancelled instead of Failed. Set fail-fast: false on a sharded job so a real failure stays a failure.

Without the step, a cancelled workflow shows as incomplete in Squally.

Without that - or after any hard kill (SIGKILL) - the reporter never gets to report the end: the run stays running until Squally's hourly check marks it incomplete once the project's run timeout (60 minutes by default) has passed. A run stopped before its first test starts sends nothing, like any run that never ran a test, and globalTimeout running out is not a cancel - that run is simply failed.

Which CI, branch and commit a run is reported under

Squally counts CI runs in its test metrics, so the reporter works out which CI system it is running on from that system's own environment:

| Provider | Recognised by | Shards grouped by | Branch | Commit | |---|---|---|---|---| | GitHub Actions | GITHUB_ACTIONS | GITHUB_RUN_ID-GITHUB_RUN_ATTEMPT | GITHUB_REF_NAME; pull requests: GITHUB_HEAD_REF | GITHUB_SHA; pull requests: the PR's head commit | | GitLab CI | GITLAB_CI | CI_PIPELINE_ID | CI_COMMIT_BRANCH; merge requests: CI_MERGE_REQUEST_SOURCE_BRANCH_NAME | CI_COMMIT_SHA; merged-results pipelines: CI_MERGE_REQUEST_SOURCE_BRANCH_SHA | | CircleCI | CIRCLECI | CIRCLE_WORKFLOW_ID | CIRCLE_BRANCH | CIRCLE_SHA1 | | Jenkins | BUILD_TAG (jenkins-…) | BUILD_TAG | CHANGE_BRANCH, else BRANCH_NAME, else GIT_BRANCH | GIT_COMMIT | | Azure Pipelines | TF_BUILD | BUILD_BUILDID | BUILD_SOURCEBRANCH; pull requests: SYSTEM_PULLREQUEST_SOURCEBRANCH | BUILD_SOURCEVERSION; pull requests: SYSTEM_PULLREQUEST_SOURCECOMMITID | | Bitbucket Pipelines | BITBUCKET_BUILD_NUMBER | BITBUCKET_BUILD_NUMBER | BITBUCKET_BRANCH | BITBUCKET_COMMIT | | Buildkite | BUILDKITE | BUILDKITE_BUILD_ID | BUILDKITE_BRANCH | BUILDKITE_COMMIT | | TeamCity | TEAMCITY_VERSION | - (set SQUALLY_CI_RUN_ID) | from git | BUILD_VCS_NUMBER | | AWS CodeBuild | CODEBUILD_BUILD_ID | CODEBUILD_BUILD_ID | CODEBUILD_WEBHOOK_HEAD_REF | CODEBUILD_RESOLVED_SOURCE_VERSION | | Travis CI | TRAVIS | TRAVIS_BUILD_ID | TRAVIS_BRANCH; pull requests: TRAVIS_PULL_REQUEST_BRANCH | TRAVIS_COMMIT; pull requests: TRAVIS_PULL_REQUEST_SHA | | Drone | DRONE | DRONE_BUILD_NUMBER | DRONE_BRANCH; pull requests: DRONE_SOURCE_BRANCH | DRONE_COMMIT_SHA | | any other CI | CI=true or CI=1 | - (set SQUALLY_CI_RUN_ID) | from git | from git |

• CI counts only as true or 1. CI=false or CI=0 is a local run - unless one of the providers above is recognised, which always wins. A run that is none of these is local. Local runs are listed with all other runs, and have numbers of their own: Squally's figures count CI runs by default, and the All · CI · Local switch shows local runs on their own, or both. A local run never posts a GitHub check, a pull-request comment or a Slack message.

• Pull requests are reported under their own head commit and source branch, not the merge commit CI checks out - that is the commit Squally's GitHub check and pull-request comment have to find. The commit message and author are that commit's too - never the "Merge … into …" message of the checkout.

• The head commit is fetched when the checkout lacks it. With actions/checkout's default fetch-depth: 1 the head commit is not in the clone, so the reporter fetches that one commit from origin (git fetch --depth=1 --no-tags), waits at most 3 seconds, and reads its message and author. It only does so in a shallow checkout - fetching into a full clone would make it shallow for the rest of your job - and never changes HEAD or any branch. If the fetch fails or takes longer, the message and author stay empty and the reporter says so once; the run and its exit code are unaffected. Alternatives: fetch-depth: 2 on the checkout step puts the head commit in the clone, so nothing is fetched, and SQUALLY_DISABLE_HEAD_COMMIT_FETCH=true turns the fetch off.

• A tag build has no branch. A detached checkout never reports HEAD as a branch name, and refs/heads/ or origin/ prefixes are removed.

• A link to the CI run (0.11.0). The reporter sends the page of the CI run, so Squally can link a run to it. It is read from each provider's own variable, sent only as an https:// URL of at most 2,048 characters, and never opened by Squally - only shown as a link:

  | Provider | Link | |---|---| | GitHub Actions | GITHUB_SERVER_URL/GITHUB_REPOSITORY/actions/runs/GITHUB_RUN_ID/attempts/GITHUB_RUN_ATTEMPT | | GitLab CI | CI_PIPELINE_URL | | Jenkins | BUILD_URL (only with Jenkins served over https) | | Buildkite | BUILDKITE_BUILD_URL | | Travis CI | TRAVIS_BUILD_WEB_URL | | Drone | DRONE_BUILD_LINK |

  Other providers send no link. It is the run's page, never one job's, and it is the same with SQUALLY_CI_RUN_ID set.

Tests marked with test.fail()

Playwright's test.fail() marks a test as expected to fail: it guards a bug you already know about, and Playwright runs the test anyway to check that it does still fail. The interesting moment is the opposite one - when someone fixes the bug, the test starts passing and Playwright turns it red so you remember to delete the marker.

The reporter passes that marker on, so Squally reads such a test the same way Playwright does:

• Marked and still failing - shown as passing, with an "expected failure" label. It counts as a pass in the test's stability, not against its failure rate, and it does not fail Squally's pull-request check.
• Marked and now passing - shown as a failure, because the marker is out of date and should be removed.
• Marked and timed out - shown as a failure. Playwright does not count a timeout as the expected failure either, so the marker is not passed on for it.

Both ways of marking a test are recognised: the test.fail() modifier and a testInfo.fail() call inside the test body. test.skip() and test.fixme() are a different statement and are not affected.

Nothing is reported for an unmarked test, so an ordinary suite is unchanged. Note also that Playwright itself treats a marked, failing test as expected, so with screenshot: "only-on-failure" it captures nothing for it - there is no artifact for the reporter to upload, which is consistent rather than missing.

Sharding

Handled automatically - no extra setup needed. Each shard is still a separate process with its own reporter instance, but Squally now correlates them itself: all shards of one CI run are grouped together and shown as one run once every shard has finished, with the flip side of that guarantee being that the run only completes once every shard has reported.

Correlation is based on an id read from your CI provider's own environment (the "Shards grouped by" column above), so it costs you nothing on every provider in that table except TeamCity. On TeamCity, on a CI system not in the table, or when you shard locally across several separate --shard invocations, set SQUALLY_CI_RUN_ID to the same value for every shard of one run - anything unique per CI run and shared by its shards works, e.g. a build number your CI system already exposes.

Several jobs in one CI run need their own key (0.11.0). The id is shared by everything in one CI run, so two jobs that are not shards of each other - a GitHub Actions matrix over browsers run without --shard, or an api and a ui suite - would report into the same Squally run. Give each job its own SQUALLY_CI_JOB_KEY, shared only by that job's own shards:

jobs:
  test:
    strategy:
      matrix:
        browser: [chromium, firefox]
    env:
      SQUALLY_CI_JOB_KEY: ${{ matrix.browser }}
    steps:
      - run: npx playwright test --project=${{ matrix.browser }}

The reporter puts the key into the id - <GITHUB_RUN_ID>-<key>-<GITHUB_RUN_ATTEMPT> on GitHub Actions, <provider id>-<key> elsewhere - so each job is its own run in Squally, and a re-run of one job links to that job's own earlier attempt. The key may hold letters, digits, ., _ and -, at most 64 characters; an invalid key is reported once and ignored, and your test run is unaffected. Without a key the id is exactly what it was before 0.11.0. A suite that is sharded and split into jobs uses the same key for all shards of one job: SQUALLY_CI_JOB_KEY: ${{ matrix.suite }} beside --shard=${{ matrix.shard }}/2.

Replaces the hand-built id. Until 0.11.0 this section suggested SQUALLY_CI_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}-${{ matrix.browser }}. It still groups each job correctly, but Squally cannot tell its attempts apart, so a re-run is never linked to the attempt before it. Switch to SQUALLY_CI_JOB_KEY. An explicit SQUALLY_CI_RUN_ID still wins for the id; the key is then only sent alongside it.

When two jobs collide anyway, Squally refuses the second one at the start (squally-app with block 1b, reporter 0.11.0). The reporter prints one block, sends nothing more for that job, and leaves your test run and its exit code alone:

• ci_run_collision - another job already reports under this CI run id (and shard). Give every job its own SQUALLY_CI_JOB_KEY.
• ci_run_mixed - the id would put a sharded job and an unsharded one, or shards of different totals, into one run. The same fix.

Before 0.11.0 the second job was handed the first job's run instead, and its results were refused once that run completed; a reporter older than 0.11.0 still behaves that way, and prints "Squally has already completed run …".

Re-runs. On GitHub Actions every re-run attempt is its own run in Squally, linked to the attempt before it ("Re-run of #N"). "Re-run failed jobs" runs only the shards that did not pass; that attempt waits only for those shards, completes once they have reported, and passes or fails on the tests they ran - the reused shards are neither waited for nor counted, and the run page shows them as "passed in #N · not re-run". A re-run shard that never reports still leaves the run incomplete after the run timeout.

Other CI systems re-run a job under the id of the run it belongs to - a GitLab job retry, an Azure Pipelines re-run, a Bitbucket re-run of failed steps, a Buildkite retry, a Travis CI restart. Since 0.11.0 Squally refuses such a retry at the start with ci_run_collision: the slot is taken by the first attempt. SQUALLY_CI_JOB_KEY does not help here - the retry is the same job - so a retry on these systems is not reported; the first attempt stays the run. A Drone restart is a new build with a new number, and so a new run in Squally, not linked to the one it repeats.

playwright merge-reports is not part of this and is not recommended for getting a single run in Squally - the automatic correlation above is the supported path, and produces correct per-shard timing/branch/commit information that a merge job cannot (see the historical note below). Nothing stops you from using merge-reports for Playwright's own HTML report if you want that separately; just don't point this reporter at the merged result expecting Squally to treat it specially.

An earlier version of this reporter recommended playwright merge-reports as the way to get one Squally run out of a sharded job. That path has a real problem: branch, commit, commit message and author are all read on whichever machine the reporter actually runs on - during a merge that is the merge job, not the shards - and without a checkout there (and without the CI provider's own commit variables) they all come out empty. A run with no branch is in no branch's numbers and no branch filter finds it, so a "merged" run could exist in the dashboard while missing from everything Squally measures per branch: suite stability, and each test's stability, flaky rate and failure rate on that branch. The automatic correlation above reads that information from each shard's own environment instead, so this problem does not arise.

Tested revision (optional)

The tested revision is the state of your application a run tested, as opposed to the commit the run is reported under. Squally does not know it by itself - it is opt-in, with one of these two:

# The tests live in the same repo as the application: the commit this CI job
# checked out is the tested revision.
SAME_REPO_AS_APP=true

# The tests live in a separate repo (e.g. a dedicated E2E repo pointed at a
# deployed app): pass the tested revision itself - a commit SHA, tag, or
# branch name/ref.
TESTED_REVISION=v1.2.0

• SAME_REPO_AS_APP=true sends the commit your job checked out: your CI provider's own variable for it (GITHUB_SHA, CI_COMMIT_SHA, CIRCLE_SHA1, GIT_COMMIT, BUILD_SOURCEVERSION, BITBUCKET_COMMIT, BUILDKITE_COMMIT, BUILD_VCS_NUMBER, CODEBUILD_RESOLVED_SOURCE_VERSION, TRAVIS_COMMIT, DRONE_COMMIT_SHA), or git rev-parse HEAD where there is none - including local runs. Only a commit SHA is sent; if neither yields one, the run has no tested revision.
• On a pull request the two commits differ, on purpose. The run is reported under the pull request's head commit (see the provider table above) - that is what Squally's check and PR comment need. The tested revision is the merge commit CI checked out, because that is what the tests ran against. On a Bitbucket pull request and a GitHub pull_request_target run it comes from git rev-parse HEAD.
• TESTED_REVISION wins when both are set.
• Neither variable is required - without them, test results are still sent and stored exactly as before.

The value is normalised before it is stored, so that the same revision written two ways still compares equal: it is lowercased, and a leading v or a refs/heads/ prefix is removed. The example above is exactly such a case - v1.2.0 is stored as 1.2.0. The reporter says so once per run when it happens, so a value that looks different in the dashboard is not a mystery.

Troubleshooting

Set SQUALLY_DEBUG=1 to print detail that is otherwise kept quiet - the browser-console extraction (how many messages were found, or that an attempt's trace held none) and, since 0.11.0, a CI run link that was not sent because it was not an https:// URL. Everything that is not ordinary (a trace that cannot be read, an unsupported schema version, a rejected upload) is printed without it.

Support

Found a bug or have a question? Please open an issue on GitHub Issues - that's the primary way to report problems or ask questions. For anything you would rather not post publicly, write to [email protected].

License

MIT - see LICENSE.