@qaguardian/playwright-score

Deterministic, AI-free quality score for Playwright specs (sqs-v1).

Website: qaguardian.com/open-source/playwright-score
Built by: QA Guardian — managed Playwright E2E (AI drafts, engineers verify, you own the code)

Lint + score Playwright tests against community best practices via eslint-plugin-playwright plus suite-level metrics (locator ratio, assertion-delegation tracing, Page Object Model import resolution, ...).

The score never calls an LLM. AI may generate or repair code using findings; rules grade code.

See METHODOLOGY.md for the frozen formula, VALIDATION.md for real-world results against public repos (cal.com, Grafana, Mattermost, ...) and a re-runnable script to reproduce them, or the full product write-up on the landing page.

Install

npm install -D @qaguardian/playwright-score

Package: @qaguardian/playwright-score on npm.

CLI

# One-shot (scoped package — use -p so the playwright-score binary is resolved)
npx -p @qaguardian/playwright-score playwright-score ./tests --profile standard --threshold 80

# After local install
npx playwright-score ./flow.spec.ts --format json --out report.json

| Flag | Description | |---|---| | --profile standard | Scoring profile (default, and only, profile) | | --threshold <n> | Pass bar 0–100 | | --format text\|json\|markdown\|sarif | Output format | | --out <file> | Write report to file | | --version | Print version |

Exit codes: 0 pass · 1 below threshold or no files matched · 2 tool error

Library

import { scorePaths } from '@qaguardian/playwright-score';

const result = await scorePaths({
  paths: ['tests/login.spec.ts'],
  profile: 'standard',
  threshold: 80,
});

console.log(result.score, result.grade, result.pass, result.findings);

GitHub Action

Drop this into a workflow to gate PRs on the score, post a job summary, and keep a sticky PR comment with the full findings up to date:

- uses: qa-guardian/playwright-score@v1
  with:
    paths: tests e2e
    # threshold: 80        # defaults to 80
    mode: gate              # gate: fail CI below threshold · warn: report only

| Input | Description | Default | |---|---|---| | paths | Space-separated paths/globs to score | tests | | profile | standard (default, and only, profile) | standard | | threshold | Pass bar 0–100 | 80 | | mode | gate (fail CI below threshold) | warn (report only) | gate | | comment | Post/update a sticky PR comment | true | | version | @qaguardian/playwright-score version to run | latest | | github-token | Token for the PR comment | ${{ github.token }} |

Outputs: score, grade, pass — usable by downstream steps (e.g. a custom badge, a Slack notification on regression, etc).

Prefer a raw CLI call, or a non-GitHub CI system? See the CLI section above — same score, same exit codes:

- name: Playwright Spec Score
  run: npx -p @qaguardian/playwright-score playwright-score ./tests --profile standard --threshold 80 --format text

QA Guardian integration

QA Guardian's own codegen pipeline (playwright_runner) dogfoods this package for the standard profile, layering its own private house-rules gate on top internally — that layer isn't part of this package (it's product-specific, e.g. "timeouts must be exactly 2000 or 20000ms", not Playwright best practice) and isn't published here. playwright_runner sets:

| Env | Values | Default | |---|---|---| | SPEC_SCORE_MODE | off | warn | gate | warn | | SPEC_SCORE_THRESHOLD | 0–100 | 80 |

• warn: log score; never fail the run; findings still inject into heal/generate repair prompts
• gate: fail the run when score < threshold
• After AI generate, specs below threshold get one automatic heal pass with score findings
• Heal prompts always include score findings when the scorer reports issues

# Local from monorepo
cd playwright-score && npm run build
cd ../playwright_runner && npm install
SPEC_SCORE_MODE=warn node ...

Development

npm install
npm run build
npm test
npm run score -- ./fixtures/good-standard.spec.ts

License

MIT