🚀 CementicTest CLI

Turn plain English into production-ready Playwright tests — instantly.

No setup headaches. No framework confusion. Just run and automate.

⚡ Run instantly (no install)

npx @cementic/cementic-test

Works even if your npm environment isn’t perfectly configured.

💡 What this does

CementicTest converts:

👉 Plain English 👉 Into structured test cases 👉 Into a Playwright framework workspace 👉 Then runs them

All from your terminal.

🧠 Why it’s different

• No framework setup required
• No Playwright expertise needed
• No manual scripting to get started
• AI-assisted test generation
• Built-in framework layout with pages, flows, assertions, and specs
• Works from day one

🧪 What you can do

• Generate test cases (manual or AI)
• Capture real websites and generate tests
• Convert cases → JSON → Playwright code
• Run tests instantly
• Scan any web app for accessibility issues (axe-core + AI vision)
• Generate CI workflows
• View reports (Playwright + Allure + Accessibility)

⚡ 60-second quick start

npx @cementic/cementic-test

Then:

ct new demo
cd demo

ct tc --feature "Login form" --count 2
ct normalize ./cases --and-gen
ct test

Done. You just created and ran automation.

🌐 Generate tests from a real website

ct tc url https://mini-bank.testamplify.com/login --ai --feature "Login" --count 2

This will:

• Capture the page
• Understand elements
• Generate realistic test scenarios
• Produce runnable Playwright framework assets

Recent behavior improvements:

• Generic verification prompts now stay scoped to what you asked for. For example, verify the login page loads successfully generates one focused page-verification scenario instead of silently expanding into a three-test auth suite.
• Generic Login prompts still keep the richer auth coverage when you actually want form and credential behavior, so a narrow page-load check no longer breaks broader auth generation.
• If generation fails to produce runnable specs, the pipeline now reports that as a real failure instead of pretending the run was merely skipped.

If no LLM key is configured, capture-aware generation now falls back to deterministic capture-backed scenarios instead of dropping to generic templates.

Important:

• Live capture resolves Playwright from your current project first
• If you are inside a project created with ct new, ct tc url --ai uses that project's local @playwright/test
• If Playwright is missing or Chromium is not installed, the CLI now prints the exact next step instead of failing with a package-resolution stack trace

♿ Scan any app for accessibility issues

# Interactive — prompts for auth, routes, depth
ct scan https://myapp.com

# Fast mode — axe only, no AI cost
ct scan https://myapp.com --routes /,/dashboard --depth fast --ci

# Full mode — axe + vision + AI clustering
ct scan https://myapp.com --routes /,/dashboard,/profile --depth full --ci

# Open last report
ct scan report

Three scanning layers:

| Layer | What | When | |---|---|---| | axe-core | WCAG 2.0/2.1/2.2 AA rule engine | Always (zero AI cost) | | AI Vision | Contrast, focus indicators, reading order, unlabelled controls, small touch targets | When ANTHROPIC_API_KEY, OPENAI_API_KEY, or GEMINI_API_KEY is set + --depth full | | AI Clustering | Groups findings into root-cause clusters with fix suggestions + corrected HTML | When any AI key is set; falls back to deterministic rule-based grouping |

Authentication support:

# Flags
ct scan https://myapp.com --user [email protected] --pass "$MY_PASSWORD" --login-url /signin

# Or reuse existing CT_VAR_* env vars (same pattern as ct gen)
[email protected] CT_VAR_PASSWORD=secret ct scan https://myapp.com --ci --routes /,/dashboard

CI/CD integration:

ct scan https://myapp.com --routes /,/dashboard --depth full --fail-on serious --no-open --ci

--fail-on controls the exit code: critical (default), serious, moderate, or minor.

Reports are written to .cementic/a11y/ as both HTML and JSON.

🧱 How it works

Plain English → Markdown Cases
           → Normalized JSON
           → Pages + Flows + Assertions + Specs
           → Test Execution

URL → axe-core + AI Vision → Clusters → A11y Report

⚠️ If install fails (important)

If you see something like:

npm ERR! Cannot read properties of null (reading 'matches')

👉 That’s an npm issue, not CementicTest.

✅ Fix instantly

Just run:

npx @cementic/cementic-test

No install needed.

🧰 Optional install

npm install -g @cementic/cementic-test

If you use the global CLI, live capture still expects Playwright to be installed in the project you are working in.

🧪 Core commands

Create project

ct new my-suite

Browser bootstrap defaults:

• ct new uses --browser-set auto by default
• auto installs the normal Playwright browser set on modern hosts
• auto switches to Chromium-only compatibility mode only on affected older macOS versions
• you can force Chromium-only installation on any machine with ct new my-suite --browser-set chromium

Generate test cases

ct tc --feature "Checkout" --count 3

Generate from live site

ct tc url <url> --ai

Common setup after scaffolding:

cd my-suite
npm install
npx playwright install chromium
ct tc url https://example.com/login --ai --feature "Login"

Normalize + generate tests

ct normalize ./cases --and-gen

That generation step writes:

• pages/ page objects
• flows/ reusable action helpers
• assertions/ reusable expectation helpers
• tests/generated/ thin specs that import the helpers

Use ct gen --vars when you want generated specs to keep env-backed test data with custom defaults:

ct gen --lang ts --vars '{"email":"[email protected]","password":"supersecret"}'

Run tests

ct test

Full workflow

ct flow

Accessibility scan

ct scan https://myapp.com
ct scan https://myapp.com --routes /,/dashboard --depth fast --ci
ct scan https://myapp.com --routes /,/dashboard,/profile --depth full --fail-on serious --ci
ct scan report                  # opens last report in browser
ct scan report --json           # prints raw JSON to stdout
ct scan report --no-open        # prints the HTML file path only

Release verification

Before publishing, run the full local gate:

npm test

Do not reuse an npm version. npm will reject republishing an existing version even if the Git tag was moved later.

If you want to inspect the prompt regression suite directly:

node --test test/prompt.spec.mjs
node --test --test-reporter=spec test/prompt.spec.mjs
node --test --test-name-pattern="counting intent" test/prompt.spec.mjs

Note: on current Node test runner versions, use --test-reporter=spec, not --reporter=spec.

v0.2.27 acceptance criteria

All of the following must be true before publishing v0.2.27:

• npm test passes from a clean checkout and includes the integration suite, test/prompt.spec.mjs, and test/scan.test.mjs
• prepublishOnly runs the same full release gate, not just build
• the release tag matches package.json
• the target npm version is not already published
• npm pack --dry-run shows the expected publishable payload, including dist/**/*, README.md, CHANGELOG.md, and scripts/postinstall-banner.cjs
• ct gen framework output coverage is verified end to end:
  • generated specs stay thin and import helpers from flows/ and assertions/
  • helper modules remain runnable when CT_VAR_* declarations are present
  • starter templates include the expanded pages/, flows/, assertions/, and tests/generated/ layout
• ct scan feature coverage:
  • ct scan --help lists all flags: --user, --pass, --login-url, --routes, --depth, --out, --fail-on, --no-open, --ci
  • ct scan report --help lists --json, --no-open, --out
  • ct scan without a URL exits with an error
  • ct scan report --no-open prints the path when a report exists
  • ct scan report --json prints valid JSON matching the report data
  • ct scan report errors when no report exists
  • report JSON preserves cluster structure, severity counts, multi-page data, and affected routes
  • --fail-on threshold logic correctly evaluates all four severity levels
  • CT_VAR_USERNAME / CT_VAR_PASSWORD env vars pass through without triggering interactive prompts in --ci mode
  • depth auto-detection documented in help output
• prompt regression suite verifies all existing intent checks (counting, presence, CTA, form, navigation, negative, error, heading)
• all existing ct gen, ct normalize, ct tc, ct new, ct ci, ct flow acceptance criteria from v0.2.20 continue to pass
• README, package version, runtime metadata strings, and changelog entries match the release being cut

Release sequence:

npm test
npm pack --dry-run
npm view @cementic/cementic-test version
git tag v0.2.27
git push origin main --tags
npm publish

Note:

• v0.2.21 introduced the accessibility scan feature set, but that tag failed CI and was not published to npm because the bundled CLI resolved package.json from the wrong path at runtime.
• v0.2.22 is the first publishable release that includes the ct scan work plus the bundled-CLI version-resolution fix.

📊 Reports

ct report              # Playwright report
ct serve               # Allure report (if installed)
ct scan report         # Accessibility report (opens HTML in browser)
ct scan report --json  # Raw JSON accessibility data

⚙️ CI (GitHub Actions)

ct ci

Generates a ready-to-run workflow.

🤖 AI support

Supports multiple providers:

• OpenAI
• Anthropic
• Gemini
• DeepSeek
• Qwen
• Kimi

Set your key:

export OPENAI_API_KEY=your_key

or

export ANTHROPIC_API_KEY=your_key
export GEMINI_API_KEY=your_key
export DEEPSEEK_API_KEY=your_key
export QWEN_API_KEY=your_key
export KIMI_API_KEY=your_key

Optional (generic / override)

export CT_LLM_API_KEY=your_key
export CT_LLM_PROVIDER=openai
export CT_LLM_MODEL=your_model
export CT_LLM_BASE_URL=https://your-api-base-url

Provider notes:

• DEEPSEEK_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, QWEN_API_KEY, KIMI_API_KEY, and OPENAI_API_KEY are all detected directly
• CT_LLM_API_KEY + CT_LLM_BASE_URL works for OpenAI-compatible endpoints
• CT_LLM_PROVIDER can explicitly select deepseek, anthropic, gemini, qwen, kimi, or openai

🛠️ Troubleshooting

ct tc url --ai says Playwright is not found

Run:

npm install
npx playwright install chromium

This command resolves Playwright from the active project directory first, not from the CLI's global install.

ct tc url --ai says Chromium is not installed

Run:

npx playwright install chromium

No AI key found

Set one of:

export OPENAI_API_KEY=your_key
export ANTHROPIC_API_KEY=your_key
export GEMINI_API_KEY=your_key
export DEEPSEEK_API_KEY=your_key
export QWEN_API_KEY=your_key
export KIMI_API_KEY=your_key

Or use:

export CT_LLM_API_KEY=your_key
export CT_LLM_BASE_URL=https://your-api-base-url

Legacy macOS project setup

On macOS 13 and older, ct new automatically switches to a legacy-compatible Playwright path and installs Chromium only during browser setup to avoid unsupported WebKit failures. This behavior is conditional for affected older macOS versions only. Newer macOS releases and other platforms keep the normal Playwright install path by default.

ct scan says Playwright is not found

ct scan reuses the same project-first Playwright resolution as ct tc url. Run:

npm install
npx playwright install chromium

ct scan with vision says no API key

Vision scanning requires ANTHROPIC_API_KEY, OPENAI_API_KEY, or GEMINI_API_KEY. Without a key, depth defaults to fast (axe only).

Scaffold modes

ct new currently implements --mode greenfield only. If you pass an unsupported mode such as --mode enhance, the CLI exits with an explicit error instead of silently ignoring it.

Browser install profiles

ct new supports:

• --browser-set auto for host-aware browser installation
• --browser-set all to install the normal Playwright browser set
• --browser-set chromium to install Chromium only
• --no-browsers to skip browser installation entirely

📁 Output structure

project/
  cases/
  assertions/
  flows/
  pages/
  tests/generated/
  .cementic/normalized/
  .cementic/a11y/               ← accessibility scan reports
    scan-myapp-com-2026-04-15.html
    scan-myapp-com-2026-04-15.json
    last-scan.json

ct gen now treats the project like a small framework workspace:

• pages/ for page objects that stay user-editable
• flows/ for reusable flow helpers generated from normalized cases
• assertions/ for reusable assertion helpers generated from normalized cases
• tests/generated/ for thin generated specs that import the helpers instead of repeating the whole workflow inline

🎯 Who this is for

• QA engineers moving into automation
• Beginners learning Playwright
• Teams needing fast test generation
• Developers who want speed over setup

🚀 Philosophy

Meaning → Structure → System → Resilience

CementicTest flips automation from:

Code → Tests

to:

Intent → Tests

🔗 Links

• 🌐 Website: https://cementic.testamplify.io/
• 💬 Community: https://t.me/+Wbx7oK7ivqgxZGJh

🧪 Status

• CLI: ✅ Production-ready
• Web App (Agentic UI): 🚧 Coming soon

💬 Final note

You don’t need to install anything. You don’t need to configure anything.

Just run:

npx @cementic/cementic-test

And start automating.

Changelog

See CHANGELOG.md for the full release history.

v0.2.22

• publishable follow-up to the ct scan release after fixing bundled CLI version resolution in CI and published installs
• shared runtime version lookup used by the CLI, scan command, and report/capture metadata

v0.2.21

• added ct scan <url> — multi-page accessibility scanning with axe-core + AI vision
• three scanning layers: axe-core (always), AI vision (when API key set), AI clustering (with deterministic fallback)
• interactive and CI modes with --routes, --depth, --fail-on, --user, --pass, --ci flags
• CT_VAR_USERNAME / CT_VAR_EMAIL / CT_VAR_PASSWORD env var reuse for authenticated scanning
• HTML + JSON reports to .cementic/a11y/ with ct scan report subcommand
• multi-LLM vision support: Anthropic, OpenAI, Gemini — no new npm dependencies
• note: this tag failed CI and was not published to npm; use v0.2.22 as the working release

v0.2.26

• fixed passive correct URL verification prompts so capture-backed AI analysis now emits toHaveURL(...) assertions instead of trying to match invented text on the page
• added regression coverage for Verify the login page loads successfully with the correct URL so URL-verification prompts stay stable through normalize and gen

v0.2.25

• fixed passive login-page verification prompts so they stay scoped to one focused presence scenario while generic Login prompts still preserve broader auth coverage
• fixed the prompt regressions from the failed v0.2.24 release attempt and revalidated the CLI gate before retagging

v0.2.20

• fixed auth/form intent classification so explicit prompts like empty fields and wrong password no longer collapse into generic page-load fallback coverage
• fixed deterministic auth fallback so it generates only the requested auth scenarios instead of always emitting a canned three-case bundle
• added regression coverage for auth scenario-cap and fallback-scope behavior

v0.2.19

• fixed double navigation so same-URL navigate steps are suppressed
• fixed literal-text fallback assertions to use getByText(...) from quoted step text
• fixed duplicate assertion deduplication within a single scenario
• fixed cleared-field assertions to target the named field with toHaveValue('')
• fixed unused expect import in specs that have no assertions
• fixed ct:url metadata injection in the --capture-only path
• made capture report version string dynamic (no manual update needed per release)

v0.2.18

• stripped wrapping quotes from captured label selectors before generating getByLabel(...) locators
• added resilient fallbacks for label-based auth field and select interactions
• fixed wrong-password negative generation so only the target field is invalid and leave-empty steps emit fill('')

v0.2.17

• fixed ct gen URL assertions so redirect and stay-on-page regexes now come from the exact intent text, including secure area -> /secure/ and login page -> /login/
• stripped leaked backticks from generated regex literals and added a post-generation Playwright validation pass that repairs known typos such as .fil()
• fixed negative auth generation so the field under test uses a deliberately invalid literal instead of reusing valid CT_VAR_* credentials

v0.2.16

• fixed ct gen so unquoted and backticked CT_VAR_* fill steps now generate runnable .fill() calls instead of TODO comments
• made generated auth button clicks more resilient for generic phrases like Click the login button
• added release guards so publish only proceeds when the Git tag matches package.json and the npm version has not already been published

v0.2.15

• fixed capture semantics so <a href="..."> elements are classified as links unless they explicitly expose role="button"
• made CTA-like presence selectors resilient by generating Locator.or(...) fallbacks across button and link roles when the intent is ambiguous
• tightened intent label cleanup so trailing filler phrases like on the page do not leak into generated locator text
• updated capture artifact metadata and capture user agent version strings to 0.2.15

v0.2.14

• hardened the release gate so npm test now includes the prompt regression suite and prepublishOnly runs the full test gate
• fixed the prompt test harness so it executes the local dist/cli.js build instead of depending on whichever global ct happens to be on PATH
• added deterministic capture-analysis fallback for ct tc url --ai when no LLM key is configured or remote analysis fails
• hardened auth, navigation, negative-state, heading, and count scenario fallback behavior so capture-driven generation still produces intent-aligned Playwright output offline
• added smoke coverage for ct new, ct ci, ct flow --no-run, ct report, and both ct serve execution paths
• replaced the old browser bootstrap heuristic in ct new with explicit browser install profiles: auto, all, and chromium
• kept the compatibility downgrade conditional to affected older macOS versions only, while preserving the normal Playwright install path for newer machines
• changed ct new --mode handling to reject unsupported scaffold modes explicitly instead of silently behaving like greenfield

v0.2.13

• replaced the old Rule 10 prompt guidance with a Playwright knowledge base aligned to official Playwright docs and web-first assertion patterns
• aligned locator priority with Playwright best practices: getByRole -> getByLabel -> getByPlaceholder -> getByText -> locator
• made counting intents generate toHaveCount() or .count() instead of intent-text locators
• enforced that intent wording is never reused as locator text
• made negative assertions generate .not. or toBeHidden() patterns
• made error assertions prefer getByRole('alert') and heading assertions prefer getByRole('heading')
• added on-demand Playwright doc fetching for advanced categories such as uploads, mocking, accessibility, viewports, auth, downloads, dialogs, frames, and popups
• added the 12-check prompt regression suite in test/prompt.spec.mjs

v0.2.12

• made both AI generation paths intent-first so markdown and capture-based scenario generation follow the same scope and presence-only rules
• tightened capture-based scenario generation to preserve presence-only checks as visibility assertions, honor intent-based selector fallback, and avoid generic 'value' placeholders
• added regression coverage for the new markdown prompt contract and capture-path intent enforcement
• breaking change: none

v0.2.11

• AI no longer writes 'value' as a step input and now prefers matching CT_VAR_* references or test-{fieldname} fallbacks
• fixed ct gen so generic fill steps read matching CT_VAR_* values from the environment and emit env-backed constants only when they are actually needed
• fixed generated toHaveValue(...) assertions so they reuse the same emitted fill value for selector-backed form fields
• added a generated spec header comment that lists required CT_VAR_* variables with inline usage examples
• breaking change: none

v0.2.10

• fixed ct gen so selector-hinted fields like #username and #password infer matching CT_VAR_* constants instead of leaving .fill('value') placeholders behind
• fixed generated toHaveValue(...) assertions so they reuse the same CT_VAR_* binding when the related fill step was variablized
• added regression coverage for selector-driven auth generation to keep fill calls and value assertions aligned

v0.2.9

• added CT_VAR_* extraction in ct gen for input and select test data so generated specs use env-backed constants instead of hardcoded values
• added ct gen --vars '{...}' to override generated fallback values without editing the emitted spec files by hand
• added regression coverage for variableized generated specs, selector-hint preservation, and select-option extraction

v0.2.8

• fixed TypeScript generator identifier sanitization so scenario titles containing em dashes and other non-identifier characters no longer break generated POM class names or spec imports
• added regression coverage for generated TypeScript output from scenario titles like Login-002 — User can enter credentials — before submission

v0.2.7

• fixed project-first Playwright resolution for ct tc url --ai, including global and npx CLI usage
• added clearer capture failure categories and setup guidance for missing Playwright, missing browsers, and page-load failures
• centralized AI provider detection so tc --ai and capture analysis recognize the same providers and env vars
• fixed missing DeepSeek guidance in the CLI help path
• changed legacy macOS browser setup to install Chromium only by default and avoid WebKit unsupported failures

v0.2.6

• added an install-time banner with links to the website and community
• surfaced project links and web app status near the top of the README
• added a standalone CONTRIBUTING.md
• added a CODE_OF_CONDUCT.md

v0.2.5

• replaced the legacy URL scraping path with full Playwright-based live page capture
• integrated the POC capture and AI analysis flow into ct tc url --ai
• added capture artifacts at .cementic/capture/capture-*.json
• added preview spec output at tests/preview/spec-preview-*.spec.cjs
• preserved capture-generated selector hints and exact playwright assertions through normalize and gen
• added --headed and --capture-only support for the URL capture flow
• expanded capture-aware AI provider support to DeepSeek, Anthropic, Gemini, Qwen, Kimi, and OpenAI-compatible endpoints

Current limitations

• generator assertions and step mapping are heuristic
• there is no built-in self-healing test loop yet
• there is no first-class Playwright CLI integration yet
• POM generation intentionally avoids overwriting existing page objects, so existing pages may diverge from later normalized cases
• flow and assertion helpers are generated per normalized case today, so reuse is structural but not yet merged across related scenarios automatically

Contributing

Contribution guidelines live in CONTRIBUTING.md.