easy-jev-natural-test
v0.1.0
Published
Run natural-language test case trees with Jev and Playwright
Readme
@hartk/natural-test
Run natural-language E2E cases directly from a testCaseTree with TypeSafe Jev and Playwright. The primary output is a per-case result, not generated test code.
Install
From another local project, install the package directory or the tarball produced by pnpm pack:
pnpm add file:/absolute/path/to/packages/natural-test playwright
pnpm exec playwright install chromiumAfter the package is published, pnpm add @hartk/natural-test playwright can be used instead. The package is not published by this repository change.
Node.js 20 or newer is required. Configure TYPESAFE_API_KEY on the server or pass apiKey to runTestCaseTree. Do not call this package from browser-side code.
Use
import { runTestCaseTree } from '@hartk/natural-test';
const result = await runTestCaseTree({
url: 'http://localhost:3000/login',
tree: {
nodes: [
{ id: 'root', parentId: null, kind: 'root', title: '登录' },
{
id: 'login-error', parentId: 'root', kind: 'case', type: 'e2e',
title: '错误密码提示',
steps: ['在邮箱输入框填写用户邮箱', '在密码输入框填写错误密码', '点击登录按钮'],
inputData: '{"邮箱":"[email protected]","密码":"wrong-password"}',
expectedResult: '页面显示密码错误提示',
},
],
},
traceDir: './test-artifacts',
});
console.log(result.cases[0].outcome); // passed | failed | blockedThe input shape is a structural subset of hartk's TestCaseTree; other projects can use it without importing hartk. Only kind: 'case' and type: 'e2e' nodes run. Each case gets a fresh Playwright browser context. storageState or beforeCase can establish authentication. The URL origin is the only allowed origin by default; set allowedOrigins if a case intentionally leaves it.
Jev selects an operation and a target from the currently observed controls. It cannot provide selectors, JavaScript, or input text. Input values come from inputData, quoted step text, or the caller's resolveInput callback. Unknown values block the case. A separate Jev judgment checks whether the final visible page state supports expectedResult; uncertain evidence produces blocked, not passed. The default action and verification probability thresholds are provisional; evaluate them on representative cases before relying on them as a release gate.
DecisionProvider is injectable, so projects can use another TypeSafe transport or test the runner without network calls. traceDir saves a Playwright trace for each case. The result includes stable caseId, outcome, action log, reason, observed page state, verification probabilities, and trace path.
Current boundaries
The runner supports click, fill, and native select on visible controls. It does not yet handle frames, shadow roots, file uploads, drag and drop, arbitrary keyboard widgets, or visual assertions. A multi-action natural-language step has a bounded action budget. If a case needs unsupported actions or its expected result cannot be observed in page text, title, or URL, it returns blocked for review.
