prompttest

v1.5.2

Published

8 hours ago

Ultra-fast, zero-code autonomous mobile testing & visual QA brain for Android & iOS

⚡ PromptTest Mobile

Ultra-fast, zero-code autonomous mobile testing & visual QA brain for Android & iOS.
The zero-setup, zero-instrumentation alternative to Appium & Detox for React Native, Expo, Flutter, and Native Android.

[!NOTE] Looking for the Python LLM prompt evaluator? That project is decodingchris/prompttest.
This is PromptTest Mobile — the autonomous mobile application QA and visual regression testing engine for native Android & iOS mobile applications. Available on NPM as prompttest-mobile and prompttest.

PromptTest Mobile enables QA engineers, mobile developers, and autonomous agents to test Android applications using plain-English natural language scripts. It completely bypasses heavy server frameworks (Appium, Selenium, WebDriver) in favor of high-speed native ADB streams, expectation-driven polling, and an autonomous state-graph exploration engine.

🖥️ Prefer a Visual Desktop IDE? Try PromptTest Studio

If you prefer an intuitive desktop application over the command line, check out PromptTest Studio:

📱 Real-Time Device Mirroring: Low-latency screen streaming with instant click, drag, and hardware navigation.
🎯 Visual Element Inspector: Point and click to inspect native views with instant auto-generated plain-English assertions.
📸 Visual Regression & Exclude Masks: Pixel-level baseline comparisons with draggable exclude masks for dynamic areas (clocks, battery, banners).
🚀 100% Local-First & Air-Gapped: Runs entirely on your machine over local ADB with zero cloud dependencies.

👉 Download PromptTest Studio for Windows (macOS & Linux coming soon)

📑 Quick Navigation

🌟 Why PromptTest?

📝 Plain-English Test Scripts: Write tests in human language without writing fragile XPath selectors or boilerplate glue code.
⚡ Zero-Setup & Zero-Instrumentation: Connects directly over native ADB (USB or Wi-Fi). No SDKs, test dependencies, or code modifications required in your target app.
🤖 Autonomous State-Graph DFS Explorer: Crawls apps autonomously, detects bottom-tab navigation hubs, traverses nested screens, and automatically triages defects without human intervention.
💡 Dynamic Self-Healing Locators: Resilient heuristic matching automatically adapts to changing dynamic counts (e.g. auto-resolving "Present (2)" to "Present (5)").
🛡 Built-In Safety Engine: Guards production and staging apps by blocking destructive actions (e.g., "Delete Account", "Discard Changes") during autonomous crawls.
📊 Standalone HTML & JUnit Reports: Generates executive test reports with failure-only visual screenshots and actionable error diffs.
🌐 Built for Modern Mobile Frameworks: Native support for React Native, Expo, Flutter, and Native Android (Jetpack Compose / Views).

🚀 Quick Start & CLI Workflows

Prerequisites

Node.js 18.0.0+
Android ADB installed and accessible in your system PATH
An Android device (USB or Wi-Fi) or emulator with USB Debugging enabled

Install in Your Mobile Project

You can run PromptTest instantly with npx or install it as a dev dependency:

# In your React Native, Expo, Flutter, or Android project:
npm install --save-dev prompttest

⚖️ Licensing note: PromptTest is source-available under BSL 1.1 — free for personal, educational, open-source, and small-team use (< $100k revenue and < 10 employees). Larger organizations and commercial QA agencies need a commercial license. It converts to Apache 2.0 in 2030. See License.

5 Core CLI Workflows

1. Environment Diagnostic Check

Before running tests, verify your ADB connectivity, connected devices, and permissions:

npx prompttest doctor

2. Zero-Code Autonomous Exploration (AI App Crawl)

Crawl bottom tabs, lists, and forms automatically to discover bugs, crashes, or React Native red-screens without writing a single line of test code:

npx prompttest explore com.yourcompany.app

Tip: explore is its own top-level command. Pass --max-screens=30 or --safety-mode=strict to customize.

3. Interactive Record & Replay

Record your natural interactions on a physical phone directly into a reusable test spec:

npx prompttest record specs/login.txt

Tap buttons and type in input fields on your phone; PromptTest auto-generates conversational English test steps.
Append mode: Add --append to add more steps to an existing spec without overwriting.
Safety backup: If the target file already exists, PromptTest automatically creates a .bak backup before modifying.

4. Run Plain-English Test Specs

Execute test specifications with fail-fast validation and locator self-healing:

npx prompttest run specs/login.txt com.yourcompany.app

Power flags:

--heal: Automatically self-heal altered counts and dynamic locators.
--video: Record an MP4 video of the execution session.
--screenshots: Capture high-resolution visual evidence at every step.
--fresh: Cold-restart the target app before execution begins.
--embed-screenshots: Embed screenshots directly into a standalone, shareable HTML report.

5. Interactive Live REPL Playground

Experiment with commands live in your terminal against your connected device:

npx prompttest repl com.yourcompany.app

⚠️ CLI Command Disambiguation Table

To avoid common syntax errors, remember that explore, record, and repl are standalone commands:

📦 Recommended `package.json` Scripts

Add these convenient shortcuts to your project's package.json:

"scripts": {
  "test:mobile": "prompttest run specs/smoke.txt com.yourcompany.app --heal",
  "test:explore": "prompttest explore com.yourcompany.app --max-screens=25",
  "test:record": "prompttest record specs/new_flow.txt",
  "test:doctor": "prompttest doctor"
}

📂 Outputs, Reports & Screenshots

Whenever PromptTest runs (run, explore, or record), all outputs are automatically organized inside an output/ folder at your project root:

your-mobile-project/
├── node_modules/
├── specs/
│   └── login.txt
├── output/                               <-- 📂 Created automatically
│   ├── login-report.html                 <-- 🌐 Interactive visual HTML report
│   ├── login-junit.xml                   <-- 🤖 CI/CD JUnit test results
│   ├── login-report.md                   <-- 📝 Markdown summary for PR comments
│   ├── login-results.json                <-- 📊 Structured raw JSON execution metrics
│   ├── step_1_tap_sign_in.png            <-- 📸 High-res visual screenshots
│   ├── login-recording.mp4               <-- 🎥 Full MP4 video (when using --video)
│   └── screenshots/                      <-- 📸 Screen transition photos from record sessions

Viewing & Sharing Reports:

Interactive HTML Dashboard (output/<spec>-report.html): Double-click to open in any browser. Features step-by-step audit timelines, latency timings, and failure triage bundles.
CI/CD Integration (output/<spec>-junit.xml): Standard JUnit format natively recognized by GitHub Actions, GitLab CI, Jenkins, and CircleCI.
Zero-Dependency Sharing (--embed-screenshots): Generates a single standalone HTML report with all images inlined via Base64 data URIs. Email or Slack it directly to teammates without missing image links!

✍️ Writing Plain-English Tests

Test specifications are simple text files containing numbered steps written in conversational English.

Example Spec (`specs/login_and_dashboard.txt`)

# 1. Authentication Flow
1. Type '[email protected]' into 'Email Address'
2. Type 'Teacher@1234' into 'Password'
3. Tap 'Sign In'
4. Wait 3s

# 2. Dashboard Verification
5. Verify 'Apex Coaching Academy' is visible
6. Verify 'Dr. Ramesh Sharma' is visible
7. Verify 'My Batches' is visible

# 3. Batches Module & Navigation
8. Tap 'Batches'
9. Verify 'Batches & Schedules' is visible
10. Tap 'Grade 10 Mathematics'
11. Verify 'Weekly Schedule' is visible
12. Press back

# 4. Sign Out Flow
13. Tap 'Profile'
14. Tap 'Sign Out'
15. Tap 'Sign Out'
16. Verify 'Welcome Back' is visible

Run the Spec:

npx prompttest run specs/login_and_dashboard.txt com.yourcompany.app

📖 Syntax Reference

🛠 CLI Command Reference

Primary Execution Modes

# 1. Deterministic Spec Runner
npx prompttest run specs/flow.txt <package>

# Cold-restart app before suite begins
npx prompttest run specs/flow.txt <package> --fresh

# Capture high-resolution visual screenshots on every step
npx prompttest run specs/flow.txt <package> --screenshots

# Dry-run validation (checks syntax without touching device)
npx prompttest run specs/flow.txt --dry-run

# Display formatted table of parsed steps
npx prompttest run specs/flow.txt --list-steps

# Run data-driven iterations
npx prompttest run specs/flow.txt <package> --iterations 5

# Run against a managed device pool with concurrency leasing
npx prompttest run specs/flow.txt <package> --device-pool emulator-5554,emulator-5556

# Visual regression testing against gold baseline images
npx prompttest run specs/flow.txt --save-baseline
npx prompttest run specs/flow.txt --compare-baseline --baseline-threshold=0.02

Autonomous State-Graph Explorer (`explore`)

# Autonomous exploration with Strict safety policy
npx prompttest explore <package>

# Set custom screen discovery and step interaction limits
npx prompttest explore <package> --max-screens=30 --step-budget=60

# Protect custom sensitive action keywords from being clicked
npx prompttest explore <package> --safety-blacklist="Wipe,Revoke,Transfer"

Interactive Record & Replay (`record`)

# Record gestures, taps, and inputs directly on device into a spec
npx prompttest record specs/recorded_flow.txt

Device Management & Wi-Fi Debugging

# List all connected devices, emulators, and serial numbers
npx prompttest devices

# Connect to physical device wirelessly over Wi-Fi
npx prompttest wifi 192.168.1.50

# Inspect active screen hierarchy and detected components
npx prompttest status

# Run comprehensive environment diagnostic check
npx prompttest doctor

# View offline learned component memory graph
npx prompttest memory <package>

🎯 Platform Support, Expectations & Boundaries

To maintain honest expectations, here is what PromptTest currently supports, what is under active development, and its architectural boundaries:

⚠️ Common Pitfalls & How to Solve Them

1. Hardware Back Button Minimizing the App

What happens: Running Press back while on the app's root dashboard or home tab tells the Android OS to minimize or exit the app.
How to solve it:
- In specs, tap the in-app back icon/button (e.g. Tap 'Back' or Tap '<') instead of the hardware key on top-level screens.
- In autonomous exploration (explore), PromptTest's built-in Package Jail Guard automatically detects if the app was backgrounded and restores it.
- Add the --fresh flag when running specs to cold-start your app cleanly before tests.

2. Android OS Permission Dialogs ("Allow Notifications / Location")

What happens: System dialogs belong to Android OS (com.android.permissioncontroller), not your app, and can appear unexpectedly on new installs.

How to solve it:

Use conditional handling in your spec:

Tap 'While using the app' (if present)
Tap 'Allow' (if present)

Or auto-grant permissions via ADB before running tests:

adb shell pm grant com.yourcompany.app android.permission.POST_NOTIFICATIONS

3. Off-Screen Items in Long Lists (FlatList / RecyclerView)

What happens: Mobile frameworks only render visible items on screen to save memory. Elements located further down the page are not in the hierarchy yet.

How to solve it:

Scroll before tapping:

Scroll down
Verify 'Save Changes' is visible
Tap 'Save Changes'

4. Layout Animations & Shimmer Settling

What happens: Tapping an element during a layout animation or skeleton fade-in can cause touch coordinates to miss while elements shift.
How to solve it:
- Add a brief settling pause: Wait 500ms or assert an anchor element first: Verify 'Dashboard' is visible.
- Disable animations on test devices to run tests 2x faster:
```
adb shell settings put global window_animation_scale 0
adb shell settings put global transition_animation_scale 0
adb shell settings put global animator_duration_scale 0
```

5. WebViews & In-App Browsers (OAuth & Payment Gateways)

What happens: In-app web pages (like Google Sign-In or Stripe) expose rendered text to ADB, but not internal HTML DOM tags or CSS selectors.
How to solve it:
- Use plain text matching (Tap 'Sign in with Google'). Deep DOM selector manipulation inside WebViews is not supported.

6. Multiple Devices Connected

What happens: If a physical phone and an emulator are both plugged in, ADB doesn't know which one to target.

How to solve it:

Target a specific device serial using --serial:

npx prompttest run specs/login.txt com.app --serial=emulator-5554

💡 Pro-Tips & Best Practices

1. The "Record & Refine" Workflow

prompttest record captures your natural physical device interactions and generates plain-English test steps in real time—scaffolding 90% of your test boilerplate in seconds.
QA Best Practice: After recording, do a quick 30-second review of the generated .txt spec to fine-tune timings or add custom Verify assertions.

Preview without touching your device:

npx prompttest run specs/flow.txt --dry-run

2. Deterministic Clean States (`--fresh`)

Prevent "already-logged-in" test pollution by adding --fresh to cold-start the target app before execution:
```
npx prompttest run specs/login.txt com.yourcompany.app --fresh
```

3. Zero-Maintenance Dynamic Badges (`--heal`)

When notification badges or counts change dynamically (e.g. 'Cart (1)' vs 'Cart (3)'), pass --heal to allow PromptTest's heuristic engine to self-heal the locator without failing.

4. Device & Screen Resolution Agnostic Portability

PromptTest binds interactions to semantic accessibility labels and proportional gestures—not fragile hardware pixel coordinates. Specs recorded on a phone seamlessly run on foldables and tablets.

5. 🛡️ Enterprise Safety Guardrails & Custom Blacklists

Autonomous exploration is safe by default, but enterprise applications often have company-specific sensitive keywords (e.g. "Deactivate", "Transfer Funds", "Revoke Access").

Default Protection: PromptTest automatically detects and blocks destructive actions like "Delete", "Wipe", "Remove", and "Discard Changes".

Add Your Own Sensitive Keywords: You can supply your own custom blocked keywords to run alongside the defaults:

# Via CLI flag:
npx prompttest explore com.yourcompany.app --safety-blacklist="Transfer,Deactivate,Revoke,Unsubscribe"

Or configure once in .prompttestrc.json:

{
  "safety": {
    "mode": "strict",
    "customBlacklist": ["Transfer", "Deactivate", "Revoke", "Unsubscribe"]
  }
}

💻 Programmatic TypeScript SDK

PromptTest exports a full programmatic SDK for embedding into Node.js test runners or custom CI scripts:

import { createPromptRunner, createAndroidDriver } from 'prompttest';

// Initialize native ADB driver for connected device
const driver = createAndroidDriver('DEVICE_SERIAL');
const runner = createPromptRunner(driver);

// Execute spec and retrieve structured execution report
const report = await runner.runSpec('specs/login.txt', 'com.example.app', {
  fresh: true,
  screenshots: 'failure-only',
});

console.log(`Execution complete: ${report.passedSteps}/${report.totalSteps} passed.`);

🐳 Docker Container Deployment

PromptTest is containerized with Android platform-tools and headless runtime support:

# Build container image
docker build -t prompttest .

# Execute test spec inside container sharing host ADB daemon
docker run --rm --net=host \
  -v $(pwd)/specs:/app/specs \
  -v $(pwd)/output:/app/output \
  prompttest run specs/flow.txt com.example.app

📊 Proven Real-Device Benchmarks

Tested and verified against live physical Android devices running complex multi-role enterprise apps:

| Test Suite | Total Steps | Passed | Failed | Pass Rate | Execution Time | | :-------------------------------- | :---------: | :----: | :----: | :-------: | :------------: | | CoachConnect Enterprise Suite | 52 | 52 | 0 | 100% | 47.8s | | Govindam Multilingual Suite | 8 | 8 | 0 | 100% | 35.1s | | Calculator Operations Suite | 5 | 5 | 0 | 100% | 19.6s | | Automated Vitest Test Matrix | 294 | 294 | 0 | 100% | 14.8s |

For deep architectural details and driver design, see docs/ARCHITECTURE.md included in the package.

📄 License

PromptTest is licensed under the Business Source License 1.1 (BSL 1.1), and automatically converts to the permissive Apache License 2.0 on the Change Date: September 14, 2030.

✅ Free to use — no license required for:

Personal, educational, academic, and open-source projects
Evaluation use
Internal software testing within organizations that have both under $100,000 USD in annual gross revenue and fewer than 10 employees

💼 Commercial license required for:

Use within organizations exceeding either of the free-tier thresholds (revenue or headcount)
Testing agencies providing commercial QA services to third-party clients
Deployment into production commercial CI/CD pipelines
Offering PromptTest (modified or unmodified) as a hosted or managed QA/cloud testing service

See the full LICENSE and Terms & Conditions for exact terms.

Commercial inquiries and enterprise licensing: [email protected] or open an issue at github.com/shriramsingh/prompttest-community.

Published

Vulnerabilities

Links

Maintainers

Keywords

Readme

⚡ PromptTest Mobile

🖥️ Prefer a Visual Desktop IDE? Try PromptTest Studio

📑 Quick Navigation

🌟 Why PromptTest?

🚀 Quick Start & CLI Workflows

Prerequisites

Install in Your Mobile Project

5 Core CLI Workflows

1. Environment Diagnostic Check

2. Zero-Code Autonomous Exploration (AI App Crawl)

3. Interactive Record & Replay

4. Run Plain-English Test Specs

5. Interactive Live REPL Playground

⚠️ CLI Command Disambiguation Table

📦 Recommended package.json Scripts

📂 Outputs, Reports & Screenshots

Viewing & Sharing Reports:

✍️ Writing Plain-English Tests

Example Spec (specs/login_and_dashboard.txt)

Run the Spec:

📖 Syntax Reference

🛠 CLI Command Reference

Primary Execution Modes

Autonomous State-Graph Explorer (explore)

Interactive Record & Replay (record)

Device Management & Wi-Fi Debugging

🎯 Platform Support, Expectations & Boundaries

⚠️ Common Pitfalls & How to Solve Them

1. Hardware Back Button Minimizing the App

2. Android OS Permission Dialogs ("Allow Notifications / Location")

3. Off-Screen Items in Long Lists (FlatList / RecyclerView)

4. Layout Animations & Shimmer Settling

5. WebViews & In-App Browsers (OAuth & Payment Gateways)

6. Multiple Devices Connected

💡 Pro-Tips & Best Practices

1. The "Record & Refine" Workflow

2. Deterministic Clean States (--fresh)

3. Zero-Maintenance Dynamic Badges (--heal)

4. Device & Screen Resolution Agnostic Portability

5. 🛡️ Enterprise Safety Guardrails & Custom Blacklists

💻 Programmatic TypeScript SDK

🐳 Docker Container Deployment

📊 Proven Real-Device Benchmarks

📄 License

📦 Recommended `package.json` Scripts

Example Spec (`specs/login_and_dashboard.txt`)

Autonomous State-Graph Explorer (`explore`)

Interactive Record & Replay (`record`)

2. Deterministic Clean States (`--fresh`)

3. Zero-Maintenance Dynamic Badges (`--heal`)