VSCode AutoTest

An AI-assisted end-to-end testing framework for VS Code extensions.

Users provide a YAML test plan. The framework launches VS Code, executes actions, verifies outcomes, captures screenshots, and writes structured reports.

Goal: Use declarative YAML test plans to automate VS Code extension E2E testing, with Copilot CLI able to run, analyze, and help fix test plans directly.

Quick start

# Install dependencies
npm install

# Build
npm run build

# Validate a test plan
npx autotest validate test-plans/java-maven.yaml

# Run a test plan (default output: test-results/<plan-name>/)
npx autotest run test-plans/java-maven.yaml

# Run with an explicit output directory
npx autotest run test-plans/java-maven.yaml --output test-results/java-maven

# Run all plans and generate an aggregate summary
npx autotest run-all test-plans --exclude java-fresh-import

# Opt in to per-case pass audits / failure RCA and matrix-level clustering
npx autotest run-all test-plans --analysis-mode case

# Capture a reusable evidence bundle without calling an LLM
npx autotest run test-plans/java-maven.yaml --analysis-mode evidence-only

# Re-analyze existing test results
npx autotest analyze test-results --analysis-mode case --report-only

Configurable logging

Console output remains enabled by default. File logging is opt-in and independent of --analysis-mode; enabling logs does not enable case analysis or change actions, retries, test verdicts, exit-code policy or legacy report fields.

# Build this checkout before using newly added CLI options
npm run build
node dist/cli/index.js run test-plans/java-maven.yaml --logs

# Choose a log directory; --log-output also enables logging
node dist/cli/index.js run test-plans/java-maven.yaml --log-output run-logs/java-maven

# run-all creates a separate log directory for each plan
node dist/cli/index.js run-all test-plans --log-output run-logs

The equivalent top-level YAML configuration is:

logging:
  enabled: true
  outputDir: "../run-logs/my-case"

logging.outputDir is relative to the YAML file. CLI paths are relative to the current directory and override YAML settings. --logs enables the YAML directory or defaults to <case-output>/logs; --no-logs overrides YAML logging, but does not remove evidence artifacts requested by --analysis-mode. Configuring logging: {} also enables the default log directory.

SDK callers can pass logging: { enabled: true, outputDir: "run-logs/my-case" } in TestRunnerOptions; these fields override the plan's logging configuration. SDK paths are relative to the current directory. If the SDK has no case outputDir, its default log directory is ./logs.

Each enabled case saves:

• autotest.log: the complete stdout/stderr produced during TestRunner.run(), including driver/action/verifier/LLM output, with terminal output preserved. CLI setup/validation and the run-all aggregate summary remain terminal output.
• runner-launch.log: timestamped startup stages, including version resolution, extension installation, Electron launch and workbench readiness.
• runner-failure.log: fatal exception stack, bounded cause/aggregate chains and transport/status fields, including failures with empty messages.
• environment.json: runtime/extension metadata, log source mappings and explicit diagnostic collection errors.
• vscode-N.log and jdtls-N.log: available VS Code/extension-host and JDT LS logs. Component logs retain bounded, redacted excerpts rather than full files; VS Code collection is limited to 20 files and each excerpt to 64 KiB.

Saved logs use the existing secret/user-path redaction. Startup and component diagnostics are also referenced by evidence manifests when evidence mode is active. Reusing a log directory replaces AutoTest-owned files from the previous run, not unrelated files. Give concurrent runs different directories; their console captures are isolated even when they share a Node process. Invalid configuration or a console-log write failure is reported explicitly; a logging failure does not rewrite the saved test verdict.

Generic artifact collection

Artifact collection is a separate opt-in feature. logging saves AutoTest's own output; artifacts archives explicitly selected files; evidence: tail supplies bounded excerpts to the existing analysis pipeline. AutoTest does not need to know an extension ID, language-server directory or external tool.

Put shared defaults in an artifacts section in a YAML/JSON file, for example ci-artifacts.yaml. The same section is supported directly in a test plan:

artifacts:
  enabled: true
  limits:
    maxFiles: 1000
    maxFileBytes: 52428800
    maxTotalBytes: 209715200
  sources:
    - id: ide
      root: userData
      include: ['logs\**\*']
      destination: logs\ide
      evidence: tail
    - id: diagnostics
      root: workspace
      include: ['.autotest\**\*.log', '.autotest\**\*.json']
      destination: diagnostics\workspace
      optional: true
      evidence: tail
    - id: preparation
      root: {env: AUTOTEST_STAGING_DIR}
      include: ['*-version.log', '*.json']
      destination: diagnostics\ci
      optional: true
    - id: console
      root: {env: AUTOTEST_STAGING_DIR}
      include: ['console.log']
      destination: logs\console
      phase: collect
      optional: true
    - id: native-reports
      root: {env: HOME, path: 'Library\Logs\DiagnosticReports'}
      include: ['*.ips', '*.crash']
      destination: logs\crashes
      platforms: [darwin]
      modifiedSince: run-start
      phase: collect
      optional: true

autotest run test-plans\example.yaml --output test-results\example --logs --artifacts-config ci-artifacts.yaml --analysis-mode case

# Run in the CI always() step, after external log writers have closed.
autotest collect --output test-results\example --artifacts-config ci-artifacts.yaml

run-all --artifacts-config ci-artifacts.yaml applies the same defaults to every case, with separate archives under <output>/<plan-name>. Precedence is shared file defaults, then plan fields, then SDK/CLI overrides. Source arrays replace the previous array, including []; they never concatenate. Limit fields merge individually. --no-artifacts disables collection without disabling logging or analysis. If a plan overrides artifact settings, pass that same plan with collect --plan test-plans\example.yaml so collection uses the same definitions.

| Field | Behavior | |-------|----------| | root: workspace | Actual isolated workspace, not the original fixture; single-file runs use its containing directory | | root: userData | Actual VS Code user-data directory, including paths known before an early launch failure | | root: output | Case output; collector-owned archive trees and its index are excluded from traversal | | root: {path: ...} | Explicit directory; relative paths use the plan/shared-file directory, or cwd for SDK options | | root: {env: NAME, path: ...} | Resolve one named environment variable at collection time; optional path is a safe relative subdirectory | | include / exclude | Root-relative globs; / and \ separators, **, braces and hidden files are supported | | destination | Safe directory inside the case output; the complete root-relative source path is preserved | | phase: run | Default: archive after VS Code shutdown, before temporary workspace deletion | | phase: collect | Defer until the explicit post-run command; useful for console, display and other external writers | | platforms | Optional list of win32, linux, darwin; other platforms are visibly skipped | | optional: true | Missing directory or no matching files is recorded but not a collection failure | | modifiedSince: run-start | Select only files modified since the persisted run start; without one, record the source as unavailable rather than include historical files | | format: text | Default: complete UTF-8 file, with the existing secret/user-path redaction; invalid UTF-8 or NUL bytes are an explicit error | | format: binary | Explicit unmodified binary archive; no redaction or LLM input, so only select safe files | | evidence: tail | Include a bounded text tail in case/evidence-only modes or configured diagnostic logs; default is none |

Source IDs must be unique; source destination trees must not overlap. Traversal, symlink/junction escapes, framework-owned result paths and overwriting unowned files are rejected. Only declared roots are inspected: the collector does not dump the environment, scan home directories automatically or run setup commands. CI/fixtures still install tools and generate version/diagnostic files; GitHub Actions still uploads the final case directory.

The default limits are the values shown above. maxFiles and maxTotalBytes include already archived files. maxFileBytes bounds source reads; oversized files are omitted, never silently truncated. artifacts/manifest.json records each source's status, missing/deferred sources, omission counts, errors, original and stored sizes, SHA-256 hashes, redaction and output-relative file references. Error details are bounded with explicit errorsTruncated counts.

collect creates output if necessary but never clears it, overwrites existing archives, edits results.json or updates saved case analysis. Repeated calls validate existing hashes and add only new files. Already archived runtime sources remain usable after their workspace disappears, with previous errors and omissions preserved if the source cannot be rechecked. A changed source definition requires a new ID. collect exits 1 for an incomplete/failed collection, separately from the original test exit code. run records optional report.artifacts metadata without changing its test verdict or exit policy.

Output is initialized once before opening the runner logger. Keep pre-run and externally redirected console files outside --output, for example in AUTOTEST_STAGING_DIR, because startup still clears case output. Probe capture precedes shutdown; artifact archival follows shutdown and precedes workspace cleanup. Signal cleanup shares the same idempotent finalization. A hard kill, runner loss or reports generated after the final collect cannot guarantee full in-process capture; available external files can still be collected/uploaded.

In this opt-in mode, legacy automatic Java/JDT/bundled-JAR discovery is disabled. Declare product-specific paths in consumer configuration instead. Existing YAML, logging flags, Java evidence adapters and report defaults remain unchanged when artifacts is absent or disabled.

Full archives do not inflate LLM prompts. Text evidence selects files round-robin across declared sources, at most 20 files / 64 KiB per file / 256 KiB total. tailTruncated and artifactEvidenceOmitted expose evidence bounds. The evidence manifest links to the separate artifact index and selected logs, rather than embedding all files. LLM source/error summaries are also bounded, with omission counts; the persisted manifest remains complete. Passing-case audits, failed-case RCA and matrix summaries retain their existing behavior.

import { loadTestPlan, TestRunner, ArtifactCollector, loadArtifactConfig } from "@vscjava/vscode-autotest";

const runner = new TestRunner(loadTestPlan("test-plans/example.yaml"), {
  outputDir: "test-results/example",
  artifactsConfig: "ci-artifacts.yaml",
  // artifacts: { enabled: false } overrides shared and plan settings.
});
await runner.run();

// The same incremental collector used by the post-run CLI.
new ArtifactCollector("test-results/example", loadArtifactConfig("ci-artifacts.yaml")).collect();

Requirements

• Node.js >= 22
• JDK installed for Java extension tests
• The vscode-java repository cloned locally when a test plan references its sample projects

Core architecture

┌─────────────────────────────────────────────────────┐
│                  Test Plan (YAML)                    │
│   Describes steps and expected outcomes; no locators │
└──────────────────┬──────────────────────────────────┘
                   │  planParser.ts
                   ▼
┌─────────────────────────────────────────────────────┐
│              TestRunner (orchestrator)               │
│  Launch VS Code → execute steps → evidence → report   │
│                                                     │
│  ┌──────────────────┐  ┌─────────────────────────┐  │
│  │  ActionResolver   │  │     StepVerifier        │  │
│  │  Action → Driver  │  │  Deterministic checks   │  │
│  │  (50+ regex)      │  │  (10+ strategies)       │  │
│  └────────┬─────────┘  └──────────┬──────────────┘  │
│           │                       │                  │
│           │              ┌────────┴────────┐         │
│           │              │    LLMClient    │         │
│           │              │  Azure OpenAI   │         │
│           │              │case + matrix AI │         │
│           │              └─────────────────┘         │
│           │              ┌─────────────────┐         │
│           │              │EvidenceCollector│         │
│           │              │ evidence bundle │         │
│           │              └─────────────────┘         │
└───────────┼──────────────────────────────────────────┘
            ▼
┌─────────────────────────────────────────────────────┐
│           VscodeDriver (operation SDK)               │
│  Playwright Electron + @vscode/test-electron         │
│  Workspace isolation · event waits · process cleanup │
└──────────────────┬──────────────────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────────────────┐
│           Playwright Electron Runtime                │
│  Launches the VS Code process and exposes Page APIs  │
└─────────────────────────────────────────────────────┘

Writing test plans

Basic structure

name: "My test"
setup:
  extension: "my-extension"       # Marketplace extension ID; also installed as the primary extension
  extensionPath: "./path/to/extension"
  extensions: ["publisher.other-extension"]
  vsix: ["./dist/my-extension.vsix"]
  vscodeVersion: "stable"
  workspace: "./test-workspace"   # Relative to the test plan file
  # file: "./Single.java"         # Single-file mode without a workspace
  # repos: [{ url: "https://github.com/org/repo.git", path: "./repo", branch: "main" }]
  timeout: 60
  settings:                       # Injected into VS Code user settings
    editor.fontSize: 14
  workspaceSettings:              # Written to <workspace>/.vscode/settings.json
    java.configuration.updateBuildConfiguration: "automatic"
  workspaceTrust: "disabled"      # disabled | trusted | untrusted
  mockOpenDialog:                 # Optional mocked native open/save dialog responses
    - ["~/lib/example.jar"]

steps:
  - id: "step-1"
    action: "run command My Extension: Hello"
    verifyNotification: "Hello World!"

Path resolution: extensionPath, extensionPaths, localExtensions, workspace, file, vsix, repos[].path, and non-~/ mockOpenDialog paths are resolved relative to the test plan file, not the current working directory. ~/ means the temporary workspace root at runtime.

Supported actions

ActionResolver uses a deterministic regex dictionary. Unmatched actions are executed as Command Palette text.

| Category | Action syntax | Description | |----------|---------------|-------------| | Commands / keys | run command <name>, selectCommand <name>, executeVSCodeCommand <id> [jsonArg], pressKey <key>, pressTerminalKey <key> | Command Palette, VS Code command IDs, and keyboard input | | Files / editor | open file <name>, saveFile, insertLineInFile <path> <line> <text>, deleteFile <path>, typeInEditor <text>, typeAndTriggerSnippet <word> | Open/save/delete files and edit text; prefer insertLineInFile for LS-sensitive code changes | | Navigation / code intelligence | waitForLanguageServer, goToLine <n>, goToEndOfLine, findText <text>, navigateToError <n>, applyCodeAction <label>, renameSymbol <newName>, organizeImports, triggerCompletion, triggerCompletionAt <position>, hoverOnText <text>, clickHoverAction <label>, dismissHover | Java LS, Problems, Code Actions, completion, rename, and hover. Completion positions support endOfLine, endOfMethod, and line <n> [column <m>]. Use clickHoverAction to follow links inside the hover popup (e.g. "Go to super implementation"). | | Workbench UI | click side tab <name>, collapseSidebarSection <name>, collapseWorkspaceRoot, select <name> option, selectOptionByIndex <n>, wait <n> seconds | Side bar, Quick Pick, and static waits | | TreeView | click <name> tree item, expandTreeItem <name>, doubleClick <name> tree item, clickTreeItemAction <item> <label>, contextMenu <item> <menuLabel>, createNewFile <folder> <name>, openDependencyExplorer | Tree node click/expand/double-click, inline actions, context menus, and Java Dependencies. Quote arguments that contain spaces, for example contextMenu "Maven Dependencies" "Add JAR". | | Quick Input / Dialog | fillQuickInput <text>, fillAnyInput <text>, typeInQuickInput <text>, confirmQuickInput, dismissQuickInput, waitForDialog [seconds], clickDialogButton <label>, tryClickDialogButton <label>, confirmDialog, tryClickButton <label> | Quick Input, inline rename, and modal dialogs | | Debug / Test Runner | startDebugSession, stopDebugSession, setBreakpoint <line>, debugStepOver, debugStepInto, debugStepOut, openTestExplorer, waitForTestDiscovery <name> [timeout]s, runAllTests, runTestsWithProfile <profile>, clickCodeLens <label> | Java Debugger and Java Test Runner scenarios |

Supported verification fields

| Field | Type | Description | |-------|------|-------------| | verify | string | Natural-language expected outcome used by the existing screenshot re-check and case analysis | | verifyFile | object | File existence/content checks with path, exists, and contains | | verifyNotification | string | Notification text match | | verifyEditor | object | Editor content match, commonly contains | | verifyProblems | object | Problems count checks with errors, warnings, atLeast, and polling | | verifyCompletion | object | Completion list checks with notEmpty, contains, and excludes | | verifyQuickInput | object | Quick Input validation checks with noError, messageContains, and messageExcludes | | verifyDialog | object | Modal dialog visibility/content with visible and contains | | verifyTreeItem | object | Tree item appearance/disappearance with name, visible, exact, optional visible count, and optional 1-based level (aria-level) | | verifyEditorTab | object | Editor tab title appearance; optional active: true requires that tab to be selected | | verifyWebview | object | Active webview text checks with contains and notContains | | verifyOutputChannel | object | Output channel text checks with channel, contains, and notContains | | verifyTerminal | object | Terminal text checks with contains and notContains |

When the Output panel is hidden, output channel checks open it through the Command Palette rather than a platform-dependent panel shortcut in the active editor.

State-driven hover links

Use hoverAndClickAction "<text>" "<label>" to hover editor text, wait for an actual visible command link, and click it once. A loading popup is not readiness. If the popup disappears before the click, the operation re-hovers the target. It does not repeatedly restart an existing loading popup. DOM observation starts before the hover is triggered and records whether a popup appeared, so even a popup dismissed between driver polls is recoverable.

The operation uses step.timeout (or the framework's default verification budget) as one shared execution deadline, including recovery. There is no separate link timeout, and recovery does not restart the deadline. Existing hoverOnText and clickHoverAction retain their previous behavior.

- id: "navigate-to-super"
  action: 'hoverAndClickAction "greet" "Go to Super Implementation"'
  verifyEditorTab:
    title: "Base.java"
    active: true

Verify the target editor contents after the active-tab check has completed. Intermediate hover-open, hover-lost, hover-restored, and hover-action-ready screenshots are diagnostic evidence, not verdicts.

Test isolation and screenshots

Workspace isolation

Each run copies workspace into a temporary autotest-workspace/ location and cleans it up afterwards. The original workspace is never modified.

Screenshot strategy

Every step captures screenshots under the output directory's screenshots/ subdirectory. The default output path is test-results/<plan-name>/screenshots/; use --output to change the output root.

| Step status | Screenshot files | |-------------|------------------| | pass | NN_<stepId>_before.png + NN_<stepId>_after.png | | fail / error | NN_<stepId>_before.png + NN_<stepId>_after.png or NN_<stepId>_error.png |

The default legacy mode keeps the existing report and screenshot behavior unchanged. --analysis-mode case and --analysis-mode evidence-only additionally produce a generic evidence bundle under evidence/. A bundled test-only probe reads diagnostics through vscode.languages.getDiagnostics() and records extension/runtime metadata. The Node collector adds bounded, secret-redacted logs and component metadata; Java runs currently include a legacy optional adapter for JDT LS logs and bundled JDT/Lombok artifacts. With generic artifacts enabled, only explicitly declared product files are collected. The probe is loaded only for these opt-in modes and is not part of the extension under test.

In case and evidence-only modes, each step and retry attempt also records:

• verification: per-check expected values, the bounded values actually read by the verifier, observation timestamps, and pass / fail / error / skipped / not-run status. Checks after a fail-fast failure remain not-run. A step with no executed structured assertion is not-configured, not proof of its natural-language expectation; an action error before verification is not-run.
• screenshots: timestamped, output-relative references for before, after (action), verified (after deterministic verification), error, and intermediate sub images. The original screenshot field remains unchanged. NN_<stepId>_verified.png is additional evidence; it does not replace the action-time image.
• collectionErrors: screenshot, evidence-collection, and step-model failures. A malformed step-model response is recorded as unavailable, not as a successful audit.

Observations come from the checks themselves, not a second read of a potentially changed UI. File content assertions in evidence modes use one read for the verdict, excerpt, and SHA-256 hash. Text is limited to 4,096 characters and arrays to 100 entries / 4,096 total characters, with truncated fields explicitly marked. Per-check failure and exception reasons are redacted and limited to 4,096 characters, with reasonTruncated: true when shortened. Evidence-mode step/attempt reasons and stored model reasoning/suggestions are also redacted before persistence; legacy reason text is unchanged. Driver helpers that return only a boolean record that boolean, not an invented count or full UI snapshot. Problems counts describe the status bar; they do not prove that indexing has finished or that diagnostics apply to the latest document version.

Step screenshot verification in case mode receives the action-time image, the separately labeled post-verification image when available, and structured observations. Verifiers may change the UI themselves: completion verification reads items and then dismisses the popup, so its absence in the later image is not by itself evidence of failure. The existing downgrade threshold, retry behavior, and verdict policy are unchanged.

Failed-attempt diagnostics have distinct paths under evidence/diagnostics/<step-key>/attempt-N.json; evidence.artifactPath and manifest stepId / attempt metadata identify them without filename guessing. Successful retries retain earlier failures. Case analysis selects screenshots from those exact attempts, including the failed step's own final attempt even when later steps run, instead of mixing images from different attempts. Within the four-image budget, failure and recovery evidence take priority over intermediate sub-screenshots; the overall final state is also retained.

Process management

• The VS Code user-data directory is cleared before each launch to avoid restoring old windows.
• Ctrl+C automatically shuts down the VS Code process.
• close() includes retries for Windows file locks.

Analysis modes (optional)

Evidence-backed case analysis is opt-in so existing plans, reports, verdicts, and exit codes remain compatible:

| Mode | Behavior | |------|----------| | legacy | Default. Keeps the existing step screenshot verification and report shape; does not load the probe or write an evidence bundle. | | case | Keeps step verification, writes the evidence bundle, then performs one case-level pass audit or failure RCA. The case analysis is advisory and never changes the runner verdict. | | evidence-only | Writes the same evidence bundle without any LLM calls. |

Case analysis follows one framework-agnostic pattern: reconstruct the complete scenario and attempt history, identify the earliest divergence, cite logs/diagnostics/screenshots, split direct failures from cascading failures, record evidence gaps, and recommend a confirming experiment. Passing cases are audited for no-op actions, stale state, weak assertions, hidden errors, and other false-pass risks. Results are stored under the optional top-level analysis field and, when successful, in analysis/case-analysis.json.

Each case-analysis request defaults to a 4,000-token completion budget. Set AUTOTEST_CASE_ANALYSIS_MAX_TOKENS to override it in the CLI, or pass caseAnalysisMaxTokens to the SDK's LLMClient. Precedence is SDK option, then environment variable, then the default. The selected value must be a positive safe integer; environment values use decimal digits, with optional surrounding whitespace.

import { LLMClient } from "@vscjava/vscode-autotest";

const llm = new LLMClient({ caseAnalysisMaxTokens: 6000 });

This setting applies only to case analysis, not step verification or aggregate summaries. Invalid values fail case analysis before sending the request. The runner records these errors, as well as truncated responses, in analysis.error without changing the test verdict.

run-all --analysis-mode case and analyze --analysis-mode case aggregate those case analyses using evidence-backed root-cause groups across plans and platforms. Reports created by older AutoTest versions remain readable; cases without the new analysis field fall back to their full failed-step reasons.

Both commands share the same summary.md layout in case mode:

1. AI Analysis — TL;DR: key conclusions, affected cases/platforms, priority actions, and important uncertainty, shown before the results table.
2. E2E Test Results: the existing result table and totals.
3. Detailed Analysis: analysis coverage, evidence-backed problem groups from the aggregate model, saved case diagnoses/audits, and folded original execution failures.

One aggregate LLM request produces both the TL;DR and detailed analysis. Each problem should identify its affected cases/platforms, earliest supported divergence, direct and cascading failures, evidence, uncertainty, and a confirming action. Recorded failures and runtime signals accompany the model-generated case diagnoses; matching fingerprints are only candidate groups, not proof of a common cause. The TL;DR stays short, while detailed analysis is no longer subject to a shared 500-word limit. Identical recorded observations are supplied once with per-case/step references to avoid inflating the aggregate input; this deduplication does not establish shared causality.

Case-mode aggregation defaults to an 8,000-token completion budget. Configure it with LLMClientOptions.aggregateAnalysisMaxTokens or AUTOTEST_AGGREGATE_ANALYSIS_MAX_TOKENS; precedence is SDK option, environment variable, then default. Both summarizeCaseResults() and summarizeCaseResultsStructured() use this setting. It is validated only when either aggregate method is called, using the same positive-safe-integer and decimal-environment-value rules as the case budget. Step verification, individual case analysis, and legacy summarizeResults() budgets are unchanged.

const llm = new LLMClient({
  caseAnalysisMaxTokens: 4000,
  aggregateAnalysisMaxTokens: 16000,
});

The configured budget must be supported by the selected Azure deployment. It can include reasoning tokens for reasoning models; 8,000 is a default, not a guarantee against truncation. AutoTest does not automatically increase the budget or retry the aggregate request.

If aggregate analysis is disabled, unconfigured, or fails (including invalid configuration, truncated, or malformed responses), the TL;DR explicitly reports that it is unavailable. The report still renders saved per-case RCA and pass audits without another model call: failure diagnoses and audit warnings are expanded as a fallback, without guessing cross-case groups. Coverage and individual analysis errors remain visible. Model assessments and confidence are labeled as opinions, and suspected false passes do not change verdicts.

The case-mode results table identifies platforms and links to the corresponding saved analysis. Case details include the model's hypothesis, cited evidence, recorded diagnostic snapshots, direct/cascading step mappings, confirming actions, and evidence gaps. Diagnostic snapshots are folded after the hypotheses and confirming actions. Original failure/crash reasons are retained in a folded section without the 150-character cut; HTML and terminal control characters are escaped or removed for safe display. This does not change stored case results or command exit behavior.

Legacy and evidence-only report layouts are unchanged. SDK callers can use LLMClient.summarizeCaseResultsStructured() for an AggregateAnalysis containing tldr and details Markdown strings; it rejects unavailable or invalid analysis. The existing string-returning summary methods remain compatible.

- id: "check-ls"
  action: "waitForLanguageServer"
  verify: "The status bar shows the Java language server is ready"
  verifyProblems:
    errors: 0

Enable it with environment variables:

export AZURE_OPENAI_ENDPOINT=https://<resource>.openai.azure.com/
export AZURE_OPENAI_API_KEY=<key>
export AZURE_OPENAI_DEPLOYMENT=gpt-4.1       # Optional, default: gpt-4.1
export AZURE_OPENAI_API_VERSION=2024-12-01-preview
export AUTOTEST_CASE_ANALYSIS_MAX_TOKENS=4000 # Optional, default: 4000
export AUTOTEST_AGGREGATE_ANALYSIS_MAX_TOKENS=8000 # Optional, case-aggregate default: 8000

• If Azure OpenAI is not configured, case mode still writes evidence and records analysis.error; the runner verdict is unchanged.
• --no-llm disables LLM calls. In case mode, evidence collection still runs.

TreeView inline actions

VS Code TreeView inline actions, such as Run / Add / New icons on the right side of a row, are usually visible only after hover, focus, or selection. Do not click them with hard-coded coordinates. Use clickTreeItemAction <item> <label>:

- action: "expandTreeItem Lifecycle"
- action: "clickTreeItemAction compile Run"

If the target view is pushed off screen by Explorer, Outline, Timeline, Java Projects, or other sections, collapse space-consuming sections before focusing the target view:

- action: "collapseWorkspaceRoot"
- action: "collapseSidebarSection OUTLINE"
- action: "collapseSidebarSection TIMELINE"
- action: "run command Maven: Focus on Maven Projects View"

Copilot CLI integration

This repository provides AGENTS.md, so Copilot CLI can run tests directly from this directory:

> run the java-maven test

Copilot CLI runs npx autotest run, reads results and screenshots, and analyzes failures.

See AGENTS.md.

CLI commands

| Command | Description | |---------|-------------| | npx autotest run <plan> | Run one YAML test plan; default output is test-results/<plan-name>/ | | npx autotest run-all <dir> | Run all .yaml/.yml test plans in a directory and generate summary.md | | npx autotest analyze <dir> | Scan existing results.json files and regenerate summary / LLM analysis | | npx autotest validate <plan> | Validate YAML test plan format | | npx autotest collect --output <case-dir> --artifacts-config <file> | Incrementally archive available declared files without editing test results |

Common options:

| Option | Commands | Description | |--------|----------|-------------| | --output <dir> | run / run-all / analyze | Set the output directory | | --analysis-mode <mode> | run / run-all / analyze | Select legacy (default), case, or evidence-only | | --no-llm | run / run-all / analyze | Skip LLM calls; opt-in evidence collection remains enabled | | --report-only | analyze | Write the summary and exit 0 even when cases failed | | --vsix <paths> | run / run-all | Comma-separated VSIX paths appended to setup.vsix | | --override <kv...> | run / run-all | Override setup fields, for example --override extensionPath=../../vscode-java | | --exclude <plans> | run-all | Comma-separated plan names; defaults to excluding java-fresh-import | | --artifacts-config <file> | run / run-all / collect | Shared artifact defaults; plan fields override defaults | | --no-artifacts | run / run-all | Disable artifact collection independently of logging and analysis | | --plan <file> | collect | Use the plan's artifact overrides, or its standalone artifact settings |

Existing test plans

| File | Steps | Scenario | |------|-------|----------| | annotation-completion-before.yaml | 6 | Annotation completion baseline | | java-annotation-completion-bug.yaml | 6 | Annotation completion regression | | java-basic-editing.yaml | 21 | Basic editing, snippet, Code Action, Rename, Import, Explorer | | java-debugger.yaml | 9 | Breakpoints, debug start, stepping, stop | | java-dependency-viewer.yaml | 7 | Java Dependencies / TreeView | | java-extension-pack.yaml | 3 | Java Extension Pack / Configure Classpath | | java-fresh-import.yaml | 3 | Fresh import / Spring Petclinic | | java-gradle-delegate-test.yaml | 15 | Gradle delegate test | | java-gradle-java25.yaml | 6 | Gradle Java 25 | | java-gradle.yaml | 7 | Gradle LS, completion, navigation, editing | | java-maven-java25.yaml | 6 | Maven Java 25 | | java-maven-multimodule.yaml | 5 | Maven multi-module | | java-maven-resolve-type.yaml | 6 | Maven resolve type / Code Action | | java-maven.yaml | 7 | Maven LS, completion, navigation, editing, diagnostics | | java-new-file-snippet.yaml | 4 | New Java file + class snippet | | java-single-file.yaml | 6 | Single Java file | | java-single-no-workspace.yaml | 6 | Single Java file without a workspace | | java-test-runner.yaml | 6 | Java Test Runner | | java-unicode-classname-789.yaml | 6 | Unicode class name regression | | maven-workspace-trust.yaml | 5 | Maven workspace trust |

Wiki scenario coverage

| Wiki scenario | Status | Notes | |---------------|--------|-------| | Basic #1-5 | Covered | Existing test plan | | Basic #6-8 (completion/import/rename) | Covered | Existing test plan | | Basic #9 (New Java File snippet) | Covered | Existing test plan | | Maven | Covered | Existing test plan | | Maven Multimodule | Covered | Existing test plan | | Gradle | Covered | Existing test plan | | Maven Java 25 | Covered | Existing test plan | | Gradle Java 25 | Covered | Existing test plan | | Single file | Covered | Existing test plan | | Single file without workspace | Covered | file single-file mode | | Fresh import (spring-petclinic) | Covered | Requires the project to be cloned first | | Debugger for Java | Covered | Existing test plan | | Java Test Runner | Covered | Existing test plan | | Maven for Java | Covered | Existing test plan | | Java Dependency Viewer | Covered | Existing test plan | | Java Extension Pack | Covered | Webview internals remain limited |

Project structure

autotest/
├── .github/
│   ├── copilot-instructions.md  # Repository-wide Copilot instructions
│   └── instructions/             # Scoped Copilot instruction files
│       ├── action-dsl.instructions.md
│       ├── driver-operations.instructions.md
│       ├── test-plans.instructions.md
│       └── verifiers.instructions.md
├── src/
│   ├── drivers/
│   │   ├── vscodeDriver.ts    # VS Code lifecycle, workspace isolation, shared Driver state
│   │   └── operations/         # Function-specific Driver operation mixins
│   │       ├── _shared.ts          # Shared selectors, key constants, helpers
│   │       ├── commandOperations.ts
│   │       ├── debugOperations.ts
│   │       ├── dialogOperations.ts
│   │       ├── editorOperations.ts
│   │       ├── fileExplorerOperations.ts
│   │       ├── hoverOperations.ts
│   │       ├── languageServerOperations.ts
│   │       ├── quickInputOperations.ts
│   │       ├── snapshotOperations.ts
│   │       ├── testRunnerOperations.ts
│   │       ├── treeOperations.ts
│   │       └── verificationOperations.ts
│   ├── operators/
│   │   ├── actionResolver.ts   # Action → Driver calls (50+ regex)
│   │   ├── defaults.ts         # Shared timeout / poll-interval constants
│   │   ├── evidenceCollector.ts # Diagnostics, runtime metadata, and log evidence
│   │   ├── stepVerifier.ts     # Deterministic verification (10+ strategies)
│   │   ├── verifierUtils.ts    # pollUntil + verify result helpers
│   │   ├── llmClient.ts        # Azure OpenAI client (evidence-based root-cause analysis)
│   │   ├── planParser.ts       # YAML test plan parser (paths relative to plan file)
│   │   └── testRunner.ts       # Orchestrator (launch → execute → screenshot → report)
│   ├── cli/
│   │   └── index.ts            # CLI entry point (run / run-all / analyze / validate)
│   ├── types.ts                # Core type definitions
│   └── index.ts                # SDK exports
├── test-plans/                  # YAML test plans
├── test-results/                # Test output (one subdirectory per plan)
│   └── <plan-name>/
│       ├── results.json
│       ├── diagnostics/
│       └── screenshots/
├── probe-extension/             # Bundled test-only VS Code diagnostics bridge
├── AGENTS.md                    # Copilot CLI integration guide
├── CONTRIBUTING.md              # Contributor workflow and design rules
├── docs/
│   ├── architecture.md          # Architecture
│   ├── implementation-plan.md   # Implementation plan
│   └── ROADMAP.md               # Roadmap
└── package.json

Related documents

• Architecture
• Contributing
• Copilot instructions
• Implementation plan
• Roadmap