iosense-widget-eval-harness

Standalone static evaluator for AI-generated IOsense widget code. Scores a widget + configuration-panel pair against the platform's own architecture, UNS-binding, and production-readiness rules — with no MCP or runtime dependency. Works as a CI gate, a benchmark/ranking tool across many candidates, and a feedback loop for iterative widget development (human-driven via --watch, or agent-driven via structured JSON output).

Quick start

npx iosense-widget-eval-harness .

That's the whole common case: point it at a widget project root (containing src/components/{Name}/ + src/components/{Name}Configuration/), get a table report and an exit code. Every other flag is opt-in.

iosense-widget-eval-harness <targetDir>

  --format <table|json|junit|llm>   default: table (llm = markdown debug report for LLM/agent fix loops)
  --fail-below <n>               default: 70
  --category <list>              comma-separated: architecture,uns-binding,production
  --run-build                    opt-in, enables production/bundle-size (shells out to build)
  --build-command <cmd>          override default "npm run build:bundle"
  --watch                        re-evaluate on file change, live delta report
  --out <file>                   write report to file
  --quiet

Exit code is non-zero if there's a blocking (error-severity) finding, or the score is below --fail-below — whichever triggers first.

What it checks

Four rule categories — 71 rules as of v1.5.0 — sourced from the platform's own widget architecture/setup/build-ship docs, the design-sdk component manifests and official examples, and calibrated against a real shipped production widget — not invented:

• architecture — paired widget/configuration folder structure, window.ReactWidgets self-registration, props contract (config/data/onEvent for the widget; config/onChange/unsTree/isLoadingTree/onLoadWorkspaces/resolveUNSValue for the configurator), no direct fetch/mini-engine import/envelope mutation in the widget, no apiConfig/data/render/style/time fields sneaking into the built envelope. Since v1.1.0 also: no leftover console.*/debugger, undefined-config handling (first mount), no any in the Props contract, loading/empty-state branching on data, and id= (not key=) on the design-sdk components that migrated in SDK v0.4. Since v1.2.0 also: the full time-mode contract — timeConfig local/fixed shape (rolling window presets vs literal epoch bounds), timeTabConfig pairing, comparison-vs-shifts exclusivity, TimeTabConfiguration usage (required onChange, no custom time UI, GTP globalTimepickers passthrough), and the TIME_CHANGE drilldown payload contract.

• uns-binding — {{uns:wsId://path}} marker format, dynamicBindingPathList shape and presence, UNSPathInput wiring (onOpen, resolveUNSValue piping), single useUNSTree() call, getValue/getSeriesData usage instead of raw data[] indexing.

• production — webpack output.library/libraryTarget must be absent, React must be externalized (design-sdk must not be), widget and configuration must be separate webpack entries, optional gzip bundle-size check (--run-build).

• design-system (since v1.4.0) — design-sdk compatibility, grounded in the SDK inventory (docs/design-sdk-inventory.md): deprecated components (ChartSwitcher → Chart views), ghost components the SDK docs reference but the registry lacks, canonical import subpaths, chart composition anti-patterns (no Card-wrapped charts, bare rules, series/shift conflict, PieChart {name,y} shape), Modal/Accordion/DatePicker grammar, onChange({name,value}) metadata misuse, .fds-* CSS overrides (warning — tracked upgrade debt, kept non-blocking for widget-construction flexibility), hardcoded style values vs tokens — plus "A replacement for <x> in Widget is found in @faclon-labs/design-sdk" adoption warnings for raw HTML controls, refined by <input type>. Since v1.5.0 also: performance/functionality patterns from the official SDK examples — chart data must be memoized (useMemo, not inline .map() in the JSX prop, which forces a full Highcharts redraw every render), ChartTimeProvider required for shift/comparison modes, tooltipCategories alongside comparison/shift, and exportChart needs onChartReady.

Every component pair is also classified into widget types (chart, pie-chart, gauge-kpi, data-point, interactive, ui-generic — additive, data-driven, extensible) — surfaced as perComponent[name].detectedTypes and used by type-conditional rules (chart empty-state handling, drilldown TIME_CHANGE wiring, interactive widgets must emit events).

Scoring

Every finding belongs to a category (above) and a scope — widget, configuration, or shared (structural/webpack checks that don't belong to either side). Both are scored independently: result.categories.architecture.score, result.scopes.widget.score, result.scopes.configuration.score. A candidate can nail the widget and botch the config panel (or vice versa) and that's visible directly, not averaged away.

result.perComponent[name] breaks this down further per {Name}/{Name}Configuration pair, for target directories with more than one widget.

Any error-severity finding sets hasBlockingErrors = true, independent of the numeric score — see docs/decisions/0002-scoring-model.md for why both gates exist.

Library API

import { evaluateWidget, compareResults, createEvaluator } from 'iosense-widget-eval-harness';

const result = await evaluateWidget('./my-widget'); // one-shot (CI, benchmark)
// result.score, result.categories, result.scopes, result.perComponent, result.findings
// result.meta.durationMs — evaluation time

const candidates = await Promise.all(dirs.map((d) => evaluateWidget(d)));
const ranked = compareResults(candidates); // sorted desc by score, ties broken sensibly

// Loop mode (agent fix cycles): parse once, refresh only changed files per iteration —
// much faster than repeated evaluateWidget() calls on non-trivial repos.
const evaluator = createEvaluator('./my-widget');
let report = await evaluator.evaluate();
// ...apply fixes to some files...
evaluator.refresh(changedFilePaths);
report = await evaluator.evaluate();
evaluator.dispose();

Using it as a loop manager

When the loop's consumer is an LLM/agent, use --format llm: a markdown debug report that includes, per finding, the violated platform contract, the offending code snippet (flagged line marked), the mechanical fix with example, and fix ordering — plus a "Constraints to preserve" section listing every passing rule so the agent doesn't fix one finding by introducing another violation. Also available programmatically as renderLlmReport(result, failBelow).

When the consumer is code (a CI parser, a custom script), use --format json — every Finding carries an optional fix: { summary, example? } wherever the violation has a mechanical correction, mirroring the platform's own review_code review/production/fix mode split. For local human iteration, --watch re-runs on file change and prints a score/findings delta against the previous run.

Extending (v2+)

import { registerRule } from 'iosense-widget-eval-harness';

registerRule({
  id: 'design-system/no-raw-button',
  category: 'design-system', // extend the Category union for custom categories
  scope: 'widget',
  severity: 'warning',
  weight: 1,
  description: '...',
  check(ctx) { /* ... */ return []; },
});

The rule engine, scoring, and CLI are category-agnostic — adding a category is registering more rules, not a core change.