🖥️ Screen Control Runtime (SCR)

A production-grade runtime that lets AI agents safely observe and control graphical applications.

Quick Start • Architecture • Project Structure • Development • Contributing

Overview

SCR is infrastructure for autonomous AI agents. It turns natural-language or programmatic instructions into verified, replayable action sequences on real graphical interfaces — browser today, desktop and mobile next.

It's built to be the shared execution layer underneath any agent or client: Claude, ChatGPT, Codex, Folk, or your own MCP-based agent — not tied to a single model or product.

✨ Features

| | | |---|---| | 🔒 Type-Safe | Full TypeScript support with strict type checking end to end | | 🧩 Modular Architecture | Composable modules with clear separation of concerns (planner, engine, observer, verifier) | | ⚡ Event-Driven | Reactive event system for state changes and executed actions | | 🛡️ Production Ready | Comprehensive error handling, structured logging, and test coverage | | 🔌 Extensible | Plugin-based architecture for custom targets and custom actions | | 🌐 Cross-Platform | Chromium today; Android, Desktop, iOS Simulator, and Cloud VM backends on the roadmap |

📦 Installation

pnpm add @scr-runtime/runtime

npm install @scr-runtime/runtime
# or
yarn add @scr-runtime/runtime

🚀 Quick Start

import { SCR } from '@scr-runtime/runtime';

const scr = new SCR({
  target: 'chromium',
  session: {
    id: 'my-session',
  },
});

await scr.start();

🏗️ Architecture

SCR turns an instruction into a verified action through a single pipeline:

Instruction ──▶ Planner ──▶ Execution Engine ──▶ Execution Backend ──▶ Observer ──▶ Verifier
                    ▲                                                                  │
                    └──────────────────────── feedback / retry ──────────────────────┘

• Planner — turns high-level instructions into a concrete, ordered action plan
• Execution Engine — runs the plan against a session, handling retries and errors
• Execution Backend — the target-specific driver (Chromium first, more coming)
• Observer — captures the resulting screen/DOM state
• Verifier — confirms the action had the intended effect before moving on

📁 Project Structure

src/
├── contracts/      # Type definitions and interfaces
├── runtime/        # Core runtime implementation
├── engine/         # Execution engine
├── planner/        # Action planning
├── observer/       # Screen observation
├── verifier/       # State verification
├── actions/        # Action implementations
├── sessions/       # Session management
├── memory/         # Memory and state persistence
├── registry/        # Component registry
├── targets/        # Target implementations
│   └── chromium/   # Chromium browser target
├── sdk/            # Public SDK
├── cli/            # Command-line interface
├── events/         # Event system
└── utils/          # Utility functions

🛠️ Development

Prerequisites

• Node.js >= 22.0.0
• pnpm >= 9.0.0

Setup

pnpm install

Common commands

| Command | Description | |---|---| | pnpm build | Build the project | | pnpm test | Run the test suite | | pnpm lint | Lint the codebase | | pnpm format | Format code with Prettier | | pnpm docs | Generate documentation |

🗺️ Roadmap

• [x] Chromium execution backend
• [ ] Android backend
• [ ] Desktop backend
• [ ] Remote browser / desktop backend
• [ ] iOS Simulator backend
• [ ] Cloud VM scaling

🤝 Contributing

Issues and pull requests are welcome. Please open an issue first to discuss any significant change before submitting a PR.

📄 License

Released under the MIT License.