ShellX AI

TypeScript library for Android device automation via WebSocket

Features • Installation • Quick Start • API Reference • Examples

Summary

ShellX AI is a TypeScript library that enables programmatic control of Android devices through a WebSocket connection. It provides a simple, type-safe API for:

• UI Automation - Click, input, swipe, press keys, wait for elements
• Element Finding - Find UI elements by text, ID, class, or coordinates
• Shell Commands - Execute shell commands and get output
• Device Info - Get screen info, app info, screenshots
• Batch Operations - Execute multiple actions in sequence

Features

• 🎯 Simple API - Intuitive methods for common automation tasks
• 🔄 Auto Retry - Built-in retry logic for robust operations
• 📝 Type-Safe - Full TypeScript support with comprehensive types
• 🌍 Cross-Platform - Works in Node.js and browser environments
• 📸 Screenshots - Capture screenshots with customizable options
• 🔧 Shell Commands - Execute commands with real-time output
• 🛠️ Batch Actions - Chain multiple operations together

Installation

npm install shellx-ai

Requirements

• Node.js >= 14.0.0
• TypeScript >= 4.0.0 (recommended)
• Android device connected to ShellX service

Optional Dependencies

For Node.js environment, install ws:

npm install ws  # Optional, for Node.js WebSocket support

For browser environments, no additional dependencies needed.

Quick Start

1. Basic Example

import { ShellX } from 'shellx-ai';

// Create ShellX instance
const shellx = new ShellX({
  deviceId: 'your-device-id'  // Replace with your device ID
});

// Wait for connection
await shellx.ready();

// Click element by text
await shellx.click('Settings');

// Get screen info
const screen = await shellx.getScreenInfo();
console.log(`Screen: ${screen.width}x${screen.height}`);

// Execute shell command
const result = await shellx.command('getprop ro.build.version.release');
console.log('Android version:', result.output);

2. Using Environment Variables

Create a .env file:

DEVICE_ID=your-device-id

Then use it in your code:

import { ShellX } from 'shellx-ai';
import dotenv from 'dotenv';

dotenv.config();

const shellx = new ShellX({
  deviceId: process.env.DEVICE_ID
});

await shellx.ready();
// Start automating...

3. With Connection Events

const shellx = new ShellX({
  deviceId: 'your-device-id',
  onOpen: () => console.log('✅ Connected'),
  onClose: () => console.log('❌ Disconnected'),
  onError: (error) => console.error('⚠️ Error:', error)
});

await shellx.ready();

API Reference

ShellX Class

The main class for Android automation.

Constructor

new ShellX(options: ShellXOptions)

Options:

| Parameter | Type | Required | Default | Description | |-----------|------|----------|---------|-------------| | deviceId | string | Yes | - | Device ID (UUID or identifier) | | timeout | number | No | 5000 | Connection timeout in ms | | reconnect | boolean | No | true | Enable auto-reconnect | | reconnectMaxAttempts | number | No | 5 | Max reconnect attempts | | logLevel | LogLevel | No | INFO | Log level (0=NONE, 1=ERROR, 2=WARN, 3=INFO, 4=DEBUG) | | onOpen | () => void | No | - | Callback when connection opens | | onClose | () => void | No | - | Callback when connection closes | | onError | (error?) => void | No | - | Callback on error | | onMessage | (msg) => void | No | - | Callback on message |

Methods

Connection

| Method | Description | Returns | |--------|-------------|---------| | ready() | Wait for connection to be ready | Promise<void> | | getClient() | Get underlying ConnectionClient | ConnectionClient |

UI Actions

| Method | Description | Example | |--------|-------------|---------| | click(selector) | Click element | await shellx.click('Submit') | | input(data) | Input text | await shellx.input({ text: 'Hello' }) | | swipe(data) | Swipe gesture | await shellx.swipe({ fromX: 500, fromY: 1000, toX: 500, toY: 500 }) | | press(key) | Press key | await shellx.press('BACK') | | wait(selector) | Wait for element | await shellx.wait('Loading') | | find(selector) | Find elements | await shellx.find('Button', { multiple: true }) |

Device Operations

| Method | Description | Example | |--------|-------------|---------| | command(cmd) | Execute shell command | await shellx.command('ls -la') | | getScreenInfo() | Get screen info | await shellx.getScreenInfo() | | takeScreenshot() | Take screenshot | await shellx.takeScreenshot({ format: 'png' }) | | getAppInfo(pkg) | Get app info | await shellx.getAppInfo('com.example.app') | | getAppList() | Get app list | await shellx.getAppList() | | clipboard(data) | Clipboard operations | await shellx.clipboard({ text: 'Hello' }) |

Advanced

| Method | Description | Example | |--------|-------------|---------| | executeActions(actions) | Execute multiple actions | await shellx.executeActions([{ text: 'OK' }, { cmd: 'ls' }]) | | sendRawMessage(msg) | Send raw WebSocket message | await shellx.sendRawMessage({ screenInfo: {} }) |

Type Definitions

Click

Click on an element by text, ID, or coordinates.

Simplified form (click by text):

await shellx.click('Submit');
await shellx.click('Submit', { clickType: 'long' });

Full form:

await shellx.click({
  text: 'Submit',           // Click by text
  elementId: 'btn123',      // Or by element ID
  resourceId: 'submit_btn', // Or by resource ID
  x: 500,                   // Or by coordinates
  y: 1000,
  clickType: 'single',      // 'single' | 'double' | 'long' | 'normal'
  timeout: 5000
});

Input

Input text into a field.

await shellx.input({
  elementId: 'field123',
  text: 'Hello World',
  clear: true,              // Clear field before input
  hideKeyboard: true        // Hide keyboard after input
});

Swipe

Perform swipe gesture.

await shellx.swipe({
  fromX: 500,
  fromY: 1000,
  toX: 500,
  toY: 500,
  duration: 800            // Duration in ms
});

Press

Press hardware key.

Simplified:

await shellx.press('BACK');
await shellx.press('HOME');
await shellx.press('MENU', { longPress: true });

Full:

await shellx.press({
  key: 'BACK',
  longPress: false
});

Wait

Wait for an element condition.

Simplified:

await shellx.wait('Submit');
await shellx.wait('Loading', { timeout: 10000 });

Full:

await shellx.wait({
  text: 'Submit',
  condition: 'visible',      // 'visible' | 'gone' | 'enabled'
  timeout: 5000
});

Find

Find UI elements.

Simplified:

const result = await shellx.find('Button');
const all = await shellx.find('Button', { multiple: true, maxResults: 10 });

Full:

const result = await shellx.find({
  text: 'Submit',
  multiple: false,
  maxResults: 100,
  pressClick: true          // Auto-click after find
});

Command

Execute shell command.

Simplified:

const result = await shellx.command('ls -la');
const result = await shellx.command('ls -la', { timeout: 5000 });

Full:

const result = await shellx.command({
  cmd: 'ls -la',
  timeout: 10000,
  wait: 1000               // Wait after command
});

Screenshot

Take screenshot.

const screenshot = await shellx.takeScreenshot({
  format: 'png',            // 'png' | 'jpeg'
  quality: 100,             // 0-100 for JPEG
  saveToFile: true
});

Clipboard

Clipboard operations.

// Get clipboard
const result = await shellx.clipboard({ get: true });

// Set clipboard
await shellx.clipboard({ text: 'Hello' });

// Paste clipboard
await shellx.clipboard({ paste: true });

ExecuteActions (Batch)

Execute multiple actions in sequence.

const result = await shellx.executeActions([
  { text: 'Settings' },                    // Click
  { cmd: 'ls -la' },                       // Command
  { fromX: 500, fromY: 1000, toX: 500, toY: 500 },  // Swipe
  { key: 'BACK' }                          // Press
]);

console.log(`Success: ${result.successCount}/${result.results.length}`);

Complete Working Example

Here's a complete example that you can run directly:

// File: example.ts
import { ShellX } from 'shellx-ai';

async function main() {
  // 1. Create ShellX instance
  const shellx = new ShellX({
    deviceId: process.env.DEVICE_ID || 'your-device-id',
    onOpen: () => console.log('✅ Connected!')
  });

  try {
    // 2. Wait for connection
    await shellx.ready();
    console.log('📱 Connected to device');

    // 3. Get screen info
    const screen = await shellx.getScreenInfo();
    console.log(`Screen: ${screen.width}x${screen.height}`);

    // 4. Click element
    await shellx.click('Settings');
    console.log('✅ Clicked Settings');

    // 5. Press back
    await shellx.press('BACK');
    console.log('✅ Pressed BACK');

    // 6. Execute command
    const result = await shellx.command('getprop ro.build.version.release');
    console.log(`Android: ${result.output.trim()}`);

    // 7. Take screenshot
    const screenshot = await shellx.takeScreenshot({ format: 'png' });
    console.log(`Screenshot: ${screenshot.imagePath}`);

    // 8. Batch operations
    const batch = await shellx.executeActions([
      { text: 'Settings' },
      { cmd: 'wm size' }
    ]);
    console.log(`Batch: ${batch.successCount}/${batch.results.length} successful`);

    console.log('🎉 All operations completed!');
  } catch (error) {
    console.error('❌ Error:', error);
  }
}

main();

Compile and run:

# Install dependencies
npm install shellx-ai dotenv

# Create .env file
echo "DEVICE_ID=your-device-id" > .env

# Run with tsx
npx tsx example.ts

Response Types

All operations return a result object with the following structure:

interface Result {
  success: boolean;      // Operation success status
  error?: string;        // Error message if failed
  duration: number;      // Operation duration in ms
  timestamp: number;     // Operation timestamp
  // ... additional fields specific to each operation
}

CommandResult

{
  success: boolean;
  output: string;        // Command output
  error?: string;
  exitCode?: number;
  duration: number;
  cmd: string;
  timestamp: number;
}

FindResult

{
  success: boolean;
  found: boolean;
  count: number;         // Number of elements found
  elements: Array<{
    id: string;
    text: string;
    class: string;
    left: number;
    top: number;
    right: number;
    bottom: number;
    visible: boolean;
    clickable: boolean;
  }>;
  duration: number;
  timestamp: number;
}

ScreenInfoResult

{
  success: boolean;
  width: number;
  height: number;
  density: number;
  screenOn: boolean;
  screenUnlocked: boolean;
  foregroundApp?: string;
  foregroundActivity?: string;
  model?: string;
  androidVersion?: string;
  manufacturer?: string;
  duration: number;
  timestamp: number;
}

Logging

Control logging level:

const shellx = new ShellX({
  deviceId: 'your-device-id',
  logLevel: 3  // 0=NONE, 1=ERROR, 2=WARN, 3=INFO, 4=DEBUG
});

// Or at runtime
shellx.setLogLevel(4);         // Enable debug logging
shellx.enableDebugLogging();   // Shortcut
shellx.disableLogging();       // Disable all logging

Error Handling

try {
  await shellx.click('Settings');
} catch (error) {
  if (error instanceof Error) {
    console.error('Click failed:', error.message);
  }
}

// Or check result
const result = await shellx.click('Settings');
if (!result.success) {
  console.error('Failed:', result.error);
}

Advanced Usage

Wait for Multiple Elements

const result = await shellx.waitAnyElement([
  { text: 'OK' },
  { text: 'Confirm' },
  { text: 'Submit' }
], 10000);

Scroll to Find Element

const element = await shellx.scrollToFindElement(
  { text: 'Target Item' },
  5,      // max scroll attempts
  'down'  // direction
);

Find with Retry

const element = await shellx.findElementWithRetry(
  { text: 'Submit', visible: true },
  3,      // max retries
  1000    // retry delay
);

Platform Support

Node.js

npm install shellx-ai ws

import { ShellX } from 'shellx-ai';
const shellx = new ShellX({ deviceId: 'device-id' });

Browser

npm install shellx-ai

import { ShellX } from 'shellx-ai';
const shellx = new ShellX({ deviceId: 'device-id' });

Troubleshooting

Connection Issues

const shellx = new ShellX({
  deviceId: 'your-device-id',
  timeout: 10000,           // Increase timeout
  reconnect: true,          // Enable auto-reconnect
  reconnectMaxAttempts: 10, // More attempts
  onError: (error) => console.error('Connection error:', error)
});

Element Not Found

// Wait for element first
await shellx.wait('Submit', { timeout: 10000 });

// Then click
await shellx.click('Submit');

Command Timeout

const result = await shellx.command({
  cmd: 'long-running-command',
  timeout: 30000  // 30 seconds
});

See Also

• API-GUIDE.md - Detailed API guide
• API-QUICK-REFERENCE.md - Quick reference
• example/ - Complete working examples

License

MIT License - see LICENSE file.

Support

• 📚 Documentation
• 🐛 Issues
• 💬 Discussions