Field Service Agent Package

An NPM package for integrating AI-powered field service management into mobile applications. This package provides a natural language interface for managing clients, jobs, invoices, estimates, and communications in field service businesses.

Features

• Natural Language Processing: Process voice or text commands to perform field service operations
• Client Management: Create, update, and retrieve client information
• Job Management: Create and manage work orders with scheduling
• Financial Operations: Create invoices and estimates with line items
• Communication: Initiate calls, texts, emails, and navigation
• Navigation Support: Return navigation instructions for mobile apps
• Multi-language Support: Text-to-speech in multiple languages

Installation

npm install field-service-agent field-service-sdk

Quick Start

import { FieldServiceAgent } from 'field-service-agent';
import { FieldServiceSDK } from 'field-service-sdk';

// First, initialize the SDK
const sdk = new FieldServiceSDK({
  baseUrl: 'https://api.fieldservice.com',
  token: 'your-auth-token'
});

// Then create the agent with the SDK instance
const agent = new FieldServiceAgent({
  sdk: sdk, // Pass the SDK instance
  language: 'en-US',
  timezone: 'America/New_York',
  enableDebugLogging: true,
  googlePlacesApiKey: 'your-google-places-key', // Optional
  onNavigationRequest: (action) => {
    // Handle navigation in your app
    console.log('Navigate to:', action.screen, action.params);
  },
  onCommunicationRequest: (action) => {
    // Handle communication actions in your app
    console.log('Communication:', action.method, action.recipient);
  }
});

// Initialize the agent
await agent.initialize();

// Process a text command
const response = await agent.processCommand('Create a new client named John Doe');
console.log(response);

// Process audio input directly
const audioResponse = await agent.processCommand({
  audioContent: 'base64_encoded_audio_data',
  inputEncoding: 'WEBM_OPUS', // optional, defaults to WEBM_OPUS
  inputSampleRate: 48000       // optional, defaults to 48000
});
console.log(audioResponse);

Audio Input Support

The agent can process audio input directly without requiring separate transcription. This is useful for voice-enabled applications:

// Process audio input
const response = await agent.processCommand({
  audioContent: audioBase64,      // Base64 encoded audio data
  inputEncoding: 'WEBM_OPUS',     // Audio encoding format
  inputSampleRate: 48000          // Sample rate in Hz
});

// The response includes both text and audio
if (response.audioContent) {
  // Play the audio response
  const audioBlob = base64ToBlob(response.audioContent, 'audio/mpeg');
  const audio = new Audio(URL.createObjectURL(audioBlob));
  audio.play();
}

Supported Audio Formats

• WEBM_OPUS: Default format, 48kHz sample rate
• M4A: Alternative format if needed
• Other formats supported by the Vertex AI API

Context Awareness

The agent can be made aware of what the user is currently viewing in your app. This enables more intuitive commands that operate on the current context.

Setting Current Screen Context

// When user navigates to a client details screen
agent.setCurrentScreen('Client', 'client-123');

// When user navigates to a job details screen
agent.setCurrentScreen('Job', 'job-456');

// When user navigates to an invoice
agent.setCurrentScreen('Invoice', 'invoice-789');

// When user navigates to an estimate
agent.setCurrentScreen('Estimate', 'estimate-012');

// Clear context when leaving detail screens
agent.setCurrentScreen(undefined, undefined);

Context-Aware Commands

When context is set, users can issue commands that automatically apply to the current screen:

// While viewing a client
agent.setCurrentScreen('Client', 'client-123');
await agent.processCommand('Create a new job'); // Creates job for client-123
await agent.processCommand('Update email to [email protected]'); // Updates client-123's email

// While viewing a job
agent.setCurrentScreen('Job', 'job-456');
await agent.processCommand('Create an invoice'); // Creates invoice for job-456
await agent.processCommand('Add estimate'); // Creates estimate for job-456

Passing Context with Commands

You can also pass context directly with each command:

const response = await agent.processCommand(
  'Create an invoice',
  {
    currentScreenType: 'Job',
    currentRecordId: 'job-789'
  }
);

Conversation History & Follow-up Questions

The agent maintains conversation history within a session, enabling natural follow-up questions and contextual responses.

Automatic Conversation Tracking

Every command and response is automatically tracked:

// First command
const response1 = await agent.processCommand('Show me clients named John');
// Response: "I found 3 clients named John: John Doe, John Smith, John Williams. Which one would you like to see?"

// Follow-up command - the agent knows this is about the previous question
const response2 = await agent.processCommand('The second one');
// The agent understands "the second one" refers to John Smith from the previous response

Managing Conversation History

// Clear history to start a new conversation
agent.clearConversationHistory();

// Get the full conversation history
const history = agent.getConversationHistory();
console.log(history);
// [
//   { id: 'msg_123_user', text: 'Show me clients named John', isUser: true },
//   { id: 'msg_124_agent', text: 'I found 3 clients...', isUser: false },
//   { id: 'msg_125_user', text: 'The second one', isUser: true },
//   { id: 'msg_126_agent', text: 'Opening client details...', isUser: false }
// ]

// Get only recent messages
const recentMessages = agent.getRecentMessages(5);

Handling Follow-up Questions

The agent automatically detects when it needs more information:

const response = await agent.processCommand('Create an invoice for John Doe');

if (response.followUpQuestion) {
  // The agent is asking a follow-up question
  console.log(response.message); // "John Doe has multiple jobs. Which job is this invoice for?"
  
  // Show options to user and get their selection
  const followUp = await agent.processCommand('The plumbing repair job');
  // The agent will now create the invoice for the correct job
}

Multi-turn Conversations

// Complex multi-step task
await agent.processCommand('I need to create an invoice');
// "Which client is this invoice for?"

await agent.processCommand('John Doe');
// "John Doe has 2 jobs: Plumbing Repair and Kitchen Remodel. Which job is this invoice for?"

await agent.processCommand('The kitchen remodel');
// "What items should I add to the invoice?"

await agent.processCommand('Labor for 8 hours at $50 per hour');
// "I created invoice #123 for John Doe's Kitchen Remodel job with 1 item totaling $400.00."

Configuration

The FieldServiceAgent constructor accepts the following configuration options:

| Option | Type | Required | Description | |--------|------|----------|-------------| | authToken | string | Yes | Authentication token for the Field Service API | | baseUrl | string | Yes | Base URL for the Field Service API | | language | string | No | Language for text-to-speech (default: 'en-US') | | timezone | string | No | Timezone for scheduling (default: system timezone) | | enableDebugLogging | boolean | No | Enable debug logging (default: false) | | onNavigationRequest | function | No | Callback for navigation actions | | onCommunicationRequest | function | No | Callback for communication actions |

Usage Examples

Creating a Client

const response = await agent.processCommand(
  'Create a new client named Sarah Johnson with phone 555-1234'
);

if (response.success) {
  console.log('Client created:', response.result.data.client);
}

Creating a Job

// Update context to current client
agent.updateContext({
  currentClient: { id: 'client-123', name: 'Sarah Johnson' }
});

const response = await agent.processCommand(
  'Create a new job for carpet cleaning scheduled for tomorrow at 2pm'
);

Creating an Invoice

// Update context to current job
agent.updateContext({
  currentJob: { id: 'job-456', title: 'Carpet Cleaning' }
});

const response = await agent.processCommand(
  'Create an invoice for carpet cleaning $50 per room, 3 rooms'
);

Handling Navigation

const agent = new FieldServiceAgent({
  // ... other config
  onNavigationRequest: (action) => {
    switch (action.screen) {
      case 'job_details':
        navigation.navigate('JobDetailsScreen', { jobId: action.params.id });
        break;
      case 'client_details':
        navigation.navigate('ClientDetailsScreen', { clientId: action.params.id });
        break;
      // ... handle other screens
    }
  }
});

await agent.processCommand('Open job details for carpet cleaning');

Handling Communications

const agent = new FieldServiceAgent({
  // ... other config
  onCommunicationRequest: (action) => {
    switch (action.method) {
      case 'call':
        Linking.openURL(`tel:${action.recipient.phone}`);
        break;
      case 'sms':
        Linking.openURL(`sms:${action.recipient.phone}`);
        break;
      case 'email':
        const { subject, body, attachmentUrl } = action.content;
        // Handle email with attachment
        break;
      case 'navigation':
        // Open navigation app
        const address = action.recipient.address;
        const app = action.navigationApp; // 'waze', 'google_maps', 'apple_maps'
        // Handle navigation
        break;
    }
  }
});

await agent.processCommand('Call Sarah Johnson');
await agent.processCommand('Email the invoice to the client');

Response Structure

All commands return an AgentResponse object:

interface AgentResponse {
  success: boolean;
  message: string;
  functionCalled?: string;
  functionArgs?: any;
  result?: {
    success: boolean;
    message: string;
    data?: any;
    action?: NavigationAction | CommunicationAction;
    requiresUserAction?: boolean;
  };
  error?: string;
  debugInfo?: any; // Only when enableDebugLogging is true
}

Action Types

Navigation Actions

interface NavigationAction {
  type: 'navigation';
  screen: 'job_details' | 'client_details' | 'invoice_details' | 
          'estimate_details' | 'calendar_daily' | 'calendar_weekly';
  params: {
    id?: string;
    // ... other params
  };
}

Communication Actions

interface CommunicationAction {
  type: 'communication';
  method: 'call' | 'sms' | 'email' | 'navigation';
  recipient: {
    id: string;
    name: string;
    phone?: string;
    email?: string;
    address?: string;
  };
  content?: {
    subject?: string;
    body?: string;
    attachmentUrl?: string;
  };
  navigationApp?: 'waze' | 'google_maps' | 'apple_maps';
}

Context Management

Update the agent's context when the user navigates or selects items:

// When user views a client
agent.updateContext({
  currentScreenType: 'Client',
  currentRecordId: 'client-123',
  currentClient: { id: 'client-123', name: 'Sarah Johnson' }
});

// When user views a job
agent.updateContext({
  currentScreenType: 'Job',
  currentRecordId: 'job-456',
  currentJob: { id: 'job-456', title: 'Carpet Cleaning', clientId: 'client-123' }
});

Text-to-Speech

Generate audio for agent responses:

const audioBuffer = await agent.getTextToSpeechAudio(
  'Invoice created successfully',
  'en-US'
);

// Play audio in your app
const audio = new Audio(URL.createObjectURL(new Blob([audioBuffer])));
await audio.play();

Available Commands

The agent understands natural language commands for:

• Clients: "Create a new client...", "Update client phone number...", "Show client details"
• Jobs: "Create a job for...", "Schedule for tomorrow at 2pm", "Update job status"
• Invoices: "Create an invoice for...", "Email invoice to client", "List all invoices"
• Estimates: "Create an estimate for...", "Email estimate to client"
• Communication: "Call client", "Text client", "Email client", "Navigate to client"
• Calendar: "Show today's schedule", "Show weekly calendar"

Error Handling

try {
  const response = await agent.processCommand(command);
  
  if (!response.success) {
    console.error('Command failed:', response.message);
    if (response.error) {
      console.error('Error details:', response.error);
    }
  }
} catch (error) {
  console.error('Agent error:', error);
}

API Reference

Main Methods

processCommand(command)

Process a text or audio command.

Parameters:

• command: Can be either:
  • string: Text command to process
  • object: Audio input with:
    • audioContent (string): Base64 encoded audio data
    • inputEncoding (string, optional): Audio format (default: 'WEBM_OPUS')
    • inputSampleRate (number, optional): Sample rate in Hz (default: 48000)

Returns: Promise<AgentResponse> with:

• success (boolean): Whether the command was processed successfully
• message (string): Response message
• audioContent (string, optional): Base64 encoded audio response
• functionCalled (string, optional): Name of function executed
• functionArgs (any, optional): Arguments passed to function

processVoiceCommand(command, context?)

Process a text command with simplified response for TTS. Use processCommand with audio input for direct audio processing.

processVoiceInput(audioContent, inputEncoding?, inputSampleRate?, context?)

Process audio input directly. This method is now deprecated - use processCommand with an audio object instead.

setCurrentScreen(screenType?, recordId?)

Set the current context for the agent.

setLanguage(language)

Set the agent's response language (e.g., 'en-US', 'es-ES', 'fr-FR').

setVoiceName(voiceName)

Set the TTS voice name for audio responses.

TypeScript Support

This package includes full TypeScript definitions. Import types as needed:

import { 
  FieldServiceAgent,
  AgentResponse,
  NavigationAction,
  CommunicationAction,
  FieldServiceAgentConfig
} from 'field-service-agent';

License

MIT

Support

For issues and feature requests, please visit the GitHub repository.