@micdrop/gemini
v1.3.0
Published
Google Gemini implementation for @micdrop/server
Readme
@micdrop/gemini
Micdrop website | Documentation
Google Gemini implementation for @micdrop/server.
This package provides an agent (GeminiAgent), a speech-to-text (GeminiSTT) and a text-to-speech (GeminiTTS) running on the Gemini API. GeminiLive is a realtime model running on the Gemini Live API: it hears the user and answers with its own voice, in place of the three. GeminiClassifier answers typed questions about each turn of the user with the Decisions API.
Installation
npm install @micdrop/geminiGemini Agent, STT and TTS
import { GeminiAgent, GeminiSTT, GeminiTTS } from '@micdrop/gemini'
import { MicdropServer } from '@micdrop/server'
const apiKey = process.env.GEMINI_API_KEY || ''
new MicdropServer(socket, {
agent: new GeminiAgent({
apiKey,
model: 'gemini-3.8-flash', // Default model
thinkingLevel: 'low', // Optional, the lower the sooner the first word
systemPrompt: 'You are a helpful assistant',
}),
stt: new GeminiSTT({
apiKey,
model: 'gemini-3.5-transcribe-live', // Default model
language: 'en-US', // Optional, detected when left out
}),
tts: new GeminiTTS({
apiKey,
model: 'gemini-3.8-flash-tts', // Default model
voice: 'Kore', // Default voice
}),
})Each one can be combined with the providers of other packages.
Gemini Live
Usage with MicdropServer
import { GeminiLive } from '@micdrop/gemini'
import { MicdropServer } from '@micdrop/server'
const realtime = new GeminiLive({
apiKey: process.env.GEMINI_API_KEY || '',
model: 'gemini-3.8-live', // Default model
systemPrompt: 'You are a helpful assistant',
voice: 'Kore', // Optional, a prebuilt voice
// Advanced features (optional)
autoEndCall: true, // Automatically end call when user requests
})
new MicdropServer(socket, {
realtime,
generateFirstMessage: true,
})The Micdrop client detects when the user speaks, so the automatic activity detection of Gemini is turned off. Tools are added with addTool() right after creating the model, since Gemini reads them when the session opens. When the user interrupts the answer while a tool runs, its result goes in the conversation and reaches the model silently, so it keeps quiet about it.
Events
| Event | Payload | Description |
| ----------------------- | ------------------------- | ---------------------------------------------------------------------- |
| Audio | Buffer | A chunk of the voice of the model, PCM 16 bits, 24 kHz, mono. |
| PartialMessage | string | The transcript of the answer so far. |
| Message | MicdropConversationItem | A message, a tool call or a tool result was added to the conversation. |
| ToolCall | MicdropToolCall | A tool declared with emitOutput ran, with its parameters and output. |
| CancelLastUserMessage | none | The last user message was dropped because it carried no intent. |
| SkipAnswer | none | The model gave no spoken answer. |
| EndCall | none | The model decided that the call is over. |
| Failed | none | The connection could not be restored after its retries. |
gemini-3.8-live-extended-thinking reasons before answering and needs a thinkingLevel, such as 'low'.
Gemini Classifier
Classifies each turn of the user with the Decisions API: yes or no questions, choices and scores answered with calibrated probabilities by gemini-decision-1, on text, images, audio, video or PDF.
import { choice, GeminiClassifier, noul } from '@micdrop/gemini'
import { MicdropServer } from '@micdrop/server'
const classifier = new GeminiClassifier({
apiKey: process.env.GEMINI_API_KEY || '',
questions: {
intent: choice('What does the user in `turn` want?', {
billing: 'A charge, an invoice, a refund',
outage: 'The service is down or slow',
other: 'Anything else',
}),
wantsHuman: noul('Does the user in `turn` ask for a human?'),
},
})
new MicdropServer(socket, {
stt,
agent,
tts,
classifier,
classifierOptions: { waitBeforeAnswer: true },
})Documentation
Read full documentation of the Gemini integration for Micdrop on the website.
License
MIT
Author
Originally developed for Raconte.ai, created and open sourced by Godefroy de Compreignac
