iobroker.assistant
v1.13.0
Published
LLM assistant for ioBroker — ask about and control any state via an LLM with tool-calling
Maintainers
Readme
ioBroker.assistant
An LLM-based assistant for ioBroker. Ask questions in natural language and let the assistant read or control any ioBroker state — it uses an LLM with tool-calling over the native ioBroker API, so there is no rigid rule engine and no virtual-device tree.
Status: early proof-of-concept. Text in → answer out. Audio (satellites, STT/TTS) and on-device wake word are planned next (see Roadmap).
What works today
- Provider: OpenAI (and OpenAI-compatible endpoints via Base URL) or Anthropic (Claude).
- The LLM can call these tools, all backed by the native adapter API:
list_rooms,list_functions— readenum.rooms/enum.functionsfind_states({ room?, func?, query? })— find states + current valuesget_state({ id })— read one valueset_state({ id, value })— control a device (can be disabled in settings)
- Text interface via two states:
- write your question to
assistant.0.text.request - read the answer from
assistant.0.text.response
- write your question to
- Or from a script:
sendTo('assistant.0', 'ask', { text: 'Wie warm ist es im Wohnzimmer?' }, cb)
Configuration
In the adapter admin (Instances → assistant → ⚙):
| Setting | Meaning |
|--------------------------|---------------------------------------------------|
| Provider | openai or anthropic |
| Model | e.g. gpt-4o-mini, gpt-4o, claude-sonnet-4-6 |
| API key | your provider API key |
| Base URL | optional endpoint override (e.g. Groq) |
| Allow controlling states | if off, the assistant is read-only |
| System prompt | persona / behaviour |
Voice satellites (Hannah)
The adapter runs a UDP voice server that speaks the Hannah satellite protocol
(0x01 control / 0x02 mic audio / 0x03 TTS), so an existing Hannah Pi/ESP satellite can
talk to it directly. STT → LLM → TTS runs in the adapter; the satellite only captures audio,
detects the wake word and plays the reply.
On the adapter side: open the Voice tab, tick Enable voice server, pick the STT/TTS
providers + credentials, keep the port (default 7775). When the instance starts the log shows:
Voice server listening on UDP 7775Recognised text and the reply also appear in assistant.0.text.request / .text.response, with
the origin in assistant.0.text.querySource (satellite name, chat, or empty for a direct state write).
Announcements / TTS: write to assistant.0.tts.text to speak on all satellites, or to
assistant.0.satellites.<id>.tts for one. The value is spoken as text (via the configured TTS engine),
or — if it is a URL/path to an audio file (.mp3/.wav/…) — played back.
Playing audio files needs ffmpeg on the host (Windows and Linux): Linux →
sudo apt install ffmpeg; Windows → install ffmpeg and add it to thePATH, or enter its full path under Path to ffmpeg on the Voice tab. Without either, Download ffmpeg on the Voice tab fetches a static build (ffmpeg-static, 20–30 MB) into the data directory of the instance. Plain-text TTS does not need it — only audio-file playback (mp3/wav/…) does.
Start a Hannah satellite pointed at this adapter
Match the audio rates to your device (--sample-rate = mic, --tts-rate = speaker); list devices
and supported rates with python3 -c "import pyaudio; p=pyaudio.PyAudio(); [print(i, p.get_device_info_by_index(i)) for i in range(p.get_device_count())]".
The stock Hannah satellite locates the server via MQTT discovery, so it needs a broker reachable
and one retained discovery message. Point --host at the ioBroker host explicitly:
# 1. publish the adapter address once (retained) so the satellite finds it:
mosquitto_pub -h <broker-ip> -t hannah/server -r -m '{"host":"<iobroker-ip>","port":7775}'
# 2. start the satellite (venv), --broker = MQTT broker, --host = this adapter:
/opt/Hannah/satellite-pi/venv/bin/python3 /opt/Hannah/satellite-pi/satellite.py \
--device wohnzimmer --room Wohnzimmer \
--broker <broker-ip> --host <iobroker-ip> --port 7775 \
--mic 0 --speaker 0 --sample-rate 16000 --tts-rate 48000Success looks like Registrierung bestätigt (ACK empfangen) in the satellite log and
Satellite registered: wohnzimmer in the adapter log. Then say the wake word → speak → the answer
is spoken back.
Without an MQTT broker
The stock Hannah satellite always connects to MQTT (even with --host) and exits if no broker is
reachable. To run fully broker-free, skip that one call — in satellite.py,
_resolve_hannah_address():
if self.cfg.hannah_host:
self._hannah_addr = (self.cfg.hannah_host, self.cfg.hannah_port)
# self._mqtt.connect() # ← comment out to run without a broker (disables MQTT status/LWT only)
else:
self._hannah_addr = self._mqtt.connect()Registration, audio and TTS all run over UDP, so only the (optional) MQTT status reporting is lost.
Then start with just --host (no --broker needed):
/opt/Hannah/satellite-pi/venv/bin/python3 /opt/Hannah/satellite-pi/satellite.py \
--device wohnzimmer --room Wohnzimmer --host <iobroker-ip> --port 7775 \
--mic 0 --speaker 0 --sample-rate 16000 --tts-rate 48000Native Node.js satellite
A standalone Node.js satellite is available — no ioBroker, no MQTT broker required. On a Pi (or any Linux/Windows/macOS box) with a mic + speaker:
npx @iobroker/assistant-satellite # writes a default config, then edit "host"
npx @iobroker/assistant-satellite config.jsonIt runs the wake word (OpenWakeWord) on the device and streams to this adapter's voice server. See
@iobroker/assistant-satellite for setup, the
check diagnostics and install (systemd service).
Wyoming endpoint (experimental)
Wyoming is the open voice protocol from the Home Assistant /
Rhasspy project (JSONL events over TCP), used by wyoming-satellite and other Rhasspy-style
services.
ESPHome voice devices (including the Home Assistant Voice PE) do not speak Wyoming — they speak the ESPHome native API. Use the transport below for those.
Enable Also accept Wyoming clients (TCP) in the Voice tab (default port 10700). The adapter then
exposes a Wyoming server that bridges to the same pipeline: audio-start/audio-chunk/audio-stop →
STT → answer → TTS streamed back as audio-* (plus a transcript event); describe → info;
synthesize → TTS.
The protocol framing is unit-tested, but interop with a real
wyoming-satelliteis still experimental — please report what works. Uses the same STT/TTS providers as the UDP voice server.
ESPHome voice satellites
Devices that run the ESPHome voice assistant — the ThirdReality Voice & Music Assistant (Dev Edition), the Home Assistant Voice PE, or any box running linux-voice-assistant — do not connect to a server: they are one, waiting on TCP 6053. So here the adapter is the client and dials them, the way Home Assistant would. Wake word, echo cancellation and playback stay on the device; the adapter does speech recognition, the answer and speech synthesis.
Enable Also drive ESPHome voice satellites in the Voice tab and add one row per device (address, optional port, room). Nothing needs to be installed or flashed on the device, and no ESPHome tooling is involved — "ESPHome native API" is just the protocol name.
Two things are different from the other transports:
- The adapter decides when you stopped talking. These devices stream until the server stops them, so end-of-speech detection runs here — tune it with End of speech after (ms of silence) if replies cut you off or the assistant waits too long.
- The spoken reply is fetched, not pushed. The device plays a URL, so the adapter runs a small
HTTP server (Media server port, default
8099) that serves the clip for a couple of minutes. It must be reachable from the device; the address is derived from each device connection, and only needs setting by hand behind NAT/Docker/VLAN.
Announcements (tts.text, satellites.<id>.tts, timers and alarms) reach these satellites too, and
each one shows up under assistant.0.satellites.* like any other.
Developed against the ThirdReality speaker's firmware sources and covered by a loopback test against a fake device; feedback from real hardware is welcome.
Wake words: which model for which satellite
The wake word is always detected on the satellite, so the model format depends on the satellite
type. There are two model families, and both can come as .tflite — they are not interchangeable:
a model trained for one does not run on the other. To use the same phrase on both kinds of satellites,
train it twice.
| Satellite | Engine | Custom model | Where it goes |
|-----------------------------------------------------------------------------------|------------------------------------------------------------|-----------------------------------------------------------------------|-----------------------------------------------------------------------------------|
| ESPHome (ThirdReality speaker) | microWakeWord | .tflite, optionally with its .json | this adapter: Satellites tab → ⚙ → Upload own wake word |
| Native satellite (assistant-satellite adapter, @iobroker/assistant-satellite) | openWakeWord | one self-contained .onnx | the satellite's settings: upload, then pick it in wakewordModel (or a URL/path) |
| Hannah Pi satellite (UDP) | openWakeWord | .onnx (.tflite only with inference_framework: tflite on armv7l) | --wakeword-model <path> |
| Wyoming clients | whatever the client runs | — | on the client; not managed by this adapter |
microWakeWord (ESPHome)
The model takes 40 spectrogram features per frame: input
[1, n, 40], quantized (int8 in, uint8 out) — like the official models in esphome/micro-wake-word-models.Train your own with microWakeWord's
notebooks/basic_training_notebook.ipynb(Python 3.10, a GPU helps a lot). The result istrained_models/wakeword/tflite_stream_state_internal_quant/stream_state_internal_quant.tflite— rename it after the phrase, e.g.vita_nova.tflitebecomes "Vita Nova".The
.jsonis optional. Without it the adapter takes the phrase from the file name, the language fromvoiceLanguage, the cutoff0.97and a sliding window of5. Provide one to set a differentmicro.probability_cutoff:{ "type": "micro", "wake_word": "Vita Nova", "model": "vita_nova.tflite", "trained_languages": ["de"], "version": 2, "micro": { "probability_cutoff": 0.97, "sliding_window_size": 5 } }The upload checks the model and rejects anything the device cannot run, with the reason — an openWakeWord model is named as such.
The device downloads the model from the media server when you activate the chip; at most two wake words are active at a time. A downloaded model stays on the device (the API cannot delete it). A device whose firmware does not take external wake words does not list the uploaded one.
openWakeWord (native and Hannah satellites)
- The model takes speech embeddings of 96 values, e.g. input
[1, 16, 96], float — which is why it does not run on ESPHome devices. - openWakeWord's training notebook exports
.onnxand.tflite. The native satellite needs the.onnxas one file — a model split into.onnx+.onnx.datais not supported. - Built into the native satellite:
hey_jarvis,alexa,hey_mycroft,hey_rhasspy; up to three wake words at once.
Roadmap
- Text assistant (done) — LLM + tool-calling over ioBroker states.
- Fast-path for common commands (on/off/timer) without an LLM round-trip.
- TTS / STT engines (Polly / Azure / OpenAI / AWS Transcribe) as adapter modules + config.
- Satellite endpoint — UDP audio + MQTT control, so ESP/Pi satellites talk to the adapter directly.
- Wake word — trained/managed via ioBroker, running on the device.
- Wyoming server endpoint — accept
wyoming-satelliteand Rhasspy-style clients. - ESPHome satellites (done) — drive ThirdReality / HA Voice PE / linux-voice-assistant devices.
Changelog
1.13.0 (2026-11-20)
- (@GermanBluefox) Music from Spotify: "play Bohemian Rhapsody by Queen", "spiel das Album … in der Küche", "spiel meine Playlist …" find the song, album, artist or playlist and play it on a Spotify Connect device (table "Spotify players" on the Services tab, needs spotify-premium 3.3.0 or newer); LLM tools
play_musicandmusic_player, messageplayMusic - (@GermanBluefox) Music clips on the TV: "spiel den Clip Bohemian Rhapsody von Queen", "play the music video … by …", "включи клип …" search the clip on YouTube and play it on the YouTube player; so does a song for a TV
1.12.1 (2026-11-19)
- (@GermanBluefox) Voice per answer language: the language is a drop-down now. A language picked from the suggestions was saved as its label (e.g. "de – Deutsch") and the row was never used; such rows get the language code on the first start
- (@GermanBluefox) Spoken requests: the speech recognition no longer waits for the device names it gets as hints (up to 3 s on large installations each time a minute had passed since the last request); the last list is used while a new one is built
- (@GermanBluefox) Request details: the speech recognition step shows the language it really got (that of the wake word) and the ioBroker.voice instance instead of the unrelated Vosk model; the speech synthesis step shows the voice, model and language that really spoke (a row of "voice per answer language" or the reserve) instead of the main voice of the settings
1.12.0 (2026-11-19)
- (@GermanBluefox) Device commands can be understood by the adapter smart-text2command instead of the built-in rules (tab Commands, "Device commands understood by"): it runs only the actions approved there and asks back when the room or the device is unclear; the next utterance of the same satellite or chat answers. What it does not understand goes on to the local and the cloud model; timers, alarms, time, weather and volume stay with the built-in rules
- (@GermanBluefox) New first tab "Flow": the way of a request as a diagram (sources → speech recognition → local decision → local LLM → cloud LLM / pi-agent → speech output), each step can be set up with a click
1.11.0 (2026-11-18)
- (@sadam6752-tech) ESPHome satellites: a language per wake word (tab Satellites, "Language per wake word", e.g. Alisa → Russian, Hey Bro → German): speech recognition, the reply voice and the follow-ups of that conversation use it, and pi-agent asks its yes/no questions in it
- (@sadam6752-tech) ESPHome satellites on an ESP32: two wake words can be active at once (the firmware reports a limit of one, but runs every wake word it is sent)
- (@sadam6752-tech) ioBroker.voice gets the language of the speech (its model Canary 1B v2 uses it, Parakeet ignores it) and only the hints in the alphabet of the language
- (@sadam6752-tech) Whisper (OpenAI and compatible servers) gets only the device and room names in Latin letters as hints for a language written in them: Russian names pulled German speech towards Cyrillic letters
1.10.0 (2026-11-17)
- (@GermanBluefox) ESPHome satellites: the adapter ships microWakeWord models ("ioBroker", "Hey Bro", "Alisa"); one click in the satellite settings puts one on the device and switches it on, replacing an older version of the same word
- (@GermanBluefox) Wake words from a training run get a short name with their date, e.g.
261103_iobrokerinstead of "20261103 085427 9b0f Microwakeword Iobroker" - (@GermanBluefox) A wake word selected right after it was uploaded is no longer dropped as unknown
Older changelogs can be found there
License
MIT License
Copyright (c) 2026 Denis Haev [email protected]
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
