Infer gui

A graphical user interface for local language models. It support different local backends:

• Llama.cpp server
• Koboldcpp
• Ollama

This interface uses no hidden magic: everything is explicit.

Features

• Template editor
• Gbnf grammars support
• Multimodal support

Screenshots

Templates editor

Gbnf grammar editor

Multimodal

Install

With npm

Install:

npm i -g infergui

Run:

infergui

Run a local backend and open localhost:5183 in a browser

With Python

git clone https://github.com/synw/infergui
cd infergui/dist
python3 -m http.server

Run a local backend and open localhost:8000 in a browser

Dev mode

Clone and install the dependencies:

npm i

To run in dev mode with watch autoreload:

npm run dev

Run a local backend and open localhost:5173 in a browser

To build and run:

npm run build
npm run local

Models server

To be able to switch models at runtime use the models server. It is a Koboldcpp process manager integrated in the infergui micro server. To enable the models server run the command with these parameters:

• -m=: the absolute path to your models directory, that contains the gguf files
• -k=: the absolute path to your Koboldcpp command

Example:

infergui -m=/home/me/my/models -k=/home/me/bin/koboldcpp

With these parameters the frontend will display a list of available models and let the user switch to any model at anytime.