genkitx-transformers-plugin

A Genkit plugin that integrates local AI models via the Hugging Face Transformers.js library.

Features

• Basic proof of Concept Transformers js with Genkit
• ai.generate function
• prompt and messages
• text generation model

Todo

• Streaming
• Flows
• Schema (Output)
• Support for more model types
• Type Saftey
• Better integrate with genkit

Installation

To install the plugin, run the following command in your Genkit project:

npm install genkitx-transformers-plugin

Ensure you have @huggingface/transformers and genkit installed as peer dependencies.

Usage

Add the plugin and specify the models. Currently only supports generate method. Transformers js generation config can be specified.

import { genkit } from "genkit";
import { transformers } from "../src/index.js";

const ai = genkit({
    plugins: [
        transformers({
            models: [
                {
                    name: "onnx-community/SmolLM2-360M-Instruct-ONNX",
                },
            ],
            device: "cpu",
            dtype: "q8",
        }),
    ],
});

const response = await ai.generate({
    model: "transformers/onnx-community/SmolLM2-360M-Instruct-ONNX",
    prompt: "What is Transformers js",
    config: {
        config: {
            temperature: 0.8,
        },
    },
});

console.log(response.text);

Why

I wanted to integrate genkit with local models that could run on the client side, but as of now realizing genkit can't run in the browser, I may have to do something else.