picoLLM Inference Engine Node.js Demos

Made in Vancouver, Canada by Picovoice

picoLLM Inference Engine

picoLLM Inference Engine is a highly accurate and cross-platform SDK optimized for running compressed large language models. picoLLM Inference Engine is:

• Accurate; picoLLM Compression improves GPTQ by significant margins
• Private; LLM inference runs 100% locally
• Cross-Platform
• Runs on CPU and GPU
• Free for open-weight models

Compatibility

• Node.js 18+
• Runs on Linux (x86_64), macOS (arm64, x86_64), Windows (x86_64, arm64), and Raspberry Pi (3, 4, 5).

Installation

Using Yarn:

yarn global add @picovoice/picollm-node-demo

or using npm:

npm install --save @picovoice/picollm-node-demo

Models

picoLLM Inference Engine supports the following open-weight models. The models are on Picovoice Console.

• DeepSeek-OCR-2
  • deepseek-ocr-2
• EmbeddingGemma
  • embeddinggemma-300m
• Gemma
  • gemma-2b
  • gemma-2b-it
  • gemma-7b
  • gemma-7b-it
• Gemma3
  • gemma-3-270m
  • gemma-3-270m-it
• Llama-2
  • llama-2-7b
  • llama-2-7b-chat
  • llama-2-13b
  • llama-2-13b-chat
  • llama-2-70b
  • llama-2-70b-chat
• Llama-3
  • llama-3-8b
  • llama-3-8b-instruct
  • llama-3-70b
  • llama-3-70b-instruct
• Llama-3.2
  • llama3.2-1b-instruct
  • llama3.2-3b-instruct
• Mistral
  • mistral-7b-v0.1
  • mistral-7b-instruct-v0.1
  • mistral-7b-instruct-v0.2
• Mixtral
  • mixtral-8x7b-v0.1
  • mixtral-8x7b-instruct-v0.1
• Phi-2
  • phi2
• Phi-3
  • phi3
• Phi-3.5
  • phi3.5
• Qwen3-VL
  • qwen3-vl-2b-it

AccessKey

AccessKey is your authentication and authorization token for deploying Picovoice SDKs, including picoLLM. Anyone who is using Picovoice needs to have a valid AccessKey. You must keep your AccessKey secret. You would need internet connectivity to validate your AccessKey with Picovoice license servers even though the LLM inference is running 100% offline and completely free for open-weight models. Everyone who signs up for Picovoice Console receives a unique AccessKey.

Usage

There are three demos available: completion, chat, and ocr. The completion demo accepts a prompt, an optional image, and a set of optional parameters and generates a single completion. It can run all models (including non-instruction-tuned), unless an image is provided, in which case only vision models can be used. The chat demo can run instruction-tuned (chat) models such as llama-3-8b-instruct, phi2, etc. The chat demo enables a back-and-forth conversation with the LLM, similar to ChatGPT. The optical character recognition (ocr) demo can be run on ocr models, such as deepseek-ocr-2. This demo extracts text from a provided image.

Completion Demo

Run the demo by entering the following in the terminal:

picollm-completion-demo --access_key ${ACCESS_KEY} --model_path ${MODEL_PATH} --prompt ${PROMPT}

Replace ${ACCESS_KEY} with yours obtained from Picovoice Console, ${MODEL_PATH} with the path to a model file downloaded from Picovoice Console, and ${PROMPT} with a prompt string.

To get information about all the available options in the demo, run the following:

picollm-completion-demo --help

Chat Demo

To run an instruction-tuned model for chat, run the following in the terminal:

picollm-chat-demo --access_key ${ACCESS_KEY} --model_path ${MODEL_PATH}

Replace ${ACCESS_KEY} with yours obtained from Picovoice Console and ${MODEL_PATH} with the path to a model file downloaded from Picovoice Console.

To get information about all the available options in the demo, run the following:

picollm-chat-demo --help

OCR Demo

To extract text from an image, run the following in the terminal:

picollm-ocr-demo --access_key ${ACCESS_KEY} --model_path ${MODEL_PATH} --image_path ${IMAGE_PATH}

Replace ${ACCESS_KEY} with yours obtained from Picovoice Console, ${MODEL_PATH} with the path to an ocr model file downloaded from Picovoice Console, and ${IMAGE_PATH} with the absolute path to an image file you wish to extract text from.

To get information about all the available options in the demo, run the following:

picollm-ocr-demo --help