starlight-pipeline

A lightweight neural-network and NLP pipeline for Starlight, inspired by TensorFlow and Keras. Supports tokenization, vectorization, dense neural networks, SGD & Adam optimizers, model saving/loading, and training progress visualization.

Features

• Chainable pipeline API
• Dense neural networks (fully connected layers)
• Optimizers: SGD and Adam
• Training progress bar (TensorFlow-style)
• Text preprocessing (tokenize, stopwords, vectorize)
• Save / Load models (.slm binary format)
• Loss functions: MSE, Cross-Entropy
• Evaluation utilities (accuracy, confusion matrix)

Installation

npm install starlight-pipeline

Dependencies:

• starlight-ml
• starlight-vec
• starlight-eval

Quick Start

Training a Sentiment Model

import fs from "fs"
import { pipeline } from "starlight-pipeline"

let raw = fs.readFileSync("train.json", "utf8")
let data = JSONParse(raw)

let texts = data.texts
let labels = data.labels

define p = pipeline()
  .tokenize()
  .removeStopwords(["the", "is", "and", "to", "of"])
  .vectorize(texts)

let inputSize = p.context.vectorizer.size

p
  .dense(inputSize, 16, "relu")
  .dense(16, 1, "sigmoid")
  .compile({ loss: "mse" })

let X = []
for (let i = 0; i < texts.length; i++) {
  X.push(p.context.vectorizer.transform(texts[i]))
}

p.fit(X, labels, {
  epochs: 300,
  lr: 0.01,
  optimizer: "adam"
})

p.saveModel("sentiment.slm")

Training Output

[██████████████████████████████----------] 62.0% Epoch 190/300 - Loss: 0.1824
...
Training complete!

Loading & Predicting

import { Pipeline } from "starlight-pipeline"

define model = Pipeline.loadModel("sentiment.slm")

let text = ask("Enter text:")
let vec = model.context.vectorizer.transform(text)
let pred = model.predict(vec)

sldeploy("Prediction:", pred[0])

Pipeline API

pipeline()

Creates a new pipeline instance.

Text Processing

.tokenize()
.removeStopwords(words)
.vectorize(trainingTexts)

Neural Layers

.dense(inputSize, outputSize, activation)

Supported activations:

• linear
• relu
• sigmoid
• softmax

Training

.fit(X, y, {
  epochs: 100,
  lr: 0.01,
  optimizer: "adam" // or "sgd"
})

Optimizers:

• SGD – simple and fast
• Adam – adaptive learning rate with momentum

Loss Functions

.compile({ loss: "mse" })

Available:

• mse
• crossEntropy

Model Persistence

Save

pipeline.saveModel("model.slm")

Load

Pipeline.loadModel("model.slm")

The .slm format stores:

• Network architecture
• Weights & biases
• Activation types
• Vectorizer vocabulary

Evaluation

pipeline.evaluate(yTrue, yPred)

Returns:

• Accuracy
• Confusion matrix
• Classification report

Example Use Cases

• Sentiment analysis
• Text classification
• Lightweight ML in CLI apps
• Educational neural networks
• Offline inference

Notes

• Designed for small to medium datasets
• Not GPU-accelerated
• Intended for learning, prototyping, and embedded ML

Roadmap

• Mini-batch training
• Dropout layers
• Learning rate schedulers
• Multi-class classification helpers
• Streaming datasets

License

MIT License