ML Bottleneck engine SDK

The planner's physics engine — decode and prefill rooflines, memory fit, parallelism search, speculative-decoding model, and benchmark calibration — as a dependency-free JavaScript library. It is built from the exact engine.js that mlbottleneck.com runs, so a third-party page gets the same numbers as the site.

Get it

| Channel | How | | --- | --- | | npm | npm install @mlbottleneck/engine → import { createEngine } from '@mlbottleneck/engine' (evidence: import snapshot from '@mlbottleneck/engine/snapshot') | | Script tag (UMD) | <script src="https://mlbottleneck.com/dist/mlbottleneck-engine.umd.js"></script> → window.MLBottleneck | | ES module | import { createEngine } from 'https://mlbottleneck.com/dist/mlbottleneck-engine.mjs' | | GitHub release | sdk-v<version> releases on the repo carry the tarball, both bundles, the TypeScript types, the evidence snapshot, and checksums | | Node | download dist/ (or the release tarball) and import('./mlbottleneck-engine.mjs') / require('./mlbottleneck-engine.umd.js') |

Benchmark evidence is optional: dist/localmaxxing-snapshot.json carries the gold rows the site calibrates against. Without it the engine still predicts from physics and reports confidence: "uncalibrated".

60-second example

<script src="https://mlbottleneck.com/dist/mlbottleneck-engine.umd.js"></script>
<script>
  const engine = MLBottleneck.createEngine();
  const result = engine.predict({
    model: 'qwen3.8_27b',          // preset key, label, or Hugging Face id
    hardware: { template: 'RTX 3090', count: 2 },
    quantization: 'Q4_K_M',        // family ("q4") or format label
    runtime: 'llama_cpp',
    promptTokens: 4096,
    outputTokens: 512
  });
  console.log(result.decode.tokensPerSecond, 'tok/s decode');
  console.log(result.prefill.tokensPerSecond, 'tok/s prefill');
  console.log(result.fits ? 'fits in VRAM' : result.warnings.join(' '));
</script>

With evidence (calibrated expectation, peers, confidence):

import { createEngine } from './mlbottleneck-engine.mjs';
const snapshot = await fetch('./localmaxxing-snapshot.json').then(r => r.json());
const engine = createEngine({ snapshot });
const { ceiling } = engine.predict({ model: 'qwen3.6_35b_a3b', hardware: 'AMD Strix Halo (Ryzen AI Max+ 395)', quantization: 'q4', runtime: 'llama_cpp' });
// ceiling.expectedTokensPerSecond  – engine rate × measured peer correction
// ceiling.optimizedTokensPerSecond – what a well-tuned run of this stack reaches
// ceiling.physicalTokensPerSecond  – zero-overhead bandwidth roofline
// ceiling.confidence               – 'strong' | 'directional' | 'uncalibrated'

predict(request)

| Field | Type | Notes | | --- | --- | --- | | model | string | object | Preset key (listModels()), or an architecture object: { totalParamsB, hiddenSize, numLayers, numHeads, numKVHeads?, intermediateSize?, isMoE?, numExperts?, activeExperts?, activeParamsB?, attentionMechanism?, useMTP? }. Add preset: 'qwen3_8b' to start from a preset and override fields. | | hardware | string | object | array | Template key(s) (listHardware()), { template, count }, or a custom device { name, memoryGB, localBandwidthGBps, computeTFlops: { float16 }, networkBandwidthGBps? }. Lookups are case/space-insensitive and accept unique partial names ('H100 SXM 80GB'). | | quantization | string | Family float32 | float16 | bfloat16 | int8 | fp8 | q6 | q5 | q4 | q3 | q2 or a format label (Q4_K_M, UD-IQ4_XS, MXFP4, NVFP4, AWQ, Q8_0, …) — formats carry their real bits-per-weight. Default q4. | | runtime | string | auto (llama.cpp on consumer GPUs, vLLM on data-center NVIDIA, SGLang on Instinct/Gaudi/TPU, MLX on Macs), llama_cpp, ollama, mlx, vllm, sglang, tensorrt_llm, exo. | | strategy | string | auto (search), pipeline, tensor, data, expert, sequence, context, hybrid_tp_pp, hybrid_tp_dp. | | batchSize | number | Concurrent sequences. decode.tokensPerSecond is the aggregate across the batch; decode.perUserTokensPerSecond (and msPerToken) is what one sequence sees. | | promptTokens, outputTokens | number | The workload. KV memory is sized for prompt + output; the decode rate is read at prompt + output/2. | | speculation | object | { method: 'mtp' | 'dflash' | 'dspark' | 'eagle3' | 'draft_model' | 'ngram' | 'suffix', tokens?, acceptance?, draftRatio? }. Omitted fields use the method's published defaults. Draft weights and draft KV count toward memory; gains shrink with batch size and context. | | kvCacheCompression | string | none, q8_kv, q4_kv (llama.cpp -ctk q8_0 / q4_0, vLLM fp8 KV). Compressed KV reads fewer bytes but the decode attention kernel pays a dequantization cost, so deep contexts do not get the full byte saving. | | cpuMoeLayers | number | MoE only: pin this many layers' routed experts to system RAM (llama.cpp --n-cpu-moe N). Default: only what memory forces. | | usage | object | { hoursPerDay, costPerKwh } for the power/cost estimate. | | includeRaw | boolean | Attach the full per-device engine output under result.raw. |

Result

{
  fits: boolean,                         // every device holds its share of weights + KV
  strategy: { key, reasoning, auto },
  decode:  { tokensPerSecond, msPerToken, perUserTokensPerSecond, withoutSpeculation, speculationMultiplier },
  prefill: { tokensPerSecond, timeToFirstTokenSeconds },
  ceiling: { physicalTokensPerSecond,      // zero-overhead bandwidth/compute roofline
             latencyBoundTokensPerSecond,  // roofline plus the irreducible per-layer/per-token floor and coordination
             optimizedTokensPerSecond,     // what the best-demonstrated kernel efficiency on this stack reaches
             expectedTokensPerSecond,      // engine rate x peer correction (stock software)
             engineTokensPerSecond, correctionFactor, confidence, peers, verifiedPeers },
  memory:  { modelSizeGB, residentWeightsGB, kvCacheGB, availableGB },
  measured: { nearest,                     // closest stock measurement on this model + hardware template (community gold run or lab stock row), or null
              labTuned },                  // closest tuned neural.download lab run on the same machine (same template and device count), or null
           // each: { tokensPerSecond (per request), aggregateTokensPerSecond, concurrency, origin: 'community'|'lab', stack,
           //         model, hardware, deviceCount, runtime, quantization, depthTokens, speculation, sameSetup, url, note }
  bottleneck: 'memory' | 'compute' | 'runtime' | 'coordination' | ...,   // devices[].coreBinding adds 'attention' for deep contexts
  power:   { watts, tdpWatts, costPerDay, costPer1KTokens },
  devices: [{ name, template, residentWeightGB, kvCacheGB, hasOverflow, overflowMode,
              decodeTokensPerSecond, prefillTokensPerSecond, rooflineTokensPerSecond,
              decodeBreakdownMs: { weightRead, kvRead, compute, runtime, draft, coordination, total } }],
  warnings: string[],
  config:  { model, quantization, quantFormat, runtime, batchSize, promptTokens, outputTokens, speculation }
}

Numbers are planning estimates: the engine is calibrated so that the median community run lands on its prediction and ~85% land within 1.5×. Show users the ceiling ladder (physical → optimized → expected) rather than a single number when you can, and measured.nearest / measured.labTuned beside it when they exist — they are real runs, with links.

The evidence snapshot carries two collections: goldCases (community runs, the calibration corpus) and labCases (the author's neural.download Intel Arc Pro lab: stock, lab-baseline, and tuned rows). Lab rows never calibrate the engine; they only surface as measured references.

Other methods

• engine.sweep(request, { levels, maxContext }) — decode/prefill/memory across prompt lengths and concurrency levels (what the site's "How it scales" charts plot).
• engine.listModels() / engine.listHardware() — catalogs with sizes, MoE flags, and supersededBy for old releases.
• engine.setEvidence(snapshot) — load or replace benchmark evidence after creation.
• engine.catalogs — the raw MODEL_PRESETS, DEVICE_TEMPLATES, FRAMEWORK_PROFILES, SPECULATION_METHODS, QUANT_FORMATS tables.
• engine.engine.* — lower-level functions with the same signatures as engine.js (calculateMetricsForConfig, findOptimalStrategy, getSpeculationPlan, buildExecutionPlan, …) for integrations that need the per-device breakdowns.

Deep links into the planner

Any site can open the full planner pre-configured (no SDK needed):

https://mlbottleneck.com/?model=qwen3.8_27b&hardware=Intel%20Arc%20Pro%20B70&count=2&format=Q4_K_M&runtime=vllm&prompt=4096&output=512&spec=mtp:3#plan

model (preset key, label, or Hugging Face id), hardware (template name, case/space-insensitive), count, quant (family), format (exact quant label), runtime, strategy, prompt, output, batch, spec (method[:draft tokens]). Unknown values are ignored; anything recognized opens the prediction step.

Versioning and releases

• package.json version is the SDK version; engine.version reports it.
• npm test rebuilds dist/ from engine.js + sdk/api.js (and stamps the page's engine.js?v=<hash> cache key), so the committed bundle is always the one that passed the suite.
• Bump the version when the engine, catalogs, or API change; .github/workflows/release-sdk.yml then publishes a sdk-v<version> GitHub release with the bundles, types, evidence snapshot, and checksums, and .github/workflows/publish-npm.yml publishes dist/ to npm as @mlbottleneck/engine (name set by sdkName in package.json). It authenticates with npm trusted publishing (OIDC: the package's Trusted Publisher on npmjs.com points at steveseguin/ml-bottleneck, workflow publish-npm.yml), falling back to an NPM_TOKEN secret; a 2FA-gated token fails with EOTP, in which case publish by hand from dist/ with npm publish --access public --otp=<code>. The weekly evidence refresh updates dist/localmaxxing-snapshot.json in place without a release.

Limits

• Predictions assume the runtime's mainstream kernels (flash attention on, weights resident unless the engine models expert offload or spill).
• Network-bound multi-node setups use the template's interconnect bandwidth; pass networkBandwidthGBps per device for real fabrics.
• The evidence snapshot is community-submitted (Localmaxxing); confidence tells you how much of it applies to your stack.