sampling

Common library for model sampling.

Installation:

Run

yarn add @completion/sampling

Usage:

For the most basic usage, you can start from the following snippet:

import { createSimpleSamplingStream, defaultModelParams } from '@completion/sampling';

createSimpleSamplingStream({
  basicAuth: `Basic ${someAccessToken}`,
  prompt: "\n\nHuman: What is the meaning of life? \n\nAssistant:",
  model: {
    // The `defaultModelParams` object is provided for convenience, with
    // a set of sane defaults for model params:
    ...defaultModelParams,
    tokenLimitPerResponse: 256,
    stopSequence: ["\n\nHuman:", "\n\nAssistant:"],
    name: 'model-name',
  },
  onSampleComplete: (completion) => {
    console.log('Complete sample:', completion);
  },
  onSampleChange: (completion) => {
    console.log('Sample so far:', completion);
  },
  onError: (err) => {
    console.log('Error:', err);
  },
});

If you're using the interface with conversation and templates instead of raw prompts (not recommended for most users), use:

import { createSamplingStream, defaultModelParams } from '@completion/sampling';

// createSamplingStream returns an object containing the WebSocket instance:
const { client } = createSamplingStream({
  basicAuth: `Basic ${someAccessToken}`,
  convo: conversation,
  template: 'default',
  model: {
    // The `defaultModelParams` object is provided for convenience, with
    // a set of sane defaults for model params:
    ...defaultModelParams,
    name: 'model-name',
  },
  onSampleComplete: (data) => {
    console.log('Model says:', data.sampleResult);
  },
});

Additional options are available for the method:

createSamplingStream({
  basicAuth,
  convo,
  template,
  model,

  // A full url to be used for the WebSocket connection:
  url: 'https://foo.bar/foo/bar',

  // A WebSocket instance. This causes url above to be ignored completely:
  ws: new WebSocket('http://foo/bar),

  onSampleComplete,

  // Called every time new sampling data arrives, with the same signature as onSampleComplete:
  //
  // Returning `false` from this method will stop sampling, and cause `onSampleComplete` to be
  // called with `stopReason` set to `stopCallback`.
  onSampleChange: (data) => {},

  // Called with an Error object, if one arises during sampling:
  onError: (error) => {},

  // Called whenever the client internally logs something. If unset, `console.log/info/warn` is
  // used instead.
  onLog: (type: 'info' | 'warn', message) => {},
});

A promise-based interface is also available, which resolves once sampling is complete:

const sampleResult = await createSamplingStreamPromise({
  /* same options as above */,
  rejectOnError: true
});

If provided, callback methods will still be called, before the promise returns/rejects (if appropriate).