@promptrepo/score

Calculate confidence scores for OpenAI JSON outputs. Transform black box responses into reliable data with granular field-level scoring.

Why @promptrepo/score?

• 📊 Precise Scoring: Convert OpenAI's logprobs into confidence metrics for each JSON field
• 🚀 Simple Integration: Works directly with OpenAI's completion and chat endpoints
• ⚡ Lightweight: Zero dependencies for scoring logic

Installation

npm install @promptrepo/score
npm install openai  # Required for OpenAI API examples

Model Requirements

This package requires OpenAI models that support both logprobs and structured JSON output:

• For chat completion: gpt-3.5-turbo or gpt-4 or gpt-4o
• For completion: gpt-3.5-turbo-instruct

Basic Usage

With OpenAI Chat Completion

import { calculateConfidenceScores } from '@promptrepo/score';
import OpenAI from 'openai';

const openai = new OpenAI({
    apiKey: 'sk-...' // Replace with your OpenAI API key
});

// Make an API call
const response = await openai.chat.completions.create({
    model: "gpt-3.5-turbo",
    messages: [
        {
            role: "system",
            content: "You are a helpful assistant that returns JSON responses."
        },
        {
            role: "user",
            content: "Extract product information from: 'Product: Premium Widget, Price: $29.99'"
        }
    ],
    response_format: { type: "json_object" },
    logprobs: true,
    max_tokens: 500,
    temperature: 0
});

// Parse JSON response
const jsonOutput = JSON.parse(response.choices[0].message.content);

// Raw OpenAI output
console.log('OpenAI output:', jsonOutput);
// {
//   "Product name": "Premium Widget",
//   "Price": "29.99"
// }

// Calculate confidence scores
const result = calculateConfidenceScores(jsonOutput, response.choices[0].logprobs.content);

// @promptrepo/score output with confidence scores
console.log('Confidence scores:', result);
// {
//   "Product name": { value: "Premium Widget", score: 0.95 },
//   "Price": { value: "29.99", score: 0.92 }
// }

Nested Structures

import { calculateConfidenceScores } from '@promptrepo/score';
import OpenAI from 'openai';

const openai = new OpenAI({
    apiKey: 'sk-...' // Replace with your OpenAI API key
});

// Make an API call
const response = await openai.chat.completions.create({
    model: "gpt-3.5-turbo",
    messages: [
        {
            role: "system",
            content: "You are a helpful assistant that returns JSON responses with nested product information."
        },
        {
            role: "user",
            content: "Extract detailed product information from: 'Product: Premium Widget, Price: $29.99, Weight: 1.5kg, Dimensions: 10x20x30cm'"
        }
    ],
    response_format: { type: "json_object" },
    logprobs: true,
    max_tokens: 500,
    temperature: 0
});

// Parse JSON response
const jsonOutput = JSON.parse(response.choices[0].message.content);

// Raw OpenAI output
console.log('OpenAI output:', jsonOutput);
// {
//   product: {
//     details: {
//       name: "Premium Widget",
//       price: "29.99"
//     },
//     specifications: {
//       weight: "1.5kg",
//       dimensions: "10x20x30cm"
//     }
//   }
// }

// Calculate confidence scores
const result = calculateConfidenceScores(jsonOutput, response.choices[0].logprobs.content);

// @promptrepo/score output with confidence scores
console.log('Confidence scores:', result);
// {
//   product: {
//     value: {
//       details: {
//         value: {
//           name: { value: "Premium Widget", score: 0.95 },
//           price: { value: "29.99", score: 0.92 }
//         },
//         score: 0.93
//       },
//       specifications: {
//         value: {
//           weight: { value: "1.5kg", score: 0.94 },
//           dimensions: { value: "10x20x30cm", score: 0.91 }
//         },
//         score: 0.92
//       }
//     },
//     score: 0.92
//   }
// }