@learning-commons/evaluators

TypeScript SDK for Learning Commons educational text complexity evaluators.

Installation

npm install @learning-commons/evaluators ai

The SDK uses the Vercel AI SDK (ai) as its LLM interface. You also need to install the provider adapter(s) for the LLM(s) you use:

npm install @ai-sdk/openai   # for OpenAI
npm install @ai-sdk/google   # for Google Gemini
npm install @ai-sdk/anthropic  # for Anthropic

Quick Start

import { VocabularyEvaluator } from '@learning-commons/evaluators';

const evaluator = new VocabularyEvaluator({
  googleApiKey: process.env.GOOGLE_API_KEY,
  openaiApiKey: process.env.OPENAI_API_KEY
});

const result = await evaluator.evaluate("Your text here", "5");
console.log(result.score); // "moderately complex"

Evaluators

1. Vocabulary Evaluator

Evaluates vocabulary complexity using the Qual Text Complexity rubric (SAP).

Supported Grades: 3-12

Uses: OpenAI GPT-4o (background knowledge) + Google Gemini 2.5 Pro (grades 3–4) / OpenAI GPT-4.1 (grades 5–12)

Constructor:

const evaluator = new VocabularyEvaluator({
  googleApiKey?: string;  // Google API key (required by this evaluator)
  openaiApiKey?: string;  // OpenAI API key (required by this evaluator)
  maxRetries?: number;    // Optional - Max retry attempts (default: 2)
  telemetry?: boolean | TelemetryOptions; // Optional (default: true)
  logger?: Logger;        // Optional - Custom logger
  logLevel?: LogLevel;    // Optional - SILENT | ERROR | WARN | INFO | DEBUG (default: WARN)
});

API:

await evaluator.evaluate(text: string, grade: string)

Returns:

{
  score: 'slightly complex' | 'moderately complex' | 'very complex' | 'exceedingly complex';
  reasoning: string;
  metadata: {
    model: string;
    processingTimeMs: number;
  };
  _internal: VocabularyComplexity; // Detailed analysis
}

2. Sentence Structure Evaluator

Evaluates sentence structure complexity based on grammatical features.

Supported Grades: 3-12

Uses: OpenAI GPT-4o

Constructor:

const evaluator = new SentenceStructureEvaluator({
  openaiApiKey?: string;  // OpenAI API key (required by this evaluator)
  maxRetries?: number;    // Optional - Max retry attempts (default: 2)
  telemetry?: boolean | TelemetryOptions; // Optional (default: true)
  logger?: Logger;        // Optional - Custom logger
  logLevel?: LogLevel;    // Optional - Logging verbosity (default: WARN)
});

API:

await evaluator.evaluate(text: string, grade: string)

Returns:

{
  score: 'Slightly Complex' | 'Moderately Complex' | 'Very Complex' | 'Exceedingly Complex';
  reasoning: string;
  metadata: {
    model: string;
    processingTimeMs: number;
  };
  _internal: {
    sentenceAnalysis: SentenceAnalysis;
    features: SentenceFeatures;
    complexity: ComplexityClassification;
  };
}

3. Subject Matter Knowledge (SMK) Evaluator

Evaluates the background knowledge demands of educational texts relative to grade level. Determines how much prior subject knowledge a student needs to comprehend the text, based on the Common Core Qualitative Text Complexity Rubric.

Supported Grades: 3-12

Uses: Google Gemini 3 Flash Preview

Constructor:

const evaluator = new SmkEvaluator({
  googleApiKey?: string;  // Google API key (required by this evaluator)
  maxRetries?: number;    // Optional - Max retry attempts (default: 2)
  telemetry?: boolean | TelemetryOptions; // Optional (default: true)
  logger?: Logger;        // Optional - Custom logger
  logLevel?: LogLevel;    // Optional - Logging verbosity (default: WARN)
});

API:

await evaluator.evaluate(text: string, grade: string)

Returns:

{
  score: 'Slightly complex' | 'Moderately complex' | 'Very complex' | 'Exceedingly complex';
  reasoning: string;
  metadata: {
    model: string;
    processingTimeMs: number;
  };
  _internal: {
    identified_topics: string[];
    curriculum_check: string;
    assumptions_and_scaffolding: string;
    friction_analysis: string;
    complexity_score: 'Slightly complex' | 'Moderately complex' | 'Very complex' | 'Exceedingly complex';
    reasoning: string;
  };
}

Example:

import { SmkEvaluator } from '@learning-commons/evaluators';

const evaluator = new SmkEvaluator({
  googleApiKey: process.env.GOOGLE_API_KEY,
});

const result = await evaluator.evaluate(
  "Hydraulic propulsion works by sucking water at the bow and forcing it sternward.",
  "10"
);
console.log(result.score);          // "Very complex"
console.log(result.reasoning);
console.log(result._internal.identified_topics); // ["hydraulics", "propulsion", "physics"]

4. Conventionality Evaluator

Evaluates how explicit, literal, and straightforward a text's meaning is versus how abstract, ironic, figurative, or archaic it is for the target grade level. Based on the Common Core Qualitative Text Complexity Rubric.

Supported Grades: 3-12

Uses: Google Gemini 3 Flash Preview

Constructor:

const evaluator = new ConventionalityEvaluator({
  googleApiKey?: string;  // Google API key (required by this evaluator)
  maxRetries?: number;    // Optional - Max retry attempts (default: 2)
  telemetry?: boolean | TelemetryOptions; // Optional (default: true)
  logger?: Logger;        // Optional - Custom logger
  logLevel?: LogLevel;    // Optional - Logging verbosity (default: WARN)
});

API:

await evaluator.evaluate(text: string, grade: string)

Returns:

{
  score: 'Slightly complex' | 'Moderately complex' | 'Very complex' | 'Exceedingly complex';
  reasoning: string;
  metadata: {
    model: string;
    processingTimeMs: number;
  };
  _internal: {
    conventionality_features: string[];
    grade_context: string;
    instructional_insights: string;
    complexity_score: 'Slightly complex' | 'Moderately complex' | 'Very complex' | 'Exceedingly complex';
    reasoning: string;
  };
}

Example:

import { ConventionalityEvaluator } from '@learning-commons/evaluators';

const evaluator = new ConventionalityEvaluator({
  googleApiKey: process.env.GOOGLE_API_KEY,
});

const result = await evaluator.evaluate(
  "The author uses sustained irony to critique societal norms throughout the passage.",
  "10"
);
console.log(result.score);          // "Very complex"
console.log(result.reasoning);
console.log(result._internal.conventionality_features); // ["sustained irony", ...]

5. Text Complexity Evaluator

Composite evaluator that analyzes vocabulary, sentence structure, subject matter knowledge, and conventionality complexity in parallel.

Supported Grades: 3-12

Uses: Google Gemini 2.5 Pro + Google Gemini 3 Flash Preview + OpenAI GPT-4o (composite)

Constructor:

const evaluator = new TextComplexityEvaluator({
  googleApiKey?: string;  // Google API key (required by this evaluator)
  openaiApiKey?: string;  // OpenAI API key (required by this evaluator)
  maxRetries?: number;    // Optional - Max retry attempts (default: 2)
  telemetry?: boolean | TelemetryOptions; // Optional (default: true)
  logger?: Logger;        // Optional - Custom logger
  logLevel?: LogLevel;    // Optional - Logging verbosity (default: WARN)
});

API:

await evaluator.evaluate(text: string, grade: string)

Returns:

{
  vocabulary: EvaluationResult<TextComplexityLevel> | { error: Error };
  sentenceStructure: EvaluationResult<TextComplexityLevel> | { error: Error };
  subjectMatterKnowledge: EvaluationResult<TextComplexityLevel> | { error: Error };
  conventionality: EvaluationResult<TextComplexityLevel> | { error: Error };
}

Each sub-evaluator result is either a full EvaluationResult or { error: Error } if that evaluator failed. An error is only thrown if all four fail.

Example:

import { TextComplexityEvaluator } from '@learning-commons/evaluators';

const evaluator = new TextComplexityEvaluator({
  googleApiKey: process.env.GOOGLE_API_KEY,
  openaiApiKey: process.env.OPENAI_API_KEY,
});

const result = await evaluator.evaluate("Your text here", "6");

if (!('error' in result.vocabulary)) {
  console.log('Vocabulary:', result.vocabulary.score);
}
if (!('error' in result.sentenceStructure)) {
  console.log('Sentence structure:', result.sentenceStructure.score);
}
if (!('error' in result.subjectMatterKnowledge)) {
  console.log('Subject matter knowledge:', result.subjectMatterKnowledge.score);
}
if (!('error' in result.conventionality)) {
  console.log('Conventionality:', result.conventionality.score);
}

6. Grade Level Appropriateness Evaluator

Determines appropriate grade level for text.

No grade parameter required - evaluates what grade the text is appropriate for.

Uses: Google Gemini 2.5 Pro

Constructor:

const evaluator = new GradeLevelAppropriatenessEvaluator({
  googleApiKey?: string;  // Google API key (required by this evaluator)
  maxRetries?: number;    // Optional - Max retry attempts (default: 2)
  telemetry?: boolean | TelemetryOptions; // Optional (default: true)
  logger?: Logger;        // Optional - Custom logger
  logLevel?: LogLevel;    // Optional - Logging verbosity (default: WARN)
});

API:

await evaluator.evaluate(text: string)

Returns:

{
  score: string; // e.g., 'K-1', '2-3', '4-5', '6-8', '9-10', '11-CCR'
  reasoning: string;
  metadata: {
    model: string;
    processingTimeMs: number;
  };
  _internal: {
    grade: string;
    alternative_grade: string;
    scaffolding_needed: string;
    reasoning: string;
  };
}

Batch CSV Evaluation

For evaluating many texts at once, the SDK ships a CLI tool that reads a CSV file, runs all evaluators in a group, and produces CSV and HTML reports.

# Run from the directory containing your CSV
npx evaluators-batch

The CLI will prompt for your CSV path, API keys, and output directory, then process all rows in parallel with real-time progress.

See src/batch/README.md for full documentation.

Error Handling

The SDK provides specific error types to help you handle different scenarios:

import {
  ConfigurationError,
  ValidationError,
  APIError,
  AuthenticationError,
  RateLimitError,
  NetworkError,
  TimeoutError,
} from '@learning-commons/evaluators';

try {
  const evaluator = new VocabularyEvaluator({ googleApiKey, openaiApiKey });
  const result = await evaluator.evaluate(text, grade);
} catch (error) {
  if (error instanceof ConfigurationError) {
    // Missing or invalid API keys — fix your config
    console.error('Configuration error:', error.message);
  } else if (error instanceof ValidationError) {
    // Invalid input (text too short, invalid grade, etc.)
    console.error('Invalid input:', error.message);
  } else if (error instanceof AuthenticationError) {
    // Invalid API keys
    console.error('Check your API keys:', error.message);
  } else if (error instanceof RateLimitError) {
    // Rate limit exceeded - wait and retry
    console.error('Rate limited. Retry after:', error.retryAfter);
  } else if (error instanceof NetworkError) {
    // Network connectivity issues
    console.error('Network error:', error.message);
  } else if (error instanceof APIError) {
    // Other API errors
    console.error('API error:', error.message, 'Status:', error.statusCode);
  }
}

Logging

Control logging verbosity with logLevel:

import { VocabularyEvaluator, LogLevel } from '@learning-commons/evaluators';

const evaluator = new VocabularyEvaluator({
  googleApiKey: '...',
  openaiApiKey: '...',
  logLevel: LogLevel.INFO, // SILENT | ERROR | WARN | INFO | DEBUG
});

Or provide a custom logger:

import type { Logger } from '@learning-commons/evaluators';

const customLogger: Logger = {
  debug: (msg, ctx) => myLogger.debug(msg, ctx),
  info: (msg, ctx) => myLogger.info(msg, ctx),
  warn: (msg, ctx) => myLogger.warn(msg, ctx),
  error: (msg, ctx) => myLogger.error(msg, ctx),
};

const evaluator = new VocabularyEvaluator({
  googleApiKey: '...',
  openaiApiKey: '...',
  logger: customLogger,
});

Telemetry & Privacy

See docs/telemetry.md for telemetry configuration and privacy information.

Configuration Options

All evaluators use the same BaseEvaluatorConfig interface:

interface BaseEvaluatorConfig {
  googleApiKey?: string;  // Google API key (required by some evaluators)
  openaiApiKey?: string;  // OpenAI API key (required by some evaluators)
  maxRetries?: number;    // Max API retry attempts (default: 2)
  telemetry?: boolean | TelemetryOptions; // Telemetry config (default: true)
  logger?: Logger;        // Custom logger (optional)
  logLevel?: LogLevel;    // Console log level (default: WARN)
  partnerKey?: string;    // Learning Commons partner key for authenticated telemetry (optional)
}

Note: Which API keys are required depends on the evaluator. The SDK validates required keys at runtime based on the evaluator's metadata:

• Vocabulary: Requires both googleApiKey and openaiApiKey
• Sentence Structure: Requires openaiApiKey only
• Subject Matter Knowledge: Requires googleApiKey only
• Conventionality: Requires googleApiKey only
• Text Complexity: Requires both googleApiKey and openaiApiKey
• Grade Level Appropriateness: Requires googleApiKey only

License

MIT