@zai/streaming-parser

An async generator-based streaming JSON parser for real-time field tracking. Built on top of @streamparser/json with a clean, functional API that provides accumulated parse state as JSON fields are completed during streaming.

Features

• 🔄 Real-time field tracking - Get accumulated state as each top-level field completes
• 🔧 Async generator API - Bidirectional communication with next(chunk) and yielded state
• 🧩 Chunk-based streaming - Process JSON data as it arrives in chunks
• 🔒 TypeScript support - Fully typed discriminated union for type-safe state access
• 🧪 Well tested - Comprehensive test suite with 18 test cases
• 🚀 High performance - Built on the fast @streamparser/json library
• ⚡ Simple build - TypeScript-only build with clean ESM output

Installation

pnpm add @zai/streaming-parser

Basic Usage

import { parseStreamingJSON } from '@zai/streaming-parser';

async function example() {
  const parser = parseStreamingJSON();
  
  // Initialize the parser
  let result = await parser.next();
  console.log(result.value.status); // 'ready'
  
  // Send chunks and get accumulated state
  result = await parser.next('{"name": ');
  console.log(result.value.status); // 'ready' (no fields completed yet)
  
  result = await parser.next('"John", "age": ');
  console.log(result.value.status); // 'parsing'
  console.log(result.value.completedFields); // ['name']
  
  result = await parser.next('30}');
  console.log(result.value.status); // 'complete'
  console.log(result.value.completedFields); // ['name', 'age']
  
  // Type-safe access to parsedData with discriminated union
  if (result.value.status === 'complete') {
    console.log(result.value.parsedData); // { name: 'John', age: 30 }
  }
}

Web Component Generation Example

Perfect for tracking AI-generated web components:

import { parseStreamingJSON } from '@zai/streaming-parser';

async function trackWebComponentGeneration() {
  const parser = parseStreamingJSON();
  await parser.next(); // Initialize
  
  let result = await parser.next('{"name": "WeatherWidget",');
  console.log(`✅ Fields: ${result.value.completedFields.join(', ')}`); // name
  
  result = await parser.next(' "html": "<div>Weather content</div>",');
  console.log(`✅ Fields: ${result.value.completedFields.join(', ')}`); // name, html
  
  result = await parser.next(' "css": ".widget { color: blue; }"}');
  console.log(`🎉 Complete! Fields: ${result.value.completedFields.join(', ')}`); // name, html, css
  
  // Type-safe access to final data
  if (result.value.status === 'complete') {
    console.log('Final data:', result.value.parsedData);
  }
}

Processing Multiple Chunks

For processing multiple chunks in sequence:

import { parseStreamingJSON } from '@zai/streaming-parser';

async function processChunks() {
  const chunks = ['{"name": ', '"John", "age": ', '30}'];
  const parser = parseStreamingJSON();
  await parser.next(); // Initialize
  
  for (const chunk of chunks) {
    const result = await parser.next(chunk);
    console.log(`Status: ${result.value.status}`);
    console.log(`Completed fields: ${result.value.completedFields.join(', ')}`);
    
    if (result.value.status === 'complete') {
      console.log('Final data:', result.value.parsedData);
      break;
    } else if (result.value.status === 'error') {
      console.error('Parse error:', result.value.error);
      break;
    }
  }
}

API Reference

Types

StreamingParseState (Discriminated Union)

type StreamingParseState = 
  | {
      /** Parser is ready to receive first chunk */
      status: 'ready';
      /** Array of completed top-level field names */
      completedFields: string[];
    }
  | {
      /** Parser is actively parsing and has completed some fields */
      status: 'parsing';
      /** Array of completed top-level field names */
      completedFields: string[];
      /** Name of field currently being processed (if any) */
      inProgressField?: string;
    }
  | {
      /** Parsing completed successfully */
      status: 'complete';
      /** Array of completed top-level field names */
      completedFields: string[];
      /** Complete parsed JSON data */
      parsedData: any;
    }
  | {
      /** Parser encountered an error */
      status: 'error';
      /** Array of completed top-level field names */
      completedFields: string[];
      /** Name of field currently being processed (if any) */
      inProgressField?: string;
      /** Parse error */
      error: Error;
    };

Functions

parseStreamingJSON()

Returns an async generator for streaming JSON parsing.

const parser = parseStreamingJSON();

// Initialize
const initResult = await parser.next();

// Send chunks
const result = await parser.next('{"field": "value"}');

Generator Protocol:

• First next(): Initialize, returns { status: 'ready', completedFields: [] }
• Subsequent next(chunk): Process chunk, returns current accumulated state
• When status is 'complete' or 'error', the generator is done

Status States

• ready: Parser initialized, no fields completed yet
• parsing: Actively parsing, some fields may be completed, inProgressField may indicate current field being parsed
• complete: Parsing finished successfully, parsedData available
• error: Parsing failed, error object available, may preserve inProgressField from when error occurred

Type-Safe Usage with Discriminated Union

The discriminated union provides excellent TypeScript support:

const result = await parser.next(chunk);

// TypeScript narrows the type based on status
switch (result.value.status) {
  case 'ready':
    // Only completedFields is available
    console.log('Ready:', result.value.completedFields);
    break;
    
  case 'parsing':
    // completedFields and optionally inProgressField
    console.log('Parsing:', result.value.completedFields);
    if (result.value.inProgressField) {
      console.log('In progress:', result.value.inProgressField);
    }
    break;
    
  case 'complete':
    // completedFields and parsedData are guaranteed
    console.log('Complete:', result.value.parsedData);
    break;
    
  case 'error':
    // completedFields and error are guaranteed
    console.error('Error:', result.value.error.message);
    break;
}

Real-Time Field Progress Tracking

The parser provides real-time feedback about which field is currently being parsed:

import { parseStreamingJSON } from '@zai/streaming-parser';

async function trackFieldProgress() {
  const parser = parseStreamingJSON();
  await parser.next(); // Initialize
  
  // Start parsing
  let result = await parser.next('{"username": ');
  console.log(result.value.status); // 'parsing'
  if (result.value.status === 'parsing') {
    console.log(result.value.inProgressField); // 'username'
    console.log(result.value.completedFields); // []
  }
  
  // Continue with partial value
  result = await parser.next('"john_');
  if (result.value.status === 'parsing') {
    console.log(result.value.inProgressField); // 'username' (still parsing)
  }
  
  // Complete first field
  result = await parser.next('doe", ');
  if (result.value.status === 'parsing') {
    console.log(result.value.inProgressField); // undefined (field completed)
    console.log(result.value.completedFields); // ['username']
  }
  
  // Start next field
  result = await parser.next('"email": ');
  if (result.value.status === 'parsing') {
    console.log(result.value.inProgressField); // 'email'
    console.log(result.value.completedFields); // ['username']
  }
}

Advanced Usage

Error Handling

const parser = parseStreamingJSON();
await parser.next(); // Initialize

const result = await parser.next('{"name": invalid}');
if (result.value.status === 'error') {
  console.error('Parse error:', result.value.error.message);
  console.log('Fields completed before error:', result.value.completedFields);
}

Streaming Progress Tracking

const parser = parseStreamingJSON();
await parser.next();

// Track progress as chunks arrive
const chunks = ['{"field1": "val1", ', '"field2": "val2", ', '"field3": "val3"}'];
for (const chunk of chunks) {
  const result = await parser.next(chunk);
  const progress = (result.value.completedFields.length / 3) * 100;
  console.log(`Progress: ${progress}%`);
  
  if (result.value.status === 'complete') {
    console.log('Final result:', result.value.parsedData);
    break;
  }
}

Use Cases

• AI Content Generation: Track field completion during streaming AI responses
• Real-time Progress: Show progress bars for large JSON downloads
• Incremental Processing: Process data as soon as fields become available
• Error Recovery: Know exactly where parsing failed and what was completed
• Live Dashboards: Update UI elements as data streams in

Development

Building

cd packages/streaming-parser
pnpm build

Testing

Run the comprehensive test suite:

cd packages/streaming-parser
pnpm test

The package includes 18 test cases covering:

• Basic async generator functionality
• Streaming in multiple chunks
• Complex nested objects and arrays
• Error handling and recovery
• Real-time field progress tracking (inProgressField)
• State management and immutability
• Type safety with discriminated unions

License

MIT