Overview

Batchy is a small utility library that manages batch requests for you. It ships with a small TypeScript library that lets you define batch requests and exposes API hooks to process the processed responses.

• OpenAI Compatible
• Wonderful TypeScript Library
• Complete TypeSafety with Zod

Batchy does three things

1. easily define asynchronous processes that should be batched,
2. receive webhook updates when the processing has finished,
3. get the whole request and response object via the API.

Example

import { z } from 'zod'
import { BatchyClient, createTask } from 'batchy'

const batchy = new BatchyClient({
  endpoint: 'cloud.batchy.ai',
  apiKey: 'SSSSH',
})

// Create a task
const summarize = createTask({
  metadata: z.object({
    document_id: z.string(),
    paragraph_id: z.string(),
  }),
  response: z.object({
    summary: z.string(),
  }),
  cache: '1d',
})

// Submit a request
const id = await batchy.submit(summarize)({
  metadata: {
    document_id: 'doc#1',
    paragraph_id: 'par#1',
  },
  model: 'gpt-4o',
  messages: [
    {
      role: 'system',
      message: 'Summarize the following text and return output in JSON format!',
    },
    {
      role: 'user',
      message: '.....paragraph.....',
    },
  ],
  temperature: 0.2,
  max_tokens: 4960,
})

// Get the completed response.
const result = await batchy.get(summarize)({ id })

// Get the response via Webhook Trigger
const app = express()

app.get('/health', async (req, res) => {
  res.send('OK')
})

app.get('/webhook', async (req, res) => {
  const result = batchy.parse(summarize)(req)
})

https://github.com/joelhooks/inngest-resend-example/blob/main/src/inngest/functions/user-created.ts

Inngest Example

const completedAction = await step.waitForEvent('user/created.document', {
  timeout: '1m',
  if: 'event.user.email == async.user.email',
})