@inferencesh/app — build inference.sh apps in node.js

app framework for building inference.sh apps in node.js — file handling, output metadata, and storage utilities.

this is the app-side sdk. for the client-side sdk (calling apps, agents, file uploads), see @inferencesh/sdk.

installation

npm install @inferencesh/app

what's included

| export | description | |--------|-------------| | File | smart file reference — downloads urls, caches locally, resolves paths, serializes for the engine | | StorageDir | standard storage directory constants (DATA, TEMP, CACHE) | | download() | download a url to a directory with caching | | textMeta, imageMeta, videoMeta, audioMeta, rawMeta | output metadata factories for usage-based pricing | | createStreamSchema, pcm16, media, Live | live fields for stream functions — values that flow over the socket while the task runs | | meta() | per-function credentials and annotations (<Name>Meta exports) |

file handling

the File class handles downloading, caching, and path resolution — the node.js equivalent of the python sdk's File class.

reading input files

import { File } from "@inferencesh/app";

async run(inputData) {
  // Input files come as URLs — File downloads and caches them
  const file = await File.from(inputData.image);
  console.log(file.path);        // /home/.cache/inferencesh/files/abc123/image.jpg
  console.log(file.contentType);  // image/jpeg
  console.log(file.size);        // 102400
}

returning output files

import { File } from "@inferencesh/app";

async run(inputData) {
  const outputPath = "/tmp/result.png";
  await generateImage(inputData.prompt, outputPath);

  // File.fromPath is sync — no download needed for local files
  return { image: File.fromPath(outputPath) };
}

the engine reads path from the serialized output and uploads it to cdn automatically.

how File serializes

File implements toJSON() so it works seamlessly with JSON.stringify:

const file = File.fromPath("/tmp/output.png");
JSON.stringify(file);
// {"path":"/tmp/output.png","content_type":"image/png","size":102400,"filename":"output.png"}

construction options

// From URL (downloads and caches)
const file = await File.from("https://example.com/image.jpg");

// From local path
const file = await File.from("/tmp/output.png");

// From options object
const file = await File.from({ path: "/tmp/output.png", contentType: "image/png" });

// From engine-style data (snake_case)
const file = await File.from({ uri: "https://...", content_type: "image/jpeg" });

// Sync from local path (no download)
const file = File.fromPath("/tmp/output.png");

caching

downloaded files are cached at ~/.cache/inferencesh/files/{url_hash}/{filename}. set FILE_CACHE_DIR to override.

storage directories

import { StorageDir, ensureDir } from "@inferencesh/app";

// Standard directories available on inference.sh workers
StorageDir.DATA   // "/app/data"  — persistent storage
StorageDir.TEMP   // "/app/tmp"   — cleaned between runs
StorageDir.CACHE  // "/app/cache" — persists, may be evicted

// Ensure directory exists
const dataDir = ensureDir(StorageDir.DATA);

download utility

import { download, StorageDir } from "@inferencesh/app";

// Download to a specific directory with caching
const modelPath = await download(
  "https://huggingface.co/org/model/resolve/main/weights.bin",
  StorageDir.CACHE
);
// Returns: /app/cache/{hash}/weights.bin

skips download if the file already exists in the target directory (except for TEMP).

output metadata

report what your app processes and generates for usage-based pricing:

import { textMeta, imageMeta, videoMeta, audioMeta } from "@inferencesh/app";

// LLM app
async run(inputData) {
  const result = await this.llm.generate(inputData.prompt);
  return {
    response: result.text,
    output_meta: {
      inputs: [textMeta({ tokens: result.promptTokens })],
      outputs: [textMeta({ tokens: result.completionTokens })],
    },
  };
}

// Image generation app
async run(inputData) {
  const image = await this.model.generate(inputData.prompt);
  return {
    image: File.fromPath(image.path),
    output_meta: {
      outputs: [imageMeta({ width: 1024, height: 1024, steps: 20 })],
    },
  };
}

// Video generation app
async run(inputData) {
  return {
    video: File.fromPath(videoPath),
    output_meta: {
      outputs: [videoMeta({ resolution: "1080p", seconds: 5.0, fps: 30 })],
    },
  };
}

meta types

| factory | key fields | |---------|-----------| | textMeta | tokens | | imageMeta | width, height, resolution_mp, steps, count | | videoMeta | width, height, resolution, seconds, fps | | audioMeta | seconds, sample_rate | | rawMeta | cost (dollar cents) |

all meta types support an optional extra field for app-specific pricing factors.

full app example

import { z } from "zod";
import { File, textMeta } from "@inferencesh/app";

export const RunInput = z.object({
  prompt: z.string().describe("Input prompt"),
  image: z.string().optional().describe("Optional image URL"),
});

export const RunOutput = z.object({
  result: z.string(),
  processedImage: z.any().optional(),
});

export class App {
  async setup(config) {
    this.model = await loadModel();
  }

  async run(inputData) {
    // Handle input files
    let imageFile;
    if (inputData.image) {
      imageFile = await File.from(inputData.image);
    }

    const result = await this.model.process({
      prompt: inputData.prompt,
      imagePath: imageFile?.path,
    });

    return {
      result: result.text,
      processedImage: result.outputPath
        ? File.fromPath(result.outputPath)
        : undefined,
      output_meta: {
        inputs: [textMeta({ tokens: result.inputTokens })],
        outputs: [textMeta({ tokens: result.outputTokens })],
      },
    };
  }
}

stream functions

a stream function takes its input and a socket (async talk(inputData, socket)), runs until it returns, and its return value is the result. some input and output fields are live: their values travel over the socket while the task runs instead of in the request body. declare them with createStreamSchema; in the json schema they become {"type": "array", "format": "stream", "items": ...} and are never required.

import { z } from "zod";
import { createStreamSchema, pcm16, Live } from "@inferencesh/app";

export const TalkInput = z.object({
  effect: z.enum(["none", "robot"]).default("none"),   // ordinary: set at the start, may change live
  audio: createStreamSchema(z, pcm16(z, 16000)),      // live media: binary frames
});

export const TalkOutput = z.object({
  audio: createStreamSchema(z, pcm16(z, 16000)),
  frames: z.number().default(0),
});

export class App {
  async talk(input, socket) {
    const live = new Live(socket, input, TalkInput, TalkOutput);
    let n = 0;
    for await (const { field, value } of live) {
      if (field === "audio") {          // value is a Buffer of PCM samples
        n++;
        await live.send({ audio: value });
      }
      // an ordinary field (effect) is already set on input when it changes
    }
    return { frames: n };
  }
}

on the wire, a binary frame is one item of the schema's single binary live field (pcm16(z, rate, channels) or media(z, contentType)); a json text frame is a partial object keyed by field name: an item of a live field ({"events": {"type": "interrupt"}}) or a new value for an ordinary field ({"effect": "robot"}). frames that do not fit the input schema are answered with {"error": {"field", "message"}} and skipped. live.send({...}) sends the output binary field as a binary frame and everything else as one json frame; unknown output fields throw.

progress from a stream function

a stream function may also be an async generator (async *dictate(input, socket)). each yield is a progress snapshot of the task's output: it replaces the task's output and reaches clients as a task update (sse /tasks/{id}/stream), not over the socket — the socket carries the live fields. yield cumulative snapshots; the last yield is the result, so output_meta goes there, and a generator that ends without yielding fails the task (NoOutput).

export const DictateInput = z.object({
  audio: createStreamSchema(z, pcm16(z, 16000)),   // mic frames arrive over the socket
});

export const DictateOutput = z.object({
  text: z.string().default(""),
  partial: z.boolean().default(true),
});

export class App {
  async *dictate(input, socket) {
    const live = new Live(socket, input, DictateInput, DictateOutput);
    let window = Buffer.alloc(0);
    for await (const { field, value } of live) {
      if (field === "audio") {
        window = Buffer.concat([window, value]);
        if (window.length >= 32000) {                       // one second of 16 khz s16le
          yield { text: await this.transcribe(window) };    // the transcript so far
        }
      }
    }
    yield {
      text: await this.transcribe(window),
      partial: false,
      output_meta: { inputs: [], outputs: [] },
    };
  }
}

the client closing the socket ends the for await, so the final yield goes after the loop. both channels can be used at once: live.send({ text }) pushes a partial over the socket for the lowest latency, while the yields build the transcript the task stores.

frames dropped when the app falls behind

binary frames are perishable media, so the kernel keeps at most 256 of them queued for the app (about five seconds of 20 ms audio) and drops the oldest when the app reads slower than the client sends — latency stays bounded instead of growing for ever. text frames (commands and patches) are never dropped. socket.dropped counts the binary frames lost so far, and socket.binaryBacklog = null turns dropping off when every frame matters (a file sent in chunks).

per-function credentials and annotations

export <PascalName>Meta next to a function's input and output schemas. credentials narrows what the function needs at run time (each entry must also be in inf.yml's credentials:; a function without it gets the app-wide list). annotations are hints about what it does.

import { meta } from "@inferencesh/app";

export const CreateEventInput = z.object({ title: z.string() });
export const CreateEventOutput = z.object({ id: z.string() });
export const CreateEventMeta = meta({
  credentials: [{ key: "google.calendar.events" }],
  annotations: { readOnlyHint: false, destructiveHint: false },
});

export class App {
  async createEvent(input) { /* ... */ }
}

meta() returns its argument unchanged after the same checks as sdk-py's @function (it throws on an empty credentials list, unknown keys, wrong value types, or a credential with neither key nor provider). it gives editors the CredentialRequirement and ToolAnnotations types (add // @ts-check to a .js file to get errors, not only completions).

requirements

• node.js 18.0.0 or higher
• zero runtime dependencies

resources

• documentation — getting started guides
• app development guide — writing app logic
• client sdk — calling apps from your code
• discord — community support

license

MIT © inference.sh