gl-forecast-logger

A distributed logging and tracing system for hierarchical, context-aware logs with DAG visualization. Built with TypeScript and ClickHouse for high-volume, structured log ingestion and fast parent-child (DAG) traversal. Includes batching, context management, and is designed for custom UI exploration and analysis.

Features

• Context-based logging with parent-child (DAG) relationships
• Batching: logs are sent every 10 seconds or after 100 logs
• ClickHouse backend for fast, scalable log storage and querying
• Type-safe API using Zod schemas
• Ready for custom UI and analytics
• Browser-compatible query functions - query logs from client-side applications

Quick Start

1. Install dependencies

bun install

2. Set up your environment

Create a .env file with your ClickHouse Cloud credentials:

CLICKHOUSE_HOST=your-clickhouse-url
CLICKHOUSE_USERNAME=default
CLICKHOUSE_PASSWORD=your-password
CLICKHOUSE_DATABASE=default

3. Set up the database

bun run scripts/setup-database.ts

4. Server-side logging usage

import { createLogger } from "@tonerow/gl-forecast-logger";
import { batchSendToClickhouse } from "@tonerow/gl-forecast-logger";

const logger = createLogger({
  project: "my-project",
  send: batchSendToClickhouse({
    url: process.env.CLICKHOUSE_HOST!,
    username: process.env.CLICKHOUSE_USERNAME!,
    password: process.env.CLICKHOUSE_PASSWORD!,
    database: process.env.CLICKHOUSE_DATABASE,
  }),
});

const ctx = logger.createContext({ function: "main", version: "1.0.0" });
ctx.start();
ctx.log("Hello world!");
ctx.end();

5. Browser/client-side query usage

For browser applications that need to query logs (without Node.js dependencies):

import { queryLogs } from "@tonerow/gl-forecast-logger/browser";

const config = {
  url: "https://your-clickhouse-url",
  username: "default",
  password: "your-password",
  database: "default",
};

// Query logs
const { rows, total } = await queryLogs(config, {
  project: "my-project",
  page: 0,
  pageSize: 50,
  event_type: ["start", "end", "log"],
});

console.log(`Found ${total} logs:`, rows);

6. Query logs from command line

bun run scripts/query-logs.ts

Package Exports

This package provides different entry points for different use cases:

• @tonerow/gl-forecast-logger - Full package with server-side logging (includes Node.js dependencies)
• @tonerow/gl-forecast-logger/browser - Browser-compatible query functions only (no Node.js dependencies)
• @tonerow/gl-forecast-logger/query - Query functions only (minimal import)

Batching

• Logs are buffered and sent in batches every 10 seconds or after 100 logs (whichever comes first).
• On process exit, all remaining logs are flushed.

ClickHouse Schema

See scripts/setup-database.ts for the full schema. The table is optimized for parent-child traversal and time-based queries.

Contributing

• PRs and issues welcome!
• See src/types/log.ts for type definitions and Zod schemas.

This project was created using bun init in bun v1.2.8. Bun is a fast all-in-one JavaScript runtime.