togather-shared-events

This repository contains event schemas for the toGather event-driven backend system. We use Protocol Buffers (Protobuf) as the messaging format for events. It provides type-safe code generation for TypeScript and Python, enabling services in different languages to share a consistent event schema.

Table of Contents

1. Directory Structure
2. SDK Development
   • Adding New Proto Files
   • Linting
   • Code Generation
   • Publishing Package
3. Using Generated Code
   • TypeScript
   • Python
4. Best Practices
5. Troubleshooting

Directory Structure

Protobuf files should follow a domain/version-based structure:

proto/
└── <domain>/
    └── <version>/
        └── <event>.proto

1. Domain: The feature or module (e.g., user)
2. Version: The event schema version (e.g., v1)

Example:

proto/user/v1/UserCreated.proto

This structure ensures forward-compatibility and makes schema evolution manageable.

SDK Development

Adding New Proto Files

1. Place your .proto file in the appropriate proto/<domain>/<version> folder.
2. Define the package in the .proto file to match the folder structure.

Example for user.create.v1:

syntax = "proto3";

package user.create.v1;

message UserCreated {
  string id = 1;
  string name = 2;
  string email = 3;
  int64 created_at = 4;
}

Linting

Lint your proto files to ensure consistency and adherence to standards:

pnpm buf:lint

This checks for things like missing packages, incorrect directory structure, and other style issues.

Code Generation

Generate TypeScript and Python code from your proto files:

pnpm buf:generate

The generated code is placed in the gen/ directory:

gen/ts/...
gen/python/...

Tip: Keep buf.gen.yaml updated if you add new languages or plugins.

Publishing Package

To publish the package, run the following command:

pnpm run release

This command performs the following steps:

• Deletes the old build artifacts.
• Lints the proto files.
• Generates new code for TypeScript and Python.
• Patches the npm version.
• Publishes the package to the registry.

Note: Ensure all changes are committed and tested before running this command.

Using Generated Code

TypeScript

Dependencies:

pnpm add @bufbuild/protobuf@^2.9.0

Example usage:

import {UserCreated,UserCreatedSchema} from '@togatherlabs/event-sdk/user/v1'
import { create, toBinary, fromBinary, toJson } from '@bufbuild/protobuf';

// Create a new message
const newUser: UserCreated = create(UserCreatedSchema, {
  id: "1",
  name: "Abhiram",
  email: "[email protected]",
  createdAt: BigInt(Date.now()),
});

// Serialize to binary
const buffer = toBinary(UserCreatedSchema, newUser);

// Deserialize from binary
const decoded = fromBinary(UserCreatedSchema, buffer);

// Convert to JSON
const json = toJson(UserCreatedSchema, decoded);

Tip: Use toBinary/fromBinary for Kafka or network transmission. Use toJson/fromJson for logging or debugging.

Python

Install dependencies (for usage):

pip install protobuf togather-event-sdk 

Dependencies (for maintainers / publishing):

pip install build twine

Example usage:

from togather_event_sdk.user.v1 import UserCreated

# Create a new message
msg = UserCreated(
    id="1",
    name="Abhiram",
    email="[email protected]",
    created_at=int(1728512345123),
)

# Serialize to bytes
binary_data = msg.SerializeToString()

# Deserialize from bytes
decoded = UserCreated()
decoded.ParseFromString(binary_data)

print(decoded)

Build & Publish Python SDK (for maintainers):

pnpm python:build

This runs:

• buf generate
• Adds init.py files
• Builds the Python wheel + source tarball

Tip: Python type hints can be added with mypy-protobuf or community plugins for full static type checking.

Best Practices

1. Package and folder alignment

   • package in .proto must match folder structure (proto/<domain>/<version>).

2. Versioning

   • Always increment the version folder (v1, v2, etc.) when changing message formats to avoid breaking consumers.

3. Type safety

   • Use the generated schemas rather than manually constructing messages.
   • In TypeScript, always use create(); in Python, use the generated UserCreated class.

4. Schema evolution

   • Avoid renaming or deleting existing fields; mark them deprecated instead.
   • Use int64 for timestamps, string for IDs/emails, etc.

How to Type a Event

A Event will have many fields like

message DomainEventExample {
 /// event name (e.g. "user.account.created")
  string event_name = 1;  
  /// Schema version (e.g."v1"), reciving systems can use this to deserialize the payload accordingly with version
  string event_version = 2; 
  /// The entity or aggregate this event belongs to (for partitioning) 
  string aggregate_id = 3; 
 /// Binary-encoded event data (the inner protobuf message),will be typed check events for specific event
  bytes payload = 4;     
  /// Epoch timestamp in milliseconds when the event was created,indicate when this action was recorded by the system
  int64 timestamp = 5; 
}

How should you type a event

syntax = "proto3";

package admin.v1;

import "google/protobuf/descriptor.proto";

extend google.protobuf.FieldOptions {
  string fixed_value = 50001;
}

message Payload {
  string id = 10;          // Account ID
  string email = 11;       // Account email
  string name = 12;        // Account name
  int64 created_at = 13;   // Account creation timestamp
}

message AdminAccountCreated {
  // Common metadata fields (in every event)
   // "admin.account.created"
   string event_name = 1 [(fixed_value) = "admin.account.created"];
  
  string event_version = 2;    // "v1"
  string aggregate_id = 3;     // Account ID for partitioning
  int64 timestamp = 4;         // When event was created
  string trace_id = 5;   // For distributed tracing

  Payload payload = 9;         // Event-specific data
}

check the docs to see why this decistion was taken: docs

Troubleshooting

• ModuleNotFoundError in Python

  • Ensure gen/ folder is in your PYTHONPATH.
  • Add __init__.py files in each subfolder if needed.

• TypeScript errors with missing $typeName or UserCreated

  • Make sure you are importing the schema (UserCreatedSchema) and using create() instead of instantiating the type directly.

• Buf generation issues

  • Run pnpm buf:lint first to catch package/directory mismatches.