Pipe Protocol

A protocol for wrapping LLM tools with IPFS storage and encryption capabilities.

Problem Statement

LLM applications often face challenges with:

• Context window limitations when handling large amounts of data
• Inefficient data sharing between tools and agents
• Redundant storage of identical data
• Lack of persistent storage for tool inputs/outputs

Pipe Protocol solves these challenges by providing content-addressable storage through IPFS, allowing efficient data sharing via CIDs instead of raw data.

Features

• 🔄 Store tool inputs and outputs on IPFS with content-addressable storage
• 📊 Automatic schema generation for stored data
• 🔒 Configurable storage scopes (private/public/machine/user)
• 🎯 Token counting and limiting for LLM context management
• 📌 Configurable pinning options for data persistence
• 🔐 Built-in encryption support with flexible access policies
• 🪝 Pre and post-store hooks for custom data processing
• 🛠️ Tool wrapping with enhanced capabilities
• 📦 Record and Bundle support for schema and data pairing

Installation

npm install pipe-protocol

Quick Start

Basic Usage

Here's a simple example showing how to wrap an OpenAI tool with Pipe Protocol:

import { Pipe } from 'pipe-protocol';
import OpenAI from 'openai';
import type { ChatCompletionTool } from 'openai/resources/chat/completions';

// Initialize OpenAI and Pipe
const openai = new OpenAI();

// Option 1: Use default in-memory IPFS node (recommended for getting started)
const pipe = new Pipe();

// Option 2: Configure custom IPFS node
// const pipe = new Pipe({
//   ipfs: {
//     endpoint: 'http://localhost:5001',
//     timeout: 5000,
//     scope: 'private',
//     pin: true
//   }
// });

// Define a tool that generates Fibonacci numbers
const fibonacciTool = {
  name: 'generate_fibonacci',
  description: 'Generate a Fibonacci sequence of given length',
  parameters: {
    type: 'object',
    properties: {
      n: {
        type: 'number',
        description: 'Number of Fibonacci numbers to generate (must be 2 or greater)'
      }
    },
    required: ['n']
  },
  call: async (args: { n: number }) => {
    const sequence = [0, 1];
    while (sequence.length < args.n) {
      sequence.push(sequence[sequence.length - 1] + sequence[sequence.length - 2]);
    }
    return sequence;
  }
};

// Wrap the tool with Pipe
const wrappedTools = pipe.wrap([fibonacciTool]);

// Format for OpenAI
const openAITools: ChatCompletionTool[] = wrappedTools.map(tool => ({
  type: 'function',
  function: {
    name: tool.name,
    description: tool.description,
    parameters: tool.parameters
  }
}));

// Use with OpenAI
const completion = await openai.chat.completions.create({
  model: 'gpt-4',
  messages: [{ role: 'user', content: 'Generate the first 10 Fibonacci numbers' }],
  tools: openAITools
});

// Execute tool call if made
const toolCalls = completion.choices[0].message.tool_calls;
if (toolCalls) {
  const toolCall = toolCalls[0];
  const args = JSON.parse(toolCall.function.arguments);
  const result = await wrappedTools[0].call(args);
  
  // Access stored data and schema
  console.log('Result CID:', result.cid);
  console.log('Schema CID:', result.schemaCid);
  
  // Retrieve stored data
  const storedData = await pipe.retrieve(result.cid);
  console.log('Retrieved data:', storedData);
}

Tool Interface

Each tool should implement this interface:

interface Tool {
  name: string;
  description: string;
  parameters: {
    type: 'object';
    properties: Record<string, any>;
    required?: string[];
  };
  returns?: {
    type: string;
    description?: string;
  };
  call: (...args: any[]) => any;
}

Adding Hooks

You can add pre and post-processing hooks for custom data handling:

const pipe = new Pipe({
  hooks: [
    {
      name: 'validator',
      type: 'beforeStore',
      handler: async (data) => {
        // Validate data before storage
        if (!data) throw new Error('Invalid data');
        return data;
      }
    },
    {
      name: 'logger',
      type: 'afterStore',
      handler: async (data) => {
        // Log after storage, includes CID and metadata
        console.log('Stored with CID:', data.cid);
        console.log('Metadata:', data.metadata);
        console.log('Schema CID:', data.schemaCid);
        return data;
      }
    }
  ]
});

Configuration Options

const pipe = new Pipe({
  ipfs: {
    endpoint: 'http://localhost:5001', // IPFS node endpoint
    timeout: 5000, // Request timeout
    scope: 'private', // Default scope for data
    pin: true // Auto-pin data to IPFS
  },
  defaults: {
    maxTokens: 1000, // Maximum number of tokens for tool outputs
    storeResult: true, // Automatically store tool results
    generateSchema: true, // Auto-generate schemas
    scope: 'private',
    pin: true
  },
  hooks: [], // Array of pre/post store hooks
  storage: 'memory', // Storage type ('memory' or 'persistent')
  storageConfig: {
    directory: '/tmp/ipfs' // Directory for persistent storage
  },
  enableNetworking: false, // Enable/disable networking
  listenAddresses: [], // Libp2p listen addresses
  bootstrapList: [] // Libp2p bootstrap list
});

Storage Scopes

Pipe Protocol supports four storage scopes:

• private: Data accessible only within the local context
• public: Data that can be shared and replicated across nodes
• machine: Data specific to the current machine/environment
• user: Data specific to the current user

API Documentation

For detailed API documentation, visit our documentation site.

Contributing

Please read CONTRIBUTING.md for details on our code of conduct and the process for submitting pull requests.

License

This project is licensed under the MIT License - see the LICENSE file for details.