@hauska-sdk/retrieval

CNS Protocol Retrieval SDK - IPFS document retrieval with gated access, encryption, watermarking, and access logging.

Installation

npm install @hauska-sdk/retrieval

Quick Start

import { RetrievalSDK } from "@hauska-sdk/retrieval";
import { VDASDK } from "@hauska-sdk/vda";
import { PostgreSQLStorageAdapter } from "@hauska-sdk/adapters-storage-postgres";
import { WalletManager } from "@hauska-sdk/wallet";
import { Pool } from "pg";

// Initialize VDA SDK (required for access verification)
const pool = new Pool({
  connectionString: process.env.DATABASE_URL,
});

const vdaSDK = new VDASDK({
  storageAdapter: new PostgreSQLStorageAdapter({
    pool,
    autoMigrate: true,
  }),
  walletManager: new WalletManager(),
});

// Initialize Retrieval SDK
const retrievalSDK = new RetrievalSDK({
  pinata: {
    pinataJwt: process.env.PINATA_JWT!,
    pinataGateway: "https://gateway.pinata.cloud",
  },
  vdaSdk: vdaSDK,
  logHook: (level, message, data) => {
    console.log(`[${level}] ${message}`, data);
  },
});

// Upload a document
const file = Buffer.from("Document content");
const uploadResult = await retrievalSDK.uploadDocument(file, {
  encrypt: true,
  vdaId: "vda-123",
  metadata: {
    name: "property-deed.pdf",
    keyvalues: {
      propertyId: "prop-123",
    },
  },
});

console.log(`Document uploaded: ${uploadResult.cid}`);

// Retrieve a document with access verification
const document = await retrievalSDK.retrieveDocument(
  uploadResult.cid,
  "0x1234567890123456789012345678901234567890",
  {
    decrypt: true,
    watermark: true, // Watermark for access pass viewers
  }
);

console.log(`Document retrieved: ${document.size} bytes`);

Features

• ✅ IPFS Document Storage - Upload and retrieve documents via Pinata
• ✅ Encryption/Decryption - AES-256-GCM encryption for secure document storage
• ✅ Gated Access Control - VDA-based access verification before document retrieval
• ✅ Permission Enforcement - Fine-grained permissions (view, download, write, annotate)
• ✅ PDF Watermarking - Automatic watermarking for access pass viewers
• ✅ Access Logging - Track document access for audit trails and analytics
• ✅ Gateway Proxy - Express/Fastify middleware for HTTP document access
• ✅ Retry Logic - Automatic retry for network failures
• ✅ Logging Hooks - Optional monitoring and debugging support

Gated Access Flow

The Retrieval SDK enforces access control using VDAs (Verified Digital Assets). Here's how it works:

1. Document Upload
   └─> Upload to IPFS (Pinata)
   └─> Store CID in VDA metadata (ipfsCid)

2. Document Retrieval Request
   └─> Verify VDA Access
       ├─> Check direct ownership
       ├─> Check access passes
       ├─> Verify permissions
       └─> Check expiry/revocation

3. Access Granted
   └─> Fetch from IPFS
   └─> Decrypt (if encrypted)
   └─> Apply watermark (if access via access pass)
   └─> Log access
   └─> Return document

4. Access Denied
   └─> Return 402 (Payment Required) or 403 (Forbidden)

Access Types

• Direct Ownership - Wallet owns the VDA directly
• Access Pass - Wallet has a valid, non-expired, non-revoked access pass
• No Access - Returns 402 Payment Required

Permission Levels

• view - Can view the document
• download - Can download the document
• write - Can modify the document
• annotate - Can add annotations

API Reference

RetrievalSDK

Main SDK class that provides all document retrieval functionality.

Constructor

new RetrievalSDK(config: RetrievalSDKConfig)

Configuration:

interface RetrievalSDKConfig {
  pinata: PinataConfig;                    // Required: Pinata configuration
  vdaSdk: VDASDK;                          // Required: VDA SDK instance
  pinataClient?: PinataClient;             // Optional: Custom Pinata client
  accessVerification?: VDAAccessVerificationService; // Optional: Custom access verification
  accessLogging?: AccessLoggingConfig;     // Optional: Access logging configuration
  logHook?: LogHook;                       // Optional: Logging hook
  getEncryptionKey?: (cid: string, wallet: string) => Promise<Buffer | undefined>; // Optional: Encryption key provider
  getVDAIdFromCID?: (cid: string) => Promise<string | null>; // Optional: Custom VDA ID lookup
}

Methods

uploadDocument(file: Buffer, options?: UploadDocumentOptions): Promise<UploadDocumentResult>

Upload a document to IPFS via Pinata.

const result = await retrievalSDK.uploadDocument(file, {
  encrypt: true,                    // Encrypt before upload
  encryptionKey: key,               // Optional: Custom encryption key
  pin: true,                         // Pin to Pinata
  metadata: {
    name: "document.pdf",
    keyvalues: {
      vdaId: "vda-123",
      spoke: "real-estate",
    },
  },
  vdaId: "vda-123",                  // Associate with VDA
  spoke: "real-estate",
});

console.log(`CID: ${result.cid}`);
console.log(`Encrypted: ${result.encrypted}`);
if (result.encryptionKey) {
  // Store encryption key securely
  await storeEncryptionKey(result.cid, result.encryptionKey);
}

fetchDocument(cid: string, options?: FetchDocumentOptions): Promise<FetchDocumentResult>

Fetch a document from IPFS via Pinata gateway.

const document = await retrievalSDK.fetchDocument("QmTest123", {
  decrypt: true,
  encryptionKey: await getEncryptionKey("QmTest123"),
});

console.log(`Size: ${document.size} bytes`);
console.log(`Content Type: ${document.contentType}`);
console.log(`Decrypted: ${document.decrypted}`);

verifyAccess(cid: string, wallet: string, requiredPermissions?: PermissionType[]): Promise<VDAAccessVerificationResult>

Verify if a wallet has access to a document.

const result = await retrievalSDK.verifyAccess(
  "QmTest123",
  "0x1234567890123456789012345678901234567890",
  ["view", "download"]
);

if (result.hasAccess) {
  console.log(`Access type: ${result.accessType}`); // "direct" or "access-pass"
  console.log(`VDA ID: ${result.vdaId}`);
} else {
  console.log(`Access denied: ${result.reason}`);
}

retrieveDocument(cid: string, wallet: string, options?: RetrieveDocumentOptions): Promise<FetchDocumentResult>

Retrieve a document with full access verification, optional watermarking, and access logging.

const document = await retrievalSDK.retrieveDocument(
  "QmTest123",
  "0x1234567890123456789012345678901234567890",
  {
    requiredPermissions: ["view"],
    decrypt: true,
    watermark: true,                 // Watermark for access pass viewers
    operation: "view",
  }
);

Options:

interface RetrieveDocumentOptions {
  requiredPermissions?: PermissionType[];  // Default: ["view"]
  decrypt?: boolean;                       // Default: false
  watermark?: boolean;                     // Default: false (only for access pass viewers)
  operation?: string;                     // Operation type for logging
}

getAccessLogs(cid: string, options?: { limit?: number; offset?: number }): Promise<AccessLogQueryResult>

Get access logs for a document (requires access logging configuration).

const logs = await retrievalSDK.getAccessLogs("QmTest123", {
  limit: 50,
  offset: 0,
});

console.log(`Total accesses: ${logs.total}`);
logs.logs.forEach((log) => {
  console.log(`${log.wallet} accessed at ${new Date(log.timestamp)}`);
});

getAccessLogsByWallet(wallet: string, options?: { limit?: number; offset?: number }): Promise<AccessLogQueryResult>

Get access logs for a wallet.

const logs = await retrievalSDK.getAccessLogsByWallet(
  "0x1234567890123456789012345678901234567890",
  { limit: 100 }
);

Pinata Integration

The Retrieval SDK uses Pinata for IPFS storage and retrieval.

Setup

1. Create a Pinata account at https://pinata.cloud
2. Generate a JWT token in the Pinata dashboard
3. Configure the SDK with your JWT:

const retrievalSDK = new RetrievalSDK({
  pinata: {
    pinataJwt: process.env.PINATA_JWT!,
    pinataGateway: "https://gateway.pinata.cloud", // Optional: Custom gateway
  },
  vdaSdk: vdaSDK,
});

Pinata Features Used

• File Upload - pinFileToIPFS() for document storage
• File Retrieval - Gateway for document fetching
• Metadata - Store VDA ID and document metadata
• Pinning - Ensure documents remain available

Custom Gateway

You can use a custom Pinata gateway or public IPFS gateway:

const retrievalSDK = new RetrievalSDK({
  pinata: {
    pinataJwt: process.env.PINATA_JWT!,
    pinataGateway: "https://ipfs.io", // Public gateway
  },
  vdaSdk: vdaSDK,
});

Code Examples

Complete Document Upload Flow

import { RetrievalSDK } from "@hauska-sdk/retrieval";
import { VDASDK } from "@hauska-sdk/vda";
import fs from "fs";

// Initialize SDKs
const vdaSDK = new VDASDK({ /* ... */ });
const retrievalSDK = new RetrievalSDK({
  pinata: { pinataJwt: process.env.PINATA_JWT! },
  vdaSdk: vdaSDK,
});

// 1. Mint a VDA for the document
const vda = await vdaSDK.mint({
  assetType: "deed",
  address: "123 Main St, Schertz, TX 78154",
  ownerWallet: ownerWallet,
  spoke: "real-estate",
});

// 2. Read document file
const file = fs.readFileSync("property-deed.pdf");

// 3. Upload document to IPFS
const uploadResult = await retrievalSDK.uploadDocument(file, {
  encrypt: true,                    // Encrypt for security
  vdaId: vda.id,                    // Associate with VDA
  metadata: {
    name: "property-deed.pdf",
    keyvalues: {
      propertyId: "prop-123",
      documentType: "deed",
    },
  },
  spoke: "real-estate",
});

// 4. Update VDA with IPFS CID
await vdaSDK.mint({
  ...vda.metadata,
  ipfsCid: uploadResult.cid,
});

// 5. Store encryption key securely
await storeEncryptionKey(uploadResult.cid, uploadResult.encryptionKey!);

console.log(`Document uploaded: ${uploadResult.cid}`);

Gated Document Retrieval

// Retrieve document with access verification
const document = await retrievalSDK.retrieveDocument(
  "QmTest123",
  userWallet,
  {
    requiredPermissions: ["view"],
    decrypt: true,
    watermark: true,                 // Watermark for access pass viewers
  }
);

// Save document
fs.writeFileSync("retrieved-document.pdf", document.content);

Express Gateway Proxy

import express from "express";
import { createGatewayRoute } from "@hauska-sdk/retrieval";
import { RetrievalSDK } from "@hauska-sdk/retrieval";

const app = express();
const retrievalSDK = new RetrievalSDK({ /* ... */ });

// Create gated document route
const documentRoute = createGatewayRoute(
  {
    accessVerificationService: retrievalSDK.getAccessVerification(),
    pinataClient: retrievalSDK.getPinataClient(),
    extractWallet: (req) => req.headers["x-wallet-address"] as string,
    getEncryptionKey: async (cid, wallet) => {
      return await getEncryptionKey(cid);
    },
  },
  {
    requiredPermissions: ["view"],
    autoDecrypt: true,
  }
);

app.use("/documents", documentRoute);

// Access document: GET /documents/:cid
// Header: x-wallet-address: 0x...

Access Logging

import { AccessLoggingService } from "@hauska-sdk/retrieval";
import { PostgreSQLStorageAdapter } from "@hauska-sdk/adapters-storage-postgres";

// Configure access logging
const retrievalSDK = new RetrievalSDK({
  pinata: { pinataJwt: process.env.PINATA_JWT! },
  vdaSdk: vdaSDK,
  accessLogging: {
    storageAdapter: new PostgreSQLAccessLogAdapter({ /* ... */ }),
    analyticsService: {
      trackAccess: async (log) => {
        // Send to analytics service
        await analytics.track("document_access", log);
      },
    },
  },
});

// Access logs are automatically recorded when using retrieveDocument()
const document = await retrievalSDK.retrieveDocument(cid, wallet);

// Query access logs
const logs = await retrievalSDK.getAccessLogs(cid);
console.log(`Document accessed ${logs.total} times`);

Watermarking

// Watermarking is automatic for access pass viewers
const document = await retrievalSDK.retrieveDocument(
  cid,
  wallet,
  {
    watermark: true, // Only watermarks for access pass viewers, not owners
  }
);

// Watermark includes:
// - Viewer wallet address
// - Timestamp
// - Access pass ID

Encryption

The Retrieval SDK supports AES-256-GCM encryption for secure document storage.

Encrypting Documents

// Upload with encryption
const result = await retrievalSDK.uploadDocument(file, {
  encrypt: true,                    // Auto-generates encryption key
});

// Or provide your own key
const key = generateEncryptionKey();
const result = await retrievalSDK.uploadDocument(file, {
  encrypt: true,
  encryptionKey: key,
});

// Store encryption key securely
await storeEncryptionKey(result.cid, result.encryptionKey || key);

Decrypting Documents

// Provide encryption key for decryption
const document = await retrievalSDK.fetchDocument(cid, {
  decrypt: true,
  encryptionKey: await getEncryptionKey(cid),
});

Encryption Key Management

// Store encryption key (example)
async function storeEncryptionKey(cid: string, key: Buffer): Promise<void> {
  // Store in secure key management service
  await keyManagementService.store(cid, key.toString("hex"));
}

// Retrieve encryption key
async function getEncryptionKey(cid: string): Promise<Buffer | undefined> {
  const keyHex = await keyManagementService.get(cid);
  return keyHex ? Buffer.from(keyHex, "hex") : undefined;
}

// Use with SDK
const retrievalSDK = new RetrievalSDK({
  pinata: { pinataJwt: process.env.PINATA_JWT! },
  vdaSdk: vdaSDK,
  getEncryptionKey: async (cid, wallet) => {
    // Only return key if wallet has access
    const hasAccess = await retrievalSDK.verifyAccess(cid, wallet);
    if (hasAccess.hasAccess) {
      return await getEncryptionKey(cid);
    }
    return undefined;
  },
});

Gateway Proxy

The Retrieval SDK provides Express/Fastify middleware for HTTP document access.

Basic Setup

import express from "express";
import { createGatewayRoute } from "@hauska-sdk/retrieval";

const app = express();
const retrievalSDK = new RetrievalSDK({ /* ... */ });

// Create gated document route
app.use(
  "/documents",
  ...createGatewayRoute(
    {
      accessVerificationService: retrievalSDK.getAccessVerification(),
      pinataClient: retrievalSDK.getPinataClient(),
      extractWallet: (req) => req.headers["x-wallet-address"] as string,
    },
    {
      requiredPermissions: ["view"],
    }
  )
);

// Access: GET /documents/:cid
// Header: x-wallet-address: 0x...

Custom Wallet Extraction

import { createGatewayRoute, defaultExtractWallet } from "@hauska-sdk/retrieval";

// Extract wallet from JWT token
function extractWalletFromJWT(req: Request): string | undefined {
  const token = req.headers.authorization?.replace("Bearer ", "");
  if (token) {
    const decoded = jwt.verify(token, process.env.JWT_SECRET!);
    return decoded.wallet;
  }
  return undefined;
}

app.use(
  "/documents",
  ...createGatewayRoute(
    {
      accessVerificationService: retrievalSDK.getAccessVerification(),
      pinataClient: retrievalSDK.getPinataClient(),
      extractWallet: extractWalletFromJWT,
    },
    { requiredPermissions: ["view"] }
  )
);

Operation-Specific Permissions

import { createOperationMiddleware } from "@hauska-sdk/retrieval";

// View endpoint (requires "view" permission)
app.get(
  "/documents/:cid/view",
  createOperationMiddleware(config, "view"),
  createDocumentRoute(config)
);

// Download endpoint (requires "download" permission)
app.get(
  "/documents/:cid/download",
  createOperationMiddleware(config, "download"),
  createDocumentRoute(config)
);

Troubleshooting

"Document not found" Error

Problem: CID doesn't exist on IPFS or Pinata gateway.

Solutions:

1. Verify the CID is correct
2. Check if document was pinned to Pinata
3. Verify Pinata gateway is accessible
4. Check network connectivity

// Verify CID exists
try {
  const metadata = await pinataClient.getFileMetadata(cid);
  if (!metadata) {
    console.error("Document not found in Pinata");
  }
} catch (error) {
  console.error("Error checking document:", error);
}

"Access denied" Error

Problem: Wallet doesn't have access to the document.

Solutions:

1. Verify wallet owns the VDA or has a valid access pass
2. Check access pass hasn't expired
3. Check access pass hasn't been revoked
4. Verify required permissions are granted

// Debug access verification
const result = await retrievalSDK.verifyAccess(cid, wallet);
console.log("Access result:", {
  hasAccess: result.hasAccess,
  accessType: result.accessType,
  reason: result.reason,
  vdaId: result.vdaId,
});

"Encryption key required" Error

Problem: Document is encrypted but no encryption key provided.

Solutions:

1. Provide encryption key in fetch options
2. Configure getEncryptionKey in SDK config
3. Verify encryption key is correct

// Provide encryption key
const document = await retrievalSDK.fetchDocument(cid, {
  decrypt: true,
  encryptionKey: await getEncryptionKey(cid),
});

"VDA ID not found for CID" Error

Problem: VDA ID cannot be resolved from CID.

Solutions:

1. Ensure VDA has ipfsCid metadata set
2. Provide getVDAIdFromCID function in config
3. Verify Pinata metadata includes vdaId keyvalue

// Custom VDA ID lookup
const retrievalSDK = new RetrievalSDK({
  pinata: { pinataJwt: process.env.PINATA_JWT! },
  vdaSdk: vdaSDK,
  getVDAIdFromCID: async (cid) => {
    // Query database for VDA with this CID
    const vda = await db.query("SELECT id FROM vdas WHERE ipfs_cid = $1", [cid]);
    return vda?.id || null;
  },
});

Network/Retry Errors

Problem: Network failures when fetching from IPFS gateway.

Solutions:

1. SDK automatically retries (3 attempts with exponential backoff)
2. Check network connectivity
3. Try different gateway
4. Verify Pinata service status

// Use custom gateway with retry
const retrievalSDK = new RetrievalSDK({
  pinata: {
    pinataJwt: process.env.PINATA_JWT!,
    pinataGateway: "https://ipfs.io", // Public gateway fallback
  },
  vdaSdk: vdaSDK,
});

Watermarking Not Working

Problem: Watermarks not appearing on PDFs.

Solutions:

1. Verify document is a PDF (watermarking only works for PDFs)
2. Ensure watermark: true is set in retrieveDocument options
3. Verify access is via access pass (owners don't get watermarked)
4. Check PDF is valid and not corrupted

// Verify PDF before watermarking
import { isPDF } from "@hauska-sdk/retrieval";

const document = await retrievalSDK.fetchDocument(cid);
if (isPDF(document.content)) {
  // Watermarking will work
  const watermarked = await retrievalSDK.retrieveDocument(cid, wallet, {
    watermark: true,
  });
}

Performance Tips

1. Use encryption keys efficiently:

   // Cache encryption keys
   const keyCache = new Map<string, Buffer>();
   const getEncryptionKey = async (cid: string) => {
     if (keyCache.has(cid)) {
       return keyCache.get(cid);
     }
     const key = await fetchEncryptionKey(cid);
     keyCache.set(cid, key);
     return key;
   };

2. Batch access log queries:

   // Query multiple CIDs at once
   const cids = ["Qm1", "Qm2", "Qm3"];
   const logs = await Promise.all(
     cids.map((cid) => retrievalSDK.getAccessLogs(cid))
   );

3. Use pagination for large result sets:

   const logs = await retrievalSDK.getAccessLogs(cid, {
     limit: 50,
     offset: 0,
   });

License

MIT