@lineai/municipal-intel

Access municipal planning applications, building permits, and construction activity data from major US cities.

Overview

@lineai/municipal-intel provides a unified interface to access local government data from major US cities. Built with TypeScript for maximum type safety, it features a hybrid architecture with built-in sources and runtime extensibility.

Features

AI-First Design

• Discovery API: Comprehensive discovery methods for AI assistants
• Strong Typing: TypeScript autocomplete for municipality IDs and search parameters
• Search Capabilities: Dynamic capability detection per municipality
• Self-Documenting: Rich metadata about available datasets and fields

Data Access

• Unified API: Single interface for multiple municipal data sources
• Real-time Data: Access live permit and planning application data
• Clean Project URLs: Each project includes a shareable URL for direct access
• URL-based Retrieval: Get projects directly by URL for bookmarking and sharing
• Transparent Adjustments: Best-effort queries with clear reporting of any modifications
• Schema Validation: Zod schemas ensure data consistency and catch API changes

Developer Experience

• TypeScript Native: Built-in sources with full type safety
• Runtime Extensible: Add new cities dynamically without package updates
• Universal Socrata Token: Single token works across all Socrata portals
• Rate Limiting: Built-in rate limiting and retry logic

Supported Cities

Built-in Sources (Ready to Use)

• California: San Francisco, Los Angeles
• New York: New York City (all 5 boroughs)

Runtime Registration (Add Your Own)

• Any Socrata Portal: Add any city with Socrata-based open data
• Custom APIs: Register custom municipal APIs
• Extensible: No package updates needed for new cities

See docs/ADD_NEW_SOURCE.md for adding new sources.

Installation

npm install @lineai/municipal-intel

Quick Start

import { createMunicipalIntel } from '@lineai/municipal-intel';

// Create client instance
const municipal = createMunicipalIntel({
  debug: true
});

// Set universal Socrata token (works for all cities)
municipal.setSocrataToken(process.env.SOCRATA_TOKEN);

// 🤖 AI Discovery: Find what's available
const municipalities = municipal.getAvailableMunicipalities();
console.log('Available municipalities:', municipalities);

// Check search capabilities for San Francisco
const capabilities = municipal.getSearchCapabilities('sf');
console.log('SF supports:', capabilities.supportedFilters);

// Search for construction permits in San Francisco
const results = await municipal.search({
  municipalityId: 'sf',        // Type-safe municipality ID
  keywords: ['renovation'],
  minValue: 100000,
  limit: 10
});

console.log(`Found ${results.total} projects`);
results.projects.forEach(project => {
  console.log(project.description); // Natural language description
  console.log('Project URL:', project.url); // Clean, shareable URL
  console.log('Raw data:', project.rawData); // Full original data
});

// 🔗 Get project by URL (great for bookmarking/sharing)
const projectUrl = 'https://municipal-intel.lineai.com/projects/sf/buildingPermits/2024-001';
const project = await municipal.getByUrl(projectUrl);
if (project) {
  console.log('Retrieved project:', project.description);
}

// Check for any query adjustments
if (results.adjustments.length > 0) {
  console.log('Query adjustments:', results.adjustments);
}

Authentication

Universal Socrata Token (Recommended)

Get a single free app token that works across all Socrata portals:

1. Visit any Socrata portal (e.g., data.sfgov.org)
2. Sign up → Developer Settings → Create App Token
3. Use this same token for all cities!

const municipal = createMunicipalIntel();
municipal.setSocrataToken(process.env.SOCRATA_TOKEN);

Rate limits:

• With token: 1000 requests/hour per portal
• Without token: Shared pool, very limited

API Reference

🤖 AI Discovery Methods

// Get all available municipalities with their datasets
const municipalities = municipal.getAvailableMunicipalities();
// Returns: [{ id: 'sf', name: 'San Francisco', state: 'CA', datasets: [...] }]

// Check what search capabilities a municipality supports
const capabilities = municipal.getSearchCapabilities('sf');
// Returns: { supportedFilters: ['minValue', 'submitDate', ...], limitations: [...] }

// Get detailed field schema for a dataset
const schema = municipal.getDatasetSchema('sf', 'buildingPermits');
// Returns: [{ name: 'permit_number', type: 'string', searchable: true }]

Search Projects

const results = await municipal.search({
  // Location filters  
  municipalityId: 'sf',             // Type-safe municipality ('sf' | 'nyc' | 'la')
  addresses: ['Market Street'],     // By address
  zipCodes: ['94102'],             // By ZIP code
  
  // Project filters
  types: ['permit', 'planning'],    // Project types
  statuses: ['approved', 'issued'], // Current status
  keywords: ['renovation'],         // Keywords
  
  // Date filters
  submitDateFrom: new Date('2024-01-01'),
  submitDateTo: new Date('2024-12-31'),
  
  // Value filters (if supported by municipality)
  minValue: 50000,
  maxValue: 1000000,
  
  // Pagination
  limit: 50,
  offset: 0,
  
  // Sorting
  sortBy: 'submitDate',
  sortOrder: 'desc'
});

// Check for query adjustments
console.log('Adjustments:', results.adjustments);

Get Specific Project

const project = await municipal.getProject('sf', 'sf-202401234567');

List Available Sources

// All sources
const allSources = municipal.getSources();

// Filter by criteria
const apiSources = municipal.getSources({ 
  type: 'api', 
  priority: 'high' 
});

// By state
const caSources = municipal.getSources({ state: 'ca' });

Health Checks

const health = await municipal.healthCheck('sf');
console.log(`Status: ${health.status}, Latency: ${health.latency}ms`);

Runtime Source Registration

Add new cities without updating the package:

// Register a new Florida city
municipal.registerSource({
  id: 'miami',
  name: 'Miami-Dade County',
  state: 'FL',
  type: 'api',
  api: {
    type: 'socrata',
    baseUrl: 'https://opendata.miamidade.gov',
    datasets: {
      buildingPermits: {
        endpoint: '/resource/8wbx-tpnc.json',
        name: 'Building Permits',
        fields: ['permit_number', 'status', 'issue_date', 'address', 'valuation', 'description'],
        fieldMappings: {
          id: 'permit_number',
          status: 'status',
          submitDate: 'issue_date',
          address: 'address',
          value: 'valuation',
          description: 'description'
        }
      }
    }
  },
  priority: 'high'
});

// Now you can search Miami data
const miamiResults = await municipal.search({ municipalityId: 'miami' });

// Remove runtime source
municipal.unregisterSource('miami');

Query Adjustments & Transparency

The library implements a best-effort approach with full transparency when handling different data sources. When a requested filter cannot be applied to a particular source, the library will continue the search and report what adjustments were made.

Adjustments Array

All search responses include an adjustments array that reports any modifications made to your query:

interface MunicipalSearchResponse {
  projects: MunicipalProject[];
  total: number;
  page: number;
  pageSize: number;
  hasMore: boolean;
  adjustments: string[];  // Query modifications made during search
}

Common Adjustments

Value Filter Skipping: Some sources don't have cost/value fields

const results = await municipal.search({
  sources: ['sf', 'nyc'],
  minValue: 100000
});

// Results:
// {
//   projects: [...],
//   total: 1234,
//   adjustments: [
//     "NYC: Skipped minValue filter - no value field available in dataset"
//   ]
// }

Field Type Conversions: Socrata sources store numbers as text, requiring automatic casting

// The library automatically converts text to numbers for comparisons
// SF: estimated_cost field contains "500000" (string) 
// Query: WHERE estimated_cost::number >= 100000
// No adjustment needed - handled transparently

Adjustment Principles

• Silent Success: No adjustments reported when everything works normally
• Transparent Failures: Only report when something unexpected happens
• Continue on Errors: Skip problematic filters rather than fail entire search
• AI-Friendly: Clean responses for automated processing

Best Practices

For AI Assistants:

const results = await municipal.search(params);

if (results.adjustments.length > 0) {
  // Inform user about limitations
  console.log('Note: ' + results.adjustments.join('; '));
}

For Applications:

// Always check adjustments for user feedback
if (results.adjustments.length > 0) {
  showWarning('Some filters were not available for all sources');
}

Data Types

MunicipalProject (AI-First Design)

interface MunicipalProject {
  // Core fields optimized for AI consumption
  id: string;              // Unique identifier (e.g., "sf-2024-001234")
  source: string;          // Source municipality ID (e.g., "sf", "nyc")
  description: string;     // Natural language description with full context
  rawData: any;           // Complete original API response
  lastUpdated: Date;      // When data was last fetched
}

Key Design Principles:

• AI-Optimized: Natural language descriptions instead of failed field normalization
• Complete Context: Descriptions include full geographic and municipal context
• Raw Data Preserved: Full original API response available for detailed analysis
• Lightweight: Reduced payload size by ~70% vs. previous complex schema

Examples

Monitor New Permits

// Get recent permits from San Francisco
const recent = await municipal.search({
  municipalityId: 'sf',
  types: ['permit'],
  submitDateFrom: new Date(Date.now() - 7 * 24 * 60 * 60 * 1000), // Last 7 days
  sortBy: 'submitDate',
  sortOrder: 'desc'
});

// AI-friendly output with natural language descriptions
recent.projects.forEach(project => {
  console.log(project.description);
  // Example: "Residential alteration permit for kitchen renovation at 123 Main St, San Francisco, CA 94102. Filed 2024-01-15, issued 2024-02-01. Estimated cost: $75,000."
});

// Monitor multiple municipalities
const municipalities = municipal.getAvailableMunicipalities();
for (const municipality of municipalities) {
  const permits = await municipal.search({
    municipalityId: municipality.id,
    limit: 5
  });
  console.log(`\n${municipality.name}: ${permits.total} recent projects`);
  permits.projects.forEach(p => console.log(`  • ${p.description}`));
}

Construction Projects by Value

// High-value construction projects in SF
const bigProjects = await municipal.search({
  municipalityId: 'sf',
  minValue: 1000000,
  sortBy: 'value',
  sortOrder: 'desc'
});

// Natural language descriptions include value context automatically
bigProjects.projects.forEach(project => {
  console.log(project.description);
  // Example: "New construction permit for 45-unit residential building at 456 Market St, San Francisco, CA 94105. Filed 2024-01-10, approved 2024-03-15. Estimated cost: $12,500,000. Developer: XYZ Development Corp."
  
  // Access raw data for detailed analysis
  const rawValue = project.rawData.estimated_cost;
  const submitter = project.rawData.applicant_name;
});

// Check capabilities
const municipalities = municipal.getAvailableMunicipalities();
for (const municipality of municipalities) {
  const capabilities = municipal.getSearchCapabilities(municipality.id);
  if (capabilities.supportedFilters.includes('minValue')) {
    console.log(`${municipality.name} supports value filtering`);
  }
}

Geographic Search

// Projects in specific SF area
const localProjects = await municipal.search({
  municipalityId: 'sf',
  addresses: ['Market Street', 'Mission Street'],
  zipCodes: ['94102', '94103'],
  limit: 20
});

// Descriptions include full geographic context
localProjects.projects.forEach(project => {
  console.log(project.description);
  // Example: "Commercial tenant improvement at 789 Mission St, San Francisco, CA 94103. Restaurant buildout permit filed 2024-02-20, under review. Estimated cost: $150,000."
});

// Cross-municipality geographic search
const addressSearch = async (streetName: string) => {
  const municipalities = municipal.getAvailableMunicipalities();
  for (const municipality of municipalities) {
    const results = await municipal.search({
      municipalityId: municipality.id,
      addresses: [streetName],
      limit: 3
    });
    console.log(`\n${municipality.name}: ${results.total} projects on ${streetName}`);
    results.projects.forEach(p => {
      console.log(`  • ${p.description}`);
    });
  }
};

await addressSearch('Main Street');

Environment Variables

# .env
SOCRATA_TOKEN=your-universal-socrata-app-token

Note: A single Socrata token works across all Socrata portals (San Francisco, NYC, Miami, etc.)

Error Handling

import { MunicipalDataError, RateLimitError } from '@lineai/municipal-intel';

try {
  const results = await municipal.search({ sources: ['sf'] });
} catch (error) {
  if (error instanceof RateLimitError) {
    console.log('Rate limited, retry after:', error.details?.resetTime);
  } else if (error instanceof MunicipalDataError) {
    console.log('API error:', error.message, 'Source:', error.source);
  }
}

Troubleshooting

Value Filtering Issues

Problem: Getting 400 errors when using minValue parameter

// This might fail with "Invalid SoQL query" error
const results = await municipal.search({
  municipalityId: 'sf',
  minValue: 100000  // 400 error
});

Solution: The library automatically handles this by converting text fields to numbers. If you see this error, ensure you're using the latest version.

Explanation: Socrata APIs store numeric values as text strings (e.g., "500000" instead of 500000). The library automatically applies ::number casting for all numeric comparisons.

Missing Value Fields

Problem: Some sources don't return value information

const results = await municipal.search({
  municipalityId: 'nyc',
  minValue: 100000
});
// NYC projects show value: null

Solution: Check the adjustments array for information about skipped filters:

if (results.adjustments.length > 0) {
  console.log('Filters adjusted:', results.adjustments);
  // Output: ["NYC: Skipped minValue filter - no value field available in dataset"]
}

Rate Limiting

Problem: Getting rate limited frequently Solution: Use a Socrata app token:

municipal.setSocrataToken(process.env.SOCRATA_TOKEN);
// Increases limit from ~100/hour to 1000/hour per portal

Field Mappings

Problem: Custom sources not returning expected data Solution: Verify field mappings match actual API response:

1. Check the API samples in /api-samples/ directory
2. Ensure your fieldMappings use correct field names at the dataset level
3. Use the library's field discovery tools

// Check available fields for your source
const source = municipal.getSource('your-city');
console.log('Available fields:', source.api?.datasets?.buildingPermits?.fields);

// Check what fields are mapped
const fieldMappings = source.api?.datasets?.buildingPermits?.fieldMappings;
console.log('Field mappings:', fieldMappings);

// Use schema discovery to validate
const schema = municipal.getDatasetSchema('your-city', 'buildingPermits');
console.log('Schema:', schema);

Development

# Install dependencies
yarn install

# Build
yarn build

# Test
yarn test:unit

# Lint and fix
yarn fix

Contributing

1. Fork the repository
2. Create a feature branch
3. Add tests for new functionality
4. Ensure all tests pass
5. Submit a pull request

Documentation

• AI-First Design Philosophy - Complete guide to the natural language description approach
• Adding New Sources - How to add new cities and data sources
• API Guide - Technical implementation details

License

MIT

Related Packages

• @lineai/gov-deals - Federal government contracts and opportunities