Lit Compiler Server

A production-ready TypeScript/Lit web component compiler service with advanced validation and scalability features. Used by the Halix Platform to compile and validate custom elements.

Features

Core Compilation

• ✅ TypeScript to JavaScript compilation using esbuild
• ✅ Import rewriting to esm.sh CDN URLs
• ✅ Source maps for debugging
• ✅ Tree shaking and optimization

Validation (Optional)

• ✅ Import validation: Verify esm.sh package availability
• ✅ Type validation: Static TypeScript type checking with npm package resolution
• ✅ Runtime validation: Execute code in headless browser to catch runtime errors

Performance & Scalability

• ✅ Dependency caching: LRU cache for npm packages (saves 5-15s per compilation)
• ✅ Result caching: Cache runtime validation results
• ✅ Concurrent processing: Controlled concurrency for resource-intensive operations
• ✅ Queue system ready: Documentation for Bull/Redis-based job queues

Developer Experience

• ✅ Comprehensive unit tests: 94+ tests with Vitest
• ✅ Integration tests: Full API endpoint testing
• ✅ CI/CD: Bitbucket Pipelines with Docker deployment
• ✅ Monitoring: Health checks and detailed logging
• ✅ Documentation: Extensive guides for deployment and scaling

Quick Start

Prerequisites

• Node.js 20+
• npm
• Playwright (for runtime validation)

Installation

cd lit-compiler-server
npm install
npx playwright install chromium

Development

# Run tests
npm test

# Build
npm run build

# Start server
npm start

Server runs at http://localhost:3000

Basic Usage

curl -X POST http://localhost:3000/compile \
  -H "Content-Type: application/json" \
  -d '{
    "sourceCode": "import { LitElement, html } from \"lit\"; export class MyElement extends LitElement { render() { return html`<div>Hello</div>`; } }"
  }'

With Validation

curl -X POST "http://localhost:3000/compile?validateTypes=true&validateRuntime=true" \
  -H "Content-Type: application/json" \
  -d '{
    "sourceCode": "..."
  }'

API Reference

POST /compile

Synchronous compilation endpoint

Query Parameters:

• validateTypes (boolean): Enable TypeScript type validation
• validateRuntime (boolean): Enable runtime validation in headless browser
• runtimeTimeout (number): Timeout for runtime validation in ms (default: 5000)

Request Body:

{
  "sourceCode": "string"
}

Response (Success):

{
  "success": true,
  "compiledCode": "string",
  "sourceMap": "string",
  "warnings": ["string"],
  "stats": {
    "compilationTime": 1234,
    "typeValidationTime": 5678,
    "runtimeValidationTime": 3456
  }
}

Response (Error):

{
  "success": false,
  "error": "string",
  "details": ["string"]
}

Architecture

┌─────────────────┐
│   Client        │
│  (Angular)      │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Express API    │
│  (index.ts)     │
└────────┬────────┘
         │
         ▼
    ┌────┴────┐
    │         │
    ▼         ▼
┌─────┐   ┌─────────┐
│esbuild   │Validation│
│Compiler  │  Layer  │
└─────┘   └────┬────┘
               │
     ┌─────────┼─────────┐
     │         │         │
     ▼         ▼         ▼
┌────────┐ ┌──────┐ ┌──────────┐
│ Import │ │ Type │ │ Runtime  │
│Validator│ │Check │ │Validator │
└────────┘ └──┬───┘ └────┬─────┘
              │          │
              ▼          ▼
         ┌────────┐  ┌──────────┐
         │npm+tsc │  │Playwright│
         │ Cache  │  │ Browser  │
         └────────┘  └──────────┘

Validation Features

Import Validation

• Checks esm.sh package availability
• Validates semver version formats
• HTTP HEAD requests with timeout protection

Type Validation

• Downloads npm packages and type definitions
• Runs TypeScript compiler (tsc --noEmit)
• Filters errors to show only user code issues
• Caches dependencies by hash (LRU eviction)
• Configurable error suppression (e.g., TS2564)

Runtime Validation

• Executes compiled code in Chromium headless browser
• Catches module loading errors
• Detects console errors and uncaught exceptions
• Validates against browser DOM API
• Result caching for identical code

Configuration

TypeScript Compiler Options

Located in src/validate-types.ts:

{
  target: 'ES2022',
  module: 'ESNext',
  strict: true,
  noImplicitAny: false,  // Allows flexibility with 'any' types
  experimentalDecorators: true,
  // ... more options
}

Suppressed Errors

// src/validate-types.ts
const SUPPRESSED_ERROR_CODES = new Set([
  'TS2564', // Property has no initializer
  'TS2307', // Cannot find module (for private packages)
]);

Cache Limits

// src/validate-types.ts
const MAX_CACHE_ENTRIES = 50;
const MAX_CACHE_SIZE_GB = 10;

Documentation

| Document | Description | |----------|-------------| | TYPE-VALIDATION.md | Deep dive into type validation system | | INTERNAL-QUEUE-SETUP.md | Phase 1: Internal resource queues (30-45 min, no API changes) | | SCALABILITY.md | Comprehensive scaling guide with queue architecture options | | QUEUE-QUICKSTART.md | Phase 2: Full async queue with Bull/Redis (4-6 hours) | | CLIENT-INTEGRATION.md | Angular client examples for async API (Phase 2) |

Scaling to Production

Current Capacity

• Single instance: ~50-100 concurrent requests
• Bottlenecks: npm installs, Playwright browsers, long request times

Scaling Options

Option 1: Internal Resource Queue (Recommended First Step)

For low-to-medium traffic where "compilation is fast enough to wait":

What it does:

• Internal queues control resource usage (no API changes needed)
• Protects against npm rate limiting (max 3 concurrent installs)
• Prevents memory issues with Playwright (max 2 concurrent browsers)
• Fair queuing (FIFO)

Setup time: 30-45 minutes
Cost: $0 (no new infrastructure)
See: INTERNAL-QUEUE-SETUP.md

Option 2: Full Async Queue (For High Traffic)

For production multi-user environments (100+ users), implement full async queue system:

┌─────────┐    ┌──────────┐    ┌───────┐    ┌────────┐
│ Client  │───▶│   API    │───▶│ Redis │───▶│Workers │
│ Angular │◀───│ Express  │◀───│ Queue │◀───│  (x5)  │
└─────────┘    └──────────┘    └───────┘    └────────┘

Benefits:

• Non-blocking API responses (instant)
• Horizontal scaling (10+ workers)
• Resource protection (controlled concurrency)
• Job priorities and rate limiting
• npm rate limit protection

Setup Time: 4-6 hours (see QUEUE-QUICKSTART.md)

Cost: ~$65-300/month for managed Redis + workers

Quick Wins (No Queue)

1. Increase concurrency limits in environment variables
2. Add more memory to the server (Playwright uses ~200MB per instance)
3. Optimize cache eviction (increase MAX_CACHE_ENTRIES)
4. Load balancing across multiple server instances (shared cache directory)

Testing

Unit Tests

npm test                    # Run all tests
npm run test:watch         # Watch mode
npm run test:ui            # Vitest UI
npm run test:coverage      # Coverage report

Test Structure

tests/
  unit/
    validate-source-code.test.ts
    validate-version-format.test.ts
    validate-esm-imports.test.ts
    validate-runtime.test.ts
    validate-types-filtering.test.ts
    import-rewriting.test.ts
  integration/
    compile-endpoint.test.ts
    type-validation.test.ts

Current Coverage: 94 tests passing

Deployment

Docker

# Build image
docker build -t lit-compiler-server .

# Run container
docker run -p 3000:3000 \
  -e NODE_ENV=production \
  lit-compiler-server

Docker Compose

docker-compose up

AWS ECS

See DEPLOYMENT.md for full AWS setup guide.

Bitbucket Pipelines

Configured for automated testing and deployment:

• ✅ Build on every commit
• ✅ Run tests with detailed reporting
• ✅ Build Docker image
• ✅ Deploy to ECS (manual trigger)

See CICD-SETUP.md for configuration.

Monitoring

Health Check

curl http://localhost:3000/health

Logs

# Development
npm start

# Production (PM2)
pm2 logs lit-compiler-server

# Docker
docker logs <container-id>

Metrics to Monitor

• Request rate and response times
• Cache hit/miss ratio
• Queue length (if using queue system)
• Memory usage (especially with Playwright)
• Disk usage (.package-cache directory)
• npm install failure rate

Troubleshooting

Common Issues

Problem: Type validation fails with "Cannot find module"

• Solution: Check if package exists on npm, verify version string

Problem: Runtime validation timeout

• Solution: Increase runtimeTimeout parameter or check for infinite loops

Problem: High memory usage

• Solution: Reduce Playwright concurrency, implement queue system

Problem: npm rate limiting

• Solution: Reduce npm install concurrency, implement queue with semaphores

Problem: Cache growing too large

• Solution: Adjust MAX_CACHE_ENTRIES or MAX_CACHE_SIZE_GB

Debug Mode

Set environment variable for verbose logging:

DEBUG=* npm start

Development

Project Structure

src/
  index.ts                    # Main Express server
  validate-esm-imports.ts     # Import validation
  validate-types.ts           # Type checking with npm
  validate-runtime.ts         # Playwright validation
  validate-source-code.ts     # Security checks
tests/
  unit/                       # Unit tests
  integration/                # Integration tests
docs/                         # Documentation
dist/                         # Compiled output
.package-cache/              # npm package cache

Adding New Features

1. Create implementation in src/
2. Add unit tests in tests/unit/
3. Add integration tests in tests/integration/
4. Update documentation
5. Run full test suite

Code Style

• TypeScript strict mode
• ESLint + Prettier (TODO: add configs)
• Explicit return types
• Comprehensive error handling

Contributing

1. Fork the repository
2. Create a feature branch
3. Write tests for new features
4. Ensure all tests pass: npm test
5. Submit a pull request

License

[Your License Here]

Support

For issues and questions:

• GitHub Issues: [Your Repo URL]
• Documentation: See /docs folder
• Email: [Your Email]

Changelog

v1.0.0 (Current)

• ✅ Core compilation with esbuild
• ✅ Import rewriting to esm.sh
• ✅ Type validation with npm packages
• ✅ Runtime validation with Playwright
• ✅ Dependency caching with LRU eviction
• ✅ Comprehensive test suite
• ✅ CI/CD with Bitbucket Pipelines
• ✅ Docker deployment support
• ✅ Scalability documentation

Roadmap

• [ ] Implement Bull/Redis queue system
• [ ] Add webhook support for job completion
• [ ] Implement job batching
• [ ] Add Prometheus metrics
• [ ] GraphQL API option
• [ ] WebSocket support for real-time progress
• [ ] User authentication and rate limiting
• [ ] Result persistence to S3
• [ ] Multi-region deployment guide