Steel Product Search & PDF Processing System

A comprehensive system for processing PDFs, extracting steel specifications, and searching product databases. The system combines AWS Textract for PDF processing, intelligent steel specification parsing, and advanced product search capabilities.

🚀 Quick Start

1. Install Dependencies

npm install

2. Start the Servers

You need to run three servers for full functionality:

# Terminal 1: AWS Textract Server (port 3000)
node server-aws.js

# Terminal 2: Steel Parser Server (port 3044)
node steel-parser-server.js

# Terminal 3: Product Search Server (port 3043)
node search-server.js

3. Launch the Application

Open http://localhost:3000/rectangle-text-extractor.html in your browser.

🏗️ System Architecture

Core Components

1. PDF Processing (rectangle-text-extractor.html)

   • Interactive PDF viewer with rectangle selection
   • Text extraction using AWS Textract
   • Steel specification parsing
   • Product search integration

2. Steel Parser (steelParser.js)

   • Parses steel specifications from text
   • Handles OCR corrections and normalization
   • Supports 20+ steel shapes and grades

3. Product Search (ProductSearcher.js)

   • Smart search with dimension matching
   • Regular search with fallback logic
   • Subset matching with tolerance

4. AWS Integration (server-aws.js)

   • PDF upload to S3
   • AWS Textract processing
   • Caching system for processed results

🔧 Steel Parser System

Script vs Server Architecture

The steel parser has both script and server implementations that share the same core logic:

Shared Code

• steelParser.js - Core parsing logic used by both script and server
• steel-parser-tests.js - Comprehensive test suite (274 tests)

Script Usage

# Direct parsing (returns JSON)
node steelParser.js "W12X26 A992 20FT"

Server Usage

# Start server on port 3044
node steel-parser-server.js

# API endpoint
POST http://localhost:3044/parse
Content-Type: application/json
{"text": "W12X26 A992 20FT"}

Steel Parser Features

• 20+ Steel Shapes: W-beams, HSS, Angles, Plates, Channels, etc.
• OCR Correction: Fixes common OCR errors (B→8, l→X, etc.)
• Grade Normalization: A-36 → A36, A572 GR50 → A572-50
• Dimension Parsing: Handles fractions, mixed fractions, decimals
• Length Conversion: Feet to inches, SRL/DRL support

🔍 Product Search System

Script vs Server Architecture

The product search has both script and server implementations:

Shared Code

• ProductSearcher.js - Core search logic used by both script and server
• test-product-searcher.js - Test suite with real product data
• product_search_data.json - Product database (50,000+ products)

Script Usage

# CLI search
node search-products.js "W" "A992" "12X26" "240"

Server Usage

# Start server on port 3043
node search-server.js

# API endpoint
POST http://localhost:3043/search
Content-Type: application/json
{"shape": "W", "grade": "A992", "dims": "12X26", "length": "240"}

Search Features

• Smart Search: Dimension subset matching with frequency counting
• Regular Search: Fallback with multiple matching strategies
• Tolerance Matching: 0.01 tolerance for numeric dimensions
• Grade Matching: Flexible grade comparison
• Length Matching: SRL/DRL support with tolerance

🧪 Testing

Steel Parser Tests

# Run comprehensive test suite (274 tests)
node steel-parser-tests.js

Product Search Tests

# Run product search tests with real data
node test-product-searcher.js

Standard Length Tests

# Test steel industry standard lengths
node test-standard-lengths.js

📁 Project Structure

Core Files

• rectangle-text-extractor.html - Main PDF processing application
• server-aws.js - AWS Textract server (port 3000)
• steel-parser-server.js - Steel parsing API (port 3044)
• search-server.js - Product search API (port 3043)

Shared Libraries

• steelParser.js - Steel specification parsing logic
• ProductSearcher.js - Product search algorithms
• product_search_data.json - Product database

CLI Tools

• search-products.js - Command-line product search
• temp-smart-search.js - Smart search testing utility
• export-products-to-json.js - Product data export utility

Test Suites

• steel-parser-tests.js - Steel parser test suite
• test-product-searcher.js - Product search test suite
• test-standard-lengths.js - Length validation tests
• Test-Data/ - Test data files
• test-results/ - Test output results

Data Directories

• processed-pdf/ - Cached PDF processing results
• parser-training/ - Steel parser training data

🔧 AWS Setup

Environment Variables

Create a .env file with your AWS credentials:

# AWS Configuration
AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=your_access_key_id
AWS_SECRET_ACCESS_KEY=your_secret_access_key
S3_BUCKET_NAME=your_s3_bucket_name

# Server Configuration
PORT=3000

AWS Services Required

• S3: PDF file storage
• Textract: Document text extraction
• IAM: Access permissions

🚀 Extending the System

Adding New Steel Shapes

1. Update steelParser.js with new shape patterns
2. Add test cases to steel-parser-tests.js
3. Update training data in parser-training/

Adding New Search Algorithms

1. Extend ProductSearcher.js with new methods
2. Add tests to test-product-searcher.js
3. Update API endpoints in search-server.js

Adding New Product Data

1. Use export-products-to-json.js to export from database
2. Update product_search_data.json
3. Restart search server to reload data

📊 Performance Notes

• Caching: Processed PDFs are cached in processed-pdf/
• Memory: Product database loads ~50,000 products into memory
• Concurrency: All servers support concurrent requests
• Optimization: Smart search uses subset matching for efficiency

🐛 Troubleshooting

Common Issues

• Port conflicts: Ensure ports 3000, 3043, 3044 are available
• AWS credentials: Verify .env file and IAM permissions
• Memory issues: Large PDFs may require increased Node.js memory
• CORS errors: Serve HTML files through a web server, not file://

Debug Tools

• Use browser developer tools for frontend debugging
• Check server console logs for API errors
• Use temp-smart-search.js for search algorithm debugging
• Review test results in test-results/ directory