Upstrima Document Processing Module

A comprehensive document processing pipeline for the Upstrima platform that enables semantic search and RAG (Retrieval-Augmented Generation) capabilities. The module supports both general documents (PDF, DOCX, TXT) and specialized engineering file formats (LAS).

Features

• Document Processing Pipeline

  • Chunking and embedding generation
  • Vector store management with FAISS
  • Support for multiple file formats
  • Metadata extraction and storage

• Semantic Search

  • Cross-document search capabilities
  • Relevance scoring
  • Configurable result limits
  • Context-aware results

• RAG Support

  • Context generation for LLMs
  • Token limit management
  • Document attribution

• Engineering File Support

  • LAS (Log ASCII Standard) file processing
  • Well log visualization
  • Curve data analysis
  • Zone identification

Installation

1. Install required packages:

pip install -r requirements.txt

2. Set up environment variables (or use defaults from config.py):

export DOC_PROC_DATABASE_URL="postgresql://user:password@localhost/documents_db"
export DOC_PROC_LOG_LEVEL="INFO"
export DOC_PROC_MAX_FILE_SIZE=52428800  # 50MB

3. Initialize the module:

from fastapi import FastAPI
from api.modules.document_processing import init_module

app = FastAPI()
init_module(app)

Usage

Process a Document

import requests

# Process a general document
with open('document.pdf', 'rb') as f:
    response = requests.post(
        'http://localhost:8000/documents/process/',
        files={'file': f}
    )
document_id = response.json()['document_id']

# Process a LAS file
with open('well_log.las', 'rb') as f:
    response = requests.post(
        'http://localhost:8000/documents/process/',
        files={'file': f}
    )
well_log_id = response.json()['document_id']

Search Documents

# Semantic search across documents
response = requests.get(
    'http://localhost:8000/documents/search/',
    params={
        'query': 'your search query',
        'document_ids': ['doc1_id', 'doc2_id'],
        'top_k': 5
    }
)

Generate RAG Context

# Generate context for LLM
response = requests.get(
    'http://localhost:8000/documents/rag-context/',
    params={
        'query': 'your question',
        'document_ids': ['doc1_id', 'doc2_id'],
        'max_tokens': 2000
    }
)

Well Log Operations

# Get well log details
response = requests.get(
    f'http://localhost:8000/documents/well-logs/{well_log_id}'
)

# Identify zones
response = requests.post(
    f'http://localhost:8000/documents/well-logs/{well_log_id}/identify-zones',
    json={
        'curve_name': 'GR',
        'threshold': 75.0
    }
)

API Endpoints

Document Processing

• POST /documents/process/
  • Process a document file
  • Supports PDF, DOCX, TXT, LAS files
  • Returns document metadata and processing status

Search and RAG

• GET /documents/search/

  • Perform semantic search across documents
  • Parameters: query, document_ids, top_k

• GET /documents/rag-context/

  • Generate context for RAG applications
  • Parameters: query, document_ids, max_tokens

Well Log Operations

• GET /documents/well-logs/{well_log_id}

  • Get well log details including curves and sections

• POST /documents/well-logs/{well_log_id}/identify-zones

  • Identify zones based on curve values
  • Parameters: curve_name, threshold

Configuration

Key configuration options in config.py:

# Document processing
CHUNK_SIZE = 1000
CHUNK_OVERLAP = 200
MAX_FILE_SIZE = 50 * 1024 * 1024  # 50MB

# Embedding model
EMBEDDING_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
EMBEDDING_DEVICE = "cpu"  # or "cuda" for GPU

# Search settings
DEFAULT_TOP_K = 5
MIN_SIMILARITY_SCORE = 0.5

# RAG settings
MAX_CONTEXT_TOKENS = 2000

Database Models

• Document: Base model for all document types
• DocumentChunk: Stores document chunks and embeddings
• WellLog: Specialized model for well log data
• LogCurve: Stores well log curve data
• LogSection: Represents well log sections/zones

Development

Running Tests

pytest api/modules/document_processing/tests/

Code Style

The project follows PEP 8 guidelines. Format code using:

black api/modules/document_processing/

Error Handling

The module includes comprehensive error handling:

• File validation errors
• Processing failures
• Database errors
• Vector store management issues

Errors are logged and appropriate HTTP status codes are returned.

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Run tests
5. Submit a pull request

License

Copyright © 2024 Gain.Energy. All rights reserved.

Support

For support, contact [email protected]