OCR/openspec/changes/dual-track-document-processing/design.md

Technical Design: Dual-track Document Processing

Context

Background

The current OCR tool processes all documents through PaddleOCR, even when dealing with editable PDFs that contain extractable text. This causes:

• Unnecessary processing overhead
• Potential quality degradation from re-OCRing already digital text
• Loss of precise formatting information
• Inefficient GPU usage on documents that don't need OCR

Constraints

• RTX 4060 8GB GPU memory limitation
• Need to maintain backward compatibility with existing API
• Must support future translation features
• Should handle mixed documents (partially scanned, partially digital)

Stakeholders

• API consumers expecting consistent JSON/PDF output
• Translation system requiring structure preservation
• Performance-sensitive deployments

Goals / Non-Goals

Goals

• Intelligently route documents to appropriate processing track
• Preserve document structure for translation
• Optimize GPU usage by avoiding unnecessary OCR
• Maintain unified output format across tracks
• Reduce processing time for editable PDFs by 70%+

Non-Goals

• Implementing the actual translation engine (future phase)
• Supporting video or audio transcription
• Real-time collaborative editing
• OCR model training or fine-tuning

Decisions

Decision 1: Dual-track Architecture

What: Implement two separate processing pipelines - OCR track and Direct extraction track

Why:

• Editable PDFs don't need OCR, can be processed 10-100x faster
• Direct extraction preserves exact formatting and fonts
• OCR track remains optimal for scanned documents

Alternatives considered:

1. Single enhanced OCR pipeline: Would still waste resources on editable PDFs
2. Hybrid approach per page: Too complex, most documents are uniformly editable or scanned
3. Multiple specialized pipelines: Over-engineering for current requirements

Decision 2: UnifiedDocument Model

What: Create a standardized intermediate representation for both tracks

Why:

• Provides consistent API interface regardless of processing track
• Simplifies downstream processing (PDF generation, translation)
• Enables track switching without breaking changes

Structure:

@dataclass
class UnifiedDocument:
    document_id: str
    metadata: DocumentMetadata
    pages: List[Page]
    processing_track: Literal["ocr", "direct"]

@dataclass
class Page:
    page_number: int
    elements: List[DocumentElement]
    dimensions: Dimensions

@dataclass
class DocumentElement:
    element_id: str
    type: ElementType  # text, table, image, header, etc.
    content: Union[str, Dict, bytes]
    bbox: BoundingBox
    style: Optional[StyleInfo]
    confidence: Optional[float]  # Only for OCR track

Decision 3: PyMuPDF for Direct Extraction

What: Use PyMuPDF (fitz) library for editable PDF processing

Why:

• Mature, well-maintained library
• Excellent coordinate preservation
• Fast C++ backend
• Supports text, tables, and image extraction with positions

Alternatives considered:

1. pdfplumber: Good but slower, less precise coordinates
2. PyPDF2: Limited layout information
3. PDFMiner: Complex API, slower performance

Decision 4: Processing Track Auto-detection

What: Automatically determine optimal track based on document analysis

Detection logic:

def detect_track(file_path: Path) -> str:
    file_type = magic.from_file(file_path, mime=True)

    if file_type.startswith('image/'):
        return "ocr"

    if file_type == 'application/pdf':
        # Check if PDF has extractable text
        doc = fitz.open(file_path)
        for page in doc[:3]:  # Sample first 3 pages
            text = page.get_text()
            if len(text.strip()) < 100:  # Minimal text
                return "ocr"
        return "direct"

    if file_type in OFFICE_MIMES:
        # Convert Office to PDF first, then analyze
        pdf_path = convert_office_to_pdf(file_path)
        return detect_track(pdf_path)  # Recursive call on PDF

    return "ocr"  # Default fallback

Office Document Processing Strategy:

1. Convert Office files (Word, PPT, Excel) to PDF using LibreOffice
2. Analyze the resulting PDF for text extractability
3. Route based on PDF analysis:
   • Text-based PDF → Direct track (faster, more accurate)
   • Image-based PDF → OCR track (for scanned content in Office docs)

This approach ensures:

• Consistent processing pipeline (all documents become PDF first)
• Optimal routing based on actual content
• Significant performance improvement for editable Office documents
• Better layout preservation (no OCR errors on text content)

Decision 5: GPU Memory Management

What: Implement dynamic batch sizing and model caching for RTX 4060 8GB

Why:

• Prevents OOM errors
• Maximizes throughput
• Enables concurrent request handling

Strategy:

# Adaptive batch sizing based on available memory
batch_size = calculate_batch_size(
    available_memory=get_gpu_memory(),
    image_size=image.shape,
    model_size=MODEL_MEMORY_REQUIREMENTS
)

# Model caching to avoid reload overhead
@lru_cache(maxsize=2)
def get_model(model_type: str):
    return load_model(model_type)

Decision 6: Backward Compatibility

What: Maintain existing API while adding new capabilities

How:

• Existing endpoints continue working unchanged
• New processing_track parameter is optional
• Output format compatible with current consumers
• Gradual migration path for clients

Risks / Trade-offs

Risk 1: Mixed Content Documents

Risk: Documents with both scanned and digital pages Mitigation:

• Page-level track detection as fallback
• Confidence scoring to identify uncertain pages
• Manual override option via API

Risk 2: Direct Extraction Quality

Risk: Some PDFs have poor internal structure Mitigation:

• Fallback to OCR track if extraction quality is low
• Quality metrics: text density, structure coherence
• User-reportable quality issues

Risk 3: Memory Pressure

Risk: RTX 4060 8GB limitation with concurrent requests Mitigation:

• Request queuing system
• Dynamic batch adjustment
• CPU fallback for overflow

Trade-off 1: Processing Time vs Accuracy

• Direct extraction: Fast but depends on PDF quality
• OCR: Slower but consistent quality
• Decision: Prioritize speed for editable PDFs, accuracy for scanned

Trade-off 2: Complexity vs Flexibility

• Two tracks increase system complexity
• But enable optimal processing per document type
• Decision: Accept complexity for 10x+ performance gains

Migration Plan

Phase 1: Infrastructure (Week 1-2)

1. Deploy UnifiedDocument model
2. Implement DocumentTypeDetector
3. Add DirectExtractionEngine
4. Update logging and monitoring

Phase 2: Integration (Week 3)

1. Update OCR service with routing logic
2. Modify PDF generator for unified model
3. Add new API endpoints
4. Deploy to staging

Phase 3: Validation (Week 4)

1. A/B testing with subset of traffic
2. Performance benchmarking
3. Quality validation
4. Client integration testing

Rollback Plan

1. Feature flag to disable dual-track
2. Fallback all requests to OCR track
3. Maintain old code paths during transition
4. Database migration reversible

Open Questions

Resolved

• Q: Should we support page-level track mixing?

  • A: No, adds complexity with minimal benefit. Document-level is sufficient.

• Q: How to handle Office documents?

  • A: Convert to PDF using LibreOffice, then analyze the PDF for text extractability.
    • Text-based PDF → Direct track (editable Office docs produce text PDFs)
    • Image-based PDF → OCR track (rare case of scanned content in Office)
  • This approach provides:
    • 10x+ faster processing for typical Office documents
    • Better layout preservation (no OCR errors)
    • Consistent pipeline (all documents normalized to PDF first)

Pending

• Q: What translation services to integrate with?

  • Needs stakeholder input on cost/quality trade-offs

• Q: Should we cache extracted text for repeated processing?

  • Depends on storage costs vs reprocessing frequency

• Q: How to handle password-protected PDFs?

  • May need API parameter for passwords

Performance Targets

Direct Extraction Track

• Latency: <500ms per page
• Throughput: 100+ pages/minute
• Memory: <500MB per document

OCR Track (Optimized)

• Latency: 2-5s per page (GPU)
• Throughput: 20-30 pages/minute
• Memory: <2GB per batch

API Response Times

• Document type detection: <100ms
• Processing initiation: <200ms
• Result retrieval: <100ms

Technical Dependencies

Python Packages

# Direct extraction
PyMuPDF==1.23.x
pdfplumber==0.10.x  # Fallback/validation
python-magic-bin==0.4.x

# OCR enhancement
paddlepaddle-gpu==2.5.2
paddleocr==2.7.3

# Infrastructure
pydantic==2.x
fastapi==0.100+
redis==5.x  # For caching

System Requirements

• CUDA 11.8+ for PaddlePaddle
• libmagic for file detection
• 16GB RAM minimum
• 50GB disk for models and cache

GPU Memory Management

Background

With RTX 4060 8GB GPU constraint and large PP-StructureV3 models, GPU OOM (Out of Memory) errors can occur during intensive OCR processing. Proper memory management is critical for reliable operation.

Implementation Strategy

1. Memory Cleanup System

Location: backend/app/services/ocr_service.py

Methods:

• cleanup_gpu_memory(): Cleans GPU memory after processing
• check_gpu_memory(): Checks available memory before operations

Cleanup Strategy:

def cleanup_gpu_memory(self):
    """Clean up GPU memory using PaddlePaddle and optionally torch"""
    # Clear PaddlePaddle GPU cache (primary)
    if paddle.device.is_compiled_with_cuda():
        paddle.device.cuda.empty_cache()

    # Clear torch GPU cache if available (optional)
    if TORCH_AVAILABLE and torch.cuda.is_available():
        torch.cuda.empty_cache()
        torch.cuda.synchronize()

    # Force Python garbage collection
    gc.collect()

2. Cleanup Points

GPU memory cleanup is triggered at strategic points:

1. After OCR processing (ocr_service.py:687)

   • After completing image OCR processing

2. After layout analysis (ocr_service.py:807-808, 913-914)

   • After enhanced PP-StructureV3 processing
   • After standard structure analysis

3. After traditional processing (ocr_service.py:1105-1106)

   • After processing all pages in traditional mode

4. On error (pp_structure_enhanced.py:168-177)

   • Clean up memory when PP-StructureV3 processing fails

3. Memory Monitoring

Pre-processing checks prevent OOM errors:

def check_gpu_memory(self, required_mb: int = 2000) -> bool:
    """Check if sufficient GPU memory is available"""
    # Get free memory via torch if available
    if TORCH_AVAILABLE and torch.cuda.is_available():
        free_memory = torch.cuda.mem_get_info()[0] / 1024**2
        if free_memory < required_mb:
            # Try cleanup and re-check
            self.cleanup_gpu_memory()
            # Log warning if still insufficient
    return True  # Continue even if check fails (graceful degradation)

Memory checks before:

• OCR processing: 1500MB required
• PP-StructureV3 processing: 2000MB required

4. Optional torch Dependency

torch is not required for GPU memory management. The system uses PaddlePaddle's built-in paddle.device.cuda.empty_cache() as the primary method.

Why optional:

• Project uses PaddlePaddle which has its own CUDA implementation
• torch provides additional memory monitoring via mem_get_info()
• Gracefully degrades if torch is not installed

Import pattern:

try:
    import torch
    TORCH_AVAILABLE = True
except ImportError:
    TORCH_AVAILABLE = False

5. Benefits

• Prevents OOM errors: Regular cleanup prevents memory accumulation
• Better GPU utilization: Freed memory available for next operations
• Graceful degradation: Works without torch, continues on cleanup failures
• Debug visibility: Logs memory status for troubleshooting

6. Performance Impact

• Cleanup overhead: <50ms per operation
• Memory recovery: Typically 200-500MB per cleanup
• No impact on accuracy or output quality