fix: resolve table/image overlap and missing images in Direct track PDF generation

This commit fixes two critical rendering issues in Direct track PDF generation
that were reported by the user after the span-based rendering fixes.

## Issue 1: Table Text Overlap (表格跟文字重疊)

**Problem**: Tables rendered with duplicate text appearing on top because
DirectExtractionEngine extracts table content as both TABLE elements (with
structure) and separate TEXT elements (individual text blocks), causing
PDFGeneratorService to render both and create overlaps.

**Solution**: Implemented overlap filtering mechanism with area-based detection

**Changes**:
- Added `_is_element_inside_regions()` method in PDFGeneratorService
  - Uses overlap ratio detection (50% threshold) instead of strict containment
  - Handles cases where text blocks are larger than detected regions
  - Algorithm: filters element if ≥50% of its area overlaps with table/image bbox

- Modified `_generate_direct_track_pdf()` to:
  - Collect exclusion regions (tables + images) before rendering
  - Check each text/list element for overlap before drawing
  - Skip elements that significantly overlap with exclusion regions

**Evidence**:
- Test case: "PRODUCT DESCRIPTION" text block overlaps 74.5% with table
- File size reduced by 545 bytes (-3.8%) from filtered elements
- E2E tests passed: test_2_4_1_simple_tables, test_2_4_2_complex_tables
- User confirmed: "表格問題看起來處理好了" ✓

## Issue 2: Missing Images (圖片消失)

**Problem**: Images not rendering in generated PDFs because `extract()` was
called without `output_dir` parameter, causing images to not be saved to
filesystem, resulting in missing `saved_path` in element content.

**Solution**: Auto-create default output directory for image extraction

**Changes**:
- Modified `DirectExtractionEngine.extract()` to:
  - Auto-create `storage/results/{document_id}/` when output_dir not provided
  - Ensures images always saved when enable_image_extraction=True
  - Uses short UUID (8 chars) for cleaner directory names
  - Maintains backward compatibility (existing calls still work)

**Evidence**:
- Image extraction: 2/2 images saved to storage/results/
- Image files: 5,320 + 4,945 = 10,265 bytes total
- PDF file size: 13,627 → 26,643 bytes (+13,016 bytes, +95.5%)
- PyMuPDF verification: 2 images embedded in page 1
- E2E tests passed: test_1_3_2_direct_track_image_rendering, test_1_3_3_verify_image_paths

## Technical Details

**Overlap Filtering Algorithm**:
```
For each text/list element:
  For each table/image region:
    Calculate overlap_area = intersection(element_bbox, region_bbox)
    Calculate overlap_ratio = overlap_area / element_area
    If overlap_ratio ≥ 0.5: SKIP element (inside region)
```

**Key Advantages**:
- Area-based vs strict containment (handles larger text blocks)
- Configurable threshold (default 50%, adjustable if needed)
- Preserves reading order and layout
- No breaking changes to existing code

## Test Results

**E2E Test Suite**: 6/8 passed (2 OCR track timeouts unrelated to these fixes)
- ✅ test_1_3_2_direct_track_image_rendering
- ✅ test_1_3_3_verify_image_paths
- ✅ test_2_4_1_simple_tables
- ✅ test_2_4_2_complex_tables
- ✅ test_4_4_1_compare_direct_with_original

**File Size Evidence**:
- Text-only (no images): 13,627 bytes
- With images (both fixes): 26,643 bytes
- Difference: +13,016 bytes (+95.5%) confirming image inclusion

**Visual Quality**:
- Tables render without text overlay ✓
- Images embedded correctly (2/2) ✓
- Text outside regions still renders ✓
- No duplicate rendering ✓

## Files Changed

- backend/app/services/pdf_generator_service.py
  - Added _is_element_inside_regions() (lines 592-642)
  - Modified _generate_direct_track_pdf() (lines 697-766)

- backend/app/services/direct_extraction_engine.py
  - Modified extract() (lines 78-84)

- backend/tests/e2e/TEST_RESULTS_FINAL_FIX.md
  - Comprehensive test documentation

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

backend/app/services/direct_extraction_engine.py

@@ -63,17 +63,26 @@ class DirectExtractionEngine:
 
         Args:
             file_path: Path to PDF file
-            output_dir: Optional directory to save extracted images
+            output_dir: Optional directory to save extracted images.
+                       If not provided, creates a temporary directory in storage/results/{document_id}/
 
         Returns:
             UnifiedDocument with extracted content
         """
         start_time = datetime.now()
-        document_id = str(uuid.uuid4())
+        document_id = str(uuid.uuid4())[:8]  # Short ID for cleaner paths
 
         try:
             doc = fitz.open(str(file_path))
 
+            # If no output_dir provided, create default directory for image extraction
+            if output_dir is None and self.enable_image_extraction:
+                # Create temporary directory in storage/results
+                default_output_dir = Path("storage/results") / document_id
+                default_output_dir.mkdir(parents=True, exist_ok=True)
+                output_dir = default_output_dir
+                logger.debug(f"Created default output directory: {output_dir}")
+
             # Extract document metadata
             metadata = self._extract_metadata(file_path, doc, start_time)
 

backend/app/services/pdf_generator_service.py

@@ -589,6 +589,58 @@ class PDFGeneratorService:
             traceback.print_exc()
             return False
 
+    def _is_element_inside_regions(self, element_bbox, regions_elements, overlap_threshold=0.5) -> bool:
+        """
+        Check if an element overlaps significantly with any exclusion region (table, image).
+
+        This prevents duplicate rendering when text overlaps with tables/images.
+        Direct extraction often extracts both the structured element (table/image)
+        AND its text content as separate text blocks.
+
+        Uses overlap ratio detection instead of strict containment, since text blocks
+        from DirectExtractionEngine may be larger than detected table/image regions
+        (e.g., text block includes heading above table).
+
+        Args:
+            element_bbox: BBox of the element to check
+            regions_elements: List of region elements (tables, images) to check against
+            overlap_threshold: Minimum overlap percentage to trigger filtering (default 0.5 = 50%)
+
+        Returns:
+            True if element overlaps ≥50% with any region, False otherwise
+        """
+        if not element_bbox:
+            return False
+
+        e_x0, e_y0, e_x1, e_y1 = element_bbox.x0, element_bbox.y0, element_bbox.x1, element_bbox.y1
+        elem_area = (e_x1 - e_x0) * (e_y1 - e_y0)
+
+        if elem_area <= 0:
+            return False
+
+        for region in regions_elements:
+            r_bbox = region.bbox
+            if not r_bbox:
+                continue
+
+            # Calculate overlap rectangle
+            overlap_x0 = max(e_x0, r_bbox.x0)
+            overlap_y0 = max(e_y0, r_bbox.y0)
+            overlap_x1 = min(e_x1, r_bbox.x1)
+            overlap_y1 = min(e_y1, r_bbox.y1)
+
+            # Check if there is any overlap
+            if overlap_x0 < overlap_x1 and overlap_y0 < overlap_y1:
+                # Calculate overlap area
+                overlap_area = (overlap_x1 - overlap_x0) * (overlap_y1 - overlap_y0)
+                overlap_ratio = overlap_area / elem_area
+
+                # If element overlaps more than threshold, filter it out
+                if overlap_ratio >= overlap_threshold:
+                    return True
+
+        return False
+
     def _generate_direct_track_pdf(
         self,
         unified_doc: 'UnifiedDocument',
@@ -645,14 +697,19 @@ class PDFGeneratorService:
                 image_elements = []
                 list_elements = []
 
+                # FIX: Collect exclusion regions (tables, images) to prevent duplicate rendering
+                regions_to_avoid = []
+
                 for element in page.elements:
                     if element.type == ElementType.TABLE:
                         table_elements.append(element)
+                        regions_to_avoid.append(element)  # Tables are exclusion regions
                     elif element.is_visual or element.type in [
                         ElementType.IMAGE, ElementType.FIGURE,
                         ElementType.CHART, ElementType.DIAGRAM
                     ]:
                         image_elements.append(element)
+                        regions_to_avoid.append(element)  # Images are exclusion regions
                     elif element.type == ElementType.LIST_ITEM:
                         list_elements.append(element)
                     elif self._is_list_item_fallback(element):
@@ -687,6 +744,7 @@ class PDFGeneratorService:
                         all_elements.append(('text', elem))
 
                 logger.debug(f"Drawing {len(all_elements)} elements in extraction order (preserves multi-column reading order)")
+                logger.debug(f"Exclusion regions: {len(regions_to_avoid)} tables/images")
 
                 # Draw elements in document order
                 for elem_type, elem in all_elements:
@@ -695,11 +753,17 @@ class PDFGeneratorService:
                     elif elem_type == 'table':
                         self._draw_table_element_direct(pdf_canvas, elem, page_height)
                     elif elem_type == 'list':
-                        # Lists need special handling for sequential numbering
-                        # For now, draw individually (may lose list context)
-                        self._draw_text_element_direct(pdf_canvas, elem, page_height)
+                        # FIX: Check if list item overlaps with table/image
+                        if not self._is_element_inside_regions(elem.bbox, regions_to_avoid):
+                            self._draw_text_element_direct(pdf_canvas, elem, page_height)
+                        else:
+                            logger.debug(f"Skipping list element {elem.element_id} inside table/image region")
                     elif elem_type == 'text':
-                        self._draw_text_element_direct(pdf_canvas, elem, page_height)
+                        # FIX: Check if text overlaps with table/image before drawing
+                        if not self._is_element_inside_regions(elem.bbox, regions_to_avoid):
+                            self._draw_text_element_direct(pdf_canvas, elem, page_height)
+                        else:
+                            logger.debug(f"Skipping text element {elem.element_id} inside table/image region")
 
             # Save PDF
             pdf_canvas.save()