# Tasks: Fix OCR Track Cell Over-Detection

## Root Cause Analysis Update

**Original assumption:** PP-Structure was over-detecting cells.

**Actual root cause:** cell_boxes from `table_res_list` were being assigned to WRONG tables when HTML matching failed. The fallback used "first available" instead of bbox matching, causing:
- Table A's cell_boxes assigned to Table B
- False over-detection metrics (density 6.22 vs actual 1.65)
- Incorrect reclassification as TEXT

## Phase 1: Cell Validation Engine

- [x] 1.1 Create `cell_validation_engine.py` with metric-based validation
- [x] 1.2 Implement cell density calculation (cells per 10000px²)
- [x] 1.3 Implement average cell area calculation
- [x] 1.4 Implement cell height validation (table_height / cell_count)
- [x] 1.5 Add configurable thresholds with defaults:
  - max_cell_density: 3.0 cells/10000px²
  - min_avg_cell_area: 3000 px²
  - min_cell_height: 10px
- [ ] 1.6 Unit tests for validation functions

## Phase 2: Table Reclassification

- [x] 2.1 Implement table-to-text reclassification logic
- [x] 2.2 Preserve original text content from HTML table
- [x] 2.3 Create TEXT element with proper bbox
- [x] 2.4 Recalculate reading order after reclassification

## Phase 3: Integration

- [x] 3.1 Integrate validation into OCR service pipeline (after PP-Structure)
- [x] 3.2 Add validation before cell_boxes processing
- [x] 3.3 Add debug logging for filtered tables
- [ ] 3.4 Update processing metadata with filter statistics

## Phase 3.5: cell_boxes Matching Fix (NEW)

- [x] 3.5.1 Fix cell_boxes matching in pp_structure_enhanced.py to use bbox overlap instead of "first available"
- [x] 3.5.2 Calculate IoU between table_res cell_boxes bounding box and layout element bbox
- [x] 3.5.3 Match tables with >10% overlap, log match quality
- [x] 3.5.4 Update validate_cell_boxes to also check table bbox boundaries, not just page boundaries

**Results:**
- OLD: cell_boxes mismatch caused false over-detection (density=6.22)
- NEW: correct bbox matching (overlap=0.97-0.98), actual metrics (density=1.06-1.65)

## Phase 4: Testing

- [x] 4.1 Test with edit.pdf (sample with over-detection)
- [x] 4.2 Verify Table 3 (51 cells) - now correctly matched with density=1.65 (within threshold)
- [x] 4.3 Verify Tables 1, 2, 4 remain as tables
- [x] 4.4 Compare PDF output quality before/after
- [ ] 4.5 Regression test on other documents

## Phase 5: cell_boxes Quality Check (NEW - 2025-12-07)

**Problem:** PP-Structure's cell_boxes don't always form proper grids. Some tables have
overlapping cells (18-23% of cell pairs overlap), causing messy overlapping borders in PDF.

**Solution:** Added cell overlap quality check in `_draw_table_with_cell_boxes()`:

- [x] 5.1 Count overlapping cell pairs in cell_boxes
- [x] 5.2 Calculate overlap ratio (overlapping pairs / total pairs)
- [x] 5.3 If overlap ratio > 10%, skip cell_boxes rendering and use ReportLab Table fallback
- [x] 5.4 Text inside table regions filtered out to prevent duplicate rendering

**Test Results (task_id: 5e04bd00-a7e4-4776-8964-0a56eaf608d8):**
- Table pp3_0_3 (13 cells): 10/78 pairs (12.8%) overlap → ReportLab fallback
- Table pp3_0_6 (29 cells): 94/406 pairs (23.2%) overlap → ReportLab fallback
- Table pp3_0_7 (12 cells): No overlap issue → Grid-based line drawing
- Table pp3_0_16 (51 cells): 233/1275 pairs (18.3%) overlap → ReportLab fallback
- 26 text regions inside tables filtered out to prevent duplicate rendering

## Phase 6: Fix Double Rendering of Text Inside Tables (2025-12-07)

**Problem:** Text inside table regions was rendered twice:
1. Via layout/HTML table rendering
2. Via raw OCR text_regions (because `regions_to_avoid` excluded tables)

**Root Cause:** In `pdf_generator_service.py:1162-1169`:
```python
regions_to_avoid = [img for img in images_metadata if img.get('type') != 'table']
```
This intentionally excluded tables from filtering, causing text overlap.

**Solution:**
- [x] 6.1 Include tables in `regions_to_avoid` to filter text inside table bboxes
- [x] 6.2 Test PDF output with fix applied
- [x] 6.3 Verify no blank areas where tables should have content

**Test Results (task_id: 2d788fca-c824-492b-95cb-35f2fedf438d):**
- PDF size reduced 18% (59,793 → 48,772 bytes)
- Text content reduced 66% (14,184 → 4,829 chars) - duplicate text eliminated
- Before: "PRODUCT DESCRIPTION" appeared twice, table values duplicated
- After: Content appears only once, clean layout
- Table content preserved correctly via HTML table rendering

## Phase 7: Smart Table Rendering Based on cell_boxes Quality (2025-12-07)

**Problem:** Phase 6 fix caused content to be largely missing because all tables were
excluded from text rendering, but tables with bad cell_boxes quality had their content
rendered via ReportLab Table fallback which might not preserve text accurately.

**Solution:** Smart rendering based on cell_boxes quality:
- Good quality cell_boxes (≤10% overlap) → Filter text, render via cell_boxes
- Bad quality cell_boxes (>10% overlap) → Keep raw OCR text, draw table border only

**Implementation:**
- [x] 7.1 Add `_check_cell_boxes_quality()` to assess cell overlap ratio
- [x] 7.2 Add `_draw_table_border_only()` for border-only rendering
- [x] 7.3 Modify smart filtering in `_generate_pdf_from_data()`:
  - Good quality tables → add to `regions_to_avoid`
  - Bad quality tables → mark with `_use_border_only=True`
- [x] 7.4 Add `element_id` to `table_element` in `convert_unified_document_to_ocr_data()`
  (was missing, causing `_use_border_only` flag mismatch)
- [x] 7.5 Modify `draw_table_region()` to check `_use_border_only` flag

**Test Results (task_id: 82c7269f-aff0-493b-adac-5a87248cd949, scan.pdf):**
- Tables pp3_0_3 and pp3_0_4 identified as bad quality → border-only rendering
- Raw OCR text preserved and rendered at original positions
- PDF output: 62,998 bytes with all text content visible
- Logs confirm: `[TABLE] pp3_0_3: Drew border only (bad cell_boxes quality)`