feat: create PDF layout restoration proposal

Create new OpenSpec change proposal to fix critical PDF generation issues:

**Problems Identified**:
1. Images never saved (empty _save_image implementation)
2. Image path mismatch (saved_path vs path lookup)
3. Tables never render (fake image dependency)
4. Text style completely lost (no font/color application)

**Solution Design**:
- Phase 1: Critical fixes (images, tables)
- Phase 2: Basic style preservation
- Phase 3: Advanced layout features
- Phase 4: Testing and optimization

**Key Improvements**:
- Implement actual image saving in pp_structure_enhanced
- Fix path resolution with fallback logic
- Use table's own bbox instead of fake images
- Track-specific rendering (rich for Direct, simple for OCR)
- Preserve StyleInfo (fonts, sizes, colors)

**Implementation Tasks**:
- 10 major task groups
- 4-week timeline
- No breaking changes
- Performance target: <10% overhead

Proposal validated: openspec validate pdf-layout-restoration ✓

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
egg
2025-11-20 19:00:49 +08:00
parent a957f06588
commit cf894b076e
4 changed files with 685 additions and 0 deletions

View File

@@ -0,0 +1,361 @@
# Technical Design: PDF Layout Restoration and Preservation
## Context
### Background
The current PDF generation system loses critical layout information during the conversion process. Despite successfully extracting images, tables, and styled text in both OCR and Direct tracks, none of this information makes it to the final PDF output due to:
1. **Empty implementations**: Image saving functions are stubs
2. **Path mismatches**: Saved paths don't match expected lookup keys
3. **Fake data dependencies**: Table rendering relies on non-existent image files
4. **Format degradation**: Rich formatting is reduced to plain text blocks
### Current Issues
#### Issue 1: OCR Track Image Loss
```python
# backend/app/services/pp_structure_enhanced.py
def _save_image(self, img_data, element_id: str, result_dir: Path):
"""Save image data to file"""
# TODO: Implement image saving
pass # Lines 262, 414 - NEVER SAVES ANYTHING!
```
Result: `img_path` from PP-Structure is ignored, no image files created.
#### Issue 2: Direct Track Path Mismatch
```python
# Saves as:
element.content["saved_path"] = f"imgs/{element_id}.png" # line 745
# But converter looks for:
image_path = content.get("path") # line 180 - WRONG KEY!
```
Result: Direct track images are saved but never found.
#### Issue 3: Table Rendering Failure
```python
# Creates fake reference:
images_metadata.append({
"path": f"table_{element.element_id}.png", # DOESN'T EXIST
"bbox": element.bbox
})
# Then tries to find it:
table_image = next((img for img in images_metadata
if "table" in img.get("path", "")), None)
if not table_image:
return # ALWAYS HAPPENS - NO RENDERING!
```
#### Issue 4: Text Style Loss
```python
# Has rich data:
StyleInfo(font='Arial', size=12, flags=BOLD|ITALIC, color='#000080')
# But only uses:
c.drawString(x, y, text) # No font, size, or style applied!
```
### Constraints
- Must maintain backward compatibility with existing API
- Cannot break current OCR/Direct track separation
- Should work within existing UnifiedDocument model
- Must handle both track types appropriately
## Goals / Non-Goals
### Goals
1. **Restore image rendering**: Save and correctly reference all images
2. **Fix table layout**: Render tables using actual bbox data
3. **Preserve text formatting**: Apply fonts, sizes, colors, and styles
4. **Track-specific optimization**: Different rendering for OCR vs Direct
5. **Maintain positioning**: Accurate spatial layout preservation
### Non-Goals
- Rewriting entire PDF generation system
- Changing UnifiedDocument structure
- Modifying extraction engines
- Supporting complex vector graphics
- Interactive PDF features (forms, annotations)
## Decisions
### Decision 1: Fix Image Handling Pipeline
**What**: Implement actual image saving and correct path resolution
**Implementation**:
```python
# pp_structure_enhanced.py
def _save_image(self, img_data, element_id: str, result_dir: Path):
"""Save image data to file"""
img_dir = result_dir / "imgs"
img_dir.mkdir(parents=True, exist_ok=True)
if isinstance(img_data, (str, Path)):
# Copy existing file
src_path = Path(img_data)
dst_path = img_dir / f"{element_id}.png"
shutil.copy2(src_path, dst_path)
else:
# Save image data
dst_path = img_dir / f"{element_id}.png"
Image.fromarray(img_data).save(dst_path)
return f"imgs/{element_id}.png" # Relative path
```
**Path Resolution**:
```python
# pdf_generator_service.py - convert_unified_document_to_ocr_data
def _get_image_path(element):
"""Get image path with fallback logic"""
content = element.content
# Try multiple path locations
for key in ["saved_path", "path", "image_path"]:
if isinstance(content, dict) and key in content:
return content[key]
# Check metadata
if hasattr(element, 'metadata') and element.metadata:
return element.metadata.get('path')
return None
```
### Decision 2: Direct Table Bbox Usage
**What**: Use table element's own bbox instead of fake image references
**Current Problem**:
```python
# Creates fake image that doesn't exist
images_metadata.append({"path": f"table_{id}.png", "bbox": bbox})
# Later fails to find it and skips rendering
```
**Solution**:
```python
def draw_table_region(self, c, table_element, page_width, page_height):
"""Draw table using its own bbox"""
# Get bbox directly from element
bbox = table_element.get("bbox")
if not bbox:
# Fallback to polygon
bbox_polygon = table_element.get("bbox_polygon")
if bbox_polygon:
bbox = self._polygon_to_bbox(bbox_polygon)
if not bbox:
logger.warning(f"No bbox for table {table_element.get('element_id')}")
return
# Use bbox to position and render table
x, y, width, height = self._normalize_bbox(bbox, page_width, page_height)
self._render_table_html(c, table_element.get("content"), x, y, width, height)
```
### Decision 3: Track-Specific Rendering
**What**: Different rendering approaches based on processing track
**Implementation Strategy**:
```python
def generate_from_unified_document(self, unified_doc, output_path):
"""Generate PDF with track-specific rendering"""
track = unified_doc.metadata.processing_track
if track == "direct":
return self._generate_direct_track_pdf(unified_doc, output_path)
else: # OCR track
return self._generate_ocr_track_pdf(unified_doc, output_path)
def _generate_direct_track_pdf(self, unified_doc, output_path):
"""Rich rendering for direct track"""
# Preserve:
# - Font families, sizes, weights
# - Text colors and backgrounds
# - Precise positioning
# - Line breaks and paragraph spacing
def _generate_ocr_track_pdf(self, unified_doc, output_path):
"""Simplified rendering for OCR track"""
# Best effort with:
# - Detected layout regions
# - Estimated font sizes
# - Basic positioning
```
### Decision 4: Style Preservation System
**What**: Apply StyleInfo to text rendering
**Text Rendering Enhancement**:
```python
def _apply_text_style(self, c, style_info):
"""Apply text styling from StyleInfo"""
if not style_info:
return
# Font selection
font_name = self._map_font(style_info.font)
if style_info.flags:
if style_info.flags & BOLD and style_info.flags & ITALIC:
font_name = f"{font_name}-BoldOblique"
elif style_info.flags & BOLD:
font_name = f"{font_name}-Bold"
elif style_info.flags & ITALIC:
font_name = f"{font_name}-Oblique"
# Apply font and size
try:
c.setFont(font_name, style_info.size or 12)
except:
c.setFont("Helvetica", style_info.size or 12)
# Apply color
if style_info.color:
r, g, b = self._parse_color(style_info.color)
c.setFillColorRGB(r, g, b)
def draw_text_region_enhanced(self, c, element, page_width, page_height):
"""Enhanced text rendering with style preservation"""
# Apply style
if hasattr(element, 'style') and element.style:
self._apply_text_style(c, element.style)
# Render with line breaks
text_lines = element.content.split('\n')
for line in text_lines:
c.drawString(x, y, line)
y -= line_height
```
## Implementation Phases
### Phase 1: Critical Fixes (Immediate)
1. Implement `_save_image()` in pp_structure_enhanced.py
2. Fix path resolution in converter
3. Fix table bbox usage
4. Test with sample documents
### Phase 2: Basic Style Preservation (Week 1)
1. Implement style application for Direct track
2. Add font mapping system
3. Handle text colors
4. Preserve line breaks
### Phase 3: Advanced Layout (Week 2)
1. Implement span-level rendering
2. Add paragraph alignment
3. Handle text indentation
4. Preserve list formatting
### Phase 4: Optimization (Week 3)
1. Cache font metrics
2. Optimize image handling
3. Batch rendering operations
4. Performance testing
## Risks / Trade-offs
### Risk 1: Font Availability
**Risk**: System fonts may not match document fonts
**Mitigation**: Font mapping table with fallbacks
```python
FONT_MAPPING = {
'Arial': 'Helvetica',
'Times New Roman': 'Times-Roman',
'Courier New': 'Courier',
# ... more mappings
}
```
### Risk 2: Complex Layouts
**Risk**: Some layouts too complex to preserve perfectly
**Mitigation**: Graceful degradation with logging
- Attempt full preservation
- Fall back to simpler rendering if needed
- Log what couldn't be preserved
### Risk 3: Performance Impact
**Risk**: Style processing may slow down PDF generation
**Mitigation**:
- Cache computed styles
- Batch similar operations
- Lazy loading for images
### Trade-off: Accuracy vs Speed
- Direct track: Prioritize accuracy (users chose quality)
- OCR track: Balance accuracy with processing time
## Testing Strategy
### Unit Tests
```python
def test_image_saving():
"""Test that images are actually saved"""
def test_path_resolution():
"""Test path lookup with fallbacks"""
def test_table_bbox_rendering():
"""Test tables render without fake images"""
def test_style_application():
"""Test font/color/size application"""
```
### Integration Tests
- Process document with images → verify images in PDF
- Process document with tables → verify table layout
- Process styled document → verify formatting preserved
### Visual Regression Tests
- Generate PDFs for test documents
- Compare with expected outputs
- Flag visual differences
## Success Metrics
1. **Image Rendering**: 100% of detected images appear in PDF
2. **Table Rendering**: 100% of tables rendered with correct layout
3. **Style Preservation**:
- Direct track: 90%+ style attributes preserved
- OCR track: Basic formatting maintained
4. **Performance**: <10% increase in generation time
5. **Quality**: User satisfaction with output appearance
## Migration Plan
### Rollout Strategy
1. Deploy fixes behind feature flag
2. Test with subset of documents
3. Gradual rollout monitoring quality
4. Full deployment after validation
### Rollback Plan
- Feature flag to disable enhanced rendering
- Fallback to current implementation
- Keep old code paths during transition
## Open Questions
### Resolved
Q: Should we modify UnifiedDocument structure?
A: No, work within existing model for compatibility
Q: How to handle missing fonts?
A: Font mapping table with safe fallbacks
### Pending
Q: Should we support embedded fonts in PDFs?
- Requires investigation of PDF font embedding
Q: How to handle RTL text and vertical writing?
- May need specialized text layout engine
Q: Should we preserve hyperlinks and bookmarks?
- Depends on user requirements

View File

@@ -0,0 +1,57 @@
# PDF Layout Restoration and Preservation
## Problem
Currently, the PDF generation from both OCR and Direct extraction tracks produces documents that are **severely degraded compared to the original**, with multiple critical issues:
### 1. Images Never Appear
- **OCR track**: `pp_structure_enhanced._save_image()` is an empty implementation (lines 262, 414), so detected images are never saved
- **Direct track**: Image paths are saved as `content["saved_path"]` but converter looks for `content.get("path")`, causing a mismatch
- **Result**: All PDFs are text-only, with no images whatsoever
### 2. Tables Never Render
- Table elements use fake `table_*.png` references that don't exist as actual files
- `draw_table_region()` tries to find these non-existent images to get bbox coordinates
- When images aren't found, table rendering is skipped entirely
- **Result**: No tables appear in generated PDFs
### 3. Text Layout is Broken
- All text uses single `drawString()` call with entire block as one line
- No line breaks, paragraph alignment, or text styling preserved
- Direct track extracts `StyleInfo` but it's completely ignored during PDF generation
- **Result**: Text appears as unformatted blocks at wrong positions
### 4. Information Loss in Conversion
- Direct track data gets converted to legacy OCR format, losing rich metadata
- Span-level information (fonts, colors, styles) is discarded
- Precise positioning information is reduced to simple bboxes
## Solution
Implement proper layout preservation for PDF generation:
1. **Fix image handling**: Actually save images and use correct path references
2. **Fix table rendering**: Use element's own bbox instead of looking for fake images
3. **Preserve text formatting**: Use StyleInfo and span-level data for accurate rendering
4. **Track-specific rendering**: Different approaches for OCR vs Direct tracks
## Impact
- **User Experience**: Output PDFs will actually be usable and readable
- **Functionality**: Tables and images will finally appear in outputs
- **Quality**: Direct track PDFs will closely match original formatting
- **Performance**: No negative impact, possibly faster by avoiding unnecessary conversions
## Tasks
- Fix image saving and path references (Critical)
- Fix table rendering using actual bbox data (Critical)
- Implement track-specific PDF generation (Important)
- Preserve text styling and formatting (Important)
- Add span-level text rendering (Nice-to-have)
## Deltas
### result-export
```delta
+ image_handling: Proper image saving and path resolution
+ table_rendering: Direct bbox usage for table positioning
+ text_formatting: StyleInfo preservation and application
+ track_specific_rendering: OCR vs Direct track differentiation
```

View File

@@ -0,0 +1,88 @@
# Result Export Specification
## ADDED Requirements
### Requirement: Layout-Preserving PDF Generation
The system MUST generate PDF files that preserve the original document layout including images, tables, and text formatting.
#### Scenario: Generate PDF with images
GIVEN a document processed through OCR or Direct track
WHEN images are detected and extracted
THEN the generated PDF MUST include all images at their original positions
AND images MUST maintain their aspect ratios
AND images MUST be saved to an imgs/ subdirectory
#### Scenario: Generate PDF with tables
GIVEN a document containing tables
WHEN tables are detected and extracted
THEN the generated PDF MUST render tables with proper structure
AND tables MUST use their own bbox coordinates for positioning
AND tables MUST NOT depend on fake image references
#### Scenario: Generate PDF with styled text
GIVEN a document processed through Direct track with StyleInfo
WHEN text elements have style information
THEN the generated PDF MUST apply font families (with mapping)
AND the PDF MUST apply font sizes
AND the PDF MUST apply text colors
AND the PDF MUST apply bold/italic formatting
### Requirement: Track-Specific Rendering
The system MUST provide different rendering approaches based on the processing track.
#### Scenario: Direct track rendering
GIVEN a document processed through Direct extraction
WHEN generating a PDF
THEN the system MUST use rich formatting preservation
AND maintain precise positioning from the original
AND apply all available StyleInfo
#### Scenario: OCR track rendering
GIVEN a document processed through OCR
WHEN generating a PDF
THEN the system MUST use simplified rendering
AND apply best-effort positioning based on bbox
AND use estimated font sizes
### Requirement: Image Path Resolution
The system MUST correctly resolve image paths with fallback logic.
#### Scenario: Resolve saved image paths
GIVEN an element with image content
WHEN looking for the image path
THEN the system MUST check content["saved_path"] first
AND fallback to content["path"] if not found
AND fallback to content["image_path"] if not found
AND finally check metadata["path"]
## MODIFIED Requirements
### Requirement: PDF Generation Pipeline
The PDF generation pipeline MUST be enhanced to support layout preservation.
#### Scenario: Enhanced PDF generation
GIVEN a UnifiedDocument from either track
WHEN generating a PDF
THEN the system MUST detect the processing track
AND route to the appropriate rendering method
AND preserve as much layout information as available
### Requirement: Image Handling in PP-Structure
The PP-Structure enhanced module MUST actually save extracted images.
#### Scenario: Save PP-Structure images
GIVEN PP-Structure extracts an image with img_path
WHEN processing the image element
THEN the _save_image method MUST save the image to disk
AND return a relative path for reference
AND handle both file paths and numpy arrays
### Requirement: Table Rendering Logic
The table rendering MUST use direct bbox instead of image lookup.
#### Scenario: Render table with direct bbox
GIVEN a table element with bbox coordinates
WHEN rendering the table in PDF
THEN the system MUST use the element's own bbox
AND NOT look for non-existent table image files
AND position the table accurately based on coordinates

View File

@@ -0,0 +1,179 @@
# Implementation Tasks: PDF Layout Restoration
## Phase 1: Critical Fixes (P0 - Immediate)
### 1. Fix Image Handling
- [ ] 1.1 Implement `_save_image()` in pp_structure_enhanced.py
- [ ] 1.1.1 Create imgs subdirectory in result_dir
- [ ] 1.1.2 Handle both file path and numpy array inputs
- [ ] 1.1.3 Save with element_id as filename
- [ ] 1.1.4 Return relative path for reference
- [ ] 1.1.5 Add error handling and logging
- [ ] 1.2 Fix path resolution in pdf_generator_service.py
- [ ] 1.2.1 Create `_get_image_path()` helper with fallback logic
- [ ] 1.2.2 Check saved_path, path, image_path keys
- [ ] 1.2.3 Check metadata for path
- [ ] 1.2.4 Update convert_unified_document_to_ocr_data to use helper
- [ ] 1.3 Test image rendering
- [ ] 1.3.1 Test with OCR track document
- [ ] 1.3.2 Test with Direct track document
- [ ] 1.3.3 Verify images appear in PDF output
### 2. Fix Table Rendering
- [ ] 2.1 Remove dependency on fake image references
- [ ] 2.1.1 Stop creating fake table_*.png references
- [ ] 2.1.2 Remove image lookup in draw_table_region
- [ ] 2.2 Use direct bbox from table element
- [ ] 2.2.1 Get bbox from table_element.get("bbox")
- [ ] 2.2.2 Fallback to bbox_polygon if needed
- [ ] 2.2.3 Implement _polygon_to_bbox converter
- [ ] 2.3 Fix table HTML rendering
- [ ] 2.3.1 Parse HTML content from table element
- [ ] 2.3.2 Position table using normalized bbox
- [ ] 2.3.3 Render with proper dimensions
- [ ] 2.4 Test table rendering
- [ ] 2.4.1 Test simple tables
- [ ] 2.4.2 Test complex multi-column tables
- [ ] 2.4.3 Test with both tracks
## Phase 2: Basic Style Preservation (P1 - Week 1)
### 3. Implement Style Application System
- [ ] 3.1 Create font mapping system
- [ ] 3.1.1 Define FONT_MAPPING dictionary
- [ ] 3.1.2 Map common fonts to PDF standard fonts
- [ ] 3.1.3 Add fallback to Helvetica for unknown fonts
- [ ] 3.2 Implement _apply_text_style() method
- [ ] 3.2.1 Extract font family from StyleInfo
- [ ] 3.2.2 Handle bold/italic flags
- [ ] 3.2.3 Apply font size
- [ ] 3.2.4 Apply text color
- [ ] 3.2.5 Handle errors gracefully
- [ ] 3.3 Create color parsing utilities
- [ ] 3.3.1 Parse hex colors (#RRGGBB)
- [ ] 3.3.2 Parse RGB tuples
- [ ] 3.3.3 Convert to PDF color space
### 4. Track-Specific Rendering
- [ ] 4.1 Add track detection in generate_from_unified_document
- [ ] 4.1.1 Check unified_doc.metadata.processing_track
- [ ] 4.1.2 Route to appropriate rendering method
- [ ] 4.2 Implement _generate_direct_track_pdf
- [ ] 4.2.1 Process each page with style preservation
- [ ] 4.2.2 Apply StyleInfo to text elements
- [ ] 4.2.3 Use precise positioning
- [ ] 4.2.4 Preserve line breaks
- [ ] 4.3 Implement _generate_ocr_track_pdf
- [ ] 4.3.1 Use simplified rendering
- [ ] 4.3.2 Best-effort positioning
- [ ] 4.3.3 Estimated font sizes
- [ ] 4.4 Test track-specific rendering
- [ ] 4.4.1 Compare Direct track with original
- [ ] 4.4.2 Verify OCR track maintains quality
## Phase 3: Advanced Layout (P2 - Week 2)
### 5. Enhanced Text Rendering
- [ ] 5.1 Implement line-by-line rendering
- [ ] 5.1.1 Split text content by newlines
- [ ] 5.1.2 Calculate line height from font size
- [ ] 5.1.3 Render each line with proper spacing
- [ ] 5.2 Add paragraph handling
- [ ] 5.2.1 Detect paragraph boundaries
- [ ] 5.2.2 Apply paragraph spacing
- [ ] 5.2.3 Handle indentation
- [ ] 5.3 Implement text alignment
- [ ] 5.3.1 Support left/right/center/justify
- [ ] 5.3.2 Calculate positioning based on alignment
- [ ] 5.3.3 Apply to each text block
### 6. List Formatting
- [ ] 6.1 Detect list elements
- [ ] 6.1.1 Identify list items from metadata
- [ ] 6.1.2 Determine list type (ordered/unordered)
- [ ] 6.1.3 Extract indent level
- [ ] 6.2 Render lists with proper formatting
- [ ] 6.2.1 Add bullets/numbers
- [ ] 6.2.2 Apply indentation
- [ ] 6.2.3 Maintain list spacing
### 7. Span-Level Rendering (Advanced)
- [ ] 7.1 Extract span information from Direct track
- [ ] 7.1.1 Parse children elements for spans
- [ ] 7.1.2 Get per-span styling
- [ ] 7.1.3 Track position within line
- [ ] 7.2 Render mixed-style lines
- [ ] 7.2.1 Switch styles mid-line
- [ ] 7.2.2 Handle inline formatting
- [ ] 7.2.3 Preserve exact positioning
## Phase 4: Testing and Optimization (P2 - Week 3)
### 8. Comprehensive Testing
- [ ] 8.1 Create test suite for layout preservation
- [ ] 8.1.1 Unit tests for each component
- [ ] 8.1.2 Integration tests for full pipeline
- [ ] 8.1.3 Visual regression tests
- [ ] 8.2 Test with various document types
- [ ] 8.2.1 Scientific papers (complex layout)
- [ ] 8.2.2 Business documents (tables/charts)
- [ ] 8.2.3 Books (chapters/paragraphs)
- [ ] 8.2.4 Forms (precise positioning)
- [ ] 8.3 Performance testing
- [ ] 8.3.1 Measure generation time
- [ ] 8.3.2 Profile memory usage
- [ ] 8.3.3 Identify bottlenecks
### 9. Performance Optimization
- [ ] 9.1 Implement caching
- [ ] 9.1.1 Cache font metrics
- [ ] 9.1.2 Cache parsed styles
- [ ] 9.1.3 Reuse computed layouts
- [ ] 9.2 Optimize image handling
- [ ] 9.2.1 Lazy load images
- [ ] 9.2.2 Compress when appropriate
- [ ] 9.2.3 Stream large images
- [ ] 9.3 Batch operations
- [ ] 9.3.1 Group similar rendering ops
- [ ] 9.3.2 Minimize context switches
- [ ] 9.3.3 Use efficient data structures
### 10. Documentation and Deployment
- [ ] 10.1 Update API documentation
- [ ] 10.1.1 Document new rendering capabilities
- [ ] 10.1.2 Add examples of improved output
- [ ] 10.1.3 Note performance characteristics
- [ ] 10.2 Create migration guide
- [ ] 10.2.1 Explain improvements
- [ ] 10.2.2 Note any breaking changes
- [ ] 10.2.3 Provide rollback instructions
- [ ] 10.3 Deployment preparation
- [ ] 10.3.1 Feature flag setup
- [ ] 10.3.2 Monitoring metrics
- [ ] 10.3.3 Rollback plan
## Success Criteria
### Must Have (Phase 1)
- [x] Images appear in generated PDFs
- [x] Tables render with correct layout
- [x] No regression in existing functionality
### Should Have (Phase 2)
- [ ] Text styling preserved in Direct track
- [ ] Font sizes and colors applied
- [ ] Line breaks maintained
### Nice to Have (Phase 3-4)
- [ ] Paragraph formatting
- [ ] List rendering
- [ ] Span-level styling
- [ ] <10% performance overhead
## Timeline
- **Week 0**: Phase 1 - Critical fixes (images, tables)
- **Week 1**: Phase 2 - Basic style preservation
- **Week 2**: Phase 3 - Advanced layout features
- **Week 3**: Phase 4 - Testing and optimization
- **Week 4**: Review, documentation, and deployment