Commit Graph

19 Commits

Author SHA1 Message Date
egg
8265be1741 test 2025-12-04 18:00:37 +08:00
egg
95ae1f1bdb feat: add table detection options and scan artifact removal
- Add TableDetectionSelector component for wired/wireless/region detection
- Add CV-based table line detector module (disabled due to poor performance)
- Add scan artifact removal preprocessing step (removes faint horizontal lines)
- Add PreprocessingConfig schema with remove_scan_artifacts option
- Update frontend PreprocessingSettings with scan artifact toggle
- Integrate table detection config into ProcessingPage
- Archive extract-table-cell-boxes proposal

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-30 13:21:50 +08:00
egg
dda9621e17 feat: enhance layout preprocessing and unify image scaling proposal
Backend changes:
- Add image scaling configuration for PP-Structure processing
- Enhance layout preprocessing service with scaling support
- Update OCR service with improved memory management
- Add PP-Structure enhanced processing improvements

Frontend changes:
- Update preprocessing settings UI
- Fix processing page layout and state management
- Update API types for new parameters

Proposals:
- Archive add-layout-preprocessing proposal (completed)
- Add unify-image-scaling proposal for consistent coordinate handling

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-28 09:23:19 +08:00
egg
2861f54838 fix: prevent preview infinite loop and add document type filtering
- Remove onAutoConfigReceived callback that caused state update loop
- Add document analysis to check if file needs OCR track
- Only show preprocessing options for OCR-eligible files (images, scanned PDFs)
- Show informative message for editable PDFs that use direct text extraction
- Display text coverage percentage for editable documents

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-27 17:31:05 +08:00
egg
894d18b432 feat: add real-time preprocessing preview with side-by-side comparison
- Create PreprocessingPreview component with debounced config updates
- Show original vs preprocessed images side-by-side
- Display image quality metrics (contrast, sharpness) with quality indicators
- Add zoom controls and fullscreen view for detailed inspection
- Show auto-detected configuration when in auto mode
- Integrate preview toggle with PreprocessingSettings component
- Add i18n translations for preview panel UI

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-27 17:25:52 +08:00
egg
5982fff71c feat: add contrast/sharpen strength controls, disable binarization
Major improvements to preprocessing controls:

Backend:
- Add contrast_strength (0.5-3.0) and sharpen_strength (0.5-2.0) to PreprocessingConfig
- Auto-detection now calculates optimal strength based on image quality metrics:
  - Lower contrast → Higher contrast_strength
  - Lower edge strength → Higher sharpen_strength
- Disable binarization in auto mode (rarely beneficial)
- CLAHE clipLimit now scales with contrast_strength
- Sharpening uses unsharp mask with variable strength

Frontend:
- Add strength sliders for contrast and sharpen in manual mode
- Sliders show current value and strength level (輕微/正常/強/最強)
- Move binarize option to collapsible "進階選項" section
- Updated i18n translations for strength labels

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-27 17:18:44 +08:00
egg
01d56f84cd feat: add preprocessing UI components and integration
Frontend implementation for add-layout-preprocessing proposal:
- Add PreprocessingSettings component with mode selection (auto/manual/disabled)
- Add manual config panel (contrast, sharpen, binarize options)
- Add zh-TW translations for preprocessing UI
- Integrate with ProcessingPage task start flow
- Add preprocessing types to apiV2.ts (PreprocessingMode, PreprocessingConfig)
- Pass preprocessing options to task start API

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-27 15:21:58 +08:00
egg
59206a6ab8 feat: simplify layout model selection and archive proposals
Changes:
- Replace PP-Structure 7-slider parameter UI with simple 3-option layout model selector
- Add layout model mapping: chinese (PP-DocLayout-S), default (PubLayNet), cdla
- Add LayoutModelSelector component and zh-TW translations
- Fix "default" model behavior with sentinel value for PubLayNet
- Add gap filling service for OCR track coverage improvement
- Add PP-Structure debug utilities
- Archive completed/incomplete proposals:
  - add-ocr-track-gap-filling (complete)
  - fix-ocr-track-table-rendering (incomplete)
  - simplify-ppstructure-model-selection (22/25 tasks)
- Add new layout model tests, archive old PP-Structure param tests
- Update OpenSpec ocr-processing spec with layout model requirements

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-27 13:27:00 +08:00
egg
1afdb822c3 feat: implement hybrid image extraction and memory management
Backend:
- Add hybrid image extraction for Direct track (inline image blocks)
- Add render_inline_image_regions() fallback when OCR doesn't find images
- Add check_document_for_missing_images() for detecting missing images
- Add memory management system (MemoryGuard, ModelManager, ServicePool)
- Update pdf_generator_service to handle HYBRID processing track
- Add ElementType.LOGO for logo extraction

Frontend:
- Fix PDF viewer re-rendering issues with memoization
- Add TaskNotFound component and useTaskValidation hook
- Disable StrictMode due to react-pdf incompatibility
- Fix task detail and results page loading states

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-26 10:56:22 +08:00
egg
2312b4cd66 feat: add frontend-adjustable PP-StructureV3 parameters with comprehensive testing
Implement user-configurable PP-StructureV3 parameters to allow fine-tuning OCR behavior
from the frontend. This addresses issues with over-merging, missing small text, and
document-specific optimization needs.

Backend:
- Add PPStructureV3Params schema with 7 adjustable parameters
- Update OCR service to accept custom parameters with smart caching
- Modify /tasks/{task_id}/start endpoint to receive params in request body
- Parameter priority: custom > settings default
- Conditional caching (no cache for custom params to avoid pollution)

Frontend:
- Create PPStructureParams component with collapsible UI
- Add 3 presets: default, high-quality, fast
- Implement localStorage persistence for user parameters
- Add import/export JSON functionality
- Integrate into ProcessingPage with conditional rendering

Testing:
- Unit tests: 7/10 passing (core functionality verified)
- API integration tests for schema validation
- E2E tests with authentication support
- Performance benchmarks for memory and initialization
- Test runner script with venv activation

Environment:
- Remove duplicate backend/venv (use root venv only)
- Update test runner to use correct virtual environment

OpenSpec:
- Archive fix-pdf-coordinate-system proposal
- Archive frontend-adjustable-ppstructure-params proposal
- Create ocr-processing spec
- Update result-export spec

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-25 14:39:19 +08:00
egg
fa1abcd8e6 feat: implement layout-preserving PDF generation with table reconstruction
Major Features:
- Add PDF generation service with Chinese font support
- Parse HTML tables from PP-StructureV3 and rebuild with ReportLab
- Extract table text for translation purposes
- Auto-filter text regions inside tables to avoid overlaps

Backend Changes:
1. pdf_generator_service.py (NEW)
   - HTMLTableParser: Parse HTML tables to extract structure
   - PDFGeneratorService: Generate layout-preserving PDFs
   - Coordinate transformation: OCR (top-left) → PDF (bottom-left)
   - Font size heuristics: 75% of bbox height with width checking
   - Table reconstruction: Parse HTML → ReportLab Table
   - Image embedding: Extract bbox from filenames

2. ocr_service.py
   - Add _extract_table_text() for translation support
   - Add output_dir parameter to save images to result directory
   - Extract bbox from image filenames (img_in_table_box_x1_y1_x2_y2.jpg)

3. tasks.py
   - Update process_task_ocr to use save_results() with PDF generation
   - Fix download_pdf endpoint to use database-stored PDF paths
   - Support on-demand PDF generation from JSON

4. config.py
   - Add chinese_font_path configuration
   - Add pdf_enable_bbox_debug flag

Frontend Changes:
1. PDFViewer.tsx (NEW)
   - React PDF viewer with zoom and pagination
   - Memoized file config to prevent unnecessary reloads

2. TaskDetailPage.tsx & ResultsPage.tsx
   - Integrate PDF preview and download

3. main.tsx
   - Configure PDF.js worker via CDN

4. vite.config.ts
   - Add host: '0.0.0.0' for network access
   - Use VITE_API_URL environment variable for backend proxy

Dependencies:
- reportlab: PDF generation library
- Noto Sans SC font: Chinese character support

🤖 Generated with Claude Code
https://claude.com/claude-code

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-17 20:21:56 +08:00
egg
012da1abc4 fix: migrate UI to V2 API and fix admin dashboard
Backend fixes:
- Fix markdown generation using correct 'markdown_content' key in tasks.py
- Update admin service to return flat data structure matching frontend types
- Add task_count and failed_tasks fields to user statistics
- Fix top users endpoint to return complete user data

Frontend fixes:
- Migrate ResultsPage from V1 batch API to V2 task API with polling
- Create TaskDetailPage component with markdown preview and download buttons
- Refactor ExportPage to support multi-task selection using V2 download endpoints
- Fix login infinite refresh loop with concurrency control flags
- Create missing Checkbox UI component

New features:
- Add /tasks/:taskId route for task detail view
- Implement multi-task batch export functionality
- Add real-time task status polling (2s interval)

OpenSpec:
- Archive completed proposal 2025-11-17-fix-v2-api-ui-issues
- Create result-export and task-management specifications

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-17 08:55:50 +08:00
egg
439458c7fe fix: migrate UploadPage to V2 API and fix logout navigation
Changes:
- Add uploadFile() method to apiClientV2 for single file uploads
- Update UploadPage to use apiClientV2 instead of apiClient
- Change upload logic to iterate files and collect task IDs
- Add navigation to /login after logout in Layout component

Fixes:
- 403 Forbidden error on file upload (token mismatch between V1/V2 APIs)
- Logout button not redirecting to login page after clearing auth

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-16 19:22:36 +08:00
egg
8f94191914 feat: add admin dashboard, audit logs, token expiry check and test suite
Frontend Features:
- Add ProtectedRoute component with token expiry validation
- Create AdminDashboardPage with system statistics and user management
- Create AuditLogsPage with filtering and pagination
- Add admin-only navigation (Shield icon) for ymirliu@panjit.com.tw
- Add admin API methods to apiV2 service
- Add admin type definitions (SystemStats, AuditLog, etc.)

Token Management:
- Auto-redirect to login on token expiry
- Check authentication on route change
- Show loading state during auth check
- Admin privilege verification

Backend Testing:
- Add pytest configuration (pytest.ini)
- Create test fixtures (conftest.py)
- Add unit tests for auth, tasks, and admin endpoints
- Add integration tests for complete workflows
- Test user isolation and admin access control

Documentation:
- Add TESTING.md with comprehensive testing guide
- Include test running instructions
- Document fixtures and best practices

Routes:
- /admin - Admin dashboard (admin only)
- /admin/audit-logs - Audit logs viewer (admin only)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-16 18:01:50 +08:00
egg
ad2b832fb6 feat: complete external auth V2 migration with advanced features
This commit implements comprehensive external Azure AD authentication
with complete task management, file download, and admin monitoring systems.

## Core Features Implemented (80% Complete)

### 1. Token Auto-Refresh Mechanism 
- Backend: POST /api/v2/auth/refresh endpoint
- Frontend: Auto-refresh 5 minutes before expiration
- Auto-retry on 401 errors with seamless token refresh

### 2. File Download System 
- Three format support: JSON / Markdown / PDF
- Endpoints: GET /api/v2/tasks/{id}/download/{format}
- File access control with ownership validation
- Frontend download buttons in TaskHistoryPage

### 3. Complete Task Management 
Backend Endpoints:
- POST /api/v2/tasks/{id}/start - Start task
- POST /api/v2/tasks/{id}/cancel - Cancel task
- POST /api/v2/tasks/{id}/retry - Retry failed task
- GET /api/v2/tasks - List with filters (status, filename, date range)
- GET /api/v2/tasks/stats - User statistics

Frontend Features:
- Status-based action buttons (Start/Cancel/Retry)
- Advanced search and filtering (status, filename, date range)
- Pagination and sorting
- Task statistics dashboard (5 stat cards)

### 4. Admin Monitoring System  (Backend)
Admin APIs:
- GET /api/v2/admin/stats - System statistics
- GET /api/v2/admin/users - User list with stats
- GET /api/v2/admin/users/top - User leaderboard
- GET /api/v2/admin/audit-logs - Audit log query system
- GET /api/v2/admin/audit-logs/user/{id}/summary

Admin Features:
- Email-based admin check (ymirliu@panjit.com.tw)
- Comprehensive system metrics (users, tasks, sessions, activity)
- Audit logging service for security tracking

### 5. User Isolation & Security 
- Row-level security on all task queries
- File access control with ownership validation
- Strict user_id filtering on all operations
- Session validation and expiry checking
- Admin privilege verification

## New Files Created

Backend:
- backend/app/models/user_v2.py - User model for external auth
- backend/app/models/task.py - Task model with user isolation
- backend/app/models/session.py - Session management
- backend/app/models/audit_log.py - Audit log model
- backend/app/services/external_auth_service.py - External API client
- backend/app/services/task_service.py - Task CRUD with isolation
- backend/app/services/file_access_service.py - File access control
- backend/app/services/admin_service.py - Admin operations
- backend/app/services/audit_service.py - Audit logging
- backend/app/routers/auth_v2.py - V2 auth endpoints
- backend/app/routers/tasks.py - Task management endpoints
- backend/app/routers/admin.py - Admin endpoints
- backend/alembic/versions/5e75a59fb763_*.py - DB migration

Frontend:
- frontend/src/services/apiV2.ts - Complete V2 API client
- frontend/src/types/apiV2.ts - V2 type definitions
- frontend/src/pages/TaskHistoryPage.tsx - Task history UI

Modified Files:
- backend/app/core/deps.py - Added get_current_admin_user_v2
- backend/app/main.py - Registered admin router
- frontend/src/pages/LoginPage.tsx - V2 login integration
- frontend/src/components/Layout.tsx - User display and logout
- frontend/src/App.tsx - Added /tasks route

## Documentation
- openspec/changes/.../PROGRESS_UPDATE.md - Detailed progress report

## Pending Items (20%)
1. Database migration execution for audit_logs table
2. Frontend admin dashboard page
3. Frontend audit log viewer

## Testing Status
- Manual testing:  Authentication flow verified
- Unit tests:  Pending
- Integration tests:  Pending

## Security Enhancements
-  User isolation (row-level security)
-  File access control
-  Token expiry validation
-  Admin privilege verification
-  Audit logging infrastructure
-  Token encryption (noted, low priority)
-  Rate limiting (noted, low priority)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-14 17:19:43 +08:00
beabigegg
0f81d5e70b feat: Docker化部署 - 單容器架構轉換
將 Tool_OCR 從 macOS conda 環境轉換為 Docker 單容器部署方案。
前後端整合於同一容器,通過 Nginx 反向代理,僅對外暴露單一端口。

## 新增功能
- Docker 單容器架構(Frontend + Backend + Nginx)
- 多階段構建優化鏡像大小
- Supervisor 進程管理
- 健康檢查機制
- 完整部署文檔

## 技術細節
- 對外端口:12015(原 12010 已被佔用)
- 內部架構:Nginx(12015) → FastAPI(8000)
- 前端靜態文件由 Nginx 直接服務
- API 請求通過 Nginx 反向代理

## 系統依賴完善
- libmagic1:文件類型檢測
- LibreOffice:Office 文檔轉換
- paddlex[ocr]:PP-StructureV3 版面分析
- 中日韓字體支援

## 配置調整
- 環境變數路徑:macOS 路徑 → 容器絕對路徑
- 前端 API URL:修正為統一端口 12015
- Pip 安裝:延長超時至 600 秒,重試 5 次
- CRLF 轉換:自動處理 Windows 換行符

## 清理
- 移除臨時文檔(API_FIX_SUMMARY.md 等 7 個文檔)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-13 13:12:59 +08:00
beabigegg
57cf91271c feat: modernize frontend UI with Tailwind v4 and professional design system
BREAKING CHANGE: Migrated to Tailwind CSS v4 configuration system

Key Changes:
- Migrated from Tailwind v3 to v4 configuration system
  - Removed tailwind.config.js (incompatible with v4)
  - Updated index.css with @theme directive and oklch color space
  - Defined all custom animations directly in CSS using @keyframes

- Redesigned LoginPage with modern, enterprise-grade UI:
  - Full-screen gradient background (blue → purple → pink)
  - Floating animated orbs with blur effects
  - Glass morphism white card with backdrop-blur
  - Gradient buttons with shadow effects
  - 7 custom animations: fade-in, slide-in-right, slide-in-left, scale-in, shimmer, pulse, float

- Added shadcn/ui components:
  - alert.tsx, dialog.tsx, input.tsx, label.tsx, select.tsx, tabs.tsx

- Updated dependencies:
  - Added class-variance-authority ^0.7.0
  - Added react-markdown ^9.0.1

- Updated frontend documentation:
  - Comprehensive README.md with feature list, tech stack, project structure
  - Quick start guide and deployment instructions

Technical Details:
- Tailwind v4 uses @import "tailwindcss" instead of @tailwind directives
- All theme customization now in @theme block with CSS variables
- Color system migrated to oklch for better perceptual uniformity
- Animation definitions moved from config to CSS @layer utilities

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-13 08:55:01 +08:00
beabigegg
21bc2f92f1 feat: modernize frontend architecture with professional UI/UX design
Complete redesign of frontend interface with focus on usability, visual hierarchy, and professional appearance:

**Design System:**
- Implemented clean blue color theme (#3B82F6) with professional palette
- Created consistent spacing, shadows, and typography system
- Added reusable utility classes (page-header, section, status-badge-*)
- Removed excessive gradients and decorative effects

**Layout Architecture:**
- Redesigned main layout with 256px sidebar navigation
- Sidebar includes logo, navigation with descriptions, and user profile
- Main content area with search bar and scrollable content
- Replaced horizontal navigation with vertical sidebar pattern

**Page Redesigns:**
1. LoginPage: Split-screen design with branding (left) and clean form (right)
   - Feature highlights with icons and statistics
   - Mobile responsive design
   - Professional gradient background with subtle pattern

2. UploadPage: Added 3-step visual progress indicator
   - Better file organization with summary and status badges
   - Clear action bar with confirmation message
   - Improved file list presentation

3. ProcessingPage: Enhanced progress visualization
   - Large progress bar with percentage display
   - 4-column stats grid (Completed, Processing, Failed, Total)
   - Clean file status list with processing times

4. ResultsPage: Improved 5-column layout (2 for list, 3 for preview)
   - Added stats cards for accuracy, processing time, and text blocks
   - Better preview panel with detailed metrics
   - Export and translate action buttons

5. ExportPage: Better organization with 2-column layout
   - Visual format selection with icons (TXT, JSON, Excel, Markdown, PDF)
   - Improved form controls and option organization
   - Sticky preview sidebar showing current configuration

**Component Updates:**
- Updated Button component with proper variants
- Enhanced Card component with hover effects
- Maintained FileUpload component functionality
- Added lucide-react for modern iconography

**Technical Improvements:**
- Fixed Tailwind CSS v4 compatibility issues with @apply
- Removed decorative animations in favor of functional ones
- Improved accessibility with proper labels and ARIA attributes
- Better color contrast and readability

This redesign transforms the interface from a basic layout to a professional, enterprise-ready application with clear visual hierarchy and excellent usability.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-12 23:54:44 +08:00
beabigegg
da700721fa first 2025-11-12 22:53:17 +08:00