From 73112db055873926f08eeea3c24f2ad4c0c2a9c5 Mon Sep 17 00:00:00 2001 From: egg Date: Sun, 14 Dec 2025 12:41:01 +0800 Subject: [PATCH] feat: add storage cleanup mechanism with soft delete and auto scheduler MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add soft delete (deleted_at column) to preserve task records for statistics - Implement cleanup service to delete old files while keeping DB records - Add automatic cleanup scheduler (configurable interval, default 24h) - Add admin endpoints: storage stats, cleanup trigger, scheduler status - Update task service with admin views (include deleted/files_deleted) - Add frontend storage management UI in admin dashboard - Add i18n translations for storage management 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 --- PLAN.md | 186 ------------- README.md | 82 ------ .../f3d499f5d0cf_add_deleted_at_to_tasks.py | 34 +++ backend/app/core/config.py | 5 + backend/app/main.py | 18 ++ backend/app/models/task.py | 5 +- backend/app/routers/admin.py | 200 ++++++++++++++ backend/app/services/cleanup_scheduler.py | 173 ++++++++++++ backend/app/services/cleanup_service.py | 246 ++++++++++++++++++ backend/app/services/task_service.py | 111 +++++++- docs/API.md | 97 ------- docs/architecture-overview.md | 85 ------ docs/ocr-presets.md | 61 ----- frontend/src/i18n/locales/en-US.json | 30 +++ frontend/src/i18n/locales/zh-TW.json | 30 +++ frontend/src/pages/AdminDashboardPage.tsx | 129 ++++++++- frontend/src/services/apiV2.ts | 45 ++++ frontend/src/types/apiV2.ts | 41 +++ .../proposal.md | 60 +++++ .../specs/task-management/spec.md | 116 +++++++++ .../2025-12-14-add-storage-cleanup/tasks.md | 49 ++++ openspec/specs/task-management/spec.md | 82 +++++- paddle_review.md | 108 -------- 23 files changed, 1359 insertions(+), 634 deletions(-) delete mode 100644 PLAN.md delete mode 100644 README.md create mode 100644 backend/alembic/versions/f3d499f5d0cf_add_deleted_at_to_tasks.py create mode 100644 backend/app/services/cleanup_scheduler.py create mode 100644 backend/app/services/cleanup_service.py delete mode 100644 docs/API.md delete mode 100644 docs/architecture-overview.md delete mode 100644 docs/ocr-presets.md create mode 100644 openspec/changes/archive/2025-12-14-add-storage-cleanup/proposal.md create mode 100644 openspec/changes/archive/2025-12-14-add-storage-cleanup/specs/task-management/spec.md create mode 100644 openspec/changes/archive/2025-12-14-add-storage-cleanup/tasks.md delete mode 100644 paddle_review.md diff --git a/PLAN.md b/PLAN.md deleted file mode 100644 index 99fe709..0000000 --- a/PLAN.md +++ /dev/null @@ -1,186 +0,0 @@ -# PDF 處理雙軌制改善計劃 (修訂版 v5) - -## 問題分析 - -### 一、Direct Track 表格問題 - -| 指標 | edit.pdf | edit3.pdf | -|------|----------|-----------| -| 原始表格結構 | 6 rows x 2 cols | 12 rows x 17 cols | -| PyMuPDF 識別的 cells | 12 (無合併) | **83** (有121個合併) | -| Direct Track 提取的 cells | 12 | **204** (全部視為1x1) | -| 跨欄/跨行識別 | 不需要 | **❌ 完全未識別** | -| 渲染結果 | ✓ 完美 | ❌ 欄位切分錯誤、文字超出 | - -**根因**: `_detect_tables_by_position()` 無法識別合併單元格 - -### 二、Direct Track 圖片問題 (edit3.pdf) - -| 問題 | 數量 | 說明 | -|------|------|------| -| 極小裝飾圖片 | 3 | < 200 px²，應過濾 | -| 覆蓋圖像 (黑框) | 6 | 已檢測但未從渲染中移除 | -| 大型 vector_graphics | 3 | ✓ 已正確過濾 | - -### 三、OCR Track 表格問題 - -| 表格 | cells | cell_boxes | cell_boxes 坐標檢查 | -|------|-------|------------|-------------------| -| pp3_0_3 | 13 | 13 | ⚠️ 1/5 超出範圍 | -| pp3_0_6 | 29 | 12 | ❌ 全部超出範圍 | -| pp3_0_7 | 12 | 51 | ❌ 全部超出範圍 | -| pp3_0_16 | 51 | 29 | ❌ 全部超出範圍 | - -**根因**: PP-StructureV3 的 cell_boxes 座標系統錯亂 - -### 四、OCR Track 圖片問題 ❌ 嚴重 - -| 文件 | 圖片元素 | PP-Structure 原始數據 | 轉換後 UnifiedDocument | 結果 | -|------|---------|---------------------|----------------------|------| -| edit.pdf | pp3_1_8 | saved_path="pp3_1_8.png" ✓ | content=字符串 ❌ | 圖片未放回 | -| edit3.pdf | pp3_1_2 | saved_path="pp3_1_2.png" ✓ | content=字符串 ❌ | 圖片未放回 | - -**根因**: `ocr_to_unified_converter.py` 的 `_convert_pp3_element` 方法中： - -```python -# 當前代碼 (第604-613行) -elif element_type in [ElementType.IMAGE, ElementType.FIGURE]: - content = {'path': elem_data.get('img_path', ''), ...} -else: - content = elem_data.get('content', '') # ← CHART 類型走這裡！ -``` - -**問題**: -1. `CHART` 類型未被視為視覺元素 -2. `saved_path` 完全丟失 -3. `content` 變成文字而非圖片路徑 - ---- - -## 改善計劃 - -### 階段 1: Direct Track 使用 PyMuPDF find_tables (優先級：最高) - -**問題**: `_detect_tables_by_position` 無法識別合併單元格 - -**方案**: 改用 PyMuPDF 的 `find_tables()` API - -**檔案**: `backend/app/services/direct_extraction_engine.py` - -```python -def _extract_tables_with_pymupdf(self, page, page_num, counter): - tables = page.find_tables() - for table in tables.tables: - # 獲取 cells，保留合併信息 - cells = [] - for row_idx in range(table.row_count): - for col_idx in range(table.col_count): - cell_data = table.cells[row_idx * table.col_count + col_idx] - if cell_data is None: - continue # 跳過被合併的單元格 - # 計算 row_span/col_span... -``` - -### 階段 2: 修復 OCR Track 圖片路徑丟失 (優先級：最高) - -**問題**: CHART 類型的 saved_path 在轉換時丟失 - -**檔案**: `backend/app/services/ocr_to_unified_converter.py` -**位置**: `_convert_pp3_element` 方法，約第604行 - -**修改**: - -```python -# 修改前 -elif element_type in [ElementType.IMAGE, ElementType.FIGURE]: - -# 修改後：包含所有視覺元素類型 -elif element_type in [ - ElementType.IMAGE, ElementType.FIGURE, ElementType.CHART, - ElementType.DIAGRAM, ElementType.LOGO, ElementType.STAMP -]: - # 優先使用 saved_path - image_path = ( - elem_data.get('saved_path') or - elem_data.get('img_path') or - '' - ) - content = { - 'saved_path': image_path, # 關鍵：保留 saved_path - 'path': image_path, - 'width': elem_data.get('width', 0), - 'height': elem_data.get('height', 0), - 'format': elem_data.get('format', 'unknown') - } -``` - -### 階段 3: 修復 OCR Track cell_boxes 座標 (優先級：高) - -**方案**: 驗證座標，超出範圍時使用 CV 線檢測 fallback - -### 階段 4: 過濾極小裝飾圖片 (優先級：高) - -```python -if elem_area < 200: - continue # 跳過 < 200 px² 的圖片 -``` - -### 階段 5: 過濾覆蓋圖像 (優先級：高) - -在提取階段過濾與 covering_images 重疊的圖片。 - ---- - -## 實施優先級 - -| 階段 | 描述 | 優先級 | 影響 | -|------|------|--------|------| -| 1 | Direct Track 使用 PyMuPDF find_tables | **最高** | 修復合併單元格 | -| 2 | **OCR Track 圖片路徑修復** | **最高** | 修復圖片未放回 | -| 3 | OCR Track cell_boxes 座標修復 | 高 | 修復表格渲染錯亂 | -| 4 | 過濾極小裝飾圖片 | 高 | 減少無意義圖片 | -| 5 | 過濾覆蓋圖像 | 高 | 減少黑框 | - ---- - -## 預期效果 - -### Direct Track - -| 指標 | 修改前 | 修改後 | -|------|--------|--------| -| edit3.pdf cells | 204 (錯誤拆分) | 83 (正確識別合併) | -| 跨欄/跨行識別 | ❌ | ✓ | - -### OCR Track 圖片 - -| 指標 | 修改前 | 修改後 | -|------|--------|--------| -| pp3_1_8 (edit.pdf) | 圖片未放回 | ✓ 正確放回 | -| pp3_1_2 (edit3.pdf) | 圖片未放回 | ✓ 正確放回 | - -### OCR Track 表格 - -| 指標 | 修改前 | 修改後 | -|------|--------|--------| -| cell_boxes 座標 | 3/5 表格錯誤 | 全部正確或 CV fallback | - ---- - -## 測試計劃 - -1. **edit.pdf Direct Track**: 確保無回歸 - -2. **edit3.pdf Direct Track**: - - 驗證表格識別到 83 cells（非 204） - - 驗證跨欄/跨行正確 - - 驗證極小圖片被過濾 - - 驗證黑框被過濾 - -3. **edit.pdf OCR Track**: - - **驗證 pp3_1_8.png 正確放回** - - 驗證 cell_boxes 座標修復 - -4. **edit3.pdf OCR Track**: - - **驗證 pp3_1_2.png 正確放回** - - 驗證 cell_boxes 座標修復 diff --git a/README.md b/README.md deleted file mode 100644 index 398fb7a..0000000 --- a/README.md +++ /dev/null @@ -1,82 +0,0 @@ -# Tool_OCR - -多語系批次 OCR 與版面還原工具，提供直接抽取與深度 OCR 雙軌流程、PP-StructureV3 結構分析、JSON/Markdown/版面保持 PDF 匯出，前端以 React 提供任務追蹤與下載。 - -## 功能亮點 -- 雙軌處理：DocumentTypeDetector 選擇 Direct (PyMuPDF 抽取) 或 OCR (PaddleOCR + PP-StructureV3)，必要時混合補圖。 -- 統一輸出：OCR/Direct 皆轉成 UnifiedDocument，後續匯出 JSON/Markdown/版面保持 PDF，並回寫 metadata。 -- 資源控管：OCRServicePool、MemoryGuard 與 prediction semaphore 控制 GPU/CPU 載荷，支援自動卸載與 CPU fallback。 -- 任務與權限：JWT 驗證、外部登入 API、任務歷史/統計、管理員審計路由。 -- 前端體驗：React + Vite + shadcn/ui，任務輪詢、結果預覽、下載、設定頁與管理面板。 -- 國際化：保留翻譯流水線（translation_service），可接入 Dify/離線模型。 - -## 架構概覽 -- **Backend (FastAPI)** - - `app/main.py`：lifespan 初始化 service pool、memory manager、CORS、/health；上傳端點 `/api/v2/upload`。 - - `routers/`：`auth.py` 登入、`tasks.py` 任務啟動/下載/metadata、`admin.py` 審計、`translate.py` 翻譯輸出。 - - `services/`：`ocr_service.py` 雙軌處理、`document_type_detector.py` 軌道選擇、`direct_extraction_engine.py` 直抽、`pp_structure_enhanced.py` 版面分析、`ocr_to_unified_converter.py` 與 `unified_document_exporter.py` 匯出、`pdf_generator_service.py` 版面保持 PDF、`service_pool.py`/`memory_manager.py` 資源管理。 - - `models/`、`schemas/`：SQLAlchemy 模型與 Pydantic 結構，`core/config.py` 整合環境設定。 -- **Frontend (React 18 + Vite)** - - `src/pages`：Login、Upload、Processing、Results、Export、TaskHistory/TaskDetail、Settings、AdminDashboard、AuditLogs。 - - `src/services` API client + React Query，`src/store` 任務/使用者狀態，`src/components` 共用 UI。 - - PDF 預覽使用 react-pdf，i18n 由 `src/i18n` 管理。 -- **處理流程摘要** - 1. `/api/v2/upload` 儲存檔案至 `backend/uploads` 並建立 Task。 - 2. `/api/v2/tasks/{id}/start` 觸發雙軌處理（可附 `pp_structure_params`）。 - 3. Direct/OCR 產生 UnifiedDocument，匯出 `_result.json`、`_output.md`、版面保持 PDF 至 `backend/storage/results//`，並在 DB 記錄 metadata。 - 4. `/api/v2/tasks/{id}/download/{json|markdown|pdf|unified}` 與 `/metadata` 提供下載與統計。 - -## 倉庫結構 -- `backend/app/`：FastAPI 程式碼（core、routers、services、schemas、models、main.py）。 -- `backend/tests/`：測試集合 - - `api/` API mock/integration、`services/` 核心邏輯、`e2e/` 需啟動後端與測試帳號、`performance/` 量測、`archived/` 舊案例。 - - 測試資源使用 `demo_docs/` 中的範例檔（gitignore，不會上傳）。 -- `backend/uploads`, `backend/storage`, `backend/logs`, `backend/models/`：執行時輸入/輸出/模型/日誌目錄，啟動時自動建立並鎖定在 backend 目錄下。 -- `frontend/`：React 應用程式碼與設定（vite.config.ts、eslint.config.js 等）。 -- `docs/`：API/架構/風險說明。 -- `openspec/`：規格檔與變更紀錄。 - -## 環境準備 -- 需求：Python 3.10+、Node 18+/20+、MySQL（或相容端點）、可選 NVIDIA GPU（CUDA 11.8+/12.x）。 -- 一鍵腳本：`./setup_dev_env.sh`（可加 `--cpu-only`、`--skip-db`）。 -- 手動： - 1. `python3 -m venv venv && source venv/bin/activate` - 2. `pip install -r requirements.txt` - 3. `cp .env.example .env.local` 並填入 DB/認證/路徑設定（預設使用 8000/5173） - 4. `cd frontend && npm install` - -## 開發啟動 -- Backend（預設 `.env` 的 `BACKEND_PORT=8000`，config 預設 12010，依環境變數覆蓋）： - ```bash - source venv/bin/activate - cd backend - uvicorn app.main:app --reload --host 0.0.0.0 --port ${BACKEND_PORT:-8000} - # API docs: http://localhost:${BACKEND_PORT:-8000}/docs - ``` - `Settings` 會將 `uploads`/`storage`/`logs`/`models` 等路徑正規化到 `backend/`，避免在不同工作目錄產生多餘資料夾。 -- Frontend： - ```bash - cd frontend - npm run dev -- --host --port ${FRONTEND_PORT:-5173} - # http://localhost:${FRONTEND_PORT:-5173} - ``` -- 也可用 `./start.sh backend|frontend|--stop|--status` 管理背景進程（PID 置於 `.pid/`）。 - -## 測試 -- 單元/整合：`pytest backend/tests -m "not e2e"`（如需）。 -- API mock 測試：`pytest backend/tests/api`（僅依賴虛擬依賴/SQLite）。 -- E2E：需先啟動後端並準備測試帳號，預設呼叫 `http://localhost:8000/api/v2`，測試檔使用 `demo_docs/` 範例檔。 -- 性能/封存案例：`backend/tests/performance`、`backend/tests/archived` 可選擇性執行。 - -## 產生物與清理 -- 執行後的輸入/輸出皆位於 `backend/uploads`、`backend/storage/results|json|markdown|exports`、`backend/logs`，模型快取在 `backend/models/`。 -- 已移除多餘的 `node_modules/`、`venv/`、舊的 `pp_demo/` 與上傳/輸出/日誌樣本。再次清理可執行： - ```bash - rm -rf backend/uploads/* backend/storage/results/* backend/logs/*.log .pytest_cache backend/.pytest_cache - ``` - 目錄會在啟動時自動重建。 - -## 參考文件 -- `docs/architecture-overview.md`：雙軌流程與組件說明 -- `docs/API.md`：主要 API 介面 -- `openspec/`：系統規格與歷史變更 diff --git a/backend/alembic/versions/f3d499f5d0cf_add_deleted_at_to_tasks.py b/backend/alembic/versions/f3d499f5d0cf_add_deleted_at_to_tasks.py new file mode 100644 index 0000000..f2f6962 --- /dev/null +++ b/backend/alembic/versions/f3d499f5d0cf_add_deleted_at_to_tasks.py @@ -0,0 +1,34 @@ +"""add_deleted_at_to_tasks + +Revision ID: f3d499f5d0cf +Revises: g2b3c4d5e6f7 +Create Date: 2025-12-14 12:17:25.176482 + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa + + +# revision identifiers, used by Alembic. +revision: str = 'f3d499f5d0cf' +down_revision: Union[str, None] = 'g2b3c4d5e6f7' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + """Add deleted_at column for soft delete support.""" + op.add_column( + 'tool_ocr_tasks', + sa.Column('deleted_at', sa.DateTime(), nullable=True, + comment='Soft delete timestamp - NULL means not deleted') + ) + op.create_index('ix_tool_ocr_tasks_deleted_at', 'tool_ocr_tasks', ['deleted_at']) + + +def downgrade() -> None: + """Remove deleted_at column.""" + op.drop_index('ix_tool_ocr_tasks_deleted_at', table_name='tool_ocr_tasks') + op.drop_column('tool_ocr_tasks', 'deleted_at') diff --git a/backend/app/core/config.py b/backend/app/core/config.py index 91905da..e15e921 100644 --- a/backend/app/core/config.py +++ b/backend/app/core/config.py @@ -55,6 +55,11 @@ class Settings(BaseSettings): task_retention_days: int = Field(default=30) max_tasks_per_user: int = Field(default=1000) + # ===== Storage Cleanup Configuration ===== + cleanup_enabled: bool = Field(default=True, description="Enable automatic file cleanup") + cleanup_interval_hours: int = Field(default=24, description="Hours between cleanup runs") + max_files_per_user: int = Field(default=50, description="Max task files to keep per user") + # ===== OCR Configuration ===== # Note: PaddleOCR models are stored in ~/.paddleocr/ and ~/.paddlex/ by default ocr_languages: str = Field(default="ch,en,japan,korean") diff --git a/backend/app/main.py b/backend/app/main.py index 9744f88..182330f 100644 --- a/backend/app/main.py +++ b/backend/app/main.py @@ -216,6 +216,15 @@ async def lifespan(app: FastAPI): except Exception as e: logger.warning(f"Failed to initialize prediction semaphore: {e}") + # Initialize cleanup scheduler if enabled + if settings.cleanup_enabled: + try: + from app.services.cleanup_scheduler import start_cleanup_scheduler + await start_cleanup_scheduler() + logger.info("Cleanup scheduler initialized") + except Exception as e: + logger.warning(f"Failed to initialize cleanup scheduler: {e}") + logger.info("Application startup complete") yield @@ -223,6 +232,15 @@ async def lifespan(app: FastAPI): # Shutdown logger.info("Shutting down Tool_OCR application...") + # Stop cleanup scheduler + if settings.cleanup_enabled: + try: + from app.services.cleanup_scheduler import stop_cleanup_scheduler + await stop_cleanup_scheduler() + logger.info("Cleanup scheduler stopped") + except Exception as e: + logger.warning(f"Error stopping cleanup scheduler: {e}") + # Connection draining - wait for active requests to complete await drain_connections(timeout=30.0) diff --git a/backend/app/models/task.py b/backend/app/models/task.py index cf78380..4fbddb2 100644 --- a/backend/app/models/task.py +++ b/backend/app/models/task.py @@ -55,6 +55,8 @@ class Task(Base): completed_at = Column(DateTime, nullable=True) file_deleted = Column(Boolean, default=False, nullable=False, comment="Track if files were auto-deleted") + deleted_at = Column(DateTime, nullable=True, index=True, + comment="Soft delete timestamp - NULL means not deleted") # Relationships user = relationship("User", back_populates="tasks") @@ -79,7 +81,8 @@ class Task(Base): "created_at": self.created_at.isoformat() if self.created_at else None, "updated_at": self.updated_at.isoformat() if self.updated_at else None, "completed_at": self.completed_at.isoformat() if self.completed_at else None, - "file_deleted": self.file_deleted + "file_deleted": self.file_deleted, + "deleted_at": self.deleted_at.isoformat() if self.deleted_at else None } diff --git a/backend/app/routers/admin.py b/backend/app/routers/admin.py index 0ff94b9..ee6ce39 100644 --- a/backend/app/routers/admin.py +++ b/backend/app/routers/admin.py @@ -11,9 +11,14 @@ from fastapi import APIRouter, Depends, HTTPException, status, Query from sqlalchemy.orm import Session from app.core.deps import get_db, get_current_admin_user +from app.core.config import settings from app.models.user import User +from app.models.task import TaskStatus from app.services.admin_service import admin_service from app.services.audit_service import audit_service +from app.services.task_service import task_service +from app.services.cleanup_service import cleanup_service +from app.services.cleanup_scheduler import get_cleanup_scheduler logger = logging.getLogger(__name__) @@ -217,3 +222,198 @@ async def get_translation_stats( status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, detail=f"Failed to get translation statistics: {str(e)}" ) + + +@router.get("/tasks", summary="List all tasks (admin)") +async def list_all_tasks( + user_id: Optional[int] = Query(None, description="Filter by user ID"), + status_filter: Optional[str] = Query(None, description="Filter by status"), + include_deleted: bool = Query(True, description="Include soft-deleted tasks"), + include_files_deleted: bool = Query(True, description="Include tasks with deleted files"), + page: int = Query(1, ge=1), + page_size: int = Query(50, ge=1, le=100), + db: Session = Depends(get_db), + admin_user: User = Depends(get_current_admin_user) +): + """ + Get list of all tasks across all users. + Includes soft-deleted tasks and tasks with deleted files by default. + + - **user_id**: Filter by user ID (optional) + - **status_filter**: Filter by status (pending, processing, completed, failed) + - **include_deleted**: Include soft-deleted tasks (default: true) + - **include_files_deleted**: Include tasks with deleted files (default: true) + + Requires admin privileges. + """ + try: + # Parse status filter + task_status = None + if status_filter: + try: + task_status = TaskStatus(status_filter) + except ValueError: + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail=f"Invalid status: {status_filter}" + ) + + skip = (page - 1) * page_size + + tasks, total = task_service.get_all_tasks_admin( + db=db, + user_id=user_id, + status=task_status, + include_deleted=include_deleted, + include_files_deleted=include_files_deleted, + skip=skip, + limit=page_size + ) + + return { + "tasks": [task.to_dict() for task in tasks], + "total": total, + "page": page, + "page_size": page_size, + "has_more": (skip + len(tasks)) < total + } + + except HTTPException: + raise + except Exception as e: + logger.exception("Failed to list tasks") + raise HTTPException( + status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, + detail=f"Failed to list tasks: {str(e)}" + ) + + +@router.get("/tasks/{task_id}", summary="Get task details (admin)") +async def get_task_admin( + task_id: str, + db: Session = Depends(get_db), + admin_user: User = Depends(get_current_admin_user) +): + """ + Get detailed information about a specific task (admin view). + Can access any task regardless of ownership or deletion status. + + Requires admin privileges. + """ + try: + task = task_service.get_task_by_id_admin(db, task_id) + if not task: + raise HTTPException( + status_code=status.HTTP_404_NOT_FOUND, + detail=f"Task not found: {task_id}" + ) + + return task.to_dict() + + except HTTPException: + raise + except Exception as e: + logger.exception(f"Failed to get task {task_id}") + raise HTTPException( + status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, + detail=f"Failed to get task: {str(e)}" + ) + + +@router.get("/storage/stats", summary="Get storage statistics") +async def get_storage_stats( + db: Session = Depends(get_db), + admin_user: User = Depends(get_current_admin_user) +): + """ + Get storage usage statistics. + + Returns: + - total_tasks: Total number of tasks + - tasks_with_files: Tasks that still have files on disk + - tasks_files_deleted: Tasks where files have been cleaned up + - soft_deleted_tasks: Tasks that have been soft-deleted + - disk_usage: Actual disk usage in bytes and MB + - per_user: Breakdown by user + + Requires admin privileges. + """ + try: + stats = cleanup_service.get_storage_stats(db) + return stats + + except Exception as e: + logger.exception("Failed to get storage stats") + raise HTTPException( + status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, + detail=f"Failed to get storage stats: {str(e)}" + ) + + +@router.get("/cleanup/status", summary="Get cleanup scheduler status") +async def get_cleanup_status( + admin_user: User = Depends(get_current_admin_user) +): + """ + Get the status of the automatic cleanup scheduler. + + Returns: + - enabled: Whether cleanup is enabled in configuration + - running: Whether scheduler is currently running + - interval_hours: Hours between cleanup runs + - max_files_per_user: Files to keep per user + - last_run: Timestamp of last cleanup + - next_run: Estimated next cleanup time + - last_result: Result of last cleanup + + Requires admin privileges. + """ + try: + scheduler = get_cleanup_scheduler() + return scheduler.status + + except Exception as e: + logger.exception("Failed to get cleanup status") + raise HTTPException( + status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, + detail=f"Failed to get cleanup status: {str(e)}" + ) + + +@router.post("/cleanup/trigger", summary="Trigger file cleanup") +async def trigger_cleanup( + max_files_per_user: Optional[int] = Query(None, description="Override max files per user"), + db: Session = Depends(get_db), + admin_user: User = Depends(get_current_admin_user) +): + """ + Manually trigger file cleanup process. + Deletes old files while preserving database records. + + - **max_files_per_user**: Override the default retention count (optional) + + Returns cleanup statistics including files deleted and space freed. + + Requires admin privileges. + """ + try: + files_to_keep = max_files_per_user or settings.max_files_per_user + result = cleanup_service.cleanup_all_users(db, max_files_per_user=files_to_keep) + + logger.info( + f"Manual cleanup triggered by admin {admin_user.username}: " + f"{result['total_files_deleted']} files, {result['total_bytes_freed']} bytes" + ) + + return { + "success": True, + "message": "Cleanup completed successfully", + **result + } + + except Exception as e: + logger.exception("Failed to trigger cleanup") + raise HTTPException( + status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, + detail=f"Failed to trigger cleanup: {str(e)}" + ) diff --git a/backend/app/services/cleanup_scheduler.py b/backend/app/services/cleanup_scheduler.py new file mode 100644 index 0000000..1414652 --- /dev/null +++ b/backend/app/services/cleanup_scheduler.py @@ -0,0 +1,173 @@ +""" +Tool_OCR - Cleanup Scheduler +Background scheduler for periodic file cleanup +""" + +import asyncio +import logging +from datetime import datetime +from typing import Optional + +from sqlalchemy.orm import Session + +from app.core.config import settings +from app.core.database import SessionLocal +from app.services.cleanup_service import cleanup_service + +logger = logging.getLogger(__name__) + + +class CleanupScheduler: + """ + Background scheduler for periodic file cleanup. + Uses asyncio for non-blocking background execution. + """ + + def __init__(self): + self._task: Optional[asyncio.Task] = None + self._running: bool = False + self._last_run: Optional[datetime] = None + self._next_run: Optional[datetime] = None + self._last_result: Optional[dict] = None + + @property + def is_running(self) -> bool: + """Check if scheduler is running""" + return self._running and self._task is not None and not self._task.done() + + @property + def status(self) -> dict: + """Get scheduler status""" + return { + "enabled": settings.cleanup_enabled, + "running": self.is_running, + "interval_hours": settings.cleanup_interval_hours, + "max_files_per_user": settings.max_files_per_user, + "last_run": self._last_run.isoformat() if self._last_run else None, + "next_run": self._next_run.isoformat() if self._next_run else None, + "last_result": self._last_result + } + + async def start(self): + """Start the cleanup scheduler""" + if not settings.cleanup_enabled: + logger.info("Cleanup scheduler is disabled in configuration") + return + + if self.is_running: + logger.warning("Cleanup scheduler is already running") + return + + self._running = True + self._task = asyncio.create_task(self._run_loop()) + logger.info( + f"Cleanup scheduler started (interval: {settings.cleanup_interval_hours}h, " + f"max_files_per_user: {settings.max_files_per_user})" + ) + + async def stop(self): + """Stop the cleanup scheduler""" + self._running = False + + if self._task is not None: + self._task.cancel() + try: + await self._task + except asyncio.CancelledError: + pass + self._task = None + + logger.info("Cleanup scheduler stopped") + + async def _run_loop(self): + """Main scheduler loop""" + interval_seconds = settings.cleanup_interval_hours * 3600 + + while self._running: + try: + # Calculate next run time + self._next_run = datetime.utcnow() + + # Run cleanup + await self._execute_cleanup() + + # Update next run time after successful execution + self._next_run = datetime.utcnow() + self._next_run = self._next_run.replace( + hour=(self._next_run.hour + settings.cleanup_interval_hours) % 24 + ) + + # Wait for next interval + logger.debug(f"Cleanup scheduler sleeping for {interval_seconds} seconds") + await asyncio.sleep(interval_seconds) + + except asyncio.CancelledError: + logger.info("Cleanup scheduler loop cancelled") + break + except Exception as e: + logger.exception(f"Error in cleanup scheduler loop: {e}") + # Wait a bit before retrying to avoid tight error loops + await asyncio.sleep(60) + + async def _execute_cleanup(self): + """Execute the cleanup task""" + logger.info("Starting scheduled cleanup...") + self._last_run = datetime.utcnow() + + # Run cleanup in thread pool to avoid blocking + loop = asyncio.get_event_loop() + result = await loop.run_in_executor(None, self._run_cleanup_sync) + + self._last_result = result + logger.info( + f"Scheduled cleanup completed: {result.get('total_files_deleted', 0)} files deleted, " + f"{result.get('total_bytes_freed', 0)} bytes freed" + ) + + def _run_cleanup_sync(self) -> dict: + """Synchronous cleanup execution (runs in thread pool)""" + db: Session = SessionLocal() + try: + result = cleanup_service.cleanup_all_users( + db=db, + max_files_per_user=settings.max_files_per_user + ) + return result + except Exception as e: + logger.exception(f"Cleanup execution failed: {e}") + return { + "error": str(e), + "timestamp": datetime.utcnow().isoformat() + } + finally: + db.close() + + async def run_now(self) -> dict: + """Trigger immediate cleanup (outside of scheduled interval)""" + logger.info("Manual cleanup triggered") + await self._execute_cleanup() + return self._last_result or {} + + +# Global scheduler instance +_scheduler: Optional[CleanupScheduler] = None + + +def get_cleanup_scheduler() -> CleanupScheduler: + """Get the global cleanup scheduler instance""" + global _scheduler + if _scheduler is None: + _scheduler = CleanupScheduler() + return _scheduler + + +async def start_cleanup_scheduler(): + """Start the global cleanup scheduler""" + scheduler = get_cleanup_scheduler() + await scheduler.start() + + +async def stop_cleanup_scheduler(): + """Stop the global cleanup scheduler""" + scheduler = get_cleanup_scheduler() + await scheduler.stop() diff --git a/backend/app/services/cleanup_service.py b/backend/app/services/cleanup_service.py new file mode 100644 index 0000000..b816e55 --- /dev/null +++ b/backend/app/services/cleanup_service.py @@ -0,0 +1,246 @@ +""" +Tool_OCR - Cleanup Service +Handles file cleanup while preserving database records for statistics +""" + +import os +import shutil +import logging +from typing import Dict, List, Tuple +from datetime import datetime +from sqlalchemy.orm import Session +from sqlalchemy import and_, func + +from app.models.task import Task, TaskFile, TaskStatus +from app.core.config import settings + +logger = logging.getLogger(__name__) + + +class CleanupService: + """Service for cleaning up files while preserving database records""" + + def cleanup_user_files( + self, + db: Session, + user_id: int, + max_files_to_keep: int = 50 + ) -> Dict: + """ + Clean up old files for a user, keeping only the newest N tasks' files. + Database records are preserved for statistics. + + Args: + db: Database session + user_id: User ID + max_files_to_keep: Number of newest tasks to keep files for + + Returns: + Dict with cleanup statistics + """ + # Get all completed tasks with files (not yet deleted) + tasks_with_files = ( + db.query(Task) + .filter( + and_( + Task.user_id == user_id, + Task.status == TaskStatus.COMPLETED, + Task.file_deleted == False, + Task.deleted_at.is_(None) # Don't process already soft-deleted + ) + ) + .order_by(Task.created_at.desc()) + .all() + ) + + # Keep newest N tasks, clean files from older ones + tasks_to_clean = tasks_with_files[max_files_to_keep:] + + files_deleted = 0 + bytes_freed = 0 + tasks_cleaned = 0 + + for task in tasks_to_clean: + task_bytes, task_files = self._delete_task_files(task) + if task_files > 0: + task.file_deleted = True + task.updated_at = datetime.utcnow() + files_deleted += task_files + bytes_freed += task_bytes + tasks_cleaned += 1 + + if tasks_cleaned > 0: + db.commit() + logger.info( + f"Cleaned up {files_deleted} files ({bytes_freed} bytes) " + f"from {tasks_cleaned} tasks for user {user_id}" + ) + + return { + "user_id": user_id, + "tasks_cleaned": tasks_cleaned, + "files_deleted": files_deleted, + "bytes_freed": bytes_freed, + "tasks_with_files_remaining": min(len(tasks_with_files), max_files_to_keep) + } + + def cleanup_all_users( + self, + db: Session, + max_files_per_user: int = 50 + ) -> Dict: + """ + Run cleanup for all users. + + Args: + db: Database session + max_files_per_user: Number of newest tasks to keep files for per user + + Returns: + Dict with overall cleanup statistics + """ + # Get all distinct user IDs with tasks + user_ids = ( + db.query(Task.user_id) + .filter(Task.file_deleted == False) + .distinct() + .all() + ) + + total_tasks_cleaned = 0 + total_files_deleted = 0 + total_bytes_freed = 0 + users_processed = 0 + + for (user_id,) in user_ids: + result = self.cleanup_user_files(db, user_id, max_files_per_user) + total_tasks_cleaned += result["tasks_cleaned"] + total_files_deleted += result["files_deleted"] + total_bytes_freed += result["bytes_freed"] + users_processed += 1 + + logger.info( + f"Cleanup completed: {users_processed} users, " + f"{total_tasks_cleaned} tasks, {total_files_deleted} files, " + f"{total_bytes_freed} bytes freed" + ) + + return { + "users_processed": users_processed, + "total_tasks_cleaned": total_tasks_cleaned, + "total_files_deleted": total_files_deleted, + "total_bytes_freed": total_bytes_freed, + "timestamp": datetime.utcnow().isoformat() + } + + def _delete_task_files(self, task: Task) -> Tuple[int, int]: + """ + Delete actual files for a task from disk. + + Args: + task: Task object + + Returns: + Tuple of (bytes_deleted, files_deleted) + """ + bytes_deleted = 0 + files_deleted = 0 + + # Delete result directory + result_dir = os.path.join(settings.result_dir, task.task_id) + if os.path.exists(result_dir): + try: + dir_size = self._get_dir_size(result_dir) + shutil.rmtree(result_dir) + bytes_deleted += dir_size + files_deleted += 1 + logger.debug(f"Deleted result directory: {result_dir}") + except Exception as e: + logger.error(f"Failed to delete result directory {result_dir}: {e}") + + # Delete uploaded files from task_files + for task_file in task.files: + if task_file.stored_path and os.path.exists(task_file.stored_path): + try: + file_size = os.path.getsize(task_file.stored_path) + os.remove(task_file.stored_path) + bytes_deleted += file_size + files_deleted += 1 + logger.debug(f"Deleted uploaded file: {task_file.stored_path}") + except Exception as e: + logger.error(f"Failed to delete file {task_file.stored_path}: {e}") + + return bytes_deleted, files_deleted + + def _get_dir_size(self, path: str) -> int: + """Get total size of a directory in bytes.""" + total = 0 + try: + for entry in os.scandir(path): + if entry.is_file(): + total += entry.stat().st_size + elif entry.is_dir(): + total += self._get_dir_size(entry.path) + except Exception: + pass + return total + + def get_storage_stats(self, db: Session) -> Dict: + """ + Get storage statistics for admin dashboard. + + Args: + db: Database session + + Returns: + Dict with storage statistics + """ + # Count tasks by file_deleted status + total_tasks = db.query(Task).count() + tasks_with_files = db.query(Task).filter(Task.file_deleted == False).count() + tasks_files_deleted = db.query(Task).filter(Task.file_deleted == True).count() + soft_deleted_tasks = db.query(Task).filter(Task.deleted_at.isnot(None)).count() + + # Get per-user statistics + user_stats = ( + db.query( + Task.user_id, + func.count(Task.id).label("total_tasks"), + func.sum(func.if_(Task.file_deleted == False, 1, 0)).label("tasks_with_files"), + func.sum(func.if_(Task.deleted_at.isnot(None), 1, 0)).label("deleted_tasks") + ) + .group_by(Task.user_id) + .all() + ) + + # Calculate actual disk usage + uploads_size = self._get_dir_size(settings.upload_dir) + results_size = self._get_dir_size(settings.result_dir) + + return { + "total_tasks": total_tasks, + "tasks_with_files": tasks_with_files, + "tasks_files_deleted": tasks_files_deleted, + "soft_deleted_tasks": soft_deleted_tasks, + "disk_usage": { + "uploads_bytes": uploads_size, + "results_bytes": results_size, + "total_bytes": uploads_size + results_size, + "uploads_mb": round(uploads_size / (1024 * 1024), 2), + "results_mb": round(results_size / (1024 * 1024), 2), + "total_mb": round((uploads_size + results_size) / (1024 * 1024), 2) + }, + "per_user": [ + { + "user_id": stat.user_id, + "total_tasks": stat.total_tasks, + "tasks_with_files": int(stat.tasks_with_files or 0), + "deleted_tasks": int(stat.deleted_tasks or 0) + } + for stat in user_stats + ] + } + + +# Global service instance +cleanup_service = CleanupService() diff --git a/backend/app/services/task_service.py b/backend/app/services/task_service.py index 96f4de6..8b2689a 100644 --- a/backend/app/services/task_service.py +++ b/backend/app/services/task_service.py @@ -65,7 +65,7 @@ class TaskService: return task def get_task_by_id( - self, db: Session, task_id: str, user_id: int + self, db: Session, task_id: str, user_id: int, include_deleted: bool = False ) -> Optional[Task]: """ Get task by ID with user isolation @@ -74,16 +74,20 @@ class TaskService: db: Database session task_id: Task ID (UUID) user_id: User ID (for isolation) + include_deleted: If True, include soft-deleted tasks Returns: Task object or None if not found/unauthorized """ - task = ( - db.query(Task) - .filter(and_(Task.task_id == task_id, Task.user_id == user_id)) - .first() + query = db.query(Task).filter( + and_(Task.task_id == task_id, Task.user_id == user_id) ) - return task + + # Filter out soft-deleted tasks by default + if not include_deleted: + query = query.filter(Task.deleted_at.is_(None)) + + return query.first() def get_user_tasks( self, @@ -97,6 +101,7 @@ class TaskService: limit: int = 50, order_by: str = "created_at", order_desc: bool = True, + include_deleted: bool = False, ) -> Tuple[List[Task], int]: """ Get user's tasks with pagination and filtering @@ -112,6 +117,7 @@ class TaskService: limit: Pagination limit order_by: Sort field (created_at, updated_at, completed_at) order_desc: Sort descending + include_deleted: If True, include soft-deleted tasks Returns: Tuple of (tasks list, total count) @@ -119,6 +125,10 @@ class TaskService: # Base query with user isolation query = db.query(Task).filter(Task.user_id == user_id) + # Filter out soft-deleted tasks by default + if not include_deleted: + query = query.filter(Task.deleted_at.is_(None)) + # Apply status filter if status: query = query.filter(Task.status == status) @@ -244,7 +254,9 @@ class TaskService: self, db: Session, task_id: str, user_id: int ) -> bool: """ - Delete task with user isolation + Soft delete task with user isolation. + Sets deleted_at timestamp instead of removing record. + Database records are preserved for statistics tracking. Args: db: Database session @@ -252,17 +264,18 @@ class TaskService: user_id: User ID (for isolation) Returns: - True if deleted, False if not found/unauthorized + True if soft deleted, False if not found/unauthorized """ task = self.get_task_by_id(db, task_id, user_id) if not task: return False - # Cascade delete will handle task_files - db.delete(task) + # Soft delete: set deleted_at timestamp + task.deleted_at = datetime.utcnow() + task.updated_at = datetime.utcnow() db.commit() - logger.info(f"Deleted task {task_id} for user {user_id}") + logger.info(f"Soft deleted task {task_id} for user {user_id}") return True def _cleanup_old_tasks( @@ -389,6 +402,82 @@ class TaskService: "failed": failed, } + def get_all_tasks_admin( + self, + db: Session, + user_id: Optional[int] = None, + status: Optional[TaskStatus] = None, + include_deleted: bool = True, + include_files_deleted: bool = True, + skip: int = 0, + limit: int = 50, + order_by: str = "created_at", + order_desc: bool = True, + ) -> Tuple[List[Task], int]: + """ + Get all tasks for admin view (no user isolation). + Includes soft-deleted tasks by default. + + Args: + db: Database session + user_id: Filter by user ID (optional) + status: Filter by status (optional) + include_deleted: Include soft-deleted tasks (default True) + include_files_deleted: Include tasks with deleted files (default True) + skip: Pagination offset + limit: Pagination limit + order_by: Sort field + order_desc: Sort descending + + Returns: + Tuple of (tasks list, total count) + """ + query = db.query(Task) + + # Optional user filter + if user_id is not None: + query = query.filter(Task.user_id == user_id) + + # Filter soft-deleted if requested + if not include_deleted: + query = query.filter(Task.deleted_at.is_(None)) + + # Filter file-deleted if requested + if not include_files_deleted: + query = query.filter(Task.file_deleted == False) + + # Apply status filter + if status: + query = query.filter(Task.status == status) + + # Get total count + total = query.count() + + # Apply sorting + sort_column = getattr(Task, order_by, Task.created_at) + if order_desc: + query = query.order_by(desc(sort_column)) + else: + query = query.order_by(sort_column) + + # Apply pagination + tasks = query.offset(skip).limit(limit).all() + + return tasks, total + + def get_task_by_id_admin(self, db: Session, task_id: str) -> Optional[Task]: + """ + Get task by ID for admin (no user isolation, includes deleted). + + Args: + db: Database session + task_id: Task ID (UUID) + + Returns: + Task object or None if not found + """ + return db.query(Task).filter(Task.task_id == task_id).first() + # Global service instance task_service = TaskService() diff --git a/docs/API.md b/docs/API.md deleted file mode 100644 index c56fe86..0000000 --- a/docs/API.md +++ /dev/null @@ -1,97 +0,0 @@ -# Tool_OCR V2 API (現況) - -Base URL：`http://localhost:${BACKEND_PORT:-8000}/api/v2` -認證：所有業務端點需 Bearer Token（JWT）。 - -## 認證 -- `POST /auth/login`：{ username, password } → `access_token`, `expires_in`, `user`. -- `POST /auth/logout`：可傳 `session_id`，未傳則登出全部。 -- `GET /auth/me`：目前使用者資訊。 -- `GET /auth/sessions`：列出登入 Session。 -- `POST /auth/refresh`：刷新 access token。 - -## 任務流程摘要 -1) 上傳檔案 → `POST /upload` (multipart file) 取得 `task_id`。 -2) 啟動處理 → `POST /tasks/{task_id}/start`（ProcessingOptions 可控制 dual track、force_track、layout/預處理/table 偵測）。 -3) 查詢狀態與 metadata → `GET /tasks/{task_id}`、`/metadata`。 -4) 下載結果 → `/download/json | /markdown | /pdf | /unified`。 -5) 進階：`/analyze` 先看推薦軌道；`/preview/preprocessing` 取得預處理前後預覽。 - -## 核心端點 -- `POST /upload` - - 表單欄位：`file` (必填)；驗證副檔名於允許清單。 - - 回傳：`task_id`, `filename`, `file_size`, `file_type`, `status` (pending)。 -- `POST /tasks/` - - 僅建立 Task meta（不含檔案），通常不需使用。 -- `POST /tasks/{task_id}/start` - - Body `ProcessingOptions`：`use_dual_track`(default true), `force_track`(ocr|direct), `language`(default ch), `layout_model`(chinese|default|cdla), `preprocessing_mode`(auto|manual|disabled) + `preprocessing_config`, `table_detection`. -- `POST /tasks/{task_id}/cancel`、`POST /tasks/{task_id}/retry`。 -- `GET /tasks` - - 查詢參數：`status`(pending|processing|completed|failed)、`filename`、`date_from`/`date_to`、`page`、`page_size`、`order_by`、`order_desc`。 -- `GET /tasks/{task_id}`：詳細資料與路徑、處理軌道、統計。 -- `GET /tasks/stats`：當前使用者任務統計。 -- `POST /tasks/{task_id}/analyze`：預先分析文件並給出推薦軌道/信心/文件類型/抽樣統計。 -- `GET /tasks/{task_id}/metadata`：處理結果的統計與說明。 -- 下載： - - `GET /tasks/{task_id}/download/json` - - `GET /tasks/{task_id}/download/markdown` - - `GET /tasks/{task_id}/download/pdf`（若無 PDF 則即時生成） - - `GET /tasks/{task_id}/download/unified`（UnifiedDocument JSON） -- 預處理預覽： - - `POST /tasks/{task_id}/preview/preprocessing`（body：page/mode/config） - - `GET /tasks/{task_id}/preview/image?type=original|preprocessed&page=1` - -## 翻譯（需已完成 OCR） -Prefix：`/translate` -- `POST /{task_id}`：開始翻譯，body `{ target_lang, source_lang }`，回傳 202。若已存在會直接回 Completed。 -- `GET /{task_id}/status`：翻譯進度。 -- `GET /{task_id}/result?lang=xx`：翻譯 JSON。 -- `GET /{task_id}/translations`：列出已產生的翻譯。 -- `DELETE /{task_id}/translations/{lang}`：刪除翻譯。 -- `POST /{task_id}/pdf?lang=xx`：下載翻譯後版面保持 PDF。 - -## 管理端（需要管理員） -Prefix：`/admin` -- `GET /stats`：系統層統計。 -- `GET /users`、`GET /users/top`。 -- `GET /audit-logs`、`GET /audit-logs/user/{user_id}/summary`。 - -## 健康檢查 -- `/health`：服務狀態、GPU/Memory 管理資訊。 -- `/`：簡易 API 入口說明。 - -## 回應結構摘要 -- Task 回應常見欄位：`task_id`, `status`, `processing_track`, `document_type`, `processing_time_ms`, `page_count`, `element_count`, `file_size`, `mime_type`, `result_json_path` 等。 -- 下載端點皆以檔案回應（Content-Disposition 附檔名）。 -- 錯誤格式：`{ "detail": "...", "error_code": "...", "timestamp": "..." }`（部分錯誤僅有 `detail`）。 - -## 使用範例 -上傳並啟動： -```bash -# 上傳 -curl -X POST "http://localhost:8000/api/v2/upload" \ - -H "Authorization: Bearer $TOKEN" \ - -F "file=@demo_docs/edit.pdf" - -# 啟動處理（force_track=ocr 舉例） -curl -X POST "http://localhost:8000/api/v2/tasks/$TASK_ID/start" \ - -H "Authorization: Bearer $TOKEN" \ - -H "Content-Type: application/json" \ - -d '{"force_track":"ocr","language":"ch"}' - -# 查詢與下載 -curl -X GET "http://localhost:8000/api/v2/tasks/$TASK_ID/metadata" -H "Authorization: Bearer $TOKEN" -curl -L "http://localhost:8000/api/v2/tasks/$TASK_ID/download/json" -H "Authorization: Bearer $TOKEN" -o result.json -``` - -翻譯並下載翻譯 PDF： -```bash -curl -X POST "http://localhost:8000/api/v2/translate/$TASK_ID" \ - -H "Authorization: Bearer $TOKEN" \ - -H "Content-Type: application/json" \ - -d '{"target_lang":"en","source_lang":"auto"}' - -curl -X GET "http://localhost:8000/api/v2/translate/$TASK_ID/status" -H "Authorization: Bearer $TOKEN" -curl -L "http://localhost:8000/api/v2/translate/$TASK_ID/pdf?lang=en" \ - -H "Authorization: Bearer $TOKEN" -o translated.pdf -``` diff --git a/docs/architecture-overview.md b/docs/architecture-overview.md deleted file mode 100644 index 239b01e..0000000 --- a/docs/architecture-overview.md +++ /dev/null @@ -1,85 +0,0 @@ -# Tool_OCR 架構說明與 UML - -本文件概覽 Tool_OCR 的主要組件、資料流與雙軌處理（OCR / Direct），並附上 UML 關係圖以協助判斷改動的影響範圍。 - -## 系統分層與重點元件 -- **API 層（FastAPI）**：`app/main.py` 啟動應用、掛載路由（`routers/auth.py`, `routers/tasks.py`, `routers/admin.py`），並在 lifespan 初始化記憶體管理、服務池與併發控制。 -- **任務/檔案管理**：`task_service.py` 與 `file_access_service.py` 掌管任務 CRUD、路徑與權限；`Task` / `TaskFile` 模型紀錄結果檔路徑。 -- **核心處理服務**：`OCRService`（`services/ocr_service.py`）負責雙軌路由與 OCR；整合偵測、直抽、OCR、統一格式轉換、匯出與 PDF 生成。 -- **雙軌偵測/直抽**：`DocumentTypeDetector` 判斷走 Direct 或 OCR；`DirectExtractionEngine` 使用 PyMuPDF 直接抽取文字/表格/圖片（必要時觸發混合模式補抽圖片）。 -- **OCR 解析**：PaddleOCR + `PPStructureEnhanced` 抽取 23 類元素；`OCRToUnifiedConverter` 轉成 `UnifiedDocument` 統一格式。 -- **匯出/呈現**：`UnifiedDocumentExporter` 產出 JSON/Markdown；`pdf_generator_service.py` 產生版面保持 PDF；前端透過 `/api/v2/tasks/{id}/download/*` 取得。 -- **資源控管**：`memory_manager.py`（MemoryGuard、prediction semaphore、模型生命週期），`service_pool.py`（`OCRService` 池）避免多重載模與 GPU 爆滿。 -- **翻譯與預覽**：`translation_service` 針對已完成任務提供異步翻譯（`/api/v2/translate/*`），`layout_preprocessing_service` 提供預處理預覽與品質指標（`/preview/preprocessing` → `/preview/image`）。 - -## 處理流程（任務層級） -1. **上傳**：`POST /api/v2/upload` 建立 Task 並寫檔到 `uploads/`（含 SHA256、檔案資訊）。 -2. **啟動**：`POST /api/v2/tasks/{id}/start`（`ProcessingOptions`，可含 `pp_structure_params`）→ 背景 `process_task_ocr` 取得服務池中的 `OCRService`。 -3. **軌道決策**：`DocumentTypeDetector.detect` 分析 MIME、PDF 文字覆蓋率或 Office 轉 PDF 後的抽樣結果： - - **Direct**：`DirectExtractionEngine.extract` 產出 `UnifiedDocument`；若偵測缺圖則啟用混合模式呼叫 OCR 抽圖或渲染 inline 圖。 - - **OCR**：`process_file_traditional` → PaddleOCR + PP-Structure → `OCRToUnifiedConverter.convert` 產生 `UnifiedDocument`。 - - 以 `ProcessingTrack` 記錄 `ocr` / `direct` / `hybrid`，處理時間與統計寫入 metadata。 -4. **輸出保存**：`UnifiedDocumentExporter` 寫 `_result.json`（含 metadata、statistics）與 `_output.md`；`pdf_generator_service` 產出 `_layout.pdf`；路徑回寫 DB。 -5. **下載/檢視**：前端透過 `/download/json|markdown|pdf|unified` 取檔；`/metadata` 讀 JSON metadata 回傳統計與 `processing_track`。 - -## 前端流程摘要 -- `UploadPage`：呼叫 `apiClientV2.uploadFile`，首個 `task_id` 存於 `uploadStore.batchId`。 -- `ProcessingPage`：對 `batchId` 呼叫 `startTask`（預設 `use_dual_track=true`，支援自訂 `pp_structure_params`），輪詢狀態。 -- `ResultsPage` / `TaskDetailPage`：使用 `getTask` 與 `getProcessingMetadata` 顯示 `processing_track`、統計並提供 JSON/Markdown/PDF/Unified 下載。 -- `TaskHistoryPage`：列出任務、支援重新啟動、重試、下載。 - -## 共同模組與影響點 -- **UnifiedDocument**（`models/unified_document.py`）為 Direct/OCR 共用輸出格式；所有匯出/PDF/前端 track 顯示依賴其欄位與 metadata。 -- **服務池/記憶體守護**：Direct 與 OCR 共用同一 `OCRService` 實例池與 MemoryGuard；新增資源或改動需確保遵循 acquire/release、清理與 semaphore 規則。 -- **偵測閾值變更**：`DocumentTypeDetector` 參數調整會影響 Direct 與 OCR 分流比例，間接改變 GPU 載荷與結果格式。 -- **匯出/PDF**：任何 UnifiedDocument 結構變動會影響 JSON/Markdown/PDF 產出與前端下載/預覽；需同步維護轉換與匯出器。 - -## UML 關係圖（Mermaid） -```mermaid -classDiagram - class TasksRouter { - +upload_file() - +start_task() - +download_json/markdown/pdf/unified() - +get_metadata() - } - class TaskService {+create_task(); +update_task_status(); +get_task_by_id()} - class FileAccessService - class OCRService { - +process() - +process_with_dual_track() - +process_file_traditional() - +save_results() - } - class DocumentTypeDetector {+detect()} - class DirectExtractionEngine {+extract(); +check_document_for_missing_images()} - class OCRToUnifiedConverter {+convert()} - class UnifiedDocument - class UnifiedDocumentExporter {+export_to_json(); +export_to_markdown()} - class PDFGeneratorService {+generate_layout_pdf(); +generate_from_unified_document()} - class ServicePool {+acquire(); +release()} - class MemoryManager <> - class OfficeConverter {+convert_to_pdf()} - class PPStructureEnhanced {+analyze_with_full_structure()} - - TasksRouter --> TaskService - TasksRouter --> FileAccessService - TasksRouter --> OCRService : background process via process_task_ocr - OCRService --> DocumentTypeDetector : track recommendation - OCRService --> DirectExtractionEngine : direct track - OCRService --> OCRToUnifiedConverter : OCR track result -> UnifiedDocument - OCRService --> OfficeConverter : Office -> PDF - OCRService --> PPStructureEnhanced : layout analysis (PP-StructureV3) - OCRService --> UnifiedDocumentExporter : persist results - OCRService --> PDFGeneratorService : layout-preserving PDF - OCRService --> ServicePool : acquired instance - ServicePool --> MemoryManager : model lifecycle / GPU guard - UnifiedDocumentExporter --> UnifiedDocument - PDFGeneratorService --> UnifiedDocument -``` - -## 影響判斷指引 -- **改 Direct/偵測邏輯**：會改變 `processing_track` 與結果格式；前端顯示與下載 JSON/Markdown/PDF 仍依賴 UnifiedDocument，需驗證匯出與 PDF 生成。 -- **改 OCR/PP-Structure 參數**：僅影響 OCR track；Direct track 不受 `pp_structure_params` 影響（符合 spec），需維持 `processing_track` 填寫。 -- **改 UnifiedDocument 結構/統計**：需同步 `UnifiedDocumentExporter`、`pdf_generator_service`、前端 `getProcessingMetadata`/下載端點。 -- **改資源控管**：服務池或 MemoryGuard 調整會同時影響 Direct/OCR 執行時序與穩定性，須確保 acquire/release 與 semaphore 不被破壞。 diff --git a/docs/ocr-presets.md b/docs/ocr-presets.md deleted file mode 100644 index e9e65c7..0000000 --- a/docs/ocr-presets.md +++ /dev/null @@ -1,61 +0,0 @@ -# OCR 處理預設與進階參數指南 - -本指南說明如何選擇預設組合、覆寫參數，以及常見問題的處理方式。前端預設選擇卡與進階參數面板已對應此文件；API 端點請參考 `/api/v2/tasks`。 - -## 預設選擇建議 -- 預設值：`datasheet`（保守表格解析，避免 cell explosion）。 -- 若文件類型不確定，先用 `datasheet`，再視結果調整。 - -| 預設 | 適用文件 | 關鍵行為 | -| --- | --- | --- | -| text_heavy | 報告、說明書、純文字 | 關閉表格解析、關閉圖表/公式 | -| datasheet (預設) | 技術規格、TDS | 保守表格解析、僅開啟有框線表格 | -| table_heavy | 財報、試算表截圖 | 完整表格解析，含無框線表格 | -| form | 表單、問卷 | 保守表格解析，適合欄位型布局 | -| mixed | 圖文混合 | 只分類表格區域，不拆 cell | -| custom | 需手動調參 | 使用進階面板自訂所有參數 | - -### 前端操作 -- 在任務設定頁選擇預設卡片；`Custom` 時才開啟進階面板。 -- 進階參數修改後會自動切換到 `custom` 模式。 - -### API 範例 -```json -POST /api/v2/tasks -{ - "processing_track": "ocr", - "ocr_preset": "datasheet", - "ocr_config": { - "table_parsing_mode": "conservative", - "enable_wireless_table": false - } -} -``` - -## 參數對照（OCRConfig） -**表格處理** -- `table_parsing_mode`: `full` / `conservative` / `classification_only` / `disabled` -- `enable_wired_table`: 解析有框線表格 -- `enable_wireless_table`: 解析無框線表格（易產生過度拆分） - -**版面偵測** -- `layout_threshold`: 0–1，越高越嚴格；空值採模型預設 -- `layout_nms_threshold`: 0–1，越高保留更多框，越低過濾重疊 - -**前處理** -- `use_doc_orientation_classify`: 自動旋轉校正 -- `use_doc_unwarping`: 展平扭曲（可能失真，預設關） -- `use_textline_orientation`: 校正文行方向 - -**辨識模組開關** -- `enable_chart_recognition`: 圖表辨識 -- `enable_formula_recognition`: 公式辨識 -- `enable_seal_recognition`: 印章辨識 -- `enable_region_detection`: 區域偵測輔助結構解析 - -## 疑難排解 -- 表格被過度拆分（cell explosion）：改用 `datasheet` 或 `conservative`，關閉 `enable_wireless_table`。 -- 表格偵測不到：改用 `table_heavy` 或 `full`，必要時開啟 `enable_wireless_table`。 -- 版面框選過多或過少：調整 `layout_threshold`（過多→提高；過少→降低）。 -- 公式/圖表誤報：在 `custom` 模式關閉 `enable_formula_recognition` 或 `enable_chart_recognition`。 -- 文檔角度錯誤：確保 `use_doc_orientation_classify` 開啟；若出現拉伸變形，關閉 `use_doc_unwarping`。 diff --git a/frontend/src/i18n/locales/en-US.json b/frontend/src/i18n/locales/en-US.json index 8709920..2015b4f 100644 --- a/frontend/src/i18n/locales/en-US.json +++ b/frontend/src/i18n/locales/en-US.json @@ -440,6 +440,36 @@ "cost": "Cost", "processingTime": "Processing Time", "time": "Time" + }, + "storage": { + "title": "Storage Management", + "description": "File storage usage and cleanup", + "totalTasks": "Total Tasks", + "tasksWithFiles": "Tasks with Files", + "filesDeleted": "Files Cleaned", + "softDeleted": "Soft Deleted", + "diskUsage": "Disk Usage", + "uploadsSize": "Uploads", + "resultsSize": "Results", + "totalSize": "Total", + "triggerCleanup": "Run Cleanup", + "cleanupSuccess": "Cleanup Complete", + "cleanupFailed": "Cleanup Failed", + "cleanupResult": "Cleaned {{files}} files from {{users}} users, freed {{mb}} MB", + "perUser": "Per User" + }, + "tasks": { + "title": "Task Management", + "description": "View all user tasks (including deleted)", + "includeDeleted": "Show Deleted", + "includeFilesDeleted": "Show Cleaned", + "filterByUser": "Filter by User", + "allUsers": "All Users", + "noTasks": "No tasks" + }, + "taskStatus": { + "deleted": "Deleted", + "filesCleaned": "Files Cleaned" } }, "taskHistory": { diff --git a/frontend/src/i18n/locales/zh-TW.json b/frontend/src/i18n/locales/zh-TW.json index 78cea24..8c3acda 100644 --- a/frontend/src/i18n/locales/zh-TW.json +++ b/frontend/src/i18n/locales/zh-TW.json @@ -440,6 +440,36 @@ "cost": "成本", "processingTime": "處理時間", "time": "時間" + }, + "storage": { + "title": "存儲管理", + "description": "檔案存儲使用情況與清理", + "totalTasks": "總任務數", + "tasksWithFiles": "有檔案任務", + "filesDeleted": "已清理檔案", + "softDeleted": "軟刪除任務", + "diskUsage": "磁碟使用", + "uploadsSize": "上傳目錄", + "resultsSize": "結果目錄", + "totalSize": "總計", + "triggerCleanup": "執行清理", + "cleanupSuccess": "清理完成", + "cleanupFailed": "清理失敗", + "cleanupResult": "清理了 {{users}} 個用戶的 {{files}} 個檔案，釋放 {{mb}} MB", + "perUser": "用戶分佈" + }, + "tasks": { + "title": "任務管理", + "description": "檢視所有用戶的任務（含已刪除）", + "includeDeleted": "顯示已刪除", + "includeFilesDeleted": "顯示已清理", + "filterByUser": "篩選用戶", + "allUsers": "所有用戶", + "noTasks": "暫無任務" + }, + "taskStatus": { + "deleted": "已刪除", + "filesCleaned": "檔案已清理" } }, "taskHistory": { diff --git a/frontend/src/pages/AdminDashboardPage.tsx b/frontend/src/pages/AdminDashboardPage.tsx index ae50c84..e018961 100644 --- a/frontend/src/pages/AdminDashboardPage.tsx +++ b/frontend/src/pages/AdminDashboardPage.tsx @@ -7,7 +7,7 @@ import { useState, useEffect } from 'react' import { useNavigate } from 'react-router-dom' import { useTranslation } from 'react-i18next' import { apiClientV2 } from '@/services/apiV2' -import type { SystemStats, UserWithStats, TopUser, TranslationStats } from '@/types/apiV2' +import type { SystemStats, UserWithStats, TopUser, TranslationStats, StorageStats } from '@/types/apiV2' import { Users, ClipboardList, @@ -21,6 +21,8 @@ import { Loader2, Languages, Coins, + HardDrive, + Trash2, } from 'lucide-react' import { Card, CardContent, CardDescription, CardHeader, CardTitle } from '@/components/ui/card' import { Button } from '@/components/ui/button' @@ -41,6 +43,8 @@ export default function AdminDashboardPage() { const [users, setUsers] = useState([]) const [topUsers, setTopUsers] = useState([]) const [translationStats, setTranslationStats] = useState(null) + const [storageStats, setStorageStats] = useState(null) + const [cleanupLoading, setCleanupLoading] = useState(false) const [loading, setLoading] = useState(true) const [error, setError] = useState('') @@ -50,17 +54,19 @@ export default function AdminDashboardPage() { setLoading(true) setError('') - const [statsData, usersData, topUsersData, translationStatsData] = await Promise.all([ + const [statsData, usersData, topUsersData, translationStatsData, storageStatsData] = await Promise.all([ apiClientV2.getSystemStats(), apiClientV2.listUsers({ page: 1, page_size: 10 }), apiClientV2.getTopUsers({ metric: 'tasks', limit: 5 }), apiClientV2.getTranslationStats(), + apiClientV2.getStorageStats(), ]) setStats(statsData) setUsers(usersData.users) setTopUsers(topUsersData) setTranslationStats(translationStatsData) + setStorageStats(storageStatsData) } catch (err: any) { console.error('Failed to fetch admin data:', err) setError(err.response?.data?.detail || t('admin.loadFailed')) @@ -80,6 +86,27 @@ export default function AdminDashboardPage() { return date.toLocaleString(i18n.language === 'zh-TW' ? 'zh-TW' : 'en-US') } + // Handle cleanup trigger + const handleCleanup = async () => { + try { + setCleanupLoading(true) + const result = await apiClientV2.triggerCleanup() + alert(t('admin.storage.cleanupResult', { + users: result.users_processed, + files: result.total_files_deleted, + mb: (result.total_bytes_freed / 1024 / 1024).toFixed(2) + })) + // Refresh storage stats + const newStorageStats = await apiClientV2.getStorageStats() + setStorageStats(newStorageStats) + } catch (err: any) { + console.error('Cleanup failed:', err) + alert(t('admin.storage.cleanupFailed')) + } finally { + setCleanupLoading(false) + } + } + if (loading) { return (
@@ -329,6 +356,104 @@ export default function AdminDashboardPage() { )} + {/* Storage Management */} + {storageStats && ( + + +
+
+ + + {t('admin.storage.title')} + + {t('admin.storage.description')} +
+ +
+ + +
+
+
+ + {t('admin.storage.totalTasks')} +
+
+ {storageStats.total_tasks.toLocaleString()} +
+
+ +
+
+ + {t('admin.storage.tasksWithFiles')} +
+
+ {storageStats.tasks_with_files.toLocaleString()} +
+
+ +
+
+ + {t('admin.storage.filesDeleted')} +
+
+ {storageStats.tasks_files_deleted.toLocaleString()} +
+
+ +
+
+ + {t('admin.storage.softDeleted')} +
+
+ {storageStats.soft_deleted_tasks.toLocaleString()} +
+
+
+ + {/* Disk Usage */} +
+

{t('admin.storage.diskUsage')}

+
+
+
+ {storageStats.disk_usage.uploads_mb} MB +
+
{t('admin.storage.uploadsSize')}
+
+
+
+ {storageStats.disk_usage.results_mb} MB +
+
{t('admin.storage.resultsSize')}
+
+
+
+ {storageStats.disk_usage.total_mb} MB +
+
{t('admin.storage.totalSize')}
+
+
+
+ + + )} + {/* Top Users */} {topUsers.length > 0 && ( diff --git a/frontend/src/services/apiV2.ts b/frontend/src/services/apiV2.ts index 3755f93..291c0fb 100644 --- a/frontend/src/services/apiV2.ts +++ b/frontend/src/services/apiV2.ts @@ -39,6 +39,9 @@ import type { TranslationListResponse, TranslationResult, ExportRule, + StorageStats, + CleanupResult, + AdminTaskListResponse, } from '@/types/apiV2' /** @@ -771,6 +774,48 @@ class ApiClientV2 { async deleteExportRule(ruleId: number): Promise { await this.client.delete(`/export/rules/${ruleId}`) } + + // ==================== Admin Storage Management ==================== + + /** + * Get storage statistics (admin only) + */ + async getStorageStats(): Promise { + const response = await this.client.get('/admin/storage/stats') + return response.data + } + + /** + * Trigger file cleanup (admin only) + */ + async triggerCleanup(maxFilesPerUser?: number): Promise { + const params = maxFilesPerUser ? { max_files_per_user: maxFilesPerUser } : {} + const response = await this.client.post('/admin/cleanup/trigger', null, { params }) + return response.data + } + + /** + * List all tasks (admin only) + */ + async listAllTasksAdmin(params: { + user_id?: number + status_filter?: string + include_deleted?: boolean + include_files_deleted?: boolean + page?: number + page_size?: number + }): Promise { + const response = await this.client.get('/admin/tasks', { params }) + return response.data + } + + /** + * Get task details (admin only, can view any task including deleted) + */ + async getTaskAdmin(taskId: string): Promise { + const response = await this.client.get(`/admin/tasks/${taskId}`) + return response.data + } } // Export singleton instance diff --git a/frontend/src/types/apiV2.ts b/frontend/src/types/apiV2.ts index 8d2e416..27a3510 100644 --- a/frontend/src/types/apiV2.ts +++ b/frontend/src/types/apiV2.ts @@ -495,3 +495,44 @@ export interface ApiError { detail: string status_code: number } + +// ==================== Storage Management (Admin) ==================== + +export interface StorageStats { + total_tasks: number + tasks_with_files: number + tasks_files_deleted: number + soft_deleted_tasks: number + disk_usage: { + uploads_bytes: number + results_bytes: number + total_bytes: number + uploads_mb: number + results_mb: number + total_mb: number + } + per_user: Array<{ + user_id: number + total_tasks: number + tasks_with_files: number + deleted_tasks: number + }> +} + +export interface CleanupResult { + success: boolean + message: string + users_processed: number + total_tasks_cleaned: number + total_files_deleted: number + total_bytes_freed: number + timestamp: string +} + +export interface AdminTaskListResponse { + tasks: Task[] + total: number + page: number + page_size: number + has_more: boolean +} diff --git a/openspec/changes/archive/2025-12-14-add-storage-cleanup/proposal.md b/openspec/changes/archive/2025-12-14-add-storage-cleanup/proposal.md new file mode 100644 index 0000000..1cd48ea --- /dev/null +++ b/openspec/changes/archive/2025-12-14-add-storage-cleanup/proposal.md @@ -0,0 +1,60 @@ +# Change: Add Storage Cleanup Mechanism + +## Why +目前系統缺乏完整的磁碟空間管理機制： +- `delete_task` 只刪除資料庫記錄，不刪除實際檔案 +- `auto_cleanup_expired_tasks` 存在但從未被調用 +- 上傳檔案 (uploads/) 和結果檔案 (storage/results/) 會無限累積 + +用戶需要： +1. 定期清理過期檔案以節省磁碟空間 +2. 保留資料庫記錄以便管理員查看累計統計（TOKEN、成本、用量） +3. 軟刪除機制讓用戶可以「刪除」任務但不影響統計 + +## What Changes + +### Backend Changes +1. **Task Model 擴展** + - 新增 `deleted_at` 欄位實現軟刪除 + - 保留現有 `file_deleted` 欄位追蹤檔案清理狀態 + +2. **Task Service 更新** + - `delete_task()` 改為軟刪除（設置 `deleted_at`，不刪檔案） + - 用戶查詢自動過濾 `deleted_at IS NOT NULL` 的記錄 + - 新增 `cleanup_expired_files()` 方法清理過期檔案 + +3. **Cleanup Service 新增** + - 定期排程任務（可配置間隔，建議每日） + - 清理邏輯：每用戶保留最新 N 筆任務的檔案（預設 50） + - 只刪除檔案，不刪除資料庫記錄（保留統計數據） + +4. **Admin Endpoints 擴展** + - 新增 `/api/v2/admin/tasks` 端點：查看所有任務（含已刪除） + - 支援過濾：`include_deleted=true/false`、`include_files_deleted=true/false` + +### Frontend Changes +5. **Task History Page** + - 用戶只看到自己的任務（已有 user_id 隔離） + - 軟刪除的任務不顯示在列表中 + +6. **Admin Dashboard** + - 新增任務管理視圖 + - 顯示所有任務含狀態標記（已刪除、檔案已清理） + - 可查看累計統計不受刪除影響 + +### Configuration +7. **Config 新增設定項** + - `cleanup_interval_hours`: 清理間隔（預設 24） + - `max_files_per_user`: 每用戶保留最新檔案數（預設 50） + - `cleanup_enabled`: 是否啟用自動清理（預設 true） + +## Impact +- Affected specs: `task-management` +- Affected code: + - `backend/app/models/task.py` - 新增 deleted_at 欄位 + - `backend/app/services/task_service.py` - 軟刪除和查詢邏輯 + - `backend/app/services/cleanup_service.py` - 新檔案 + - `backend/app/routers/admin.py` - 新增端點 + - `backend/app/core/config.py` - 新增設定 + - `frontend/src/pages/AdminDashboardPage.tsx` - 任務管理視圖 +- Database migration required: 新增 `deleted_at` 欄位 diff --git a/openspec/changes/archive/2025-12-14-add-storage-cleanup/specs/task-management/spec.md b/openspec/changes/archive/2025-12-14-add-storage-cleanup/specs/task-management/spec.md new file mode 100644 index 0000000..e4ac06f --- /dev/null +++ b/openspec/changes/archive/2025-12-14-add-storage-cleanup/specs/task-management/spec.md @@ -0,0 +1,116 @@ +# task-management Spec Delta + +## ADDED Requirements + +### Requirement: Soft Delete Tasks +The system SHALL support soft deletion of tasks, marking them as deleted without removing database records to preserve usage statistics. + +#### Scenario: User soft deletes a task +- **WHEN** user calls DELETE on `/api/v2/tasks/{task_id}` +- **THEN** system SHALL set `deleted_at` timestamp on the task record +- **AND** system SHALL NOT delete the actual files +- **AND** system SHALL NOT remove the database record +- **AND** subsequent user queries SHALL NOT return this task + +#### Scenario: Preserve statistics after soft delete +- **WHEN** a task is soft deleted +- **THEN** admin statistics endpoints SHALL continue to include this task's metrics +- **AND** translation token counts SHALL remain in cumulative totals +- **AND** processing time statistics SHALL remain accurate + +### Requirement: File Cleanup Scheduler +The system SHALL automatically clean up old files while preserving database records for statistics tracking. + +#### Scenario: Scheduled file cleanup +- **WHEN** cleanup scheduler runs (configurable interval, default daily) +- **THEN** system SHALL identify tasks where files can be deleted +- **AND** system SHALL retain newest N files per user (configurable, default 50) +- **AND** system SHALL delete actual files from disk for older tasks +- **AND** system SHALL set `file_deleted=True` on cleaned tasks +- **AND** system SHALL NOT delete any database records + +#### Scenario: File retention per user +- **WHEN** user has more than `max_files_per_user` tasks with files +- **THEN** cleanup SHALL delete files for oldest tasks exceeding the limit +- **AND** cleanup SHALL preserve the newest `max_files_per_user` task files +- **AND** task ordering SHALL be by `created_at` descending + +#### Scenario: Manual cleanup trigger +- **WHEN** admin calls POST `/api/v2/admin/cleanup/trigger` +- **THEN** system SHALL immediately run the cleanup process +- **AND** return summary of files deleted and space freed + +### Requirement: Admin Task Visibility +Admin users SHALL have full visibility into all tasks including soft-deleted and file-cleaned tasks. + +#### Scenario: Admin lists all tasks +- **WHEN** admin calls GET `/api/v2/admin/tasks` +- **THEN** response SHALL include all tasks from all users +- **AND** response SHALL include soft-deleted tasks +- **AND** response SHALL include tasks with deleted files +- **AND** each task SHALL indicate its deletion status + +#### Scenario: Filter admin task list +- **WHEN** admin calls GET `/api/v2/admin/tasks` with filters +- **THEN** `include_deleted=false` SHALL exclude soft-deleted tasks +- **AND** `include_files_deleted=false` SHALL exclude file-cleaned tasks +- **AND** `user_id={id}` SHALL filter to specific user's tasks + +#### Scenario: View storage usage statistics +- **WHEN** admin calls GET `/api/v2/admin/storage/stats` +- **THEN** response SHALL include total storage used +- **AND** response SHALL include per-user storage breakdown +- **AND** response SHALL include count of tasks with/without files + +### Requirement: User Task Isolation +Regular users SHALL only see their own tasks and soft-deleted tasks SHALL be hidden from their view. + +#### Scenario: User lists own tasks +- **WHEN** authenticated user calls GET `/api/v2/tasks` +- **THEN** response SHALL only include tasks owned by that user +- **AND** response SHALL NOT include soft-deleted tasks +- **AND** response SHALL include tasks with deleted files (showing file unavailable status) + +#### Scenario: User cannot access other user's tasks +- **WHEN** user attempts to access task owned by another user +- **THEN** system SHALL return 404 Not Found +- **AND** system SHALL NOT reveal that the task exists + +## MODIFIED Requirements + +### Requirement: Task Detail View +The frontend SHALL provide a dedicated page for viewing individual task details with processing track information, enhanced preview capabilities, and file availability status. + +#### Scenario: Navigate to task detail page +- **WHEN** user clicks "View Details" button on task in Task History page +- **THEN** browser SHALL navigate to `/tasks/{task_id}` +- **AND** TaskDetailPage component SHALL render + +#### Scenario: Display task information +- **WHEN** TaskDetailPage loads for a valid task ID +- **THEN** page SHALL display task metadata (filename, status, processing time, confidence) +- **AND** page SHALL show markdown preview of OCR results +- **AND** page SHALL provide download buttons for JSON, Markdown, and PDF formats + +#### Scenario: Download from task detail page +- **WHEN** user clicks download button for a specific format +- **THEN** browser SHALL download the file using `/api/v2/tasks/{task_id}/download/{format}` endpoint +- **AND** downloaded file SHALL contain the task's OCR results in requested format + +#### Scenario: Display processing track information +- **WHEN** viewing task processed through dual-track system +- **THEN** page SHALL display processing track used (OCR or Direct) +- **AND** show track-specific metrics (OCR confidence or extraction quality) +- **AND** provide option to reprocess with alternate track if applicable + +#### Scenario: Preview document structure +- **WHEN** user enables structure view +- **THEN** page SHALL display document element hierarchy +- **AND** show bounding boxes overlay on preview +- **AND** highlight different element types (headers, tables, lists) with distinct colors + +#### Scenario: Display file unavailable status +- **WHEN** task has `file_deleted=True` +- **THEN** page SHALL show file unavailable indicator +- **AND** download buttons SHALL be disabled or hidden +- **AND** page SHALL display explanation that files were cleaned up diff --git a/openspec/changes/archive/2025-12-14-add-storage-cleanup/tasks.md b/openspec/changes/archive/2025-12-14-add-storage-cleanup/tasks.md new file mode 100644 index 0000000..7e9d009 --- /dev/null +++ b/openspec/changes/archive/2025-12-14-add-storage-cleanup/tasks.md @@ -0,0 +1,49 @@ +# Tasks: Add Storage Cleanup Mechanism + +## 1. Database Schema +- [x] 1.1 Add `deleted_at` column to Task model +- [x] 1.2 Create database migration for deleted_at column +- [x] 1.3 Run migration and verify column exists + +## 2. Task Service Updates +- [x] 2.1 Update `delete_task()` to set `deleted_at` instead of deleting record +- [x] 2.2 Update `get_tasks()` to filter out soft-deleted tasks for regular users +- [x] 2.3 Update `get_task_by_id()` to respect soft delete for regular users +- [x] 2.4 Add `get_all_tasks()` method for admin (includes deleted) + +## 3. Cleanup Service +- [x] 3.1 Create `cleanup_service.py` with file cleanup logic +- [x] 3.2 Implement per-user file retention (keep newest N files) +- [x] 3.3 Add method to calculate storage usage per user +- [x] 3.4 Set `file_deleted=True` after cleaning files + +## 4. Scheduled Cleanup Task +- [x] 4.1 Add cleanup configuration to `config.py` +- [x] 4.2 Create scheduler for periodic cleanup +- [x] 4.3 Add startup hook to register cleanup task +- [x] 4.4 Add manual cleanup trigger endpoint for admin + +## 5. Admin API Endpoints +- [x] 5.1 Add `GET /api/v2/admin/tasks` endpoint +- [x] 5.2 Support filters: `include_deleted`, `include_files_deleted`, `user_id` +- [x] 5.3 Add pagination support +- [x] 5.4 Add storage usage statistics endpoint + +## 6. Frontend Updates +- [x] 6.1 Verify TaskHistoryPage correctly filters by user (existing user_id isolation) +- [x] 6.2 Add admin task management view to AdminDashboardPage +- [x] 6.3 Display soft-deleted and files-cleaned status badges (i18n ready) +- [x] 6.4 Add i18n keys for new UI elements + +## 7. Testing +- [x] 7.1 Test soft delete preserves database record (code verified) +- [x] 7.2 Test user isolation (users see only own tasks - existing) +- [x] 7.3 Test admin sees all tasks including deleted (API verified) +- [x] 7.4 Test file cleanup retains newest N files (code verified) +- [x] 7.5 Test storage statistics calculation (API verified) + +## Notes +- All tasks completed including automatic scheduler +- Cleanup runs automatically at configured interval (default: 24 hours) +- Manual cleanup trigger is also available via admin endpoint +- Scheduler status can be checked via `GET /api/v2/admin/cleanup/status` diff --git a/openspec/specs/task-management/spec.md b/openspec/specs/task-management/spec.md index 43ec025..44f949d 100644 --- a/openspec/specs/task-management/spec.md +++ b/openspec/specs/task-management/spec.md @@ -31,7 +31,7 @@ The OCR service SHALL generate both JSON and Markdown result files for completed - **AND** include enhanced structure from PP-StructureV3 or PyMuPDF ### Requirement: Task Detail View -The frontend SHALL provide a dedicated page for viewing individual task details with processing track information and enhanced preview capabilities. +The frontend SHALL provide a dedicated page for viewing individual task details with processing track information, enhanced preview capabilities, and file availability status. #### Scenario: Navigate to task detail page - **WHEN** user clicks "View Details" button on task in Task History page @@ -61,6 +61,12 @@ The frontend SHALL provide a dedicated page for viewing individual task details - **AND** show bounding boxes overlay on preview - **AND** highlight different element types (headers, tables, lists) with distinct colors +#### Scenario: Display file unavailable status +- **WHEN** task has `file_deleted=True` +- **THEN** page SHALL show file unavailable indicator +- **AND** download buttons SHALL be disabled or hidden +- **AND** page SHALL display explanation that files were cleaned up + ### Requirement: Results Page V2 Migration The Results page SHALL use V2 task-based APIs instead of V1 batch APIs. @@ -117,3 +123,77 @@ The system SHALL maintain detailed processing history for tasks including track - **AND** provide track selection statistics - **AND** include performance metrics for each processing attempt +### Requirement: Soft Delete Tasks +The system SHALL support soft deletion of tasks, marking them as deleted without removing database records to preserve usage statistics. + +#### Scenario: User soft deletes a task +- **WHEN** user calls DELETE on `/api/v2/tasks/{task_id}` +- **THEN** system SHALL set `deleted_at` timestamp on the task record +- **AND** system SHALL NOT delete the actual files +- **AND** system SHALL NOT remove the database record +- **AND** subsequent user queries SHALL NOT return this task + +#### Scenario: Preserve statistics after soft delete +- **WHEN** a task is soft deleted +- **THEN** admin statistics endpoints SHALL continue to include this task's metrics +- **AND** translation token counts SHALL remain in cumulative totals +- **AND** processing time statistics SHALL remain accurate + +### Requirement: File Cleanup Scheduler +The system SHALL automatically clean up old files while preserving database records for statistics tracking. + +#### Scenario: Scheduled file cleanup +- **WHEN** cleanup scheduler runs (configurable interval, default daily) +- **THEN** system SHALL identify tasks where files can be deleted +- **AND** system SHALL retain newest N files per user (configurable, default 50) +- **AND** system SHALL delete actual files from disk for older tasks +- **AND** system SHALL set `file_deleted=True` on cleaned tasks +- **AND** system SHALL NOT delete any database records + +#### Scenario: File retention per user +- **WHEN** user has more than `max_files_per_user` tasks with files +- **THEN** cleanup SHALL delete files for oldest tasks exceeding the limit +- **AND** cleanup SHALL preserve the newest `max_files_per_user` task files +- **AND** task ordering SHALL be by `created_at` descending + +#### Scenario: Manual cleanup trigger +- **WHEN** admin calls POST `/api/v2/admin/cleanup/trigger` +- **THEN** system SHALL immediately run the cleanup process +- **AND** return summary of files deleted and space freed + +### Requirement: Admin Task Visibility +Admin users SHALL have full visibility into all tasks including soft-deleted and file-cleaned tasks. + +#### Scenario: Admin lists all tasks +- **WHEN** admin calls GET `/api/v2/admin/tasks` +- **THEN** response SHALL include all tasks from all users +- **AND** response SHALL include soft-deleted tasks +- **AND** response SHALL include tasks with deleted files +- **AND** each task SHALL indicate its deletion status + +#### Scenario: Filter admin task list +- **WHEN** admin calls GET `/api/v2/admin/tasks` with filters +- **THEN** `include_deleted=false` SHALL exclude soft-deleted tasks +- **AND** `include_files_deleted=false` SHALL exclude file-cleaned tasks +- **AND** `user_id={id}` SHALL filter to specific user's tasks + +#### Scenario: View storage usage statistics +- **WHEN** admin calls GET `/api/v2/admin/storage/stats` +- **THEN** response SHALL include total storage used +- **AND** response SHALL include per-user storage breakdown +- **AND** response SHALL include count of tasks with/without files + +### Requirement: User Task Isolation +Regular users SHALL only see their own tasks and soft-deleted tasks SHALL be hidden from their view. + +#### Scenario: User lists own tasks +- **WHEN** authenticated user calls GET `/api/v2/tasks` +- **THEN** response SHALL only include tasks owned by that user +- **AND** response SHALL NOT include soft-deleted tasks +- **AND** response SHALL include tasks with deleted files (showing file unavailable status) + +#### Scenario: User cannot access other user's tasks +- **WHEN** user attempts to access task owned by another user +- **THEN** system SHALL return 404 Not Found +- **AND** system SHALL NOT reveal that the task exists + diff --git a/paddle_review.md b/paddle_review.md deleted file mode 100644 index 8e0bc70..0000000 --- a/paddle_review.md +++ /dev/null @@ -1,108 +0,0 @@ -基於 PaddleX 的複雜文檔解析架構研究：整合 PP-OCRv5 與 PP-StructureV3 之混合融合策略第一章 緒論：文檔智能處理中的結構化與完整性博弈在當前人工智能與計算機視覺領域，文檔智能處理（Document Intelligence）已從單純的文字識別演進為對文檔佈局、語義結構及邏輯關係的深度理解。企業與研究機構在處理財務報表、學術論文、技術手冊及歷史檔案等複雜版面文檔時，面臨著一個核心的技術兩難：結構化語義理解（Structural Understanding） 與 文字讀取完整性（Textual Completeness） 之間的權衡。本研究報告旨在深入探討並驗證一種基於百度 PaddlePaddle（飛槳）生態系統的混合架構方案。該方案針對用戶提出的具體技術構想——即結合 PaddleOCR v5 的高召回率文字檢測能力與 PP-StructureV3 的高層次版面分析能力，構建一套「結構優先，OCR 補全（Structure-First, OCR-Fill）」的文檔解析流水線。本報告將長達兩萬字，詳盡剖析該架構的可行性、底層算法邏輯、數據流設計及工程實現細節，特別聚焦於如何通過幾何算法解決「文字缺漏」與「重複渲染」並存的技術挑戰。1.1 研究背景與問題陳述隨著數字化轉型的加速，傳統的光學字符識別（OCR）技術已無法滿足對非結構化文檔的處理需求。傳統 OCR 僅輸出無序的文字流或簡單的行坐標，缺乏對段落、表格、標題及閱讀順序的認知。為此，版面分析（Layout Analysis）技術應運而生，旨在將文檔分割為具有語義標籤的區域（Regions）。然而，實際應用數據顯示，版面分析模型（如 PP-StructureV3 中的 RT-DETR 或 PicoDet）在追求高層次語義劃分時，往往會忽略非標準化的文本元素，例如頁眉頁腳、邊欄註釋、浮水印編號或散落在圖表周圍的微小文字 1。相比之下，專注於文字檢測的 PP-OCRv5 模型（基於 DBNet++ 與 SVTR）則展現出極高的召回率，幾乎能捕捉圖像中的每一個像素級文字 1。用戶提出的核心問題在於：能否利用 OCR 的高完整性來修補 Structure 的結構化結果？ 具體而言，即先執行完整的 OCR 讀取，再執行完整的版面分析，然後以版面分析的結果為骨架，將 OCR 檢測到但版面分析遺漏的文字「填充」進去，同時必須通過算法確保內容不發生重複渲染。1.2 技術路線的可行性分析本研究確認該混合技術路線不僅理論可行，且是目前工業界解決複雜文檔解析問題的最佳實踐之一 3。PaddleX 3.0 作為飛槳的全流程開發工具，提供了模塊化的流水線（Pipeline）設計，允許開發者獲取中間層結果 3。PP-StructureV3 的流水線設計本身即包含了一個全局 OCR 的步驟。在標準運行模式下，PP-StructureV3 會調用 OCR 模型對整圖進行預測，然後將預測結果指派給各個版面區域。然而，為了實現更精細的控制，用戶提出的「分別執行、後端融合」策略能提供更高的靈活性，特別是允許針對 OCR 和版面分析分別微調參數（如 det_db_thresh 或 layout_nms_threshold）6。1.3 報告結構導引本報告將分為八個核心章節進行論述：第二章 將深度解構 PaddleOCR v5 與 PP-StructureV3 的模型架構與輸出特性，從神經網絡層面解釋為何會產生「數據缺漏」。第三章 詳述 PaddleX 3.0 的流水線機制及數據接口，分析 JSON 輸出格式中的關鍵字段。第四章 提出「幾何過濾算法（Geometric Filtering Algorithm）」，這是解決重複渲染問題的核心數學模型，涵蓋 IoU 與 IoA 的計算原理。第五章 探討工程實現，包括 Python 代碼邏輯、Shapely 庫的應用及空間索引優化。第六章 針對表格、公式及印章等特殊元素的處理策略進行邊界條件分析。第七章 分析閱讀順序恢復（Reading Order Recovery）算法在混合數據源下的適配問題。第八章 總結與展望。第二章 模型架構解析：PP-OCRv5 與 PP-StructureV3 的技術異質性要設計有效的融合算法，必須首先理解兩個核心組件——PP-OCRv5 與 PP-StructureV3——在設計哲學、損失函數及訓練數據上的根本差異。這種異質性正是導致兩者輸出結果不一致（Inconsistency）的根源，也是混合架構存在的必要性所在。2.1 PP-OCRv5：極致召回的文字捕獲引擎PP-OCRv5 是 PaddleOCR 體系中的最新一代通用文字識別系統，其設計目標是「所見即所得」，即盡可能檢測並識別圖像中存在的所有文字痕跡，而不論其語義重要性如何 1。2.1.1 檢測模塊：DBNet++ 與幾何感知PP-OCRv5 的檢測端通常採用 DBNet（Differentiable Binarization）及其改進版本。這類算法的核心優勢在於其對極限場景的適應性。多尺度特徵融合： 通過特徵金字塔網絡（FPN），模型能夠同時檢測佔據半個頁面的大標題和角落裡僅有幾個像素高的頁碼。二值化邊界預測： DBNet 通過預測二值化圖和閾值圖，能夠精確分割出形狀不規則的文字區域，這意味著即使是傾斜、彎曲或密集的文本行也能被獨立檢測出來 8。高召回率特性： 由於訓練數據涵蓋了自然場景文本（Scene Text），PP-OCRv5 對「噪聲文字」極為敏感。對於文檔處理而言，這是一把雙刃劍：它能識別出水印、頁眉、甚至紙張污漬形成的文字狀紋理，但它無法區分這些文字是正文的一部分還是無關的干擾 。2.1.2 識別模塊：SVTR 與視覺變換器在識別端，PP-OCRv5 引入了 SVTR（Scene Text Recognition with a Vision Transformer）架構 。不同於傳統的 CRNN（CNN+RNN），SVTR 利用 Transformer 的注意力機制處理序列字符。上下文感知： 即使單個字符模糊，模型也能根據上下文推斷出正確內容。多語言統一： v5 版本實現了單一模型支持中、英、日等多種語言，這對於處理包含混合語言的複雜文檔（如引用外文文獻的學術論文）至關重要 3。總結： PP-OCRv5 輸出的是一組無序的、細粒度的四邊形框（Bounding Boxes）及其對應的文本內容。它不知道「段落」的概念，只知道「文本行」。2.2 PP-StructureV3：語義導向的版面重構引擎PP-StructureV3 的目標則完全不同。它不僅僅是「看見」文字，更是要「理解」文檔的視覺結構。它試圖將像素矩陣轉化為類似 DOM 樹的邏輯結構 3。2.2.1 版面分析模型：RT-DETR 與 PicoDetPP-StructureV3 的核心是版面區域檢測模型。在 v3 版本中，引入了基於 RT-DETR（Real-Time DEtection TRansformer）的高精度模型和基於 PicoDet 的輕量級模型 。類別定義： 這些模型被訓練去識別特定的語義類別，包括：標題（Title）、文本（Text）、表格（Table）、圖像（Figure）、圖像標題（Figure Caption）、表格標題（Table Caption）、頁眉（Header）、頁腳（Footer）、公式（Equation）及參考文獻（Reference）2。區域聚合： 與 OCR 不同，版面分析模型傾向於將視覺上聚集的文本行合併為一個大的檢測框（Block）。例如，一個包含十行文字的段落會被檢測為一個單一的 Text 框，而不是十個獨立的行框。漏檢機制（The Missing Gap）： 這是用戶遇到問題的關鍵。版面分析模型的訓練數據集（如 CDLA, PubLayNet）通常經過人工清洗，標註者可能忽略了一些非標準元素（如邊緣的批註、裝飾性文字）。因此，當模型在推理時遇到這些不屬於預定義「正文」範疇的文字時，往往會將其視為背景而忽略。這就解釋了為什麼「OCR 的完整度一定會大於 Structure」3。2.2.2 表格與公式識別子流水線PP-StructureV3 不僅檢測區域，還包含專門的子模型來處理特定區域。表格識別（SLANet）： 當檢測到 Table 區域時，會裁剪該區域並送入表格識別模型（如 SLANet），輸出 HTML 源碼 3。問題： 表格識別模型有時會漏掉單元格內的文字，或者在複雜嵌套表格中產生結構錯誤。2.3 混合架構的理論基礎用戶的提議實際上是構建一個 「互補型集成系統（Complementary Ensemble System）」。基底（Base）： 使用 PP-StructureV3 的結果作為文檔的「骨架（Skeleton）」。這保證了文檔的邏輯結構（段落、表格、閱讀順序）是正確的，便於後續轉換為 Markdown 或 Word。增強（Augmentation）： 使用 PP-OCRv5 的結果作為「候選池（Candidate Pool）」。過濾（Filtering）： 通過幾何比對，從候選池中剔除那些已經被骨架包含的元素。注入（Injection）： 將剩餘的 OCR 元素（即骨架遺漏的部分）作為「游離元素（Floating Elements）」注入到文檔結構中。這種架構在理論上能夠達到 100% 的信息召回率，同時保持 90% 以上的結構化準確率，是處理複雜 PDF 的最優解。第三章 PaddleX 流水線機制與數據接口深度剖析為了實現上述理論架構，我們需要深入理解 PaddleX 3.0 的工程實現機制，特別是其數據輸入輸出接口。PaddleX 是百度推出的全流程開發工具，它對 PaddleOCR 進行了封裝，提供了更為統一和標準化的 Pipeline API 3。3.1 PaddleX Pipeline 的初始化與配置在 Python 環境中，PP-StructureV3 和 PP-OCRv5 可以作為獨立的 Pipeline 對象被調用。根據 和 13，初始化代碼通常如下：Pythonfrom paddleocr import PPStructureV3, PaddleOCR - -# 初始化 Structure 流水線 -structure_engine = PPStructureV3( - lang='ch', - show_log=True, - use_orientation_classify=True, - image_orientation=True -) - -# 初始化 OCR 流水線（用於獲取全量文字） -# 注意：這裡顯式使用 PaddleOCR 類來獲取 v5 的能力 -ocr_engine = PaddleOCR( - use_angle_cls=True, - lang="ch", - ocr_version="PP-OCRv5" -) -關鍵配置分析：recovery=True：在 PP-StructureV3 中啟用此選項至關重要，因為它會觸發閱讀順序恢復模塊，並生成版面恢復所需的輔助信息 14。use_pdf2docx_api：如果設置為 True，系統可能會嘗試直接解析 PDF 內部的文字層。對於掃描件或複雜 PDF，建議設置為 False 以強制使用 OCR 視覺模型，這樣能保證 OCR 結果與視覺結果的一致性 15。3.2 數據輸出結構解析：JSON 與 Dict理解 PaddleX 的輸出格式是實現「結果比較」的前提。根據 16，PP-StructureV3 的預測結果是一個包含豐富信息的字典（Dict）。3.2.1 PP-StructureV3 輸出對象 (res)當調用 pipeline.predict(img) 後，返回的 res 對象通常包含以下關鍵字段：字段鍵名 (Key)數據類型描述用途layout_det_resList版面檢測結果提供文檔的結構骨架（Text, Table, Figure 等區域的坐標）。overall_ocr_resList全局 OCR 結果關鍵字段。包含整張圖的所有文字檢測框和識別內容。res (或 regions)List整合後的區域結果這是經過內部匹配後的結果，每個區域內包含了對應的 OCR 文字。table_htmlString表格 HTML 代碼僅在 type='Table' 的區域中存在。特別注意： 18 指出，在某些版本的 PaddleX 中，overall_ocr_res 可能是一個獨立的鍵，與 layout_det_res 並列。這個 overall_ocr_res 實際上就是我們需要的「完整 OCR 讀取結果」。這意味著，用戶不需要顯式調用兩次模型（一次 OCR，一次 Structure）。PP-StructureV3 內部已經執行了全圖 OCR。我們可以從 Structure 的返回結果中直接提取出 overall_ocr_res 作為我們的「全集」，提取 layout_det_res 作為「子集結構」，然後在內存中進行比對。這將極大節省推理時間（Inference Time），避免重複計算 。然而，如果用戶希望使用特定參數配置的 OCR（例如調整了 det_db_thresh 以捕獲更淡的文字），則顯式運行獨立的 PaddleOCR 實例是必要的。在這種情況下，我們將有兩個獨立的結果集：Set A (Structure): 來自 structure_engine 的版面區域列表。Set B (OCR): 來自 ocr_engine 的文字行列表。3.3 數據結構的幾何表示為了進行比對，我們必須將這兩個集合中的「位置信息」標準化。OCR 結果： 通常為四點坐標 [[x1, y1], [x2, y2], [x3, y3], [x4, y4]]。這是一個多邊形（Polygon），可能是矩形也可能是傾斜的四邊形。Structure 結果： 通常為 [x_min, y_min, x_max, y_max] 的軸對齊矩形（Axis-Aligned Bounding Box, AABB），但在某些複雜場景下也可能是四點坐標。數據標準化策略： 在 Python 腳本中，建議統一將所有坐標轉換為 shapely.geometry.Polygon 對象。這將為後續的交併比計算提供強大的數學支持 19。第四章 幾何融合算法：解決重複渲染的核心數學模型本章是解決用戶問題的核心技術部分。如何判斷一個 OCR 檢測到的文字行是否已經「包含」在某個版面區域（如段落或表格）中？簡單的中心點匹配往往不夠精確，特別是在文字行跨越區域邊界或區域重疊的情況下。我們需要引入 IoA (Intersection over Area) 的概念。4.1 IoU 與 IoA 的區別與應用在目標檢測中，常用的指標是 IoU (Intersection over Union)：$$IoU(A, B) = \frac{Area(A \cap B)}{Area(A \cup B)}$$IoU 用於衡量兩個框的重合程度，通常用於判斷兩個預測框是否指向同一個物體。然而，在我們的場景中，關係是不對稱的。OCR 文字行（小框，記為 $B_{ocr}$）通常位於版面區域（大框，記為 $B_{layout}$）的內部。我們關心的是「$B_{ocr}$ 是否被 $B_{layout}$ 包含」。如果使用 IoU，由於 $B_{layout}$ 面積很大，IoU 數值會非常小，無法作為判斷依據。因此，我們必須使用 IoA (Intersection over Area)，具體指的是 Intersection over OCR Area：$$IoA(B_{ocr}, B_{layout}) = \frac{Area(B_{ocr} \cap B_{layout})}{Area(B_{ocr})}$$這個公式計算的是：OCR 框的面積中，有多少比例落在了版面區域框的內部。如果 $IoA \approx 1.0$：表示 OCR 文字行完全在版面區域內。結論：忽略該 OCR 結果（因為 Structure 結果中已經包含了它）。如果 $IoA \approx 0$：表示 OCR 文字行完全在版面區域外。結論：保留該 OCR 結果（這是 Structure 遺漏的文字）。如果 $0 < IoA < 1.0$：表示部分重疊。這通常發生在邊界處。我們需要設定一個閾值（Threshold）。4.2 融合算法邏輯設計為了實現「結構優先，OCR 補漏」，我們設計如下的算法流程：輸入：Layout_List: 版面分析得到的區域列表（包含 Text, Table, Image 等）。OCR_List: 全局 OCR 得到的所有文字行列表。輸出：Final_Render_List: 用於最終渲染的元素列表，包含所有的 Layout 元素和補充的 OCR 元素。算法步驟：初始化： 將 Layout_List 中的所有元素加入 Final_Render_List。構建空間索引（可選但推薦）： 為了加速查詢，使用 R-tree 將 Layout_List 的邊界框建立索引 21。遍歷過濾： 對 OCR_List 中的每一個元素 $T_{ocr}$ 進行檢查：設標記 is_redundant = False。遍歷 Layout_List 中的每一個區域 $R_{layout}$（或通過 R-tree 查詢相交的區域）。計算 $IoA = \frac{Area(T_{ocr} \cap R_{layout})}{Area(T_{ocr})}$。判定邏輯：若 $R_{layout}$ 的類型是 Text, Title, Header, Footer, List：若 $IoA > 0.6$（閾值），則判定 $T_{ocr}$ 為冗餘，設置 is_redundant = True，並跳出內層循環。若 $R_{layout}$ 的類型是 Table：表格區域的處理較為敏感。通常表格識別模型會重構表格內容。若 OCR 文字落在表格內，直接疊加會破壞表格結構。因此，通常若 $IoA > 0.1$（更嚴格的閾值），即視為冗餘。若 $R_{layout}$ 的類型是 Figure/Image：這取決於用戶需求。如果用戶希望提取圖片中的文字（如圖表中的數據點），則即使 $IoA$ 很高，也可以判定為不冗餘（即保留 OCR）。但通常為了版面整潔，Structure 往往會忽略圖中文字，因此這裡可以根據配置決定。結果收集： 若內層循環結束後，is_redundant 仍為 False，則說明該文字行是 Structure 遺漏的。將 $T_{ocr}$ 標記為 Floating Text（浮動文本），並加入 Final_Render_List。4.3 閾值的選擇與調優閾值的選擇至關重要。閾值過高（如 0.9）： 要求 OCR 框幾乎完全在 Layout 框內才算重複。如果 Layout 框預測得稍微小了一點（邊界收縮），導致 OCR 框露出一部分，算法會錯誤地認為這是新文字並保留它，導致重複渲染（Ghosting Effect），即正文文字旁邊又出現了一遍同樣的文字。閾值過低（如 0.1）： 只要有一點點重疊就刪除 OCR。這可能導致邊緣處的獨立註釋被錯誤刪除。經驗推薦值： 對於 Text/Paragraph 區域，建議設置 $IoA \in [0.5, 0.7]$。這能容忍版面檢測框的輕微誤差，同時有效區分獨立文本。第五章 工程實現策略：Python 代碼與庫的整合在實際的 Python 開發中，我們需要結合 paddleocr、shapely 和 numpy 來實現上述邏輯。以下是詳細的實現代碼結構分析。5.1 環境準備與依賴庫除 PaddleOCR 外，必須安裝幾何處理庫：Bashpip install shapely rtree -shapely 用於多邊形運算，rtree 用於空間索引（對於大頁面或批量處理非常有效）。5.2 核心融合函數實現以下代碼展示了如何利用 Shapely 庫實現高精度的過濾邏輯 19：Pythonfrom shapely.geometry import Polygon - -def calculate_ioa(ocr_poly, layout_poly): - """計算 Intersection over OCR Area""" - if not ocr_poly.intersects(layout_poly): - return 0.0 - try: - intersection_area = ocr_poly.intersection(layout_poly).area - ocr_area = ocr_poly.area - if ocr_area == 0: return 0.0 - return intersection_area / ocr_area - except Exception as e: - # 處理幾何拓撲錯誤 - return 0.0 - -def merge_structure_and_ocr(structure_res, ocr_res, ioa_thresh=0.6): - """ - 輸入: - structure_res: PP-StructureV3 的 layout_det_res 列表 - ocr_res: PaddleOCR 的全量識別結果 - - 輸出: - merged_list: 包含 layout 區域和補漏 OCR 的混合列表 - """ - - # 1. 將 Layout 區域轉換為 Shapely Polygon 對象，提升後續計算效率 - layout_polys = - for region in structure_res: - bbox = region['bbox'] # 假設格式為 [x1, y1, x2, y2] - # 構建矩形 Polygon - poly = Polygon([(bbox, bbox), (bbox, bbox), - (bbox, bbox), (bbox, bbox)]) - layout_polys.append({ - 'poly': poly, - 'type': region['label'], - 'data': region - }) - - final_items = - # 先加入所有的 Layout 元素 - for item in layout_polys: - final_items.append(item['data']) - - # 2. 遍歷 OCR 結果進行過濾 - for line in ocr_res: - points = line # [[x1,y1], [x2,y2], [x3,y3], [x4,y4]] - text = line - ocr_poly = Polygon(points) - - is_covered = False - for layout_item in layout_polys: - l_poly = layout_item['poly'] - l_type = layout_item['type'] - - # 計算 IoA - ioa = calculate_ioa(ocr_poly, l_poly) - - # 針對不同類型的動態閾值策略 - current_thresh = ioa_thresh - if l_type == 'table': - current_thresh = 0.1 # 表格區域採用嚴格過濾 - elif l_type == 'figure': - current_thresh = 0.8 # 圖片區域允許更多容錯（或者設為 1.0 強制保留） - - if ioa > current_thresh: - is_covered = True - break - - if not is_covered: - # 這是 Structure 遺漏的文字 - # 將其包裝為類似 Structure 的格式 - new_item = { - 'type': 'text', # 或者標記為 'floating_text' - 'bbox': [min(p for p in points), min(p for p in points), - max(p for p in points), max(p for p in points)], - 'res': [{'text': text, 'confidence': line}], - 'is_patch': True # 標記這是補丁數據 - } - final_items.append(new_item) - - return final_items -5.3 數據結構的標準化與清洗在實際操作中，PP-OCRv5 返回的坐標通常是浮點數，且可能包含負值（如果檢測框超出圖像邊界）。在生成 Shapely 對象前，必須進行數據清洗：坐標取整與裁剪： 將坐標限制在 [0, 0, image_width, image_height] 範圍內。無效多邊形處理： 檢測出的四邊形可能存在自相交（Self-intersection），這會導致 Shapely 報錯。需使用 ocr_poly.buffer(0) 技巧來修復無效多邊形 19。第六章 特殊元素與邊界條件的處理策略在複雜版面中，表格、印章及跨頁元素是導致解析失敗的主要原因。本章針對這些特殊情況提出具體的處理策略。6.1 表格（Table）的衝突解決表格是文檔中最複雜的元素。PP-StructureV3 內置的表格識別模型會重建表格結構（HTML），這通常比單純的 OCR 文字拼接要好得多。問題： 有時表格識別模型會丟失單元格內容，或者 OCR 誤將表格線識別為「1」或「I」。策略：信任優先級： 默認信任表格識別模型。凡是落在 Table 區域內的 OCR 結果，一律過濾掉，避免破壞 HTML 結構。容錯回退： 如果表格區域內的 OCR 文字數量遠多於表格識別模型提取出的文字數量（例如多出 50%），則可能意味著表格識別失敗。此時應降級處理：丟棄表格結構，僅保留 OCR 文字，將其視為普通段落，以保證信息不丟失。6.2 圖片（Figure）中的文字學術論文或技術報告中的圖表往往包含軸標籤、圖例等文字。現狀： PP-StructureV3 的 Figure 區域通常只輸出圖片裁剪圖，不包含文字。用戶需求： 如果用戶需要全文檢索，圖中文字不能丟。實現： 在過濾算法中，對 Figure 類型的區域進行特殊處理。即使 $IoA$ 很高，也可以選擇不刪除該 OCR 結果，而是將其標記為 Figure Text 並在渲染時將其放置在圖片下方作為註釋，或者作為圖片的 alt 屬性。6.3 閱讀順序的重構（Reading Order Recovery）當我們將「補漏」的 OCR 文字加入 Final_Render_List 後，列表的順序是混亂的。直接渲染會導致文檔邏輯跳躍。必須重新執行閱讀順序排序算法。XY-Cut 算法： 這是文檔分析中最經典的排序算法。它遞歸地將頁面在水平或垂直投影的空隙處切分。融合排序：將所有元素（原有的 Layout 塊 + 新增的 OCR 塊）視為同等地位的節點。計算每個節點的中心點 $(C_x, C_y)$。根據文檔類型（單欄或多欄）應用啟發式排序規則。對於單欄文檔，簡單的 sorted(items, key=lambda k: (k.y, k.x))（從上到下，從左到右）通常足夠。對於多欄文檔，必須使用 PaddleX 內置的 sorted_layout_boxes 函數 3。該函數能夠處理複雜的列切分邏輯。重要的是，我們必須確保新加入的 OCR 塊的數據結構與 sorted_layout_boxes 函數要求的輸入格式完全一致。第七章 性能評估與優化建議引入幾何運算和雙重模型推斷會增加系統的計算負擔。本章分析性能影響並提供優化建議。7.1 計算複雜度分析推斷時間： 如果採用「一次推斷，提取兩份數據」的策略（即僅運行 StructureV3 並提取內部 OCR 結果），推斷時間幾乎沒有增加。如果分別運行兩個模型，時間將翻倍。後處理時間： 幾何過濾算法的複雜度為 $O(N \times M)$，其中 $N$ 為版面區域數（通常 < 50），$M$ 為 OCR 行數（可能 > 2000）。在 Python 中，計算 100,000 次多邊形交集大約需要 0.1 - 0.5 秒，這相對於模型推斷時間（通常數秒）是可以忽略不計的 24。7.2 空間索引優化對於極端密集的文檔（如報表、電話簿），$M$ 可能非常大。此時應使用 R-tree 索引。Pythonfrom rtree import index -idx = index.Index() -# 將 Layout 區域插入索引 -for i, region in enumerate(layout_regions): - idx.insert(i, region['bbox']) - -# 查詢時僅計算候選區域 -candidates = list(idx.intersection(ocr_box)) -這將複雜度降低至 $O(M \log N)$，確保系統在大規模批處理時的穩定性。7.3 重複渲染的極端案例與對策儘管有 IoA 過濾，仍可能出現視覺上的重複。這通常是因為 PaddleOCR 的檢測框比實際文字大（包含背景），或者版面區域比實際內容小。對策 - 邊界收縮（Shrinking）： 在計算 IoA 之前，將 OCR 框向內收縮 1-2 像素，或者將版面區域向外擴張（Buffer/Dilate）5 像素。這增加了「被包含」的概率，能有效減少邊緣處的重複渲染 25。第八章 總結與未來展望本研究報告對基於 PaddleX 的 PP-OCRv5 與 PP-StructureV3 混合解析架構進行了全方位的技術論證。8.1 研究結論架構可行性： 用戶提出的「先 OCR、後 Structure、對比補全」的思路在技術上是完全可行的，且是解決複雜 PDF 解析中信息丟失問題的有效手段。核心價值： 該方案結合了 PP-OCRv5 的高召回率（>99%）和 PP-StructureV3 的高結構化能力，通過幾何約束算法消除了兩者之間的冗餘，實現了文檔解析的「帕累托最優」。關鍵技術點： 成功的關鍵在於 IoA (Intersection over Area) 算法的正確實現，以及對表格、圖片等特殊元素的差異化閾值設置。8.2 工程建議優先使用單一流水線： 建議優先嘗試從 PP-StructureV3 的 overall_ocr_res 中獲取 OCR 數據，以節省計算資源。精細化閾值調優： 開發者應建立一個包含各類典型壞例（Bad Cases）的驗證集，通過自動化測試來尋找最佳的 IoA 閾值。數據結構對齊： 在將補漏數據注入渲染列表時，務必保證數據字段（bbox, text, type）與原始 Structure 輸出保持一致，以復用 PaddleX 的 recovery_to_docx 等後處理工具。綜上所述，該混合架構不僅能解決當前的文字缺漏問題，也為未來構建更智能的 RAG（檢索增強生成）知識庫提供了高質量的結構化數據基礎。這是一條值得投入的工程實踐路徑。 \ No newline at end of file