feat: add storage cleanup mechanism with soft delete and auto scheduler
- Add soft delete (deleted_at column) to preserve task records for statistics - Implement cleanup service to delete old files while keeping DB records - Add automatic cleanup scheduler (configurable interval, default 24h) - Add admin endpoints: storage stats, cleanup trigger, scheduler status - Update task service with admin views (include deleted/files_deleted) - Add frontend storage management UI in admin dashboard - Add i18n translations for storage management 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
@@ -31,7 +31,7 @@ The OCR service SHALL generate both JSON and Markdown result files for completed
|
||||
- **AND** include enhanced structure from PP-StructureV3 or PyMuPDF
|
||||
|
||||
### Requirement: Task Detail View
|
||||
The frontend SHALL provide a dedicated page for viewing individual task details with processing track information and enhanced preview capabilities.
|
||||
The frontend SHALL provide a dedicated page for viewing individual task details with processing track information, enhanced preview capabilities, and file availability status.
|
||||
|
||||
#### Scenario: Navigate to task detail page
|
||||
- **WHEN** user clicks "View Details" button on task in Task History page
|
||||
@@ -61,6 +61,12 @@ The frontend SHALL provide a dedicated page for viewing individual task details
|
||||
- **AND** show bounding boxes overlay on preview
|
||||
- **AND** highlight different element types (headers, tables, lists) with distinct colors
|
||||
|
||||
#### Scenario: Display file unavailable status
|
||||
- **WHEN** task has `file_deleted=True`
|
||||
- **THEN** page SHALL show file unavailable indicator
|
||||
- **AND** download buttons SHALL be disabled or hidden
|
||||
- **AND** page SHALL display explanation that files were cleaned up
|
||||
|
||||
### Requirement: Results Page V2 Migration
|
||||
The Results page SHALL use V2 task-based APIs instead of V1 batch APIs.
|
||||
|
||||
@@ -117,3 +123,77 @@ The system SHALL maintain detailed processing history for tasks including track
|
||||
- **AND** provide track selection statistics
|
||||
- **AND** include performance metrics for each processing attempt
|
||||
|
||||
### Requirement: Soft Delete Tasks
|
||||
The system SHALL support soft deletion of tasks, marking them as deleted without removing database records to preserve usage statistics.
|
||||
|
||||
#### Scenario: User soft deletes a task
|
||||
- **WHEN** user calls DELETE on `/api/v2/tasks/{task_id}`
|
||||
- **THEN** system SHALL set `deleted_at` timestamp on the task record
|
||||
- **AND** system SHALL NOT delete the actual files
|
||||
- **AND** system SHALL NOT remove the database record
|
||||
- **AND** subsequent user queries SHALL NOT return this task
|
||||
|
||||
#### Scenario: Preserve statistics after soft delete
|
||||
- **WHEN** a task is soft deleted
|
||||
- **THEN** admin statistics endpoints SHALL continue to include this task's metrics
|
||||
- **AND** translation token counts SHALL remain in cumulative totals
|
||||
- **AND** processing time statistics SHALL remain accurate
|
||||
|
||||
### Requirement: File Cleanup Scheduler
|
||||
The system SHALL automatically clean up old files while preserving database records for statistics tracking.
|
||||
|
||||
#### Scenario: Scheduled file cleanup
|
||||
- **WHEN** cleanup scheduler runs (configurable interval, default daily)
|
||||
- **THEN** system SHALL identify tasks where files can be deleted
|
||||
- **AND** system SHALL retain newest N files per user (configurable, default 50)
|
||||
- **AND** system SHALL delete actual files from disk for older tasks
|
||||
- **AND** system SHALL set `file_deleted=True` on cleaned tasks
|
||||
- **AND** system SHALL NOT delete any database records
|
||||
|
||||
#### Scenario: File retention per user
|
||||
- **WHEN** user has more than `max_files_per_user` tasks with files
|
||||
- **THEN** cleanup SHALL delete files for oldest tasks exceeding the limit
|
||||
- **AND** cleanup SHALL preserve the newest `max_files_per_user` task files
|
||||
- **AND** task ordering SHALL be by `created_at` descending
|
||||
|
||||
#### Scenario: Manual cleanup trigger
|
||||
- **WHEN** admin calls POST `/api/v2/admin/cleanup/trigger`
|
||||
- **THEN** system SHALL immediately run the cleanup process
|
||||
- **AND** return summary of files deleted and space freed
|
||||
|
||||
### Requirement: Admin Task Visibility
|
||||
Admin users SHALL have full visibility into all tasks including soft-deleted and file-cleaned tasks.
|
||||
|
||||
#### Scenario: Admin lists all tasks
|
||||
- **WHEN** admin calls GET `/api/v2/admin/tasks`
|
||||
- **THEN** response SHALL include all tasks from all users
|
||||
- **AND** response SHALL include soft-deleted tasks
|
||||
- **AND** response SHALL include tasks with deleted files
|
||||
- **AND** each task SHALL indicate its deletion status
|
||||
|
||||
#### Scenario: Filter admin task list
|
||||
- **WHEN** admin calls GET `/api/v2/admin/tasks` with filters
|
||||
- **THEN** `include_deleted=false` SHALL exclude soft-deleted tasks
|
||||
- **AND** `include_files_deleted=false` SHALL exclude file-cleaned tasks
|
||||
- **AND** `user_id={id}` SHALL filter to specific user's tasks
|
||||
|
||||
#### Scenario: View storage usage statistics
|
||||
- **WHEN** admin calls GET `/api/v2/admin/storage/stats`
|
||||
- **THEN** response SHALL include total storage used
|
||||
- **AND** response SHALL include per-user storage breakdown
|
||||
- **AND** response SHALL include count of tasks with/without files
|
||||
|
||||
### Requirement: User Task Isolation
|
||||
Regular users SHALL only see their own tasks and soft-deleted tasks SHALL be hidden from their view.
|
||||
|
||||
#### Scenario: User lists own tasks
|
||||
- **WHEN** authenticated user calls GET `/api/v2/tasks`
|
||||
- **THEN** response SHALL only include tasks owned by that user
|
||||
- **AND** response SHALL NOT include soft-deleted tasks
|
||||
- **AND** response SHALL include tasks with deleted files (showing file unavailable status)
|
||||
|
||||
#### Scenario: User cannot access other user's tasks
|
||||
- **WHEN** user attempts to access task owned by another user
|
||||
- **THEN** system SHALL return 404 Not Found
|
||||
- **AND** system SHALL NOT reveal that the task exists
|
||||
|
||||
|
||||
Reference in New Issue
Block a user