Commit Graph

45 Commits

Author SHA1 Message Date
egg
fb92579331 feat: harden long-range batch queries with redis+parquet caching 2026-03-02 21:04:18 +08:00
egg
2568fd836c feat(reject-history): multi-pareto 3×2 grid with cross-filter linkage
Replace single-dimension Pareto dropdown with 6 simultaneous Pareto charts
(不良原因, PACKAGE, TYPE, WORKFLOW, 站點, 機台) in a responsive 3-column grid.
Clicking items in one Pareto cross-filters the other 5 (exclude-self logic),
and the detail table applies all dimension selections with AND logic.

Backend:
- Add batch-pareto endpoint (cache-only, no Oracle queries)
- Add _apply_cross_filter() with exclude-self pattern
- Extend view/export endpoints for multi-dimension sel_* params

Frontend:
- New ParetoGrid.vue wrapping 6 ParetoSection instances
- Simplify ParetoSection: remove dimension dropdown, keep TOP20 toggle
- Replace single-dimension state with paretoSelections reactive object
- Adaptive x-axis labels (font size, rotation, hideOverlap) for compact grid
- Responsive grid: 3-col desktop, 2-col tablet, 1-col mobile

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 14:43:35 +08:00
egg
e83d8e1a36 feat(reject-history): fix Pareto datasources, multi-select filtering, and UX enhancements
- Fix dimension Pareto datasources: PJ_TYPE/PRODUCTLINENAME from DW_MES_CONTAINER,
  WORKFLOWNAME from DW_MES_LOTWIPHISTORY via WIPTRACKINGGROUPKEYID, EQUIPMENTNAME
  from LOTREJECTHISTORY only (no WIP fallback), workcenter dimension uses WORKCENTER_GROUP
- Add multi-select Pareto click filtering with chip display and detail list integration
- Add TOP 20 display scope selector for TYPE/WORKFLOW/機台 dimensions
- Pass Pareto selection (dimension + values) through to list/export endpoints
- Enable TRACE_WORKER_ENABLED=true by default in start_server.sh and .env.example
- Archive reject-history-pareto-datasource-fix and reject-history-pareto-ux-enhancements
- Update reject-history-api and reject-history-page specs with new Pareto behaviors

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 13:23:16 +08:00
egg
cdf6f67c54 feat: relax query limits for query-tool and mid-section-defect 2026-03-02 08:20:55 +08:00
egg
c6f982ae50 test: fix E2E/integration failures, add trace pipeline E2E, remove dead migration tests
- Fix 6 Playwright strict-mode violations in query tool E2E (v-show dual-tab selectors)
- Update 5 resource history E2E tests for POST /query API restructure
- Add 22 trace pipeline E2E tests: admission control, async job queue, NDJSON streaming
- Fix 3 health endpoint tests: add circuit breaker + route cache mocks
- Fix WIP integration tests: load .env before DB module import for --run-integration
- Remove 4 dead migration test files (20 permanently-skipped tests)

Final: 1101 unit + 10 integration + 121 E2E + 23 stress = 1255 passed, 0 failed

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 08:33:07 +08:00
egg
dbe0da057c feat(trace-pipeline): memory triage, async job queue, and NDJSON streaming
Three proposals addressing the 2026-02-25 trace pipeline OOM crash (114K CIDs):

1. trace-events-memory-triage: fetchmany iterator (read_sql_df_slow_iter),
   admission control (50K CID limit for non-MSD), cache skip for large queries,
   early memory release with gc.collect()

2. trace-async-job-queue: RQ-based async jobs for queries >20K CIDs,
   separate worker process with isolated memory, frontend polling via
   useTraceProgress composable, systemd service + deploy scripts

3. trace-streaming-response: chunked Redis storage (TRACE_STREAM_BATCH_SIZE=5000),
   NDJSON stream endpoint (GET /api/trace/job/<id>/stream), frontend
   ReadableStream consumer for progressive rendering, backward-compatible
   with legacy single-key storage

All three proposals archived. 1101 tests pass, frontend builds clean.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 21:01:27 +08:00
egg
cbb943dfe5 feat(trace-pool-isolation): migrate event_fetcher/lineage_engine to slow connections + fix 51 test failures
Trace pipeline pool isolation:
- Switch event_fetcher and lineage_engine to read_sql_df_slow (non-pooled)
- Reduce EVENT_FETCHER_MAX_WORKERS 4→2, TRACE_EVENTS_MAX_WORKERS 4→2
- Add 60s timeout per batch query, cache skip for CID>10K
- Early del raw_domain_results + gc.collect() for large queries
- Increase DB_SLOW_MAX_CONCURRENT: base 3→5, dev 2→3, prod 3→5

Test fixes (51 pre-existing failures → 0):
- reject_history: WORKFLOW CSV header, strict bool validation, pareto mock path
- portal shell: remove non-existent /tmtt-defect route from tests
- conftest: add --run-stress option to skip stress/load tests by default
- migration tests: skipif baseline directory missing
- performance test: update Vite asset assertion
- wip hold: add firstname/waferdesc mock params
- template integration: add /reject-history canonical route

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 16:13:19 +08:00
egg
71c8102de6 feat: dataset cache for hold/resource history + slow connection migration
Two changes combined:

1. historical-query-slow-connection: Migrate all historical query pages
   to read_sql_df_slow with semaphore concurrency control (max 3),
   raise DB slow timeout to 300s, gunicorn timeout to 360s, and
   unify frontend timeouts to 360s for all historical pages.

2. hold-resource-history-dataset-cache: Convert hold-history and
   resource-history from multi-query to single-query + dataset cache
   pattern (L1 ProcessLevelCache + L2 Redis parquet/base64, TTL=900s).
   Replace old GET endpoints with POST /query + GET /view two-phase
   API. Frontend auto-retries on 410 cache_expired.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 13:15:02 +08:00
egg
cd061e0cfd test(msd): add unit tests for _build_detail_table structured output and CSV flatten
Covers task 14.3: verify UPSTREAM_MACHINES/UPSTREAM_MATERIALS list format,
WAFER_ROOT field, multi-reason row expansion, machine deduplication,
and CSV export flatten logic.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 11:28:28 +08:00
egg
86984cfeb1 feat: dimension pareto cache-based computation, filter propagation, and MSD events cache isolation
Reject History:
- Compute dimension pareto (package/type/workflow/workcenter/equipment) from
  cached DataFrame instead of re-querying Oracle per dimension change
- Propagate supplementary filters and trend date selection to dimension pareto
- Add staleness tracking to prevent race conditions on rapid dimension switches
- Add WORKFLOWNAME to detail and export outputs
- Fix button hover visibility with CSS specificity

MSD (製程不良追溯分析):
- Separate raw events caching from aggregation computation so changing
  loss_reasons uses EventFetcher per-domain cache (fast) and recomputes
  aggregation with current filters instead of returning stale cached results
- Exclude loss_reasons from MSD seed cache key since seed resolution does
  not use it, avoiding unnecessary Oracle re-queries
- Add suspect context panel, analysis summary, upstream station/spec filters
- Add machine bar click drill-down and filtered attribution charts

Query Tool:
- Support batch container_ids in lot CSV export (history/materials/rejects/holds)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 09:02:39 +08:00
egg
e37902861f fix(query-tool): export button hover visibility, jobs tab hide columns and export with txn history
- Increase CSS specificity for .btn-export to prevent portal-shell override on hover
- Remove RESOURCEID and CONTAINERIDS from jobs tab display columns
- Add lot_jobs_with_txn.sql joining JOB with JOBTXNHISTORY for complete export
- Route lot_jobs export through get_lot_jobs_with_history() for full transaction data

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 19:10:19 +08:00
egg
7fffa812a3 fix(query-tool): export all selected CIDs instead of single, fix hold detail float precision
Export was sending only one container_id while the detail table loaded
data for all selected CIDs (including subtree), causing "查無資料" errors.
Now sends container_ids array and uses batch service functions.

Also rounds hold detail age KPI cards to 1 decimal place.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 18:41:24 +08:00
egg
f14591c7dc feat(mid-section-defect): full-line bidirectional defect trace center with dual query mode
Transform /mid-section-defect from TMTT-only backward analysis into a full-line
bidirectional defect traceability center supporting all detection stations.

Key changes:
- Parameterized station detection: any workcenter group as detection station
- Bidirectional tracing: backward (upstream attribution) + forward (downstream reject rates)
- Dual query mode: date range OR LOT/工單/WAFER container-based seed resolution
- Multi-select filters for upstream station, equipment model (RESOURCEFAMILYNAME), and loss reasons
- Progressive 3-stage trace pipeline (seed-resolve → lineage → events) with streaming UI
- Equipment model lookup via resource cache instead of SPECNAME
- Session caching, auto-refresh, searchable MultiSelect with fuzzy matching
- Remove legacy tmtt-defect module (fully superseded)
- Archive openspec change artifacts

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 16:16:33 +08:00
egg
e5d7700b36 harden released pages and archive openspec change 2026-02-23 17:48:32 +08:00
egg
57a0b780b1 feat(query-tool): align lineage merge semantics and add tree exports 2026-02-23 07:07:36 +08:00
egg
3dc7886c90 fix(query-tool): show hold lot id via containername and align export 2026-02-22 18:32:07 +08:00
egg
bfec6b2293 feat(query-tool): align lot reject detail with reject-history layout 2026-02-22 18:17:44 +08:00
egg
97872cca97 fix(query-tool): finalize raw-material tab/export and resolve ORA-00918 2026-02-22 17:59:27 +08:00
egg
9890586191 feat(query-tool): align lineage model and tighten timeline mapping 2026-02-22 17:36:47 +08:00
egg
5e5cc487ac feat: archive filter strategy change and optimize reject-history filters 2026-02-22 12:50:05 +08:00
egg
7bf9e33cd5 feat: polish reject history UI and enhance WIP filter interactions 2026-02-22 11:54:51 +08:00
egg
9687deb9ad feat(query-tool): split serial reverse trace tab and use reverse lineage 2026-02-22 09:52:06 +08:00
egg
05d907ac72 feat(reject-history): ship report page and archive openspec change 2026-02-13 20:35:52 +08:00
egg
5b358d71c1 feat(query-tool): rewrite frontend with ECharts tree, multi-select, and modular composables
Replace the monolithic useQueryToolData composable and nested Vue component tree
with a modular architecture: useLotResolve, useLotLineage, useLotDetail, and
useEquipmentQuery. Introduce ECharts TreeChart (LR orthogonal layout) for lot
lineage visualization with multi-select support, subtree expansion, zoom/pan,
and serial number normalization. Add unified LineageEngine backend with split
descendant traversal and leaf serial number queries. Archive the query-tool-rewrite
openspec change and sync delta specs to main.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-13 15:25:00 +08:00
egg
519f8ae2f4 feat(lineage): unified LineageEngine, EventFetcher, and progressive trace API
Introduce a unified Seed→Lineage→Event pipeline replacing per-page Python
BFS with Oracle CONNECT BY NOCYCLE queries, add staged /api/trace/*
endpoints with rate limiting and L2 Redis caching, and wire progressive
frontend loading via useTraceProgress composable.

Key changes:
- Add LineageEngine (split ancestors / merge sources / full genealogy)
  with QueryBuilder bind-param safety and batched IN clauses
- Add EventFetcher with 6-domain support and L2 Redis cache
- Add trace_routes Blueprint (seed-resolve, lineage, events) with
  profile dispatch, rate limiting, and Redis TTL=300s caching
- Refactor query_tool_service to use LineageEngine and QueryBuilder,
  removing raw string interpolation (SQL injection fix)
- Add rate limits and resolve cache to query_tool_routes
- Integrate useTraceProgress into mid-section-defect with skeleton
  placeholders and fade-in transitions
- Add lineageCache and on-demand lot lineage to query-tool
- Add TraceProgressBar shared component
- Remove legacy query-tool.js static script (3k lines)
- Fix MatrixTable package column truncation (.slice(0,15) removed)
- Archive unified-lineage-engine change, add trace-progressive-ui specs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 16:30:24 +08:00
egg
c38b5f646a feat(modernization): promote deferred routes to in-scope and unify page header styles
Promote /tables, /excel-query, /query-tool, /mid-section-defect from
deferred to full shell-governed in-scope routes with canonical redirects,
content contracts, governance artifacts, and updated CI gates.

Unify all page header gradients to #667eea → #764ba2 and h1 font-size
to 24px for visual consistency across all dashboard pages. Remove
Native Route-View dev annotations from job-query, excel-query, and
query-tool headers.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 13:20:06 +08:00
egg
7cb0985b12 feat(modernization): full architecture blueprint with hardening follow-up
Implement phased modernization infrastructure for transitioning from
multi-page legacy routing to SPA portal-shell architecture, plus
post-delivery hardening fixes for policy loading, fallback consistency,
and governance drift detection.

Key changes:
- Add route contract enrichment with scope/visibility/compatibility policies
- Canonical 302 redirects from legacy direct-entry to /portal-shell/ routes
- Asset readiness enforcement and runtime fallback retirement for in-scope routes
- Shared feature-flag helpers (env > config > default) replacing duplicated _to_bool
- Defensive copy for lru_cached policy payloads preventing mutation corruption
- Unified retired-fallback response helper across app and blueprint routes
- Frontend/backend route-contract cross-validation in governance gates
- Shell CSS token fallback values for routes rendered outside shell scope
- Local-safe .env.example defaults with production recommendation comments
- Legacy contract fallback warning logging and single-hop redirect optimization

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 11:26:02 +08:00
egg
8550f6dc3e fix(hold-history): align KPI cards with trend data, improve filters and UX across pages
Use Daily Trend as single source of truth for On Hold and New Hold KPI cards
instead of separate snapshot SQL queries, eliminating value mismatches. Fix
timezone bug in default date range (toISOString UTC offset), add 1st-of-month
fallback to previous month, replace Hold Type radio buttons with select dropdown,
reorder/relabel summary cards with 累計 prefix, add job-query MultiSelect for
equipment filter, and fix heatmap chart X-axis overlap with visualMap legend.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 08:13:07 +08:00
egg
1e7f8f4498 feat: finalize no-iframe portal shell route-view migration 2026-02-11 17:07:50 +08:00
egg
ccab10bee8 feat: finalize portal no-iframe migration baseline and archive change 2026-02-11 13:25:03 +08:00
egg
f90a8a57b4 fix(security): add table_name whitelist to prevent SQL injection in table query APIs
The /api/query_table and /api/get_table_columns endpoints accepted arbitrary
table_name values that were interpolated directly into SQL f-strings. Since
api_public is true, any unauthenticated user could exploit this. Now validates
table_name and time_field against TABLES_CONFIG before reaching the database.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-11 10:44:56 +08:00
egg
e2ce75b004 fix(hold): dedup equipment cache, fix portal iframe, improve Hold dashboards
- Equipment cache: add freshness gate so only 1 Oracle query per 5-min cycle
  across 4 gunicorn workers; sync worker waits before first refresh
- Portal: add frame-busting to prevent recursive iframe nesting
- Hold Overview: remove redundant TreeMap, add Product & Future Hold Comment
  columns to LotTable
- Hold History: switch list.sql JOIN from DW_MES_LOT_V (WIP snapshot) to
  DW_MES_CONTAINER (historical master) for reliable Product data; add
  Future Hold Comment column; fix comment truncation with hover tooltip
- Page status: reorganize drawer groupings

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-11 09:01:02 +08:00
egg
be22571421 feat(wip): preserve filters between Overview and Detail with thundering-herd fix
URL is now single source of truth for filter state (workorder, lotid,
package, type, status) across WIP Overview and Detail pages. Drill-down
carries all filters + status; back button dynamically reflects Detail
changes. Backend Detail API now supports pj_type filter parameter.

Harden concurrency: add pagehide abort for MPA navigation, double-check
locking on Redis JSON parse and snapshot build to prevent thread pool
saturation during rapid page switching. Fix watchdog setsid and PID
discovery. Fix test_realtime_equipment_cache RUNCARDLOTID field mismatch.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-11 07:22:48 +08:00
egg
9a4e08810b feat(hold-history): add Hold 歷史績效 Dashboard with trend, pareto, duration, and detail views
New independent report page based on DWH.DW_MES_HOLDRELEASEHISTORY providing
historical hold/release performance analysis. Includes daily trend with Redis
caching, reason Pareto with click-to-filter, duration distribution with
click-to-filter, multi-select record type filter (new/on_hold/released),
workcenter-group mapping via memory cache, and server-side paginated detail
table. All 32 backend tests passing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-10 18:03:08 +08:00
egg
8225863a85 feat(hold-overview): add Hold Lot Overview page with TreeMap, Matrix, and cascade filtering
Provide managers with a dedicated page to analyze hold lots across all stations.
Extends existing service functions (get_hold_detail_summary, get_hold_detail_lots,
get_wip_matrix) with optional parameters for backward compatibility, adds one new
function (get_hold_overview_treemap), and registers the page in the portal navigation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-10 13:02:24 +08:00
egg
af59031f95 feat(mid-section-defect): harden with distributed lock, rate limit, filter separation, abort, SQL classification and tests
Address 6 code review findings (P0-P3): add Redis distributed lock to prevent
duplicate Oracle pipeline on cold cache, apply rate limiting to 3 high-cost
routes, separate UI filter state from committed query state, add AbortController
for request cancellation, push workcenter group classification into Oracle SQL
CASE WHEN, and add 18 route+service tests. Also add workcenter group selection
to job-query equipment selector and rename button to "查詢".

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-10 09:32:14 +08:00
egg
720e190bc6 feat(resource): migrate resource-status and resource-history from Jinja2 to Vue 3 + Vite
Rewrite both resource pages (1,697 lines vanilla JS + 3,200 lines Jinja2 templates)
as Vue 3 SFC components. Extract resource-shared/ module with shared CSS, E10 status
constants, and HierarchyTable tree component. History page charts use vue-echarts,
Status page reuses useAutoRefresh composable with 5-minute interval.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-09 18:19:32 +08:00
egg
a2653b8139 feat(wip): migrate WIP trio pages from Jinja2 to Vue 3 + Vite
Migrate /wip-overview, /wip-detail, and /hold-detail (1,941 lines vanilla JS)
to Vue 3 SFC architecture. Extract shared CSS/constants/components to
wip-shared/. Switch Pareto charts to vue-echarts with autoresize. Replace
Jinja2 template injection with frontend URL params + constant classification
for Hold Detail. Add 10-min auto-refresh + AbortController to Hold Detail.
Remove three Jinja2 templates, update Flask routes to send_from_directory.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-09 16:39:20 +08:00
egg
dcbf6dcf1f feat(tables): migrate /tables page from Jinja2 to Vue 3 + Vite
Rewrite 237-line vanilla JS + Jinja2 template into Vue 3 SFC components
(App.vue, TableCatalog.vue, DataViewer.vue, useTableData composable).
Establishes apiPost POST request pattern for pure Vite pages. Removes
templates/index.html, updates Vite entry to HTML, and Flask route to
send_from_directory. Includes sql_fragments WHERE_CLAUSE escaping fix,
updated integration tests, and OpenSpec artifact archive.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-09 14:52:14 +08:00
egg
bf7285fb51 feat(qc-gate): add QC-GATE real-time LOT status report as first pure Vue 3 + Vite page
Introduce QC-GATE station monitoring with stacked bar chart and filterable LOT table,
using Vue 3 SFC + ECharts via npm. Establishes the pure Vite page architecture pattern
(no Jinja2) for future page migration. Also removes stale design files and README.mdj.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-09 13:06:59 +08:00
egg
9b1d2edc52 feat(portal): implement dynamic drawer/page navigation management
Replace hardcoded sidebar drawer configuration with admin-manageable
dynamic system. Extend page_status.json with drawer definitions and
page assignments, add drawer CRUD API endpoints, render portal sidebar
via Jinja2 loops, and extend /admin/pages UI with drawer management.
Fix multi-worker cache invalidation via mtime-based staleness detection.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-09 11:34:04 +08:00
egg
1e6d6dbd31 feat: complete dashboard-vite parity and fix portal health/csp regressions 2026-02-09 09:22:23 +08:00
beabigegg
cf194bc3a3 chore: finalize vite migration hardening and watchdog logging 2026-02-08 22:55:38 +08:00
beabigegg
c8e225101e chore: finalize vite migration hardening and archive openspec changes 2026-02-08 20:03:36 +08:00
beabigegg
b56e80381b chore: reinitialize project with vite architecture 2026-02-08 08:30:48 +08:00