feat(lineage): unified LineageEngine, EventFetcher, and progressive trace API
Introduce a unified Seed→Lineage→Event pipeline replacing per-page Python BFS with Oracle CONNECT BY NOCYCLE queries, add staged /api/trace/* endpoints with rate limiting and L2 Redis caching, and wire progressive frontend loading via useTraceProgress composable. Key changes: - Add LineageEngine (split ancestors / merge sources / full genealogy) with QueryBuilder bind-param safety and batched IN clauses - Add EventFetcher with 6-domain support and L2 Redis cache - Add trace_routes Blueprint (seed-resolve, lineage, events) with profile dispatch, rate limiting, and Redis TTL=300s caching - Refactor query_tool_service to use LineageEngine and QueryBuilder, removing raw string interpolation (SQL injection fix) - Add rate limits and resolve cache to query_tool_routes - Integrate useTraceProgress into mid-section-defect with skeleton placeholders and fade-in transitions - Add lineageCache and on-demand lot lineage to query-tool - Add TraceProgressBar shared component - Remove legacy query-tool.js static script (3k lines) - Fix MatrixTable package column truncation (.slice(0,15) removed) - Archive unified-lineage-engine change, add trace-progressive-ui specs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
egg
@@ -24,3 +24,20 @@ The system SHALL continue to maintain full-table cache behavior for `resource` a
|
||||
- **WHEN** cache update runs for `resource` or `wip`
|
||||
- **THEN** the updater MUST retain full-table snapshot semantics and MUST NOT switch these domains to partial-only cache mode
|
||||
|
||||
### Requirement: Mid-section defect genealogy SHALL use CONNECT BY instead of Python BFS
|
||||
The mid-section-defect genealogy resolution SHALL use `LineageEngine.resolve_full_genealogy()` (CONNECT BY NOCYCLE) instead of the existing `_bfs_split_chain()` Python BFS implementation.
|
||||
|
||||
#### Scenario: Genealogy cold query performance
|
||||
- **WHEN** mid-section-defect analysis executes genealogy resolution with cache miss
|
||||
- **THEN** `LineageEngine.resolve_split_ancestors()` SHALL be called (single CONNECT BY query)
|
||||
- **THEN** response time SHALL be ≤8s (P95) for ≥50 ancestor nodes
|
||||
- **THEN** Python BFS `_bfs_split_chain()` SHALL NOT be called
|
||||
|
||||
#### Scenario: Genealogy hot query performance
|
||||
- **WHEN** mid-section-defect analysis executes genealogy resolution with L2 Redis cache hit
|
||||
- **THEN** response time SHALL be ≤1s (P95)
|
||||
|
||||
#### Scenario: Golden test result equivalence
|
||||
- **WHEN** golden test runs with ≥5 known LOTs
|
||||
- **THEN** CONNECT BY output (`child_to_parent`, `cid_to_name`) SHALL be identical to BFS output for the same inputs
|
||||
|
||||
|
||||
24
openspec/specs/event-fetcher-unified/spec.md
Normal file
24
openspec/specs/event-fetcher-unified/spec.md
Normal file
@@ -0,0 +1,24 @@
|
||||
# event-fetcher-unified Specification
|
||||
|
||||
## Purpose
|
||||
TBD - created by archiving change unified-lineage-engine. Update Purpose after archive.
|
||||
## Requirements
|
||||
### Requirement: EventFetcher SHALL provide unified cached event querying across domains
|
||||
`EventFetcher` SHALL encapsulate batch event queries with L1/L2 layered cache and rate limit bucket configuration, supporting domains: `history`, `materials`, `rejects`, `holds`, `jobs`, `upstream_history`.
|
||||
|
||||
#### Scenario: Cache miss for event domain query
|
||||
- **WHEN** `EventFetcher` is called for a domain with container IDs and no cache exists
|
||||
- **THEN** the domain query SHALL execute against Oracle via `read_sql_df()`
|
||||
- **THEN** the result SHALL be stored in L2 Redis cache with key format `evt:{domain}:{sorted_cids_hash}`
|
||||
- **THEN** L1 memory cache SHALL also be populated (aligned with `core/cache.py` LayeredCache pattern)
|
||||
|
||||
#### Scenario: Cache hit for event domain query
|
||||
- **WHEN** `EventFetcher` is called for a domain and L2 Redis cache contains a valid entry
|
||||
- **THEN** the cached result SHALL be returned without executing Oracle query
|
||||
- **THEN** DB connection pool SHALL NOT be consumed
|
||||
|
||||
#### Scenario: Rate limit bucket per domain
|
||||
- **WHEN** `EventFetcher` is used from a route handler
|
||||
- **THEN** each domain SHALL have a configurable rate limit bucket aligned with `configured_rate_limit()` pattern
|
||||
- **THEN** rate limit configuration SHALL be overridable via environment variables
|
||||
|
||||
61
openspec/specs/lineage-engine-core/spec.md
Normal file
61
openspec/specs/lineage-engine-core/spec.md
Normal file
@@ -0,0 +1,61 @@
|
||||
# lineage-engine-core Specification
|
||||
|
||||
## Purpose
|
||||
TBD - created by archiving change unified-lineage-engine. Update Purpose after archive.
|
||||
## Requirements
|
||||
### Requirement: LineageEngine SHALL provide unified split ancestor resolution via CONNECT BY NOCYCLE
|
||||
`LineageEngine.resolve_split_ancestors()` SHALL accept a list of container IDs and return the complete split ancestry graph using a single Oracle `CONNECT BY NOCYCLE` query on `DW_MES_CONTAINER.SPLITFROMID`.
|
||||
|
||||
#### Scenario: Normal split chain resolution
|
||||
- **WHEN** `resolve_split_ancestors()` is called with a list of container IDs
|
||||
- **THEN** a single SQL query using `CONNECT BY NOCYCLE` SHALL be executed against `DW_MES_CONTAINER`
|
||||
- **THEN** the result SHALL include a `child_to_parent` mapping and a `cid_to_name` mapping for all discovered ancestor nodes
|
||||
- **THEN** the traversal depth SHALL be limited to `LEVEL <= 20` (equivalent to existing BFS `bfs_round > 20` guard)
|
||||
|
||||
#### Scenario: Large input batch exceeding Oracle IN clause limit
|
||||
- **WHEN** the input `container_ids` list exceeds `ORACLE_IN_BATCH_SIZE` (1000)
|
||||
- **THEN** `QueryBuilder.add_in_condition()` SHALL batch the IDs and combine results
|
||||
- **THEN** all bind parameters SHALL use `QueryBuilder.params` (no string concatenation)
|
||||
|
||||
#### Scenario: Cyclic split references in data
|
||||
- **WHEN** `DW_MES_CONTAINER.SPLITFROMID` contains cyclic references
|
||||
- **THEN** `NOCYCLE` SHALL prevent infinite traversal
|
||||
- **THEN** the query SHALL return all non-cyclic ancestors up to `LEVEL <= 20`
|
||||
|
||||
#### Scenario: CONNECT BY performance regression
|
||||
- **WHEN** Oracle 19c execution plan for `CONNECT BY NOCYCLE` performs worse than expected
|
||||
- **THEN** the SQL file SHALL contain a commented-out recursive `WITH` (recursive subquery factoring) alternative that can be swapped in without code changes
|
||||
|
||||
### Requirement: LineageEngine SHALL provide unified merge source resolution
|
||||
`LineageEngine.resolve_merge_sources()` SHALL accept a list of container IDs and return merge source mappings from `DW_MES_PJ_COMBINEDASSYLOTS`.
|
||||
|
||||
#### Scenario: Merge source lookup
|
||||
- **WHEN** `resolve_merge_sources()` is called with container IDs
|
||||
- **THEN** the result SHALL include `{cid: [merge_source_cid, ...]}` for all containers that have merge sources
|
||||
- **THEN** all queries SHALL use `QueryBuilder` bind params
|
||||
|
||||
### Requirement: LineageEngine SHALL provide combined genealogy resolution
|
||||
`LineageEngine.resolve_full_genealogy()` SHALL combine split ancestors and merge sources into a complete genealogy graph.
|
||||
|
||||
#### Scenario: Full genealogy for a set of seed lots
|
||||
- **WHEN** `resolve_full_genealogy()` is called with seed container IDs
|
||||
- **THEN** split ancestors SHALL be resolved first via `resolve_split_ancestors()`
|
||||
- **THEN** merge sources SHALL be resolved for all discovered ancestor nodes
|
||||
- **THEN** the combined result SHALL be equivalent to the existing `_resolve_full_genealogy()` output in `mid_section_defect_service.py`
|
||||
|
||||
### Requirement: LineageEngine functions SHALL be profile-agnostic
|
||||
All `LineageEngine` public functions SHALL accept `container_ids: List[str]` and return dictionary structures without binding to any specific page logic.
|
||||
|
||||
#### Scenario: Reuse from different pages
|
||||
- **WHEN** a new page (e.g., wip-detail) needs lineage resolution
|
||||
- **THEN** it SHALL be able to call `LineageEngine` functions directly without modification
|
||||
- **THEN** no page-specific logic (profile, TMTT detection, etc.) SHALL exist in `LineageEngine`
|
||||
|
||||
### Requirement: LineageEngine SQL files SHALL reside in `sql/lineage/` directory
|
||||
New SQL files SHALL follow the existing `SQLLoader` convention under `src/mes_dashboard/sql/lineage/`.
|
||||
|
||||
#### Scenario: SQL file organization
|
||||
- **WHEN** `LineageEngine` executes queries
|
||||
- **THEN** `split_ancestors.sql` and `merge_sources.sql` SHALL be loaded via `SQLLoader.load_with_params("lineage/split_ancestors", ...)`
|
||||
- **THEN** the SQL files SHALL NOT reference `HM_LOTMOVEOUT` (48M row table no longer needed for genealogy)
|
||||
|
||||
@@ -17,3 +17,25 @@ Services consuming shared Oracle query fragments SHALL preserve existing selecte
|
||||
- **WHEN** cache services execute queries via shared fragments
|
||||
- **THEN** resulting payload structure MUST remain compatible with existing aggregation and API contracts
|
||||
|
||||
### Requirement: Lineage SQL fragments SHALL be centralized in `sql/lineage/` directory
|
||||
Split ancestor and merge source SQL queries SHALL be defined in `sql/lineage/` and shared across services via `SQLLoader`.
|
||||
|
||||
#### Scenario: Mid-section-defect lineage query
|
||||
- **WHEN** `mid_section_defect_service.py` needs split ancestry or merge source data
|
||||
- **THEN** it SHALL call `LineageEngine` which loads SQL from `sql/lineage/split_ancestors.sql` and `sql/lineage/merge_sources.sql`
|
||||
- **THEN** it SHALL NOT use `sql/mid_section_defect/split_chain.sql` or `sql/mid_section_defect/genealogy_records.sql`
|
||||
|
||||
#### Scenario: Deprecated SQL file handling
|
||||
- **WHEN** `sql/mid_section_defect/genealogy_records.sql` and `sql/mid_section_defect/split_chain.sql` are deprecated
|
||||
- **THEN** the files SHALL be marked with a deprecated comment at the top
|
||||
- **THEN** grep SHALL confirm zero `SQLLoader.load` references to these files
|
||||
- **THEN** the files SHALL be retained for one version before deletion
|
||||
|
||||
### Requirement: All user-input SQL queries SHALL use QueryBuilder bind params
|
||||
`_build_in_filter()` and `_build_in_clause()` in `query_tool_service.py` SHALL be fully replaced by `QueryBuilder.add_in_condition()`.
|
||||
|
||||
#### Scenario: Complete migration to QueryBuilder
|
||||
- **WHEN** the refactoring is complete
|
||||
- **THEN** grep for `_build_in_filter` and `_build_in_clause` SHALL return zero results
|
||||
- **THEN** all queries involving user-supplied values SHALL use `QueryBuilder.params`
|
||||
|
||||
|
||||
61
openspec/specs/query-tool-safety-hardening/spec.md
Normal file
61
openspec/specs/query-tool-safety-hardening/spec.md
Normal file
@@ -0,0 +1,61 @@
|
||||
# query-tool-safety-hardening Specification
|
||||
|
||||
## Purpose
|
||||
TBD - created by archiving change unified-lineage-engine. Update Purpose after archive.
|
||||
## Requirements
|
||||
### Requirement: query-tool resolve functions SHALL use QueryBuilder bind params for all user input
|
||||
All `resolve_lots()` family functions (`_resolve_by_lot_id`, `_resolve_by_serial_number`, `_resolve_by_work_order`) SHALL use `QueryBuilder.add_in_condition()` with bind parameters instead of `_build_in_filter()` string concatenation.
|
||||
|
||||
#### Scenario: Lot resolve with user-supplied values
|
||||
- **WHEN** a resolve function receives user-supplied lot IDs, serial numbers, or work order names
|
||||
- **THEN** the SQL query SHALL use `:p0, :p1, ...` bind parameters via `QueryBuilder`
|
||||
- **THEN** `read_sql_df()` SHALL receive `builder.params` (never an empty `{}` dict for queries with user input)
|
||||
- **THEN** `_build_in_filter()` and `_build_in_clause()` SHALL NOT be called
|
||||
|
||||
#### Scenario: Pure static SQL without user input
|
||||
- **WHEN** a query contains no user-supplied values (e.g., static lookups)
|
||||
- **THEN** empty params `{}` is acceptable
|
||||
- **THEN** no `_build_in_filter()` SHALL be used
|
||||
|
||||
#### Scenario: Zero residual references to deprecated functions
|
||||
- **WHEN** the refactoring is complete
|
||||
- **THEN** grep for `_build_in_filter` and `_build_in_clause` SHALL return zero results across the entire codebase
|
||||
|
||||
### Requirement: query-tool routes SHALL apply rate limiting
|
||||
All query-tool API endpoints SHALL apply per-client rate limiting using the existing `configured_rate_limit` mechanism.
|
||||
|
||||
#### Scenario: Resolve endpoint rate limit exceeded
|
||||
- **WHEN** a client sends more than 10 requests to query-tool resolve endpoints within 60 seconds
|
||||
- **THEN** the endpoint SHALL return HTTP 429 with a `Retry-After` header
|
||||
- **THEN** the resolve service function SHALL NOT be called
|
||||
|
||||
#### Scenario: History endpoint rate limit exceeded
|
||||
- **WHEN** a client sends more than 20 requests to query-tool history endpoints within 60 seconds
|
||||
- **THEN** the endpoint SHALL return HTTP 429 with a `Retry-After` header
|
||||
|
||||
#### Scenario: Association endpoint rate limit exceeded
|
||||
- **WHEN** a client sends more than 20 requests to query-tool association endpoints within 60 seconds
|
||||
- **THEN** the endpoint SHALL return HTTP 429 with a `Retry-After` header
|
||||
|
||||
### Requirement: query-tool routes SHALL apply response caching
|
||||
High-cost query-tool endpoints SHALL cache responses in L2 Redis.
|
||||
|
||||
#### Scenario: Resolve result caching
|
||||
- **WHEN** a resolve request succeeds
|
||||
- **THEN** the response SHALL be cached in L2 Redis with TTL = 60s
|
||||
- **THEN** subsequent identical requests within TTL SHALL return cached result without Oracle query
|
||||
|
||||
### Requirement: lot_split_merge_history SHALL support fast and full query modes
|
||||
The `lot_split_merge_history.sql` query SHALL support two modes to balance traceability completeness vs performance.
|
||||
|
||||
#### Scenario: Fast mode (default)
|
||||
- **WHEN** `full_history` query parameter is absent or `false`
|
||||
- **THEN** the SQL SHALL include `TXNDATE >= ADD_MONTHS(SYSDATE, -6)` time window and `FETCH FIRST 500 ROWS ONLY`
|
||||
- **THEN** query response time SHALL be ≤5s (P95)
|
||||
|
||||
#### Scenario: Full mode
|
||||
- **WHEN** `full_history=true` query parameter is provided
|
||||
- **THEN** the SQL SHALL NOT include time window restriction
|
||||
- **THEN** the query SHALL use `read_sql_df_slow` (120s timeout)
|
||||
- **THEN** query response time SHALL be ≤60s (P95)
|
||||
|
||||
Reference in New Issue
Block a user