feat(trace-pool-isolation): migrate event_fetcher/lineage_engine to slow connections + fix 51 test failures

Trace pipeline pool isolation:
- Switch event_fetcher and lineage_engine to read_sql_df_slow (non-pooled)
- Reduce EVENT_FETCHER_MAX_WORKERS 4→2, TRACE_EVENTS_MAX_WORKERS 4→2
- Add 60s timeout per batch query, cache skip for CID>10K
- Early del raw_domain_results + gc.collect() for large queries
- Increase DB_SLOW_MAX_CONCURRENT: base 3→5, dev 2→3, prod 3→5

Test fixes (51 pre-existing failures → 0):
- reject_history: WORKFLOW CSV header, strict bool validation, pareto mock path
- portal shell: remove non-existent /tmtt-defect route from tests
- conftest: add --run-stress option to skip stress/load tests by default
- migration tests: skipif baseline directory missing
- performance test: update Vite asset assertion
- wip hold: add firstname/waferdesc mock params
- template integration: add /reject-history canonical route

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

openspec/specs/event-fetcher-unified/spec.md

@@ -4,13 +4,14 @@
 TBD - created by archiving change unified-lineage-engine. Update Purpose after archive.
 ## Requirements
 ### Requirement: EventFetcher SHALL provide unified cached event querying across domains
-`EventFetcher` SHALL encapsulate batch event queries with L1/L2 layered cache and rate limit bucket configuration, supporting domains: `history`, `materials`, `rejects`, `holds`, `jobs`, `upstream_history`.
+`EventFetcher` SHALL encapsulate batch event queries with L1/L2 layered cache and rate limit bucket configuration, supporting domains: `history`, `materials`, `rejects`, `holds`, `jobs`, `upstream_history`, `downstream_rejects`.
 
 #### Scenario: Cache miss for event domain query
 - **WHEN** `EventFetcher` is called for a domain with container IDs and no cache exists
-- **THEN** the domain query SHALL execute against Oracle via `read_sql_df()`
-- **THEN** the result SHALL be stored in L2 Redis cache with key format `evt:{domain}:{sorted_cids_hash}`
-- **THEN** L1 memory cache SHALL also be populated (aligned with `core/cache.py` LayeredCache pattern)
+- **THEN** the domain query SHALL execute against Oracle via `read_sql_df_slow()` (non-pooled dedicated connection)
+- **THEN** each batch query SHALL use `timeout_seconds=60`
+- **THEN** the result SHALL be stored in L2 Redis cache with key format `evt:{domain}:{sorted_cids_hash}` if CID count is within cache threshold
+- **THEN** L1 memory cache SHALL also be populated if CID count is within cache threshold
 
 #### Scenario: Cache hit for event domain query
 - **WHEN** `EventFetcher` is called for a domain and L2 Redis cache contains a valid entry
@@ -22,3 +23,13 @@ TBD - created by archiving change unified-lineage-engine. Update Purpose after a
 - **THEN** each domain SHALL have a configurable rate limit bucket aligned with `configured_rate_limit()` pattern
 - **THEN** rate limit configuration SHALL be overridable via environment variables
 
+#### Scenario: Large CID set exceeds cache threshold
+- **WHEN** the normalized CID count exceeds `CACHE_SKIP_CID_THRESHOLD` (default 10000, env: `EVENT_FETCHER_CACHE_SKIP_CID_THRESHOLD`)
+- **THEN** EventFetcher SHALL skip both L1 and L2 cache writes
+- **THEN** a warning log SHALL be emitted with domain name, CID count, and threshold value
+- **THEN** the query result SHALL still be returned to the caller
+
+#### Scenario: Batch concurrency default
+- **WHEN** EventFetcher processes batches for a domain with >1000 CIDs
+- **THEN** the default `EVENT_FETCHER_MAX_WORKERS` SHALL be 2 (env: `EVENT_FETCHER_MAX_WORKERS`)
+

openspec/specs/lineage-engine-core/spec.md

@@ -59,3 +59,15 @@ New SQL files SHALL follow the existing `SQLLoader` convention under `src/mes_da
 - **THEN** `split_ancestors.sql` and `merge_sources.sql` SHALL be loaded via `SQLLoader.load_with_params("lineage/split_ancestors", ...)`
 - **THEN** the SQL files SHALL NOT reference `HM_LOTMOVEOUT` (48M row table no longer needed for genealogy)
 
+### Requirement: LineageEngine SHALL use non-pooled database connections
+All Oracle queries executed by `LineageEngine` SHALL use `read_sql_df_slow()` (dedicated non-pooled connections) instead of `read_sql_df()` (connection pool).
+
+#### Scenario: Lineage query does not consume pool connections
+- **WHEN** `LineageEngine` executes split ancestor, merge source, or other Oracle queries
+- **THEN** queries SHALL use `read_sql_df_slow()` with the default slow query timeout (300s)
+- **THEN** the shared connection pool SHALL NOT be consumed by lineage queries
+
+#### Scenario: Lineage queries respect slow query semaphore
+- **WHEN** `LineageEngine` executes queries via `read_sql_df_slow()`
+- **THEN** each query SHALL acquire and release a slot from the slow query semaphore (`DB_SLOW_MAX_CONCURRENT`)
+

openspec/specs/runtime-resilience-recovery/spec.md

@@ -10,6 +10,14 @@ The system SHALL apply database pool and timeout parameters from runtime configu
 - **WHEN** operators set pool and timeout values via environment configuration and start the service
 - **THEN** the active engine MUST use those values for pool size, overflow, wait timeout, and query call timeout
 
+#### Scenario: Slow query semaphore capacity
+- **WHEN** the service starts in production or staging configuration
+- **THEN** `DB_SLOW_MAX_CONCURRENT` SHALL default to 5 (env: `DB_SLOW_MAX_CONCURRENT`)
+- **WHEN** the service starts in development configuration
+- **THEN** `DB_SLOW_MAX_CONCURRENT` SHALL default to 3
+- **WHEN** the service starts in testing configuration
+- **THEN** `DB_SLOW_MAX_CONCURRENT` SHALL remain at 1
+
 ### Requirement: Pool Exhaustion MUST Return Retry-Aware Degraded Responses
 The system MUST return explicit degraded responses for connection pool exhaustion and include machine-readable metadata for retry/backoff behavior.
 

openspec/specs/trace-staged-api/spec.md

@@ -1,5 +1,6 @@
-## ADDED Requirements
-
+## Purpose
+Staged trace API for seed-resolve, lineage, and events pipeline with rate limiting, caching, and memory management.
+## Requirements
 ### Requirement: Staged trace API SHALL expose seed-resolve endpoint
 `POST /api/trace/seed-resolve` SHALL resolve seed lots based on the provided profile and parameters.
 
@@ -87,3 +88,27 @@ The existing analysis endpoint (GET method) SHALL internally delegate to the sta
 - **THEN** the endpoint SHALL internally execute seed-resolve → lineage → events + aggregation
 - **THEN** the response format SHALL be identical to the pre-refactoring output
 - **THEN** a golden test SHALL verify output equivalence
+
+### Requirement: Trace events endpoint SHALL limit domain concurrency
+The `/api/trace/events` endpoint SHALL use `TRACE_EVENTS_MAX_WORKERS` to control how many domains execute concurrently.
+
+#### Scenario: Default domain concurrency
+- **WHEN** the events endpoint dispatches domain queries
+- **THEN** the default `TRACE_EVENTS_MAX_WORKERS` SHALL be 2 (env: `TRACE_EVENTS_MAX_WORKERS`)
+
+### Requirement: Trace events endpoint SHALL manage memory for large queries
+The events endpoint SHALL proactively release memory after processing large CID sets.
+
+#### Scenario: Early release of grouped domain results
+- **WHEN** MSD aggregation completes using `raw_domain_results`
+- **THEN** the `raw_domain_results` reference SHALL be deleted immediately after aggregation
+- **THEN** for non-MSD profiles, `raw_domain_results` SHALL be deleted after result assembly
+
+#### Scenario: Garbage collection for large CID sets
+- **WHEN** the events endpoint completes processing and the CID count exceeds 10000
+- **THEN** `gc.collect()` SHALL be called to prompt Python garbage collection
+
+#### Scenario: Large CID set skips route-level cache
+- **WHEN** the events endpoint completes for a non-MSD profile and CID count exceeds 10000
+- **THEN** the route-level events cache write SHALL be skipped
+