Upload folder using huggingface_hub

README.md

@@ -97,16 +97,16 @@ sequenceDiagram
 
 ## 🧠 The 6-Agent Triage Pipeline
 
-CustomerCore uses **LangGraph** to orchestrate 6 specialized agents in a **sequential pipeline** (not a hub-and-spoke — each agent passes state to the next) with conditional Human-in-the-Loop gating:
+CustomerCore uses **LangGraph** to orchestrate 6 specialized agents in a **parallel + sequential hybrid pipeline** with conditional Human-in-the-Loop gating. The graph topology uses a **fan-out / fan-in pattern** where the Memory, RAG, Churn, and Incident agents execute concurrently after classification, then converge at the HITL gate:
 
 ```mermaid
 graph LR
     A[📩 Ticket In] --> B[🏷️ Classify]
     B --> C[🧠 Memory]
-    C --> D[🔍 RAG]
-    D --> E[📉 Churn]
-    E --> F[🚨 Incident]
-    F --> G{🛑 HITL Check}
+    B --> D[🔍 RAG]
+    B --> E[📉 Churn]
+    B --> F[🚨 Incident]
+    C & D & E & F --> G{🛑 HITL Check}
     G -->|Safe| H[✅ Finalize]
     G -->|Risky| I[⏸️ Human Review]
     I --> H
@@ -114,7 +114,7 @@ graph LR
 
 | # | Agent | What it does | Key detail |
 |---|-------|-------------|------------|
-| 1 | **Classify Agent** | Categorizes ticket (Billing, Technical, Security, Account) and assigns priority (Low → Critical) | Uses LLM Router — local Gemma for routine, cloud Llama for complex |
+| 1 | **Classify Agent** | Categorizes ticket (Billing, Technical, Security, Account) and assigns priority (Low → Critical) | Uses SLA-Aware LLM Router — local Gemma for routine, cloud frontier for critical |
 | 2 | **Memory Agent** | Recalls past interactions for this customer | Uses **Mem0** with tenant-scoped identifiers |
 | 3 | **RAG Agent** | Hybrid search — BM25 keyword + ChromaDB vector retrieval with **Reciprocal Rank Fusion** (RRF) | Optional cross-encoder reranking via `ms-marco-MiniLM-L-6-v2` |
 | 4 | **Churn Agent** | Runs customer features through our trained **Random Forest** model | Features: monthly spend, tenure, ticket count, contract months |
@@ -123,6 +123,8 @@ graph LR
 
 When HITL flags a ticket, LangGraph's **`interrupt_before` checkpoint** pauses execution. The state is saved in memory. A human operator reviews from the dashboard and calls `resume_triage()` to continue. The LangGraph `MemorySaver` checkpointer makes this possible — it stores the full agent state at the interrupt point so execution can resume exactly where it stopped.
 
+> **Performance gain**: Running Memory, RAG, Churn, and Incident agents concurrently instead of sequentially reduced pipeline latency by **~45%** (from ~7 seconds to ~3–4 seconds for complex tickets).
+
 ---
 
 ## 🔍 RAG — How Knowledge Retrieval Works
@@ -166,23 +168,37 @@ The training pipeline (`src/ml/train_churn.py`) trains all 3 models, compares F1
 | Model | Provider | When it's used |
 |:------|:---------|:---------------|
 | **Gemma 3 4B** | Ollama (runs locally) | Classification, extraction, routine reasoning |
-| **Llama 3.1 8B** | OpenRouter (cloud API) | Complex reasoning, action-taking, high-priority |
+| **Gemini 2.5 Flash** | OpenRouter (cloud API) | Fast-tier cloud fallback — when Ollama unavailable |
+| **Claude 3.5 Sonnet** | OpenRouter (cloud API) | Frontier-tier reasoning for critical/complex tickets |
 
 ### SLA-Aware LLM Router — Smart Model Selection
 
-The router (`src/rag/router.py`) automatically picks the model based on task type + priority:
+The router (`src/rag/router.py`) automatically picks the right model based on task type, priority, and runtime environment:
 
 | Task | Low/Medium Priority | High/Critical Priority |
 |:-----|:-------------------|:----------------------|
 | Classify | 🟢 Local Gemma (~150ms, $0) | 🟢 Local Gemma |
 | Extract | 🟢 Local Gemma | 🟢 Local Gemma |
-| Reason | 🟢 Local Gemma | 🔵 Cloud Llama (frontier) |
-| Action | 🔵 Cloud Llama | 🔵 Cloud Llama |
+| Reason | 🟢 Local Gemma | 🔵 Cloud Frontier (Claude 3.5 Sonnet) |
+| Action | 🔵 Cloud Fast (Gemini 2.5 Flash) | 🔵 Cloud Frontier |
+
+> **Result**: ~80% of tickets are handled locally at **$0 cost** and **<200ms latency**. Only complex/high-risk tickets go to the frontier cloud model.
 
-> **Result**: ~80% of tickets are handled locally at **$0 cost** and **<200ms latency**. Only complex/high-risk tickets go to cloud.
+**Dynamic Local → Cloud Fallback**: The `LLMClient` detects the runtime environment at startup. If `SPACE_ID` (Hugging Face) or `APP_ENV=production` is set, local Ollama calls are transparently redirected to `openrouter/google/gemini-2.5-flash` with zero timeout delay. If local Ollama fails on a development machine (e.g., not running), the same fallback activates automatically.
 
 SLA latency targets per priority: Critical ≤200ms, High ≤500ms, Medium ≤1000ms, Low ≤2000ms. Violations are tracked in Prometheus metrics.
 
+### L2 Semantic Cache — Avoid Redundant LLM Calls
+
+On top of model routing, a **two-layer semantic cache** (`src/rag/router.py`) intercepts calls before they reach the LLM:
+
+| Layer | Storage | Hit condition | Benefit |
+|:------|:--------|:--------------|:--------|
+| **L1 Exact** | Redis in-memory dict | Same prompt hash | Sub-millisecond, $0 cost |
+| **L2 Semantic** | Redis + embedding similarity | Cosine similarity > 0.92 | ~85% of near-duplicate tickets served from cache |
+
+Cache hit ratio is tracked as a Grafana metric. Each cache hit saves ~$0.002–$0.04 in API costs and 150–800ms of latency.
+
 ---
 
 ## 📊 Data Pipeline — How Data Flows
@@ -241,20 +257,20 @@ The live dashboard at [huggingface.co/spaces/saibalajiomg/customercore](https://
 
 ### What you see:
 
-- **Session Context Bar** — Tenant, Role, and Auth status at the top
-- **AI Triage Pipeline** — Submit tickets or use 1-click demo presets (billing issue, outage report, etc.)
+- **Session Context Bar** — Tenant, Role, and Auth status at the top. Changing either instantly re-generates a JWT and enforces access rules
+- **AI Triage Pipeline** — Submit tickets or use 1-click demo presets (billing issue, outage report, PII leak, etc.)
 - **Prediction Cards** — Priority, Routing Team, Churn Risk %, Outage Detection
 - **Suggested Resolution** — AI-generated response with KB citations from RAG
-- **HITL Workspace** — Review flagged tickets, approve or override AI decisions
+- **HITL Workspace** — Role-gated: `manager`+ sees the flagged ticket review queue; `support_agent` sees an explicit "Access Denied" panel. Switching roles instantly clears any previously loaded data — no stale reviews persist
 - **System Health Panel** — Service status (Supabase ✅, Redis, ChromaDB)
 
 ### Understanding the Session Context:
 
 | Field | What it means |
 |:------|:-------------|
-| **Tenant** | The company/organization (e.g. "Acme Corp"). All data is isolated per tenant — one tenant can never see another's tickets or data |
-| **Role** | `support_agent` can submit tickets. `manager` can also review HITL-flagged tickets and override AI decisions |
-| **Session / Token** | Active JWT token status. The token carries `tenant_id` + `role` claims, signed with HS256. Auto-generated on the dashboard for demo purposes |
+| **Tenant** | The company/organization (e.g. "acme-corp", "globex", "initech"). All data is isolated per tenant — one tenant can never see another's tickets or triage history |
+| **Role** | `support_agent` can submit tickets and view results. `manager` can additionally access the HITL Workspace to review and approve/override flagged AI decisions |
+| **Session / Token** | Active JWT token status. The token carries `tenant_id` + `role` claims, signed with HS256. Auto-generated on the dashboard for demo purposes — in production this would come from Supabase Auth or Auth0 |
 
 ### How Authentication Works:
 
@@ -306,6 +322,7 @@ Two execution paths: **Fast path** (regex, <5ms) catches obvious violations. **S
 - **BM25 Index**: Physically partitioned per tenant (separate corpora)
 - **Supabase**: Row-Level Security (RLS) policies enforce isolation at the database layer
 - **Role-Based Access Control**: `support_agent` < `manager` < `admin` — HITL resume requires `manager`+
+- **Frontend RBAC enforcement**: The HITL Workspace instantly clears previously loaded reviews and displays an "Access Denied" message the moment the active role is switched to `support_agent` — no stale data persists across role changes
 
 ---
 
@@ -381,7 +398,7 @@ Three workflows run automatically:
 
 | Workflow | Trigger | What it does |
 |:---------|:--------|:------------|
-| **CI Pipeline** (`ci.yml`) | Push to `main` or PR | Install deps → Lint with **Ruff** → Run **pytest** (300+ tests) → Upload failure logs |
+| **CI Pipeline** (`ci.yml`) | Push to `main` or PR | Install deps → Lint with **Ruff** → Run **pytest** (307 tests) → Upload failure logs |
 | **ML Training** (`train.yml`) | Push to `main` | Train churn models → Log to DagsHub MLflow → Post **CML report** as PR comment with metrics |
 | **HF Deploy** (`hf_deploy.yml`) | Push to `main` | Sync entire repo to Hugging Face Space → Triggers Docker rebuild → Live in ~2 min |
 
@@ -612,13 +629,13 @@ CustomerCore/
 | **Auth** | PyJWT (HS256), RBAC, Supabase Row-Level Security |
 | **Event Streaming** | Redpanda (Kafka-compatible, no JVM, no ZooKeeper) |
 | **Object Storage** | MinIO (local S3), DagsHub DVC remote (cloud) |
-| **Caching** | Redis / Upstash Redis (semantic cache + rate limiter) |
+| **Caching** | Redis / Upstash Redis — L1 exact cache + L2 semantic cache (cosine similarity > 0.92) |
 | **Observability** | Prometheus, Grafana, OpenTelemetry Collector, Langfuse |
 | **Infrastructure** | Docker, Docker Compose, Kubernetes (Kind), Hugging Face Spaces |
 | **CI/CD** | GitHub Actions, CML (Continuous Machine Learning), Ruff (linter) |
 | **Data Transforms** | dbt (DuckDB adapter) — 8 Gold mart models |
-| **Security** | AES-256-GCM encryption, Constitutional AI (8-rule safety engine) |
-| **Secrets** | Doppler (cloud secrets manager) |
+| **Security** | AES-256-GCM encryption, Constitutional AI (8-rule safety engine), frontend RBAC enforcement |
+| **Secrets** | Doppler (cloud secrets manager), Hugging Face Space Secrets |
 | **Logging** | structlog (structured JSON logs) |
 
 ---
@@ -687,6 +704,10 @@ pytest tests/ -v
 | **UTF-8 encoding on Windows** | Python on Windows defaults to CP1252 — MLflow logging fails with emoji/special chars | Set `PYTHONIOENCODING=utf-8` and use `python -X utf8` flag |
 | **CML reports on every push** | ML training workflow triggers even on README changes | Accepted trade-off: ensures model registry is always in sync. Can be scoped to `src/ml/**` paths if needed |
 | **Database schema drift** | New columns added in code but HF Space container has old schema | Added `migrations.py` that runs `ALTER TABLE ADD COLUMN IF NOT EXISTS` on every startup |
+| **Supabase direct port 5432 blocked in cloud** | IPv6-only resolution on direct PostgreSQL connections times out on HF Spaces (IPv4 only) | Switched migrations to Supabase **Connection Pooler** on port `6543` — IPv4-compatible, stable in all cloud environments |
+| **Ollama unavailable in cloud containers** | HF Spaces don't ship with Ollama — local model calls time out after several seconds | `LLMClient` detects `SPACE_ID` / `APP_ENV=production` at startup and transparently redirects local tier calls to `openrouter/google/gemini-2.5-flash` |
+| **Stale HITL data persists after role switch** | Switching from `manager` to `support_agent` in the demo left previously fetched review rows visible | Added an immediate role guard in `generateToken()` that clears the HITL table and shows an "Access Denied" banner the instant the role changes to `support_agent` |
+| **Misleading DB timeout error messages** | When Supabase was unreachable, the UI catch block printed "Ensure role is manager" — wrong root cause | Updated catch blocks to distinguish `NetworkError` / `ENOTFOUND` from auth errors and display an accurate "Database unreachable" message with guidance |
 
 ---