feat: integrate MLflow model registry, DagsHub metrics tracking, and local Grafana observability stack

.github/workflows/train.yml

@@ -39,6 +39,10 @@ jobs:
       uses: iterative/setup-cml@v2
 
     - name: Train model and write metrics
+      env:
+        MLFLOW_TRACKING_URI: https://dagshub.com/saibalajinamburi/CustomerCore.mlflow
+        MLFLOW_TRACKING_USERNAME: saibalajinamburi
+        MLFLOW_TRACKING_PASSWORD: ${{ secrets.DAGSHUB_TOKEN }}
       run: |
         python src/ml/train_churn.py
 

.gitignore

@@ -52,4 +52,7 @@ coverage.xml
 
 # Personal study docs — local only, not for GitHub
 BUILD_GUIDE.md
-CustomerCore_Progress_Summary.md
+CustomerCore_Progress_Summary.md
+
+# Scratch files
+scratch/

README.md

@@ -16,7 +16,7 @@ Real-time multi-tenant customer intelligence powered by streaming (Redpanda), la
 [![Tests](https://img.shields.io/badge/tests-passing-brightgreen)](https://github.com/saibalajinamburi/CustomerCore/actions)
 [![License](https://img.shields.io/badge/license-MIT-green)](https://opensource.org/licenses/MIT)
 [![EU AI Act](https://img.shields.io/badge/EU%20AI%20Act-Compliant-blue)](#responsible-ai)
-[![HF Spaces](https://img.shields.io/badge/HF%20Spaces-Live-orange)](https://huggingface.co/spaces/saibalajinamburi/customercore-api)
+[![HF Spaces](https://img.shields.io/badge/HF%20Spaces-Live-orange)](https://huggingface.co/spaces/saibalajiomg/customercore)
 
 CustomerCore is an enterprise-grade customer intelligence system designed to automate ticket classification, prioritize issues, calculate churn and SLA breach risks, recall agent memories, retrieve grounded facts, and run automated human-in-the-loop triage flows.
 
@@ -26,10 +26,12 @@ CustomerCore is an enterprise-grade customer intelligence system designed to aut
 
 | Asset | Link | Status |
 |---|---|---|
-| **Live API (HF Spaces)** | [customercore-api.hf.space](https://huggingface.co/spaces/saibalajinamburi/customercore-api/health) | ![uptime](https://img.shields.io/badge/Uptime-100%25-brightgreen) |
-| **DagsHub Repo** | [dagshub.com/saibalajinamburi/CustomerCore](https://dagshub.com/saibalajinamburi/CustomerCore) | Connected |
-| **MLflow Experiments** | [dagshub.com/saibalajinamburi/CustomerCore/mlflow](https://dagshub.com/saibalajinamburi/CustomerCore/mlflow) | 8 model experiments tracked |
-| **CI Pipeline** | [github.com/saibalajinamburi/CustomerCore/actions](https://github.com/saibalajinamburi/CustomerCore/actions) | 10-stage GHA pipeline |
+| **Live Operations Console (HF Spaces)** | [huggingface.co/spaces/saibalajiomg/customercore](https://huggingface.co/spaces/saibalajiomg/customercore) | ![uptime](https://img.shields.io/badge/Uptime-100%25-brightgreen) |
+| **GitHub Repository** | [github.com/saibalajinamburi/CustomerCore](https://github.com/saibalajinamburi/CustomerCore) | Active |
+| **DagsHub Repository & DVC** | [dagshub.com/saibalajinamburi/CustomerCore](https://dagshub.com/saibalajinamburi/CustomerCore) | Connected |
+| **MLflow Experiment Server** | [dagshub.com/saibalajinamburi/CustomerCore.mlflow](https://dagshub.com/saibalajinamburi/CustomerCore.mlflow) | Runs Tracked |
+| **MLflow Model Registry** | [dagshub.com/saibalajinamburi/CustomerCore/models](https://dagshub.com/saibalajinamburi/CustomerCore/models) | Versioned Models |
+| **CI/CD Pipeline** | [github.com/saibalajinamburi/CustomerCore/actions](https://github.com/saibalajinamburi/CustomerCore/actions) | Passing ✅ |
 
 ---
 
@@ -41,7 +43,7 @@ See [ARCHITECTURE.md](ARCHITECTURE.md) for the full system diagram and descripti
 1. **Streaming Data Flow:** Kafka-compatible Redpanda topics (`tickets`, `product`, `billing`, `incidents`) ingest event micro-batches.
 2. **Spark Lakehouse:** PySpark Structured Streaming processes raw data from Bronze to Silver layers with real-time PII masking and schema validation, writing to Apache Iceberg format.
 3. **dbt Gold Layer:** Transforms silver tables into 7 curated business marts for analytics, features, and model training.
-4. **LangGraph Triage:** A multi-agent supervisor orchestrates six specialized sub-agents (Classify, Memory, RAG, Churn, Incident, and HITL) to yield a Pydantic-validated 14-field triage output in `<2s`.
+4. **LangGraph Triage:** A stateful multi-agent supervisor orchestrates six specialized sub-agents (Classify, Memory, RAG, Churn, Incident, and HITL) to yield a Pydantic-validated 14-field triage output in `<2s`.
 
 ---
 
@@ -52,7 +54,7 @@ When a support ticket is received at `/api/v1/triage`, the LangGraph supervisor
 1. **Classify Agent:** Determines ticket category and initial priority.
 2. **Memory Agent:** Recalls customer profile and past session memories from Mem0 (Supabase pgvector).
 3. **RAG Agent:** Injects context using hybrid retrieval (Dense + BM25 sparse) from the Knowledge Base and product documentation, reranked by a cross-encoder model.
-4. **Churn Agent:** Evaluates churn risk using a LightGBM classifier.
+4. **Churn Agent:** Evaluates churn risk using a local **Random Forest Classifier** trained on client metrics.
 5. **Incident Agent:** Inspects incident patterns and flags SLA-breach risks.
 6. **HITL Agent:** Interrupts the graph and pauses execution if safety flags are violated or if classifier confidence falls below `0.65`, routing the ticket to the Supabase-backed human-in-the-loop review queue.
 7. **Finalize & Log:** Consolidates state, applies PII scrubbing, logs the audit trail, and pushes a structured trace to Langfuse.
@@ -64,19 +66,18 @@ When a support ticket is received at `/api/v1/triage`, the LangGraph supervisor
 | Layer | Technology |
 |---|---|
 | **Streaming Backbone** | Redpanda (Kafka-compatible event broker) |
-| **Lakehouse Storage** | Apache Iceberg on Cloudflare R2 |
-| **Stream Processing** | PySpark Structured Streaming |
+| **Lakehouse Storage** | Apache Iceberg on Cloudflare R2 / MinIO |
+| **Stream Processing** | PySpark Structured Streaming / PyArrow |
 | **Transformation** | dbt-core with DuckDB adapter (7 Gold marts) |
 | **Vector Database** | ChromaDB (BM25 sparse + BGE-M3 dense hybrid retrieval) |
 | **LLM Gateway** | LiteLLM routing proxy & Semantic Cache (L1/L2/L3) |
 | **Agent Framework** | LangGraph multi-agent supervisor |
 | **Long-Term Memory** | Mem0 backed by Supabase pgvector |
-| **ML Models** | 8 models (LightGBM, Isolation Forest, Prophet, etc.) |
-| **Experiment Tracking**| MLflow hosted on DagsHub |
+| **ML Models & Registry** | Random Forest Classifier, MLflow, DagsHub, DVC |
 | **LLM Observability** | Langfuse (prompt management and tracing) & LangSmith |
-| **Observability** | OpenTelemetry, Prometheus metrics, Grafana Cloud |
-| **CI/CD** | GitHub Actions with CML (Continuous Machine Learning) reporting |
-| **Hosting** | Hugging Face Spaces Docker (Inference) & Supabase |
+| **Infrastructure Observability**| OpenTelemetry, Prometheus metrics, Grafana Cloud |
+| **CI/CD & CML** | GitHub Actions with Continuous Machine Learning metrics reporting |
+| **Hosting** | Hugging Face Spaces Docker (Inference) & Supabase PostgreSQL |
 
 ---
 
@@ -97,27 +98,63 @@ pip install -r requirements.txt
 ```
 
 ### 3. Running Locally
-Run the dependent infrastructure services:
+
+#### Run base services (Redpanda, MinIO, Redis, ChromaDB, OTel):
 ```bash
 docker compose up -d
 ```
-Start the FastAPI gateway:
+
+#### Run observability console (Prometheus + Grafana):
+```bash
+docker compose -f docker-compose.yml -f docker-compose.monitoring.yml up -d
+```
+*   **Grafana Dashboard**: Open `http://localhost:3000` (pre-provisioned with metrics visualizations, no login required).
+*   **Prometheus Console**: Open `http://localhost:9090`.
+
+#### Start the FastAPI gateway:
+Using Doppler:
+```bash
+doppler run -- uvicorn src.api.main:app --reload --port 8080
+```
+Without Doppler:
 ```bash
 uvicorn src.api.main:app --reload --port 8080
 ```
-Check health:
+
+#### Check Health:
 ```bash
 curl http://localhost:8080/api/v1/health
 ```
 
 ---
 
+## ML Model Training & DagsHub Integration
+
+We train a **Random Forest Churn Predictor** locally or in CI, logging parameters, metrics, performance plots (ROC, Confusion Matrix, Feature Importance), and the model binaries directly to DagsHub's MLflow Experiments Server and Model Registry.
+
+To train the model and push to DagsHub manually:
+```bash
+# Set DagsHub credentials in your shell environment
+$env:MLFLOW_TRACKING_USERNAME="saibalajinamburi"
+$env:MLFLOW_TRACKING_PASSWORD="YOUR_DAGSHUB_TOKEN"
+$env:PYTHONIOENCODING="utf-8"
+
+# Train model and track in MLflow Model Registry
+doppler run -- python -X utf8 src/ml/train_churn.py
+
+# Push raw dataset file versioning via DVC
+doppler run -- dvc push
+```
+
+---
+
 ## Project Structure
 
 ```text
 customercore/
 ├── .github/workflows/    # CI/CD Workflows (Lint, Pytest, HF Deploy, CML Train)
 ├── infra/
+│   ├── monitoring/       # Prometheus and Grafana local config & dashboards
 │   ├── k8s/              # Kubernetes YAML manifests (Deployment, Ingress, Secrets)
 │   ├── kind-config.yaml  # Kind local multi-node cluster configuration
 │   └── otel-collector.yaml
@@ -125,14 +162,16 @@ customercore/
 │   ├── api/              # FastAPI Application endpoints and middleware
 │   ├── agent/            # LangGraph supervisor and agent nodes
 │   ├── rag/              # ChromaDB, Hybrid Retriever, Semantic Cache
-│   ├── ml/               # Model training scripts
+│   ├── ml/               # Model training scripts (Random Forest)
 │   ├── streaming/        # Redpanda Producers and PySpark pipeline
 │   ├── dbt/              # dbt transform pipeline (Gold marts)
 │   ├── db/               # Supabase database repositories and clients
 │   └── responsible_ai/   # Model cards, fairness evaluation, audit logger
 ├── tests/                # Unit and integration test suites
 ├── Dockerfile            # Default Production Dockerfile
-└── Dockerfile.hf         # Hugging Face optimized Dockerfile
+├── Dockerfile.hf         # Hugging Face optimized Dockerfile
+├── docker-compose.yml    # Standard services configuration
+└── docker-compose.monitoring.yml # Local Grafana / Prometheus observability stack
 ```
 
 ---
@@ -143,7 +182,7 @@ CustomerCore is designed with compliance in mind for the **EU AI Act (effective
 - **Article 10 (Data Governance):** Schema enforcement and automated PII masking via Presidio.
 - **Article 12 (Traceability):** Structured audit logging of all constitutional violations and inputs/outputs to Supabase PostgreSQL.
 - **Article 14 (Human Oversight):** LangGraph-native `interrupt()` gates that pause low-confidence decisions for manual human sign-off.
-- **Article 15 (Accuracy & Security):** Model card documentation for all 8 ML models (`src/responsible_ai/model_cards/`) and demographic fairness accuracy gaps calculated in `fairness.py`.
+- **Article 15 (Accuracy & Security):** Model card documentation for all ML models (`src/responsible_ai/model_cards/`) and demographic fairness accuracy gaps.
 
 ---
 
@@ -152,3 +191,4 @@ CustomerCore is designed with compliance in mind for the **EU AI Act (effective
 **Saibalaji Namburi**  
 MSc Data Analytics, University of Hildesheim  
 - GitHub: [@saibalajinamburi](https://github.com/saibalajinamburi)
+- DagsHub: [saibalajinamburi](https://dagshub.com/saibalajinamburi)

docker-compose.monitoring.yml

@@ -0,0 +1,47 @@
+version: "3.8"
+
+# ============================================================
+# CustomerCore Local Observability Stack Add-on
+# Start:  docker compose -f docker-compose.yml -f docker-compose.monitoring.yml up -d
+# Stop:   docker compose -f docker-compose.yml -f docker-compose.monitoring.yml down
+# Grafana: http://localhost:3000 (No login required - Admin access)
+# Prometheus: http://localhost:9090
+# ============================================================
+
+services:
+
+  prometheus:
+    image: prom/prometheus:latest
+    container_name: customercore_prometheus
+    volumes:
+      - ./infra/monitoring/prometheus.yml:/etc/prometheus/prometheus.yml
+    command:
+      - --config.file=/etc/prometheus/prometheus.yml
+      - --storage.tsdb.path=/prometheus
+      - --web.console.libraries=/usr/share/prometheus/console_libraries
+      - --web.console.templates=/usr/share/prometheus/consoles
+    networks:
+      - customercore
+    ports:
+      - "9090:9090"
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+
+  grafana:
+    image: grafana/grafana-oss:latest
+    container_name: customercore_grafana
+    environment:
+      - GF_AUTH_ANONYMOUS_ENABLED=true
+      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
+      - GF_AUTH_ANONYMOUS_ORG_NAME=Main Org.
+      - GF_SECURITY_ALLOW_EMBEDDING=true
+    volumes:
+      - ./infra/monitoring/grafana-datasources.yml:/etc/grafana/provisioning/datasources/datasources.yml
+      - ./infra/monitoring/grafana-dashboards.yml:/etc/grafana/provisioning/dashboards/dashboards.yml
+      - ./infra/monitoring/customercore-dashboard.json:/etc/grafana/provisioning/dashboards/customercore-dashboard.json
+    networks:
+      - customercore
+    ports:
+      - "3000:3000"
+    depends_on:
+      - prometheus

infra/monitoring/customercore-dashboard.json

@@ -0,0 +1,251 @@
+{
+  "annotations": {
+    "list": [
+      {
+        "builtIn": 1,
+        "datasource": "-- Grafana --",
+        "enable": true,
+        "hide": true,
+        "name": "Annotations & Alerts",
+        "type": "dashboard"
+      }
+    ]
+  },
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 0,
+  "id": 1,
+  "links": [],
+  "liveNow": false,
+  "panels": [
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "Prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth"
+          },
+          "unit": "reqps"
+        }
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 0
+      },
+      "id": 2,
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "Prometheus"
+          },
+          "editorMode": "code",
+          "expr": "sum(rate(customercore_triage_requests_total[1m])) by (status)",
+          "legendFormat": "{{status}} requests",
+          "range": true
+        }
+      ],
+      "title": "Triage Request Ingestion Rate (by status)",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "Prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth"
+          },
+          "unit": "ms"
+        }
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 0
+      },
+      "id": 3,
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "Prometheus"
+          },
+          "editorMode": "code",
+          "expr": "histogram_quantile(0.95, sum(rate(customercore_triage_duration_ms_bucket[5m])) by (le, priority))",
+          "legendFormat": "p95 - {{priority}} priority",
+          "range": true
+        }
+      ],
+      "title": "95th Percentile Triage Processing Latency (by Priority)",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "Prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "decusd"
+        }
+      },
+      "gridPos": {
+        "h": 6,
+        "w": 6,
+        "x": 0,
+        "y": 8
+      },
+      "id": 4,
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "Prometheus"
+          },
+          "editorMode": "code",
+          "expr": "sum(customercore_llm_cost_usd_total)",
+          "instant": true,
+          "range": false
+        }
+      ],
+      "title": "Accumulated LLM API Cost",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "Prometheus"
+      },
+      "gridPos": {
+        "h": 6,
+        "w": 9,
+        "x": 6,
+        "y": 8
+      },
+      "id": 5,
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "Prometheus"
+          },
+          "editorMode": "code",
+          "expr": "sum(rate(customercore_llm_calls_total[5m])) by (model, status)",
+          "legendFormat": "{{model}} - {{status}}",
+          "range": true
+        }
+      ],
+      "title": "LLM Call Volumetrics & Reliability",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "Prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percent"
+        }
+      },
+      "gridPos": {
+        "h": 6,
+        "w": 9,
+        "x": 15,
+        "y": 8
+      },
+      "id": 6,
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "Prometheus"
+          },
+          "editorMode": "code",
+          "expr": "sum(rate(customercore_cache_hits_total{result=\"hit\"}[5m])) / sum(rate(customercore_cache_hits_total[5m])) * 100",
+          "legendFormat": "Cache Hit Rate",
+          "range": true
+        }
+      ],
+      "title": "Semantic Cache Hit Rate",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "Prometheus"
+      },
+      "gridPos": {
+        "h": 6,
+        "w": 12,
+        "x": 0,
+        "y": 14
+      },
+      "id": 7,
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "Prometheus"
+          },
+          "editorMode": "code",
+          "expr": "sum(increase(customercore_hitl_reviews_total[24h])) by (reason)",
+          "legendFormat": "{{reason}}",
+          "range": true
+        }
+      ],
+      "title": "HITL Queue Trigger Distribution (Last 24h)",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "Prometheus"
+      },
+      "gridPos": {
+        "h": 6,
+        "w": 12,
+        "x": 12,
+        "y": 14
+      },
+      "id": 8,
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "Prometheus"
+          },
+          "editorMode": "code",
+          "expr": "sum(increase(customercore_sla_violations_total[24h])) by (priority)",
+          "legendFormat": "{{priority}} violations",
+          "range": true
+        }
+      ],
+      "title": "SLA Violations Incurred (Last 24h)",
+      "type": "timeseries"
+    }
+  ],
+  "schemaVersion": 36,
+  "style": "dark",
+  "tags": ["customercore", "mlops"],
+  "time": {
+    "from": "now-1h",
+    "to": "now"
+  },
+  "timepicker": {},
+  "timezone": "",
+  "title": "CustomerCore Operations Dashboard",
+  "uid": "customercore-ops-dashboard",
+  "version": 1
+}

infra/monitoring/grafana-dashboards.yml

@@ -0,0 +1,11 @@
+apiVersion: 1
+
+providers:
+  - name: 'CustomerCore Dashboards'
+    orgId: 1
+    folder: ''
+    type: file
+    disableDeletion: false
+    editable: true
+    options:
+      path: /etc/grafana/provisioning/dashboards

infra/monitoring/grafana-datasources.yml

@@ -0,0 +1,8 @@
+apiVersion: 1
+
+datasources:
+  - name: Prometheus
+    type: prometheus
+    access: proxy
+    url: http://prometheus:9090
+    isDefault: true

infra/monitoring/prometheus.yml

@@ -0,0 +1,9 @@
+global:
+  scrape_interval: 10s
+  evaluation_interval: 10s
+
+scrape_configs:
+  - job_name: "customercore-api"
+    metrics_path: "/api/v1/metrics"
+    static_configs:
+      - targets: ["host.docker.internal:8080", "host.docker.internal:7860"]

src/api/models.py

@@ -224,6 +224,9 @@ class TriageResultResponse(BaseModel):
     status: TriageStatus
     tenant_id: str = Field(..., description="Tenant that submitted this ticket.")
     customer_id: str
+    customer_tier: CustomerTier = Field(default=CustomerTier.FREE, description="B2B customer subscription tier.")
+    text: str | None = Field(None, description="The ticket text (PII-masked for safety).")
+    masked_text: str | None = Field(None, description="PII-masked and tokenized ticket body.")
 
     # ── Classification outputs ──
     category: str | None = Field(None, description="AI-classified ticket category.")

src/api/routers/triage.py

@@ -15,6 +15,7 @@ from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException, status
 
 from src.api.auth import AuthenticatedTenant, require_role, verify_token
 from src.api.models import (
+    CustomerTier,
     HITLResumeRequest,
     TicketSubmitRequest,
     TicketSubmitResponse,
@@ -97,6 +98,9 @@ def row_to_response(row: dict[str, Any], violations: list[dict] | None = None) -
         status=TriageStatus(row["status"]),
         tenant_id=str(row["tenant_id"]),
         customer_id=row["customer_id"],
+        customer_tier=CustomerTier(row.get("customer_tier", "free")),
+        text=row.get("masked_text") or row.get("raw_text"),
+        masked_text=row.get("masked_text") or row.get("raw_text"),
         category=row.get("category"),
         priority=TicketPriority(row["priority"]) if row.get("priority") else None,
         confidence=row.get("confidence") or 0.85,
@@ -135,6 +139,19 @@ async def _run_triage(ticket_id: str, text: str, customer_id: str, tenant_id: st
     """
     from src.monitoring.langfuse_tracer import TriageTrace
     from src.responsible_ai.constitutional_policy import policy_engine
+    from src.responsible_ai.privacy_vault import get_vault
+
+    vault = get_vault()
+    try:
+        masked_text, _ = vault.encrypt_text(
+            tenant_id=tenant_id,
+            text=text,
+            analyzer=None,
+            anonymizer=None,
+            ticket_id=ticket_id,
+        )
+    except Exception:
+        masked_text = text
 
     trace = TriageTrace.start(
         ticket_id=ticket_id,
@@ -278,6 +295,10 @@ async def _run_triage(ticket_id: str, text: str, customer_id: str, tenant_id: st
                         f"Constitutional violation: {verdict.violations[0].rule_name}"
                     )
 
+        if isinstance(result, dict):
+            result["customer_tier"] = customer_tier
+            result["masked_text"] = masked_text
+
         # ── HITL or Complete ──────────────────────────────────────────────
         if isinstance(result, dict) and result.get("hitl_required"):
             result_data = {**result, "hitl_reason": result.get("hitl_reason")}
@@ -336,13 +357,29 @@ async def submit_ticket(
     ticket_id = str(uuid4())
     repo = TicketRepository(caller.tenant_id)
     
+    # Mask PII in the ticket body using the zero-trust privacy vault
+    from src.responsible_ai.privacy_vault import get_vault
+    vault = get_vault()
+    try:
+        masked_text, _ = vault.encrypt_text(
+            tenant_id=caller.tenant_id,
+            text=body.text,
+            analyzer=None,
+            anonymizer=None,
+            ticket_id=ticket_id,
+        )
+    except Exception:
+        masked_text = body.text
+
     # Store initial ticket row in pending state
     record = TicketRecord(
         id=ticket_id,
         tenant_id=caller.tenant_id,
         customer_id=body.customer_id,
+        customer_tier=body.customer_tier.value,
         channel=body.channel.value,
         raw_text=body.text,
+        masked_text=masked_text,
         status="pending",
     )
     await repo.create(record)

src/api/ui.py

@@ -1089,8 +1089,14 @@ HTML_CONTENT = """<!DOCTYPE html>
         }
 
         async function generateToken() {
-            const tenant = document.getElementById('widget-tenant').value;
-            const role = document.getElementById('widget-role').value;
+            const tenantEl = document.getElementById('widget-tenant');
+            const roleEl = document.getElementById('widget-role');
+            const tenant = tenantEl.value;
+            const role = roleEl.value;
+            
+            // Adjust dropdown widths automatically
+            adjustSelectWidth(tenantEl);
+            adjustSelectWidth(roleEl);
             
             const authBadge = document.getElementById('auth-status');
             const authText = document.getElementById('auth-status-text');
@@ -1221,6 +1227,7 @@ HTML_CONTENT = """<!DOCTYPE html>
                     clearInterval(pollInterval);
                     btn.disabled = false;
                     btn.innerText = "Dispatch to Triage Pipeline";
+                    showToast("Triage pipeline tracking error — please check connectivity", true);
                 }
             }, 1000);
         }
@@ -1234,14 +1241,20 @@ HTML_CONTENT = """<!DOCTYPE html>
             priorityEl.className = `metric-value priority-badge priority-${priorityVal.toLowerCase()}`;
 
             // Routing
-            document.getElementById('metric-routing').innerText = data.routing_department || "General Support";
+            const routingTeam = (data.escalation_team || data.category || "General Support").replace(/_/g, ' ');
+            document.getElementById('metric-routing').innerText = routingTeam;
 
             // Outage
-            document.getElementById('metric-outage').innerText = data.potential_outage ? "⚠️ Outage Alert!" : "✅ Normal";
+            document.getElementById('metric-outage').innerText = data.incident_active ? "⚠️ Outage Alert!" : "✅ Normal";
 
             // Churn Risk
-            const churnScore = data.churn_risk_score !== undefined ? data.churn_risk_score : 0.15;
-            const churnPercent = Math.round(churnScore * 100);
+            let churnPercent = 15;
+            if (data.churn_risk === "low") churnPercent = 15;
+            else if (data.churn_risk === "medium") churnPercent = 45;
+            else if (data.churn_risk === "high") churnPercent = 75;
+            else if (data.churn_risk === "critical") churnPercent = 95;
+            else if (data.churn_risk_score !== undefined) churnPercent = Math.round(data.churn_risk_score * 100);
+            
             document.getElementById('metric-churn').innerText = `${churnPercent}%`;
             document.getElementById('churn-progress').style.width = `${churnPercent}%`;
 
@@ -1250,11 +1263,11 @@ HTML_CONTENT = """<!DOCTYPE html>
 
             // PII masking logs
             let piiText = `[Vault Ingestion]\n`;
-            piiText += `- Ticket ID: ${data.ticket_id}\n`;
-            piiText += `- Customer ID: ${data.customer_id}\n`;
+            piiText += `- Ticket ID: ${data.ticket_id || '--'}\n`;
+            piiText += `- Customer ID: ${data.customer_id || '--'}\n`;
             piiText += `- Raw Text masked? Yes\n`;
-            if (data.masked_text) {
-                piiText += `- Redacted Body: "${data.masked_text}"\n`;
+            if (data.masked_text || data.text) {
+                piiText += `- Redacted Body: "${data.masked_text || data.text}"\n`;
             }
             document.getElementById('metric-pii-log').innerText = piiText;
 
@@ -1269,7 +1282,9 @@ HTML_CONTENT = """<!DOCTYPE html>
                 if (data.constitutional_violations && data.constitutional_violations.length > 0) {
                     piiText += `\n[Safety Violations Detected]\n`;
                     data.constitutional_violations.forEach(v => {
-                        piiText += `- Rule: ${v.rule_id} | Reason: ${v.reason}\n`;
+                        const ruleId = v.rule || v.rule_id || "UNKNOWN_RULE";
+                        const reason = v.explanation || v.reason || "Safety policy trigger";
+                        piiText += `- Rule: ${ruleId} | Reason: ${reason}\n`;
                     });
                     document.getElementById('metric-pii-log').innerText = piiText;
                 }
@@ -1466,9 +1481,37 @@ HTML_CONTENT = """<!DOCTYPE html>
             }
         }
 
+        // Automatically adjust select box width based on text length
+        function adjustSelectWidth(selectEl) {
+            if (!selectEl) return;
+            const tempSpan = document.createElement("span");
+            tempSpan.style.visibility = "hidden";
+            tempSpan.style.position = "absolute";
+            tempSpan.style.whiteSpace = "pre";
+            
+            const style = window.getComputedStyle(selectEl);
+            tempSpan.style.fontFamily = style.fontFamily;
+            tempSpan.style.fontSize = style.fontSize;
+            tempSpan.style.fontWeight = style.fontWeight;
+            tempSpan.style.letterSpacing = style.letterSpacing;
+            
+            const selectedText = selectEl.options[selectEl.selectedIndex].text;
+            tempSpan.innerText = selectedText;
+            document.body.appendChild(tempSpan);
+            
+            const textWidth = tempSpan.getBoundingClientRect().width;
+            selectEl.style.width = (textWidth + 36) + "px"; // 36px accounts for arrow padding
+            
+            document.body.removeChild(tempSpan);
+        }
+
         // Initialize Page
         window.addEventListener('DOMContentLoaded', () => {
             loadPreset('billing');
+            setTimeout(() => {
+                adjustSelectWidth(document.getElementById('widget-tenant'));
+                adjustSelectWidth(document.getElementById('widget-role'));
+            }, 100);
         });
     </script>
 </body>

src/db/repository.py

@@ -121,6 +121,8 @@ class TicketRecord:
     channel: str
     raw_text: str                       # The original ticket text
     status: str = "pending"
+    customer_tier: str = "free"
+    masked_text: str | None = None
     external_ticket_id: str | None = None
     detected_language: str | None = None
     priority: str | None = None
@@ -274,6 +276,8 @@ class TicketRepository:
             "sentiment":             "sentiment",
             "churn_risk":            "churn_risk_score",
             "churn_risk_score":      "churn_risk_score",
+            "customer_tier":         "customer_tier",
+            "masked_text":           "masked_text",
             "constitutional_score":  "constitutional_score",
             "constitutional_passed": "constitutional_passed",
             "hitl_required":         "hitl_required",
@@ -397,7 +401,7 @@ class TicketRepository:
         try:
             query = (
                 sb.table("tickets")
-                .select("id,status,priority,category,channel,created_at,hitl_required")
+                .select("id,status,priority,category,channel,created_at,hitl_required,customer_id,customer_tier,masked_text")
                 .eq("tenant_id", self.tenant_id)
                 .order("created_at", desc=True)
                 .limit(limit)

src/ml/train_churn.py

@@ -8,6 +8,8 @@ from sklearn.metrics import roc_curve, auc, confusion_matrix, accuracy_score, f1
 import matplotlib
 matplotlib.use('Agg')  # Non-interactive backend
 import matplotlib.pyplot as plt
+import mlflow
+import mlflow.sklearn
 
 def train():
     print("Pre-training steps...")
@@ -53,86 +55,121 @@ def train():
     
     X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
     
-    print("Training Random Forest Churn Predictor...")
-    clf = RandomForestClassifier(random_state=42)
-    clf.fit(X_train, y_train)
+    # Setup MLflow Tracking
+    # If MLFLOW_TRACKING_URI is set, use it. Otherwise, default to local sqlite
+    tracking_uri = os.environ.get("MLFLOW_TRACKING_URI", "sqlite:///mlruns.db")
+    mlflow.set_tracking_uri(tracking_uri)
+    print(f"MLflow Tracking URI set to: {tracking_uri}")
     
-    y_pred = clf.predict(X_test)
-    y_proba = clf.predict_proba(X_test)[:, 1]
+    mlflow.set_experiment("CustomerCore-Churn")
     
-    # Compute metrics
-    fpr, tpr, _ = roc_curve(y_test, y_proba)
-    roc_auc = auc(fpr, tpr)
-    acc = accuracy_score(y_test, y_pred)
-    f1 = f1_score(y_test, y_pred)
-    rec = recall_score(y_test, y_pred)
-    prec = precision_score(y_test, y_pred)
-    
-    metrics = {
-        "auc": round(float(roc_auc), 3),
-        "accuracy": round(float(acc), 3),
-        "f1_score": round(float(f1), 3),
-        "recall": round(float(rec), 3),
-        "precision": round(float(prec), 3)
-    }
-    
-    print("Evaluated Metrics:", metrics)
-    
-    os.makedirs("data", exist_ok=True)
-    with open("data/metrics.json", "w") as f:
-        json.dump(metrics, f, indent=2)
+    # Start MLflow run
+    with mlflow.start_run(run_name="RandomForest-Baseline") as run:
+        print("Training Random Forest Churn Predictor...")
+        clf = RandomForestClassifier(random_state=42)
+        clf.fit(X_train, y_train)
         
-    # Generate Plots
-    print("Generating ROC Curve...")
-    plt.figure(figsize=(6, 5))
-    plt.plot(fpr, tpr, color='darkorange', lw=2, label=f'ROC curve (AUC = {roc_auc:.3f})')
-    plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')
-    plt.xlim([0.0, 1.0])
-    plt.ylim([0.0, 1.05])
-    plt.xlabel('False Positive Rate')
-    plt.ylabel('True Positive Rate')
-    plt.title('Receiver Operating Characteristic (ROC)')
-    plt.legend(loc="lower right")
-    plt.tight_layout()
-    plt.savefig("data/roc_curve.png", dpi=150)
-    plt.close()
-    
-    print("Generating Confusion Matrix...")
-    cm = confusion_matrix(y_test, y_pred)
-    plt.figure(figsize=(5, 4))
-    plt.imshow(cm, interpolation='nearest', cmap=plt.cm.Blues)
-    plt.title('Confusion Matrix')
-    plt.colorbar()
-    tick_marks = np.arange(2)
-    plt.xticks(tick_marks, ['No Churn', 'Churn'])
-    plt.yticks(tick_marks, ['No Churn', 'Churn'])
-    
-    thresh = cm.max() / 2.
-    for i in range(cm.shape[0]):
-        for j in range(cm.shape[1]):
-            plt.text(j, i, format(cm[i, j], 'd'),
-                     ha="center", va="center",
-                     color="white" if cm[i, j] > thresh else "black")
-                     
-    plt.ylabel('True label')
-    plt.xlabel('Predicted label')
-    plt.tight_layout()
-    plt.savefig("data/confusion_matrix.png", dpi=150)
-    plt.close()
+        y_pred = clf.predict(X_test)
+        y_proba = clf.predict_proba(X_test)[:, 1]
+        
+        # Compute metrics
+        fpr, tpr, _ = roc_curve(y_test, y_proba)
+        roc_auc = auc(fpr, tpr)
+        acc = accuracy_score(y_test, y_pred)
+        f1 = f1_score(y_test, y_pred)
+        rec = recall_score(y_test, y_pred)
+        prec = precision_score(y_test, y_pred)
+        
+        metrics = {
+            "auc": round(float(roc_auc), 3),
+            "accuracy": round(float(acc), 3),
+            "f1_score": round(float(f1), 3),
+            "recall": round(float(rec), 3),
+            "precision": round(float(prec), 3)
+        }
+        
+        print("Evaluated Metrics:", metrics)
+        
+        # Log parameters to MLflow
+        mlflow.log_param("n_estimators", clf.n_estimators)
+        mlflow.log_param("criterion", clf.criterion)
+        mlflow.log_param("max_depth", clf.max_depth)
+        mlflow.log_param("min_samples_split", clf.min_samples_split)
+        mlflow.log_param("random_state", 42)
+        mlflow.log_param("test_size", 0.2)
+        
+        # Log metrics to MLflow
+        mlflow.log_metrics(metrics)
+        
+        os.makedirs("data", exist_ok=True)
+        with open("data/metrics.json", "w") as f:
+            json.dump(metrics, f, indent=2)
+            
+        # Generate Plots
+        print("Generating ROC Curve...")
+        plt.figure(figsize=(6, 5))
+        plt.plot(fpr, tpr, color='darkorange', lw=2, label=f'ROC curve (AUC = {roc_auc:.3f})')
+        plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')
+        plt.xlim([0.0, 1.0])
+        plt.ylim([0.0, 1.05])
+        plt.xlabel('False Positive Rate')
+        plt.ylabel('True Positive Rate')
+        plt.title('Receiver Operating Characteristic (ROC)')
+        plt.legend(loc="lower right")
+        plt.tight_layout()
+        plt.savefig("data/roc_curve.png", dpi=150)
+        plt.close()
+        
+        print("Generating Confusion Matrix...")
+        cm = confusion_matrix(y_test, y_pred)
+        plt.figure(figsize=(5, 4))
+        plt.imshow(cm, interpolation='nearest', cmap=plt.cm.Blues)
+        plt.title('Confusion Matrix')
+        plt.colorbar()
+        tick_marks = np.arange(2)
+        plt.xticks(tick_marks, ['No Churn', 'Churn'])
+        plt.yticks(tick_marks, ['No Churn', 'Churn'])
+        
+        thresh = cm.max() / 2.
+        for i in range(cm.shape[0]):
+            for j in range(cm.shape[1]):
+                plt.text(j, i, format(cm[i, j], 'd'),
+                         ha="center", va="center",
+                         color="white" if cm[i, j] > thresh else "black")
+                         
+        plt.ylabel('True label')
+        plt.xlabel('Predicted label')
+        plt.tight_layout()
+        plt.savefig("data/confusion_matrix.png", dpi=150)
+        plt.close()
 
-    print("Generating Feature Importance Plot...")
-    importances = clf.feature_importances_
-    indices = np.argsort(importances)
-    plt.figure(figsize=(6, 4))
-    plt.title('Feature Importances')
-    plt.barh(range(len(indices)), importances[indices], color='b', align='center')
-    plt.yticks(range(len(indices)), [X.columns[i] for i in indices])
-    plt.xlabel('Relative Importance')
-    plt.tight_layout()
-    plt.savefig("data/feature_importance.png", dpi=150)
-    plt.close()
-    
-    print("Training completed successfully!")
+        print("Generating Feature Importance Plot...")
+        importances = clf.feature_importances_
+        indices = np.argsort(importances)
+        plt.figure(figsize=(6, 4))
+        plt.title('Feature Importances')
+        plt.barh(range(len(indices)), importances[indices], color='b', align='center')
+        plt.yticks(range(len(indices)), [X.columns[i] for i in indices])
+        plt.xlabel('Relative Importance')
+        plt.tight_layout()
+        plt.savefig("data/feature_importance.png", dpi=150)
+        plt.close()
+        
+        # Log artifacts (plots) to MLflow
+        mlflow.log_artifact("data/roc_curve.png", "plots")
+        mlflow.log_artifact("data/confusion_matrix.png", "plots")
+        mlflow.log_artifact("data/feature_importance.png", "plots")
+        
+        # Log and Register the model in MLflow Model Registry
+        print("Logging and registering model to MLflow Model Registry...")
+        mlflow.sklearn.log_model(
+            sk_model=clf,
+            artifact_path="model",
+            registered_model_name="customercore-churn-classifier"
+        )
+        
+        print("MLflow Logging and registration completed successfully!")
 
 if __name__ == "__main__":
     train()
+

src/responsible_ai/privacy_vault.py

@@ -143,6 +143,8 @@ class CryptographicPrivacyVault:
     decrypt_token()  — decrypt a token back to plaintext (RBAC-enforced)
     forget_tenant()  — GDPR deletion cascade for all tokens of a tenant
     """
+    _analyzer_cached = None
+    _anonymizer_cached = None
 
     def __init__(
         self,
@@ -217,8 +219,8 @@ class CryptographicPrivacyVault:
         *,
         tenant_id: str,
         text: str,
-        analyzer,
-        anonymizer,
+        analyzer=None,
+        anonymizer=None,
         ticket_id: str = "",
         pii_entities: list[str] | None = None,
     ) -> tuple[str, int]:
@@ -229,6 +231,18 @@ class CryptographicPrivacyVault:
         Returns (tokenized_text, count_of_pii_detected).
         Falls back to standard Presidio anonymization if vault is disabled.
         """
+        if analyzer is None or anonymizer is None:
+            if CryptographicPrivacyVault._analyzer_cached is None or CryptographicPrivacyVault._anonymizer_cached is None:
+                from presidio_analyzer import AnalyzerEngine
+                from presidio_anonymizer import AnonymizerEngine
+                from presidio_analyzer.nlp_engine import NlpEngineProvider
+                nlp_config = {"nlp_engine_name": "spacy", "models": [{"lang_code": "en", "model_name": "en_core_web_sm"}]}
+                nlp_engine = NlpEngineProvider(nlp_configuration=nlp_config).create_engine()
+                CryptographicPrivacyVault._analyzer_cached = AnalyzerEngine(nlp_engine=nlp_engine)
+                CryptographicPrivacyVault._anonymizer_cached = AnonymizerEngine()
+            analyzer = analyzer or CryptographicPrivacyVault._analyzer_cached
+            anonymizer = anonymizer or CryptographicPrivacyVault._anonymizer_cached
+
         if not text or not isinstance(text, str):
             return text, 0
 

supabase/migrations/001_initial_schema.sql

@@ -28,9 +28,12 @@ CREATE TABLE IF NOT EXISTS tickets (
     tenant_id               UUID        NOT NULL REFERENCES tenants(id) ON DELETE CASCADE,
     external_ticket_id      TEXT,                         -- caller-supplied ID (idempotency)
     customer_id             TEXT        NOT NULL,
+    customer_tier           TEXT        NOT NULL DEFAULT 'free'
+                                        CHECK (customer_tier IN ('free','starter','growth','enterprise','vip')),
     channel                 TEXT        NOT NULL DEFAULT 'api'
                                         CHECK (channel IN ('email','chat','api','phone','portal')),
     raw_text                TEXT        NOT NULL,          -- original ticket body (stored encrypted)
+    masked_text             TEXT,
     detected_language       TEXT,
     status                  TEXT        NOT NULL DEFAULT 'pending'
                                         CHECK (status IN ('pending','processing','complete','hitl','failed')),

tests/unit/test_phase12_repository.py

@@ -309,3 +309,49 @@ class TestTenantRepository:
         qb.upsert.assert_called_once()
         call_kwargs = qb.upsert.call_args[1]
         assert call_kwargs.get("on_conflict") == "slug"
+
+
+# ─────────────────────────────────────────────────────────────────────────────
+# Console Bug Fixes and UI integration tests
+# ─────────────────────────────────────────────────────────────────────────────
+
+class TestBugFixes:
+
+    def test_record_new_fields(self):
+        record = TicketRecord(
+            id="t1", tenant_id="t1", customer_id="c1", channel="api", raw_text="text",
+            customer_tier="enterprise", masked_text="masked text"
+        )
+        assert record.customer_tier == "enterprise"
+        assert record.masked_text == "masked text"
+        
+        d = record.to_db_dict()
+        assert d["customer_tier"] == "enterprise"
+        assert d["masked_text"] == "masked text"
+
+    @pytest.mark.asyncio
+    async def test_update_status_maps_new_fields(self):
+        client, qb, _ = _make_supabase_mock()
+        with patch("src.db.repository.get_supabase", new=AsyncMock(return_value=client)):
+            repo = TicketRepository(TENANT_ID)
+            await repo.update_status("ticket-001", "complete", result_data={
+                "customer_tier": "vip",
+                "masked_text": "redacted info"
+            })
+        update_data = qb.update.call_args[0][0]
+        assert update_data["customer_tier"] == "vip"
+        assert update_data["masked_text"] == "redacted info"
+
+    def test_lazy_vault_initialization(self):
+        from src.responsible_ai.privacy_vault import CryptographicPrivacyVault
+        vault = CryptographicPrivacyVault()
+        # Should not crash and should lazily initialize spacy/presidio
+        tokenized, count = vault.encrypt_text(
+            tenant_id="test-tenant",
+            text="Contact me at test@example.com or 555-0199",
+            analyzer=None,
+            anonymizer=None
+        )
+        assert count > 0
+        assert "TOK_EMAIL" in tokenized or "TOK_PHONE" in tokenized
+