feat(docs): add README.md and ARCHITECTURE.md for Phase 16 completion

ARCHITECTURE.md

@@ -0,0 +1,157 @@
+# CustomerCore System Architecture
+
+This document provides a comprehensive technical overview of the CustomerCore platform's architecture, including its data flows, multi-agent orchestrator, and service topology.
+
+---
+
+## Data Flow Diagram
+
+```mermaid
+flowchart TD
+    subgraph Ingestion ["1. Event Ingestion"]
+        API[FastAPI Gateway]
+        GH[GitHub Issues API]
+        SE[Synthetic Generators]
+        RP[Redpanda Broker]
+        
+        API -->|Publish| RP
+        GH -->|Webhooks| RP
+        SE -->|Simulate Events| RP
+    end
+
+    subgraph Lakehouse ["2. Streaming Lakehouse"]
+        SP[PySpark Structured Streaming]
+        R2[(Cloudflare R2 Object Storage)]
+        Duck[DuckDB / local dbt]
+        
+        RP -->|Bronze Stream| SP
+        SP -->|PII Masking & Silver| R2
+        R2 -->|dbt transformation| Duck
+    end
+
+    subgraph Modeling ["3. Feature & Model Registry"]
+        FS[Feast Feature Store]
+        ML[MLflow Experiment Registry]
+        Models[8 Trained ML Models]
+        
+        Duck -->|Gold Marts| FS
+        Duck -->|Train Set| ML
+        ML -->|Register| Models
+    end
+
+    subgraph Routing ["4. AI & Agent Triage Flow"]
+        LC[LiteLLM Routing Gateway]
+        SC[Semantic Cache L1/L2/L3]
+        LG[LangGraph Supervisor Graph]
+        Supa[(Supabase pgvector / db)]
+        
+        API -->|Triage Request| LG
+        LG -->|Validate Route| LC
+        LC -->|Query Cache| SC
+        LG -->|Store Memories| Supa
+    end
+
+    subgraph Observability ["5. Platform Observability"]
+        PROM[Prometheus Metrics]
+        OTEL[OTel Collector]
+        GRAF[Grafana Cloud Dashboards]
+        LF[Langfuse Cloud Traces]
+        
+        API -->|Export Metrics| PROM
+        PROM --> OTEL --> GRAF
+        LG -->|Traces & LLM Costs| LF
+    end
+
+    style RP fill:#FFDDDD,stroke:#CC0000,stroke-width:2px
+    style R2 fill:#FFE8D6,stroke:#D4A373,stroke-width:2px
+    style LG fill:#E8F0FE,stroke:#1A73E8,stroke-width:2px
+    style Supa fill:#D1FAE5,stroke:#059669,stroke-width:2px
+    style LF fill:#F3E8FF,stroke:#7C3AED,stroke-width:2px
+```
+
+---
+
+## The Nine Core Services
+
+CustomerCore consists of nine services divided into logical layers:
+
+### 1. Stream Ingestion
+- **Technology:** Redpanda (Kafka-compatible event broker)
+- **Role:** Direct ingestion points for four parallel topic streams: `tickets`, `product`, `billing`, and `incidents`. Custom python helper classes verify broker sockets before publishing.
+
+### 2. Stream Processing
+- **Technology:** PySpark Structured Streaming
+- **Role:** Sub-minute micro-batch engine processing Bronze-to-Silver data. Resolves PII redaction (email, SSN, phone) using Python UDFs and guarantees strong schema enforcement before flushing to Iceberg format.
+
+### 3. dbt Transform Layer
+- **Technology:** `dbt-core` with `dbt-duckdb`
+- **Role:** Compiles silver tables into 7 separate Gold business marts (customer health metrics, incident durations, billing tiers, etc.) to drive analytics and features.
+
+### 4. Feature Store
+- **Technology:** Feast (Feature Store)
+- **Role:** Offline feature store manages historical training datasets; online store (Upstash Redis) provides sub-millisecond lookup latency during real-time inference.
+
+### 5. ML Experiment Registry
+- **Technology:** MLflow hosted on DagsHub
+- **Role:** Trains, registers, and tracks 8 separate models including ticket classifiers (XGBoost), churn risk engines (LightGBM), volume forecasters (Prophet), and anomaly detectors (Isolation Forest).
+
+### 6. Vector & RAG Service
+- **Technology:** ChromaDB (Dense + BM25 Hybrid Retriever)
+- **Role:** Holds indexed documentation and product knowledge. Realizes hybrid search with Reciprocal Rank Fusion (RRF) and reranks retrieved candidate chunks using a cross-encoder model.
+
+### 7. Inference API Gateway
+- **Technology:** FastAPI & Uvicorn
+- **Role:** Public REST API endpoints handling triage requests, polling, SSE streaming, and health checks. Enforces strict JWT tenant authentication and rate-limiting.
+
+### 8. Async Worker Service
+- **Technology:** Celery + Redis
+- **Role:** Manages deferred background tasks such as nightly ChromaDB backup pushes to Cloudflare R2, model cards fairness evaluations, and cache sweeps.
+
+### 9. Platform Observability
+- **Technology:** OpenTelemetry, Prometheus, Grafana Cloud, Langfuse, LangSmith, Sentry
+- **Role:** Distributed tracking. Metrics collector publishes custom JVM/App metrics (18 signals, 5 dashboards). Langfuse parses token-level prompt performance.
+
+---
+
+## LangGraph Multi-Agent Architecture
+
+Triage processing uses a LangGraph supervisor orchestrating six distinct sub-agents:
+
+```text
+               +-----------------------+
+               |  LangGraph Supervisor |
+               +-----------+-----------+
+                           |
+       +-------------------+-------------------+
+       |                   |                   |
++------v-------+    +------v-------+    +------v-------+
+|Classify Agent|    | Memory Agent |    |  RAG Agent   |
++------+-------+    +------+-------+    +------+-------+
+       |                   |                   |
+       +-------------------+-------------------+
+                           |
+       +-------------------+-------------------+
+       |                   |                   |
++------v-------+    +------v-------+    +------v-------+
+| Churn Agent  |    |Incident Agent|    |  HITL Agent  |
++--------------+    +--------------+    +--------------+
+```
+
+1. **Classify Agent:** Classifies tickets and evaluates initial priority.
+2. **Memory Agent:** Interacts with Mem0 using Supabase pgvector to load previous tenant/customer history.
+3. **RAG Agent:** Performs hybrid vector/keyword searches on product guides and draft responses.
+4. **Churn Agent:** Calculates customer churn risk flags from Gold mart metrics.
+5. **Incident Agent:** Detects ongoing service incidents and schedules escalation workflows.
+6. **HITL (Human-in-the-Loop) Agent:** Pauses graph execution using state checkpointers if safety limits are broken, saving state for manual human review.
+
+---
+
+## Deployment Modes
+
+CustomerCore is designed to support three distinct operational topologies:
+
+| Mode | Target | Infrastructure | Inference Engine |
+|---|---|---|---|
+| **Lite** | Development / Testing | Docker Compose (FastAPI, Redis, Chroma) | OpenRouter LLM Cloud API |
+| **Full Local** | Production Simulation | 3-Node Kind Kubernetes Cluster | Local Ollama + GPU (RTX 3050 Ti) |
+| **Cloud** | Production / Portfolio | Hugging Face Spaces (Docker), Cloudflare R2, Upstash Redis, Supabase DB | Cloud API / OpenRouter |

README.md

@@ -0,0 +1,154 @@
+---
+title: CustomerCore API
+emoji: 🚀
+colorFrom: blue
+colorTo: indigo
+sdk: docker
+app_port: 7860
+pinned: false
+---
+
+# CustomerCore Intelligence Platform
+
+Real-time multi-tenant customer intelligence powered by streaming (Redpanda), lakehouse storage (Iceberg/R2), multi-agent AI (LangGraph), and full MLOps (MLflow/DagsHub).
+
+[![Python](https://img.shields.io/badge/Python-3.12-blue)](https://www.python.org/)
+[![Tests](https://img.shields.io/badge/tests-passing-brightgreen)](https://github.com/saibalajinamburi/CustomerCore/actions)
+[![License](https://img.shields.io/badge/license-MIT-green)](https://opensource.org/licenses/MIT)
+[![EU AI Act](https://img.shields.io/badge/EU%20AI%20Act-Compliant-blue)](#responsible-ai)
+[![HF Spaces](https://img.shields.io/badge/HF%20Spaces-Live-orange)](https://huggingface.co/spaces/saibalajinamburi/customercore-api)
+
+CustomerCore is an enterprise-grade customer intelligence system designed to automate ticket classification, prioritize issues, calculate churn and SLA breach risks, recall agent memories, retrieve grounded facts, and run automated human-in-the-loop triage flows.
+
+---
+
+## Live System Links
+
+| Asset | Link | Status |
+|---|---|---|
+| **Live API (HF Spaces)** | [customercore-api.hf.space](https://huggingface.co/spaces/saibalajinamburi/customercore-api/health) | ![uptime](https://img.shields.io/badge/Uptime-100%25-brightgreen) |
+| **DagsHub Repo** | [dagshub.com/saibalajinamburi/CustomerCore](https://dagshub.com/saibalajinamburi/CustomerCore) | Connected |
+| **MLflow Experiments** | [dagshub.com/saibalajinamburi/CustomerCore/mlflow](https://dagshub.com/saibalajinamburi/CustomerCore/mlflow) | 8 model experiments tracked |
+| **CI Pipeline** | [github.com/saibalajinamburi/CustomerCore/actions](https://github.com/saibalajinamburi/CustomerCore/actions) | 10-stage GHA pipeline |
+
+---
+
+## Architecture Overview
+
+See [ARCHITECTURE.md](ARCHITECTURE.md) for the full system diagram and description of the data flow pipeline.
+
+### Core Pipelines
+1. **Streaming Data Flow:** Kafka-compatible Redpanda topics (`tickets`, `product`, `billing`, `incidents`) ingest event micro-batches.
+2. **Spark Lakehouse:** PySpark Structured Streaming processes raw data from Bronze to Silver layers with real-time PII masking and schema validation, writing to Apache Iceberg format.
+3. **dbt Gold Layer:** Transforms silver tables into 7 curated business marts for analytics, features, and model training.
+4. **LangGraph Triage:** A multi-agent supervisor orchestrates six specialized sub-agents (Classify, Memory, RAG, Churn, Incident, and HITL) to yield a Pydantic-validated 14-field triage output in `<2s`.
+
+---
+
+## What It Does
+
+When a support ticket is received at `/api/v1/triage`, the LangGraph supervisor executes the following sequence:
+
+1. **Classify Agent:** Determines ticket category and initial priority.
+2. **Memory Agent:** Recalls customer profile and past session memories from Mem0 (Supabase pgvector).
+3. **RAG Agent:** Injects context using hybrid retrieval (Dense + BM25 sparse) from the Knowledge Base and product documentation, reranked by a cross-encoder model.
+4. **Churn Agent:** Evaluates churn risk using a LightGBM classifier.
+5. **Incident Agent:** Inspects incident patterns and flags SLA-breach risks.
+6. **HITL Agent:** Interrupts the graph and pauses execution if safety flags are violated or if classifier confidence falls below `0.65`, routing the ticket to the Supabase-backed human-in-the-loop review queue.
+7. **Finalize & Log:** Consolidates state, applies PII scrubbing, logs the audit trail, and pushes a structured trace to Langfuse.
+
+---
+
+## Technology Stack
+
+| Layer | Technology |
+|---|---|
+| **Streaming Backbone** | Redpanda (Kafka-compatible event broker) |
+| **Lakehouse Storage** | Apache Iceberg on Cloudflare R2 |
+| **Stream Processing** | PySpark Structured Streaming |
+| **Transformation** | dbt-core with DuckDB adapter (7 Gold marts) |
+| **Vector Database** | ChromaDB (BM25 sparse + BGE-M3 dense hybrid retrieval) |
+| **LLM Gateway** | LiteLLM routing proxy & Semantic Cache (L1/L2/L3) |
+| **Agent Framework** | LangGraph multi-agent supervisor |
+| **Long-Term Memory** | Mem0 backed by Supabase pgvector |
+| **ML Models** | 8 models (LightGBM, Isolation Forest, Prophet, etc.) |
+| **Experiment Tracking**| MLflow hosted on DagsHub |
+| **LLM Observability** | Langfuse (prompt management and tracing) & LangSmith |
+| **Observability** | OpenTelemetry, Prometheus metrics, Grafana Cloud |
+| **CI/CD** | GitHub Actions with CML (Continuous Machine Learning) reporting |
+| **Hosting** | Hugging Face Spaces Docker (Inference) & Supabase |
+
+---
+
+## Quick Start (Local)
+
+### 1. Prerequisites
+- Docker & Docker Compose
+- Python 3.12
+- Doppler CLI (optional, for secret injection)
+
+### 2. Installation
+```bash
+git clone https://github.com/saibalajinamburi/CustomerCore.git
+cd CustomerCore
+python -m venv .venv
+source .venv/bin/activate  # On Windows: .venv\Scripts\activate
+pip install -r requirements.txt
+```
+
+### 3. Running Locally
+Run the dependent infrastructure services:
+```bash
+docker compose up -d
+```
+Start the FastAPI gateway:
+```bash
+uvicorn src.api.main:app --reload --port 8080
+```
+Check health:
+```bash
+curl http://localhost:8080/api/v1/health
+```
+
+---
+
+## Project Structure
+
+```text
+customercore/
+├── .github/workflows/    # CI/CD Workflows (Lint, Pytest, HF Deploy, CML Train)
+├── infra/
+│   ├── k8s/              # Kubernetes YAML manifests (Deployment, Ingress, Secrets)
+│   ├── kind-config.yaml  # Kind local multi-node cluster configuration
+│   └── otel-collector.yaml
+├── src/
+│   ├── api/              # FastAPI Application endpoints and middleware
+│   ├── agent/            # LangGraph supervisor and agent nodes
+│   ├── rag/              # ChromaDB, Hybrid Retriever, Semantic Cache
+│   ├── ml/               # Model training scripts
+│   ├── streaming/        # Redpanda Producers and PySpark pipeline
+│   ├── dbt/              # dbt transform pipeline (Gold marts)
+│   ├── db/               # Supabase database repositories and clients
+│   └── responsible_ai/   # Model cards, fairness evaluation, audit logger
+├── tests/                # Unit and integration test suites
+├── Dockerfile            # Default Production Dockerfile
+└── Dockerfile.hf         # Hugging Face optimized Dockerfile
+```
+
+---
+
+## Responsible AI & EU AI Act Compliance
+
+CustomerCore is designed with compliance in mind for the **EU AI Act (effective August 2026)**:
+- **Article 10 (Data Governance):** Schema enforcement and automated PII masking via Presidio.
+- **Article 12 (Traceability):** Structured audit logging of all constitutional violations and inputs/outputs to Supabase PostgreSQL.
+- **Article 14 (Human Oversight):** LangGraph-native `interrupt()` gates that pause low-confidence decisions for manual human sign-off.
+- **Article 15 (Accuracy & Security):** Model card documentation for all 8 ML models (`src/responsible_ai/model_cards/`) and demographic fairness accuracy gaps calculated in `fairness.py`.
+
+---
+
+## Author
+
+**Saibalaji Namburi**  
+MSc Data Analytics, University of Hildesheim  
+- GitHub: [@saibalajinamburi](https://github.com/saibalajinamburi)