Upload 28 files

.dockerignore

@@ -0,0 +1,5 @@
+__pycache__/
+*.pyc
+*.db
+.git/
+.venv/

.gitignore

@@ -0,0 +1,5 @@
+__pycache__/
+*.pyc
+*.db
+.cache/
+.venv/

APPLY_V3.md

@@ -0,0 +1,58 @@
+# Jak aplikovat V3 do repozitáře `sentiment_analysis_api`
+
+Tento balíček je kompletní snapshot projektu ve stavu V3 - staví na V2 a
+přidává dynamický výběr sentiment modelu z Hugging Face Hubu a XML export
+uložených záznamů. Aplikuje se **po** V2 (tj. repozitář by měl už mít commit
+a tag `v2.0`).
+
+## Co je nové oproti V2
+
+- `processors/model_registry.py` - loader/cache pro libovolný HF
+  `text-classification` model (max 3 modely v paměti, `trust_remote_code`
+  nikdy zapnuto).
+- `processors/sentiment.py`, `pipeline.py`, `main.py` - `sentiment_model`
+  parametr v `/analyze`, `/predict`, `/ingest` (form pole).
+- Nový endpoint `GET /models` (výchozí/doporučené/cachované modely).
+- Nový endpoint `GET /records/export.xml` a `db.export_xml()` (stdlib
+  `xml.etree.ElementTree`, žádná nová závislost).
+- `static/index.html` + `app.js` - pole pro zadání modelu (s návrhy z
+  `/models`).
+- `static/records.html` - tlačítko „Export XML".
+- Aktualizovaný `README.md` a `docs/class_diagram.md`.
+
+## Postup
+
+1. Zkopíruj **veškerý obsah** této složky do kořene repozitáře (přepiš
+   `main.py`, `db.py`, `pipeline.py`, `README.md`, `docs/class_diagram.md`,
+   `static/*`; přidej nový `processors/model_registry.py`).
+2. `git status` a commit:
+
+   ```bash
+   git add -A
+   git commit -m "V3: dynamic Hugging Face model selection for sentiment analysis + XML export of stored records"
+   git tag -a v3.0 -m "V3: dynamic HF model selection + XML export"
+   ```
+
+3. Push, až budeš chtít:
+
+   ```bash
+   git push origin main
+   git push origin v3.0
+   ```
+
+## Lokální ověření před commitem (doporučeno)
+
+```bash
+pip install -r requirements.txt
+python -m spacy download en_core_web_lg
+uvicorn main:app --reload --port 8000
+```
+
+Pak vyzkoušej:
+- `GET /models` - měl by vrátit výchozí + doporučené modely
+- `POST /analyze` s `"sentiment_model": "distilbert-base-uncased-finetuned-sst-2-english"`
+  (první request tento model stáhne, může chvíli trvat)
+- `POST /analyze` s neplatným `sentiment_model` (např. `"neexistujici/model"`)
+  - očekávej HTTP 400 se srozumitelnou chybou, ne pád serveru
+- `GET /records/export.xml` - stažení/zobrazení platného XML se záznamy
+- na stránce `/static/records.html` tlačítko „Export XML"

Dockerfile

@@ -1,24 +1,38 @@
-FROM python:3.11-slim
-
-ENV PYTHONUNBUFFERED=1 \
-    PYTHONDONTWRITEBYTECODE=1 \
-    PIP_NO_CACHE_DIR=1 \
-    TRANSFORMERS_CACHE=/app/.cache
-
-WORKDIR /app
-
-# System deps (git, curl; plus basic build tools for torch if needed)
-RUN apt-get update && apt-get install -y --no-install-recommends \
-    git curl build-essential \
-    && rm -rf /var/lib/apt/lists/*
-
-COPY requirements.txt ./
-RUN pip install --upgrade pip && pip install -r requirements.txt
-
-# Copy app code + model
-COPY . .
-
-EXPOSE 8000
-
-CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
-
+FROM python:3.11-slim
+
+ENV PYTHONUNBUFFERED=1 \
+    PYTHONDONTWRITEBYTECODE=1 \
+    PIP_NO_CACHE_DIR=1 \
+    HF_HOME=/app/.cache \
+    TRANSFORMERS_CACHE=/app/.cache \
+    CONV_DB_PATH=/tmp/conversation_logs.db
+
+WORKDIR /app
+
+# System dependencies
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    git curl build-essential \
+    && rm -rf /var/lib/apt/lists/*
+
+# Python dependencies
+COPY requirements.txt ./
+RUN pip install --upgrade pip && pip install -r requirements.txt
+
+# spaCy model required by Presidio for name/location detection
+RUN python -m spacy download en_core_web_lg
+
+# Pre-download the Hugging Face models at build time so the first request is fast.
+# (Comment out to download lazily on first use instead.)
+RUN python - <<'PY'
+from transformers import pipeline
+pipeline("token-classification", model="dslim/bert-base-NER", aggregation_strategy="simple")
+pipeline("zero-shot-classification", model="facebook/bart-large-mnli")
+print("Models cached.")
+PY
+
+# Application code
+COPY . .
+
+EXPOSE 8000
+
+CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

README.md

@@ -1,8 +1,109 @@
 ---
-title: Sentiment Analysis API
+title: Conversation Data Extraction System
 sdk: docker
-emoji: 😃
+emoji: 🗂️
 colorFrom: indigo
 colorTo: purple
 app_port: 8000
----
+---
+
+# Conversation Data Extraction System
+
+A system that extracts structured information from chat conversations:
+**named entities (NER), topics and sentiment**, then **pseudonymizes** personal
+data (GDPR) and stores the structured result.
+
+## Project versions
+
+This repository is developed incrementally as part of a diploma thesis. Each
+version is tagged so the progression is visible in the git history.
+
+| Version | Tag | Description |
+|---------|-----|--------------|
+| V1 | `v1.0` | Single fine-tuned RoBERTa model, `/predict` endpoint, simple SQLite logging with regex anonymization. |
+| V2 | `v2.0` | Full extraction pipeline: batch `/ingest`, NER, zero-shot topic classification, Presidio-based anonymization, dashboard with stored records and statistics. |
+| V3 | `v3.0` | Dynamic sentiment model selection from the Hugging Face Hub at request time, plus XML export of stored records. |
+
+See [`docs/class_diagram.md`](docs/class_diagram.md) for the current architecture.
+
+## Pipeline
+
+```
+ingest → NER → topic classification → sentiment → anonymization → storage
+```
+
+| Step | Method / model |
+|------|----------------|
+| NER | `dslim/bert-base-NER` (inference) |
+| Topics | `facebook/bart-large-mnli` (zero-shot) |
+| Sentiment | fine-tuned RoBERTa (`vojmahdal/roberta-sentiment-3labels`) |
+| Anonymization | Microsoft Presidio (spaCy NER + regex), regex fallback |
+| Storage | SQLite (hash of original + anonymized text) |
+
+## Endpoints
+
+| Method | Path | Description |
+|--------|------|-------------|
+| GET | `/` | Web dashboard |
+| GET | `/health` | Service + model status |
+| GET | `/models` | Default, suggested and currently cached sentiment models |
+| POST | `/analyze` | Full pipeline on a single message |
+| POST | `/predict` | Sentiment only (backward compatible) |
+| POST | `/ingest` | Batch ingest of a CSV/JSON file |
+| GET | `/records` | Recent stored (anonymized) records |
+| GET | `/records/export.xml` | Stored records exported as XML |
+| GET | `/stats` | Aggregate statistics |
+
+### Example
+
+```bash
+curl -X POST https://<space-url>/analyze \
+  -H "Content-Type: application/json" \
+  -d '{"text": "Hi, John Smith here, my order never arrived. Email john@example.com"}'
+```
+
+```bash
+curl -X POST https://<space-url>/ingest -F "file=@sample_chats.csv"
+```
+
+```bash
+curl https://<space-url>/records/export.xml -o records.xml
+```
+
+## Choosing a sentiment model from the Hugging Face Hub
+
+Since V3, `/analyze`, `/predict` and `/ingest` accept an optional
+`sentiment_model` field (a Hugging Face repo id, e.g.
+`cardiffnlp/twitter-roberta-base-sentiment-latest`). If omitted, the
+author's fine-tuned model (`vojmahdal/roberta-sentiment-3labels`) is used.
+Requested models are downloaded and cached in memory on first use
+(`processors/model_registry.py`), with a small FIFO cache (3 models) to
+bound memory usage. `GET /models` lists the default model, a few suggested
+models, and which ones are currently cached. The web dashboard exposes this
+as an editable field with suggestions.
+
+**Security note:** loaded pipelines never use `trust_remote_code=True`, so an
+arbitrary/untrusted model id supplied by a caller cannot execute custom
+Python code inside the server process - it is limited to standard
+`transformers` text-classification inference.
+
+## Data protection
+
+The original message text is **never stored in readable form**. Only a SHA-256
+hash (for deduplication) and the anonymized text are persisted. Because the
+transformation is reversible in principle and re-identification could occur with
+additional information, the approach is **pseudonymization** under the GDPR
+(Art. 4(5)); stored data therefore remains personal data and is handled with
+data minimization in mind.
+
+## Local run
+
+```bash
+pip install -r requirements.txt
+python -m spacy download en_core_web_lg
+uvicorn main:app --reload --port 8000
+```
+
+`requirements-dev.txt` holds extra dependencies (`pandas`, `seqeval`) needed
+only by the offline helper scripts `prepare_dataset.py` and `evaluate.py` -
+not required to run the API itself.

db.py

@@ -1,158 +1,218 @@
-import os
-import sqlite3
-import threading
-import hashlib
-import re
-from datetime import datetime
-from pathlib import Path
-from typing import Any
-
-
-# ---------------------------------------------------------------------------
-# SQLite "database" for logging / anonymized storage
-# ---------------------------------------------------------------------------
-
-if os.name == "nt":
-    # Windows (locally) – save to project folder
-    DB_PATH = os.getenv(
-        "SENTIMENT_DB_PATH",
-        str((Path(__file__).resolve().parent / "sentiment_logs.db")),
-    )
-else:
-    # Linux (e.g. Hugging Face Spaces) – /tmp
-    DB_PATH = os.getenv("SENTIMENT_DB_PATH", "/tmp/sentiment_logs.db")
-
-_db_lock = threading.Lock()
-
-
-def _get_connection() -> sqlite3.Connection:
-    """
-    Get global SQLite connection; creates table on first use.
-    Uses /tmp by default so it is compatible with Hugging Face Spaces (data may be wiped on restart, which is fine for demo / thesis).
-    """
-    conn = sqlite3.connect(DB_PATH, check_same_thread=False)
-    conn.execute(
-        """
-        CREATE TABLE IF NOT EXISTS logs (
-            id INTEGER PRIMARY KEY AUTOINCREMENT,
-            created_at TEXT NOT NULL,
-            original_hash TEXT NOT NULL,
-            anonymized_text TEXT NOT NULL,
-            label TEXT NOT NULL,
-            score REAL NOT NULL,
-            client_ip TEXT
-        )
-        """
-    )
-    return conn
-
-
-_db_conn = _get_connection()
-
-
-_EMAIL_RE = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}")
-_PHONE_RE = re.compile(r"\+?\d[\d\s\-]{7,}\d")
-_DIGITS_RE = re.compile(r"\d{3,}")
-
-# very simple detection of names: sequence of 2–3 words starting with a capital letter
-_NAME_SEQ_RE = re.compile(
-    r"\b([A-ZÁČĎÉĚÍŇÓŘŠŤÚŮÝŽ][a-záčďéěíňóřšťúůýž]+(?:\s+[A-ZÁČĎÉĚÍŇÓŘŠŤÚŮÝŽ][a-záčďéěíňóřšťúůýž]+){1,2})\b"
-)
-
-
-def _anonymize_names_outside_sentence_start(text: str) -> str:
-  
-    parts = re.split(r"(\n+|[.!?])", text)
-    if len(parts) == 1:
-        return _anonymize_names_skip_first_word(text)
-
-    processed: list[str] = []
-
-    for i in range(0, len(parts), 2):
-        sentence = parts[i]
-        terminator = parts[i + 1] if i + 1 < len(parts) else ""
-        processed_sentence = _anonymize_names_skip_first_word(sentence)
-        processed.append(processed_sentence + terminator)
-
-    return "".join(processed)
-
-
-def _anonymize_names_skip_first_word(sentence: str) -> str:
-
-    if not sentence:
-        return sentence
-
-    m = re.match(r"^(\s*\S+)(.*)$", sentence, flags=re.DOTALL)
-    if not m:
-        return sentence
-
-    first_word = m.group(1)
-    rest = m.group(2)
-
-
-    rest = _NAME_SEQ_RE.sub("[NAME]", rest)
-    return first_word + rest
-
-
-def anonymize_text(text: str) -> str:
-    """
-    Very simple anonymization:
-    - e-mails -> [EMAIL]
-    - phone numbers -> [PHONE]
-    - longer sequences of digits -> [NUMBER]
-    - name-like sequences like 'Jan Novák' -> [NAME], but not at the beginning of a sentence
-    """
-    t = _EMAIL_RE.sub("[EMAIL]", text)
-    t = _PHONE_RE.sub("[PHONE]", t)
-    t = _DIGITS_RE.sub("[NUMBER]", t)
-    t = _anonymize_names_outside_sentence_start(t)
-    return t
-
-
-def log_to_db(original_text: str, label: str, score: float, client_ip: str | None) -> None:
-    """
-    Save record to SQLite:
-    - hash original text (SHA256) due to impossibility to reconstruct
-    - anonymized text according to anonymize_text
-    """
-    created_at = datetime.utcnow().isoformat(timespec="seconds") + "Z"
-    original_hash = hashlib.sha256(original_text.encode("utf-8")).hexdigest()
-    anonymized = anonymize_text(original_text)
-
-    with _db_lock:
-        _db_conn.execute(
-            """
-            INSERT INTO logs (created_at, original_hash, anonymized_text, label, score, client_ip)
-            VALUES (?, ?, ?, ?, ?, ?)
-            """,
-            (created_at, original_hash, anonymized, label, float(score), client_ip),
-        )
-        _db_conn.commit()
-
-
-def get_logs_from_db(limit: int = 100) -> list[dict[str, Any]]:
-
-    with _db_lock:
-        cur = _db_conn.execute(
-            """
-            SELECT id, created_at, anonymized_text, label, score, client_ip
-            FROM logs
-            ORDER BY id DESC
-            LIMIT ?
-            """,
-            (limit,),
-        )
-        rows = cur.fetchall()
-
-    return [
-        {
-            "id": r[0],
-            "created_at": r[1],
-            "anonymized_text": r[2],
-            "label": r[3],
-            "score": r[4],
-            "client_ip": r[5],
-        }
-        for r in rows
-    ]
-
+"""
+Database layer (SQLite).
+
+Stores the structured output of the pipeline in a pseudonymized form. The
+original text is never stored in readable form: only a SHA-256 hash (for
+deduplication) and the anonymized text are persisted, together with the
+extracted entities, topic and sentiment.
+
+SQLite was chosen for the prototype (serverless, zero-config, portable). The
+layer is intentionally small so it can be swapped for PostgreSQL in a
+production deployment without changing the rest of the application.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import os
+import sqlite3
+import threading
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any
+from xml.etree import ElementTree as ET
+
+from processors import anonymizer
+
+# ---------------------------------------------------------------------------
+# Database location: project folder on Windows, /tmp on Linux (HF Spaces).
+# ---------------------------------------------------------------------------
+if os.name == "nt":
+    DB_PATH = os.getenv(
+        "CONV_DB_PATH",
+        str((Path(__file__).resolve().parent / "conversation_logs.db")),
+    )
+else:
+    DB_PATH = os.getenv("CONV_DB_PATH", "/tmp/conversation_logs.db")
+
+_db_lock = threading.Lock()
+
+
+def _get_connection() -> sqlite3.Connection:
+    conn = sqlite3.connect(DB_PATH, check_same_thread=False)
+    conn.execute(
+        """
+        CREATE TABLE IF NOT EXISTS records (
+            id INTEGER PRIMARY KEY AUTOINCREMENT,
+            created_at TEXT NOT NULL,
+            original_hash TEXT NOT NULL,
+            anonymized_text TEXT NOT NULL,
+            entities TEXT,            -- JSON array
+            topic TEXT,
+            topic_score REAL,
+            sentiment TEXT,
+            sentiment_score REAL,
+            conversation_id TEXT,
+            source TEXT               -- 'single' or 'ingest'
+        )
+        """
+    )
+    conn.commit()
+    return conn
+
+
+_db_conn = _get_connection()
+
+
+def _utcnow() -> str:
+    return datetime.now(timezone.utc).isoformat(timespec="seconds")
+
+
+def save_record(result: dict[str, Any], source: str = "single") -> None:
+    """
+    Persist one pipeline result. The original text is hashed (not stored);
+    the anonymized text is stored. If the result does not already contain an
+    anonymized text, it is anonymized here as a safeguard.
+    """
+    original_text = result.get("text", "") or ""
+    anonymized = result.get("anonymized_text") or anonymizer.anonymize_text(original_text)
+    original_hash = hashlib.sha256(original_text.encode("utf-8")).hexdigest()
+
+    entities_json = json.dumps(result.get("entities", []), ensure_ascii=False)
+
+    with _db_lock:
+        _db_conn.execute(
+            """
+            INSERT INTO records (
+                created_at, original_hash, anonymized_text, entities,
+                topic, topic_score, sentiment, sentiment_score,
+                conversation_id, source
+            ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+            """,
+            (
+                _utcnow(),
+                original_hash,
+                anonymized,
+                entities_json,
+                result.get("topic"),
+                result.get("topic_score"),
+                result.get("sentiment"),
+                result.get("sentiment_score"),
+                result.get("conversation_id"),
+                source,
+            ),
+        )
+        _db_conn.commit()
+
+
+def get_records(limit: int = 100) -> list[dict[str, Any]]:
+    """Return the most recent records (anonymized only)."""
+    with _db_lock:
+        cur = _db_conn.execute(
+            """
+            SELECT id, created_at, anonymized_text, entities,
+                   topic, topic_score, sentiment, sentiment_score,
+                   conversation_id, source
+            FROM records
+            ORDER BY id DESC
+            LIMIT ?
+            """,
+            (limit,),
+        )
+        rows = cur.fetchall()
+
+    records = []
+    for r in rows:
+        try:
+            entities = json.loads(r[3]) if r[3] else []
+        except Exception:
+            entities = []
+        records.append(
+            {
+                "id": r[0],
+                "created_at": r[1],
+                "anonymized_text": r[2],
+                "entities": entities,
+                "topic": r[4],
+                "topic_score": r[5],
+                "sentiment": r[6],
+                "sentiment_score": r[7],
+                "conversation_id": r[8],
+                "source": r[9],
+            }
+        )
+    return records
+
+
+def stats() -> dict[str, Any]:
+    """Aggregate statistics for the dashboard (counts by sentiment / topic)."""
+    with _db_lock:
+        total = _db_conn.execute("SELECT COUNT(*) FROM records").fetchone()[0]
+        by_sentiment = _db_conn.execute(
+            "SELECT sentiment, COUNT(*) FROM records GROUP BY sentiment"
+        ).fetchall()
+        by_topic = _db_conn.execute(
+            "SELECT topic, COUNT(*) FROM records GROUP BY topic ORDER BY COUNT(*) DESC LIMIT 10"
+        ).fetchall()
+
+    return {
+        "total": total,
+        "by_sentiment": {(s or "unknown"): c for s, c in by_sentiment},
+        "by_topic": {(t or "unknown"): c for t, c in by_topic},
+    }
+
+
+def export_xml(limit: int | None = None) -> bytes:
+    """
+    Export stored (anonymized) records as XML.
+
+    Mirrors the fields returned by ``get_records``. ``limit`` caps the number
+    of most recent records exported; ``None`` exports everything.
+    """
+    query = """
+        SELECT id, created_at, anonymized_text, entities,
+               topic, topic_score, sentiment, sentiment_score,
+               conversation_id, source
+        FROM records
+        ORDER BY id DESC
+    """
+    if limit is not None:
+        query += " LIMIT ?"
+        params: tuple[Any, ...] = (limit,)
+    else:
+        params = ()
+
+    with _db_lock:
+        rows = _db_conn.execute(query, params).fetchall()
+
+    root = ET.Element("records")
+    for r in rows:
+        record_el = ET.SubElement(root, "record", id=str(r[0]))
+        ET.SubElement(record_el, "created_at").text = r[1]
+        ET.SubElement(record_el, "anonymized_text").text = r[2]
+        ET.SubElement(record_el, "topic").text = r[4]
+        ET.SubElement(record_el, "topic_score").text = (
+            str(r[5]) if r[5] is not None else None
+        )
+        ET.SubElement(record_el, "sentiment").text = r[6]
+        ET.SubElement(record_el, "sentiment_score").text = (
+            str(r[7]) if r[7] is not None else None
+        )
+        ET.SubElement(record_el, "conversation_id").text = r[8]
+        ET.SubElement(record_el, "source").text = r[9]
+
+        entities_el = ET.SubElement(record_el, "entities")
+        try:
+            entities = json.loads(r[3]) if r[3] else []
+        except Exception:
+            entities = []
+        for ent in entities:
+            ET.SubElement(
+                entities_el,
+                "entity",
+                type=str(ent.get("type", "")),
+                score=str(ent.get("score", "")),
+            ).text = ent.get("text", "")
+
+    return ET.tostring(root, encoding="utf-8", xml_declaration=True)

docs/class_diagram.md

@@ -0,0 +1,90 @@
+# Class diagram - V3
+
+Modules are shown as facade classes over their public functions. GitHub
+renders this Mermaid diagram directly. Compared to V2, this adds
+`ModelRegistry` (dynamic Hugging Face model loading/caching for sentiment
+analysis) and `export_xml()` on `Database`.
+
+```mermaid
+classDiagram
+    class API {
+        <<main.py>>
+        +home() FileResponse
+        +health() dict
+        +list_models() dict
+        +analyze(payload) dict
+        +predict(payload) dict
+        +ingest_file(file, sentiment_model) dict
+        +records(limit) list
+        +export_records_xml(limit) Response
+        +get_stats() dict
+    }
+
+    class IngestService {
+        <<ingest.py>>
+        +parse_csv(raw) list
+        +parse_json(raw) list
+        +parse_upload(filename, raw) list
+    }
+
+    class Pipeline {
+        <<pipeline.py>>
+        +process_message(text, topic_labels, sentiment_model) dict
+        +process_batch(messages, topic_labels, sentiment_model) list
+    }
+
+    class NERProcessor {
+        <<processors/ner.py>>
+        -model_name: str = "dslim/bert-base-NER"
+        +extract_entities(text) list
+        +is_ready() bool
+    }
+
+    class TopicClassifier {
+        <<processors/topics.py>>
+        -model_name: str = "facebook/bart-large-mnli"
+        +classify_topic(text, labels) dict
+        +is_ready() bool
+    }
+
+    class SentimentAnalyzer {
+        <<processors/sentiment.py>>
+        -default_model_name: str = "vojmahdal/roberta-sentiment-3labels"
+        +analyze_sentiment(text, model_id) dict
+        +is_ready() bool
+    }
+
+    class ModelRegistry {
+        <<processors/model_registry.py>>
+        -cache: dict~str, Pipeline~
+        -max_cached_models: int = 3
+        +get_pipeline(model_id, task) Pipeline
+        +cached_models() list
+    }
+
+    class Anonymizer {
+        <<processors/anonymizer.py>>
+        -backend: presidio | regex-fallback
+        +anonymize_text(text) str
+        +detect_pii(text) list
+        +backend_name() str
+    }
+
+    class Database {
+        <<db.py>>
+        +save_record(result, source) void
+        +get_records(limit) list
+        +stats() dict
+        +export_xml(limit) bytes
+    }
+
+    API --> IngestService : parses uploaded files
+    API --> Pipeline : runs analysis
+    API --> Database : reads/writes/export records
+    API --> ModelRegistry : lists cached models
+    Pipeline --> NERProcessor
+    Pipeline --> TopicClassifier
+    Pipeline --> SentimentAnalyzer
+    Pipeline --> Anonymizer
+    SentimentAnalyzer --> ModelRegistry : loads non-default HF models
+```

evaluate.py

@@ -0,0 +1,119 @@
+"""
+Evaluation scripts for the extraction system.
+
+Provides entity-level evaluation of NER (precision / recall / F1 via seqeval)
+and recall-oriented evaluation of the anonymizer. These scripts produce the
+numbers used in the "Testing" chapter of the thesis.
+
+Usage:
+    python evaluate.py ner       # evaluate NER on a small annotated sample
+    python evaluate.py anonymize # evaluate PII recall (NER vs regex)
+
+The samples here are tiny illustrative examples. For the thesis, replace them
+with a real annotated dataset (e.g. a subset of CoNLL-2003 for NER, or
+ai4privacy/pii-masking-200k for anonymization).
+"""
+
+from __future__ import annotations
+
+import sys
+
+
+# ---------------------------------------------------------------------------
+# NER evaluation (entity level, seqeval)
+# ---------------------------------------------------------------------------
+def evaluate_ner():
+    from seqeval.metrics import classification_report, f1_score, precision_score, recall_score
+    from processors import ner
+
+    # Each example: (tokens, gold BIO tags). Illustrative only.
+    samples = [
+        (
+            ["My", "name", "is", "John", "Smith", "from", "London"],
+            ["O", "O", "O", "B-PER", "I-PER", "O", "B-LOC"],
+        ),
+        (
+            ["Sarah", "works", "at", "Google", "in", "Berlin"],
+            ["B-PER", "O", "O", "B-ORG", "O", "B-LOC"],
+        ),
+    ]
+
+    y_true, y_pred = [], []
+    for tokens, gold in samples:
+        text = " ".join(tokens)
+        ents = ner.extract_entities(text)
+
+        # Build predicted BIO tags aligned to whitespace tokens.
+        pred = ["O"] * len(tokens)
+        # character offset of each token
+        offsets = []
+        pos = 0
+        for tok in tokens:
+            start = text.index(tok, pos)
+            offsets.append((start, start + len(tok)))
+            pos = start + len(tok)
+
+        for ent in ents:
+            etype = ent["type"]
+            first = True
+            for i, (s, e) in enumerate(offsets):
+                # token overlaps the entity span
+                if s >= ent["start"] and e <= ent["end"] + 1:
+                    pred[i] = ("B-" if first else "I-") + etype
+                    first = False
+
+        y_true.append(gold)
+        y_pred.append(pred)
+
+    print("=== NER evaluation (entity level) ===")
+    print(classification_report(y_true, y_pred))
+    print(f"Precision: {precision_score(y_true, y_pred):.4f}")
+    print(f"Recall:    {recall_score(y_true, y_pred):.4f}")
+    print(f"F1:        {f1_score(y_true, y_pred):.4f}")
+
+
+# ---------------------------------------------------------------------------
+# Anonymization evaluation (PII recall: NER+Presidio vs regex only)
+# ---------------------------------------------------------------------------
+def evaluate_anonymize():
+    from processors import anonymizer
+    import re
+
+    # (text, list of PII substrings that MUST be removed)
+    samples = [
+        ("My name is John Smith, email john@example.com", ["John Smith", "john@example.com"]),
+        ("Call Sarah at +1 202 555 0143", ["Sarah", "+1 202 555 0143"]),
+        ("I live in Berlin and work at Google", ["Berlin", "Google"]),
+    ]
+
+    def recall(anon_fn):
+        found, total = 0, 0
+        for text, pii_list in samples:
+            anon = anon_fn(text)
+            for pii in pii_list:
+                total += 1
+                # PII counts as removed if it no longer appears verbatim
+                if pii.lower() not in anon.lower():
+                    found += 1
+        return found / total if total else 0.0
+
+    # regex-only baseline (the original approach)
+    def regex_only(text):
+        from processors.anonymizer import _regex_anonymize
+        return _regex_anonymize(text)
+
+    print("=== Anonymization evaluation (PII recall) ===")
+    print(f"Active backend:       {anonymizer.backend_name()}")
+    print(f"Recall (regex only):  {recall(regex_only):.2%}")
+    print(f"Recall (full system): {recall(anonymizer.anonymize_text):.2%}")
+    print("\nNote: full system requires Presidio + spaCy model for name/location recall.")
+
+
+if __name__ == "__main__":
+    mode = sys.argv[1] if len(sys.argv) > 1 else "ner"
+    if mode == "ner":
+        evaluate_ner()
+    elif mode == "anonymize":
+        evaluate_anonymize()
+    else:
+        print("Usage: python evaluate.py [ner|anonymize]")

gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

ingest.py

@@ -0,0 +1,113 @@
+"""
+Ingest component.
+
+Accepts conversation data as a batch (CSV or JSON), normalizes it into a unified
+structure, and hands it over to the NLP pipeline. This is the "ingest" module
+required by the assignment.
+
+Unified message schema:
+    {
+        "conversation_id": <str | None>,
+        "speaker":         <str | None>,
+        "timestamp":       <str | None>,
+        "text":            <str>          # required
+    }
+"""
+
+from __future__ import annotations
+
+import csv
+import io
+import json
+from typing import Any
+
+# Column / key names we accept for the message text, in priority order.
+_TEXT_KEYS = ["text", "message", "content", "body", "utterance"]
+_ID_KEYS = ["conversation_id", "conv_id", "id", "dialog_id"]
+_SPEAKER_KEYS = ["speaker", "author", "role", "user"]
+_TIME_KEYS = ["timestamp", "time", "created_at", "date"]
+
+
+def _first_present(row: dict[str, Any], keys: list[str]) -> Any:
+    for k in keys:
+        if k in row and row[k] not in (None, ""):
+            return row[k]
+        # case-insensitive match
+        for rk in row:
+            if rk.lower() == k and row[rk] not in (None, ""):
+                return row[rk]
+    return None
+
+
+def _normalize_row(row: dict[str, Any]) -> dict[str, Any] | None:
+    text = _first_present(row, _TEXT_KEYS)
+    if text is None or str(text).strip() == "":
+        return None
+    return {
+        "conversation_id": _first_present(row, _ID_KEYS),
+        "speaker": _first_present(row, _SPEAKER_KEYS),
+        "timestamp": _first_present(row, _TIME_KEYS),
+        "text": str(text).strip(),
+    }
+
+
+def parse_csv(raw: bytes | str) -> list[dict[str, Any]]:
+    """Parse CSV content into a list of normalized messages."""
+    if isinstance(raw, bytes):
+        raw = raw.decode("utf-8", errors="replace")
+    reader = csv.DictReader(io.StringIO(raw))
+    messages = []
+    for row in reader:
+        norm = _normalize_row(row)
+        if norm:
+            messages.append(norm)
+    return messages
+
+
+def parse_json(raw: bytes | str) -> list[dict[str, Any]]:
+    """
+    Parse JSON content into a list of normalized messages.
+
+    Accepts either a list of objects, or an object with a "messages"/"data" list,
+    or a single object.
+    """
+    if isinstance(raw, bytes):
+        raw = raw.decode("utf-8", errors="replace")
+    data = json.loads(raw)
+
+    if isinstance(data, dict):
+        for key in ("messages", "data", "conversations", "items"):
+            if key in data and isinstance(data[key], list):
+                data = data[key]
+                break
+        else:
+            data = [data]
+
+    if not isinstance(data, list):
+        return []
+
+    messages = []
+    for row in data:
+        if isinstance(row, dict):
+            norm = _normalize_row(row)
+            if norm:
+                messages.append(norm)
+        elif isinstance(row, str) and row.strip():
+            messages.append(
+                {"conversation_id": None, "speaker": None, "timestamp": None, "text": row.strip()}
+            )
+    return messages
+
+
+def parse_upload(filename: str, raw: bytes) -> list[dict[str, Any]]:
+    """Dispatch to the right parser based on the file extension."""
+    name = (filename or "").lower()
+    if name.endswith(".json"):
+        return parse_json(raw)
+    if name.endswith(".csv") or name.endswith(".tsv") or name.endswith(".txt"):
+        return parse_csv(raw)
+    # try JSON first, then CSV, as a last resort
+    try:
+        return parse_json(raw)
+    except Exception:
+        return parse_csv(raw)

main.py

@@ -1,92 +1,226 @@
-from fastapi import FastAPI, HTTPException, Request
-from pydantic import BaseModel
-from transformers import pipeline
-from pathlib import Path
-from fastapi.responses import FileResponse
-from fastapi.staticfiles import StaticFiles
-
-from db import log_to_db, get_logs_from_db
-
-app = FastAPI(title="RoBERTa Sentiment Analysis API")
-
-MODEL_PATH = "vojmahdal/roberta-sentiment-3labels" 
-
-# Simple frontend (static files)
-BASE_DIR = Path(__file__).resolve().parent
-STATIC_DIR = BASE_DIR / "static"
-app.mount("/static", StaticFiles(directory=str(STATIC_DIR)), name="static")
-
-
-#
-# --- Simple SQLite "database" for logging/anonymized storage ---
-#
-
-try:
-    print("Loading model, please wait...")
-    sentiment_pipeline = pipeline(
-        "sentiment-analysis",
-        model=MODEL_PATH,
-        tokenizer=MODEL_PATH
-    )
-    print("Model successfully loaded!")
-except Exception as e:
-    print(f"Error loading model: {e}")
-    sentiment_pipeline = None
-
-# Definition of the request structure
-class TextRequest(BaseModel):
-    text: str
-
-@app.get("/")
-def home():
-    index_file = STATIC_DIR / "index.html"
-    if index_file.exists():
-        return FileResponse(str(index_file))
-    return {"message": "Sentiment API is running. Use POST on /predict."}
-
-@app.get("/health")
-def health():
-    return {"status": "ok", "model_loaded": sentiment_pipeline is not None}
-
-
-@app.get("/logs")
-def get_logs(limit: int = 100):
-    """
-    Return last `limit` anonymized inputs from SQLite.
-    Only anonymized_text + metadata, nikdy ne původní text.
-    """
-    return get_logs_from_db(limit=limit)
-
-@app.post("/predict")
-async def predict_sentiment(payload: TextRequest, request: Request):
-    if sentiment_pipeline is None:
-        raise HTTPException(status_code=500, detail="Model is not available.")
-    
-    if not payload.text.strip():
-        raise HTTPException(status_code=400, detail="Text cannot be empty.")
-
-    # Prediction
-    result = sentiment_pipeline(payload.text)[0]
-
-    # Try to log anonymized version into SQLite "database"
-    try:
-        client_ip = request.client.host if request.client else None
-        log_to_db(
-            original_text=payload.text,
-            label=result["label"],
-            score=float(result["score"]),
-            client_ip=client_ip,
-        )
-    except Exception as e:
-        # Logging failure must not break the API
-        print(f"Logging to DB failed: {e}")
-    
-    return {
-        "text": payload.text,
-        "label": result["label"],
-        "score": round(result["score"], 4)
-    }
-
-if __name__ == "__main__":
-    import uvicorn
-    uvicorn.run(app, host="0.0.0.0", port=8000)
+"""
+Conversation Data Extraction System - REST API.
+
+Endpoints:
+    GET  /                    -> web dashboard
+    GET  /health              -> service + model status
+    POST /analyze             -> run full pipeline on a single message
+    POST /predict             -> sentiment only (backward compatible)
+    POST /ingest               -> batch ingest of a CSV/JSON file
+    GET  /records             -> recent stored (anonymized) records
+    GET  /records/export.xml  -> stored records exported as XML
+    GET  /stats               -> aggregate statistics
+    GET  /models              -> default + suggested + currently loaded HF sentiment models
+
+The full pipeline extracts named entities, classifies the topic, evaluates
+sentiment, pseudonymizes the text and stores the structured result. Since V3,
+the sentiment step can use any Hugging Face Hub text-classification model
+selected by the caller, instead of only the fine-tuned default model.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any
+
+from fastapi import FastAPI, Form, HTTPException, UploadFile, File
+from fastapi.responses import FileResponse, Response
+from fastapi.staticfiles import StaticFiles
+from pydantic import BaseModel
+
+import db
+import ingest
+from pipeline import process_message, process_batch
+from processors import ner, topics, sentiment, anonymizer, model_registry
+
+app = FastAPI(
+    title="Conversation Data Extraction System",
+    description=(
+        "Extracts named entities, topics and sentiment from chat conversations, "
+        "pseudonymizes personal data (GDPR) and stores structured results."
+    ),
+    version="3.0.0",
+)
+
+BASE_DIR = Path(__file__).resolve().parent
+STATIC_DIR = BASE_DIR / "static"
+app.mount("/static", StaticFiles(directory=str(STATIC_DIR)), name="static")
+
+# Curated list of well-known Hugging Face sentiment/text-classification
+# models offered as suggestions in the UI. Any other model id can still be
+# supplied manually - this list is not a whitelist.
+SUGGESTED_SENTIMENT_MODELS = [
+    sentiment.DEFAULT_MODEL_NAME,
+    "cardiffnlp/twitter-roberta-base-sentiment-latest",
+    "distilbert-base-uncased-finetuned-sst-2-english",
+    "nlptown/bert-base-multilingual-uncased-sentiment",
+]
+
+
+# ---------------------------------------------------------------------------
+# Request models
+# ---------------------------------------------------------------------------
+class TextRequest(BaseModel):
+    text: str
+    topic_labels: list[str] | None = None
+    sentiment_model: str | None = None
+
+
+# ---------------------------------------------------------------------------
+# Pages
+# ---------------------------------------------------------------------------
+@app.get("/")
+def home():
+    index_file = STATIC_DIR / "index.html"
+    if index_file.exists():
+        return FileResponse(str(index_file))
+    return {"message": "Conversation Data Extraction System is running. See /docs."}
+
+
+@app.get("/health")
+def health():
+    return {
+        "status": "ok",
+        "models": {
+            "ner": {"name": ner.model_name(), "ready": ner.is_ready()},
+            "topics": {"name": topics.model_name(), "ready": topics.is_ready()},
+            "sentiment": {"name": sentiment.model_name(), "ready": sentiment.is_ready()},
+            "anonymizer": {"backend": anonymizer.backend_name()},
+        },
+    }
+
+
+@app.get("/models")
+def list_models():
+    """Default, suggested and currently warm-cached sentiment models."""
+    return {
+        "default": sentiment.DEFAULT_MODEL_NAME,
+        "suggested": SUGGESTED_SENTIMENT_MODELS,
+        "cached": model_registry.cached_models(),
+    }
+
+
+# ---------------------------------------------------------------------------
+# Core analysis
+# ---------------------------------------------------------------------------
+@app.post("/analyze")
+def analyze(payload: TextRequest):
+    """Run the full pipeline on a single message and store the result."""
+    if not payload.text or not payload.text.strip():
+        raise HTTPException(status_code=400, detail="Text cannot be empty.")
+
+    try:
+        result = process_message(
+            payload.text,
+            topic_labels=payload.topic_labels,
+            sentiment_model=payload.sentiment_model,
+        )
+    except RuntimeError as e:
+        raise HTTPException(status_code=400, detail=str(e))
+
+    try:
+        db.save_record(result, source="single")
+    except Exception as e:
+        print(f"[main] Failed to store record: {e}")
+
+    # do not return the raw text in a way that encourages storing it client-side;
+    # we return both for the immediate UI, but only anonymized is persisted.
+    return result
+
+
+@app.post("/predict")
+def predict(payload: TextRequest):
+    """
+    Backward-compatible sentiment-only endpoint.
+    Kept so existing clients of the original API keep working.
+    """
+    if not payload.text or not payload.text.strip():
+        raise HTTPException(status_code=400, detail="Text cannot be empty.")
+
+    try:
+        s = sentiment.analyze_sentiment(payload.text, model_id=payload.sentiment_model)
+    except RuntimeError as e:
+        raise HTTPException(status_code=400, detail=str(e))
+    return {"text": payload.text, "label": s["label"], "score": s["score"]}
+
+
+# ---------------------------------------------------------------------------
+# Ingest (batch)
+# ---------------------------------------------------------------------------
+@app.post("/ingest")
+async def ingest_file(
+    file: UploadFile = File(...),
+    sentiment_model: str | None = Form(None),
+):
+    """
+    Ingest a CSV or JSON file of conversations, run the full pipeline on each
+    message, store the results and return a summary.
+    """
+    raw = await file.read()
+    if not raw:
+        raise HTTPException(status_code=400, detail="Uploaded file is empty.")
+
+    try:
+        messages = ingest.parse_upload(file.filename or "", raw)
+    except Exception as e:
+        raise HTTPException(status_code=400, detail=f"Could not parse file: {e}")
+
+    if not messages:
+        raise HTTPException(
+            status_code=400,
+            detail="No messages found. Expected a 'text'/'message' column or field.",
+        )
+
+    # Cap batch size to keep the demo responsive on limited hardware.
+    MAX_BATCH = 200
+    truncated = len(messages) > MAX_BATCH
+    messages = messages[:MAX_BATCH]
+
+    try:
+        results = process_batch(messages, sentiment_model=sentiment_model)
+    except RuntimeError as e:
+        raise HTTPException(status_code=400, detail=str(e))
+
+    stored = 0
+    for r in results:
+        try:
+            db.save_record(r, source="ingest")
+            stored += 1
+        except Exception as e:
+            print(f"[main] Failed to store ingest record: {e}")
+
+    return {
+        "received": len(messages),
+        "processed": len(results),
+        "stored": stored,
+        "truncated": truncated,
+        "items": results,
+    }
+
+
+# ---------------------------------------------------------------------------
+# Stored data
+# ---------------------------------------------------------------------------
+@app.get("/records")
+def records(limit: int = 100):
+    """Return recent stored records (anonymized only)."""
+    return db.get_records(limit=limit)
+
+
+@app.get("/records/export.xml")
+def export_records_xml(limit: int | None = None):
+    """Export stored (anonymized) records as XML."""
+    xml_bytes = db.export_xml(limit=limit)
+    return Response(content=xml_bytes, media_type="application/xml")
+
+
+@app.get("/stats")
+def get_stats():
+    """Aggregate statistics for the dashboard."""
+    return db.stats()
+
+
+if __name__ == "__main__":
+    import uvicorn
+
+    uvicorn.run(app, host="0.0.0.0", port=8000)

pipeline.py

@@ -0,0 +1,91 @@
+"""
+NLP pipeline orchestration.
+
+Applies the full processing chain to a single message:
+
+    raw text
+       -> named entity recognition (NER)
+       -> topic classification
+       -> sentiment analysis
+       -> anonymization (pseudonymization) of the text
+       -> structured result
+
+This is the core component referenced in the methodology. Each step is a
+separate processor module so the components can be developed, replaced or
+scaled independently.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from processors import ner, topics, sentiment, anonymizer
+
+
+def process_message(
+    text: str,
+    topic_labels: list[str] | None = None,
+    sentiment_model: str | None = None,
+) -> dict[str, Any]:
+    """
+    Run the full pipeline on a single message and return a structured result.
+
+    The returned ``anonymized_text`` is safe to store / display; the original
+    ``text`` is returned only for the immediate response and is never persisted
+    in readable form (see db.py). ``sentiment_model`` optionally selects a
+    Hugging Face Hub model id to use instead of the default fine-tuned model.
+    """
+    text = (text or "").strip()
+    if not text:
+        return {
+            "text": "",
+            "anonymized_text": "",
+            "entities": [],
+            "topic": None,
+            "topic_score": 0.0,
+            "sentiment": None,
+            "sentiment_score": 0.0,
+        }
+
+    entities = ner.extract_entities(text)
+    topic_result = topics.classify_topic(text, labels=topic_labels)
+    sentiment_result = sentiment.analyze_sentiment(text, model_id=sentiment_model)
+    anonymized = anonymizer.anonymize_text(text)
+
+    return {
+        "text": text,
+        "anonymized_text": anonymized,
+        "entities": entities,
+        "topic": topic_result["topic"],
+        "topic_score": topic_result["score"],
+        "topic_candidates": topic_result["all"],
+        "sentiment": sentiment_result["label"],
+        "sentiment_score": sentiment_result["score"],
+    }
+
+
+def process_batch(
+    messages: list[dict[str, Any]],
+    topic_labels: list[str] | None = None,
+    sentiment_model: str | None = None,
+) -> list[dict[str, Any]]:
+    """
+    Process a list of normalized messages (from ingest).
+
+    Each input item is expected to contain at least a ``text`` field, plus
+    optional metadata (conversation_id, speaker, timestamp) which is passed
+    through to the result.
+    """
+    results: list[dict[str, Any]] = []
+    for msg in messages:
+        processed = process_message(
+            msg.get("text", ""),
+            topic_labels=topic_labels,
+            sentiment_model=sentiment_model,
+        )
+        # carry over ingest metadata
+        for key in ("conversation_id", "speaker", "timestamp"):
+            if key in msg:
+                processed[key] = msg[key]
+        results.append(processed)
+    return results

prepare_dataset.py

@@ -0,0 +1,130 @@
+"""
+Prepare a real customer-support dataset for the /ingest endpoint.
+
+The ingest endpoint expects a CSV (or JSON) with a `text` (or `message`) column.
+This script takes one of the recommended public datasets and produces a clean
+`ingest_ready.csv` with the columns: conversation_id, speaker, text.
+
+It auto-detects the dataset by its columns, so you can point it at any of:
+
+  * Customer Support on Twitter   (Kaggle: thoughtvector/customer-support-on-twitter)
+      columns include: tweet_id, author_id, inbound, text, ...
+  * Customer Support Tickets       (HF: Tobi-Bueck/customer-support-tickets)
+      columns include: subject, body, queue, priority, language, ...
+  * Tech Support Conversations     (Kaggle: steve1215rogg/...)
+      columns include: Conversation_ID, Customer_Issue, ...
+  * Generic ticket dataset         (Customer Name / Customer Email / Ticket Description)
+
+Usage:
+    python prepare_dataset.py <input.csv> [--limit 200] [--lang en]
+
+Then upload `ingest_ready.csv` on the dashboard (Batch ingest) or via:
+    curl -X POST <space-url>/ingest -F "file=@ingest_ready.csv"
+"""
+
+from __future__ import annotations
+
+import argparse
+import sys
+
+import pandas as pd
+
+
+def detect_and_extract(df: pd.DataFrame, lang: str | None) -> pd.DataFrame:
+    cols = {c.lower(): c for c in df.columns}
+
+    # --- Customer Support on Twitter ---
+    if "text" in cols and "author_id" in cols and "inbound" in cols:
+        # keep only inbound = customer messages (the real user-generated text)
+        inbound_col = cols["inbound"]
+        df = df[df[inbound_col].astype(str).str.lower().isin(["true", "1"])]
+        out = pd.DataFrame(
+            {
+                "conversation_id": df[cols.get("tweet_id", cols["text"])].astype(str),
+                "speaker": "customer",
+                "text": df[cols["text"]].astype(str),
+            }
+        )
+        return out
+
+    # --- Tobi-Bueck customer-support-tickets (subject + body) ---
+    if "body" in cols:
+        if lang and "language" in cols:
+            df = df[df[cols["language"]].astype(str).str.lower() == lang.lower()]
+        subject = df[cols["subject"]].fillna("") if "subject" in cols else ""
+        body = df[cols["body"]].fillna("")
+        text = (subject + ". " + body).str.strip(". ") if "subject" in cols else body
+        out = pd.DataFrame(
+            {
+                "conversation_id": range(1, len(df) + 1),
+                "speaker": "customer",
+                "text": text.astype(str),
+            }
+        )
+        return out
+
+    # --- Tech Support Conversations ---
+    if "customer_issue" in cols:
+        out = pd.DataFrame(
+            {
+                "conversation_id": df[cols.get("conversation_id", cols["customer_issue"])].astype(str),
+                "speaker": "customer",
+                "text": df[cols["customer_issue"]].astype(str),
+            }
+        )
+        return out
+
+    # --- Generic ticket dataset (Ticket Description) ---
+    if "ticket description" in cols or "ticket_description" in cols:
+        desc = cols.get("ticket description", cols.get("ticket_description"))
+        out = pd.DataFrame(
+            {
+                "conversation_id": df[cols.get("ticket id", cols.get("ticket_id", desc))].astype(str),
+                "speaker": "customer",
+                "text": df[desc].astype(str),
+            }
+        )
+        return out
+
+    # --- Fallback: first text-like column ---
+    for key in ("text", "message", "content", "body", "utterance"):
+        if key in cols:
+            out = pd.DataFrame(
+                {
+                    "conversation_id": range(1, len(df) + 1),
+                    "speaker": "customer",
+                    "text": df[cols[key]].astype(str),
+                }
+            )
+            return out
+
+    raise SystemExit(
+        f"Could not find a usable text column. Columns present: {list(df.columns)}"
+    )
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("input", help="Path to the downloaded dataset CSV")
+    ap.add_argument("--limit", type=int, default=200, help="Max rows to keep")
+    ap.add_argument("--lang", default="en", help="Language filter where supported")
+    ap.add_argument("--out", default="ingest_ready.csv", help="Output CSV path")
+    args = ap.parse_args()
+
+    df = pd.read_csv(args.input)
+    out = detect_and_extract(df, args.lang)
+
+    # clean up: drop empties, dedupe, trim, limit
+    out = out[out["text"].str.strip().astype(bool)]
+    out = out.drop_duplicates(subset=["text"])
+    out["text"] = out["text"].str.slice(0, 1000)  # keep messages reasonable
+    out = out.head(args.limit)
+
+    out.to_csv(args.out, index=False)
+    print(f"Wrote {len(out)} messages to {args.out}")
+    print("Preview:")
+    print(out.head(5).to_string(index=False))
+
+
+if __name__ == "__main__":
+    main()

processors/__init__.py

@@ -0,0 +1 @@
+"""NLP processors: NER, topic classification, sentiment, anonymization."""

processors/anonymizer.py

@@ -0,0 +1,162 @@
+"""
+Anonymization / pseudonymization of personal data (PII).
+
+Primary strategy: Microsoft Presidio (NER via spaCy + regex recognizers + context).
+Fallback strategy: regular expressions only (used if Presidio/spaCy are unavailable).
+
+Per GDPR terminology this is *pseudonymization*: identifiers are replaced by
+placeholders and the original text is not stored in readable form (only a hash
+is kept for deduplication). The data therefore remains personal data and must be
+handled accordingly.
+"""
+
+from __future__ import annotations
+
+import re
+from typing import Any
+
+# ---------------------------------------------------------------------------
+# Try to load Presidio. If it is not installed (or the spaCy model is missing),
+# the module gracefully degrades to a regex-only anonymizer so the system never
+# crashes on startup.
+# ---------------------------------------------------------------------------
+_PRESIDIO_AVAILABLE = False
+_analyzer = None
+_anonymizer = None
+
+# spaCy models to try, from most to least accurate. The Docker image installs
+# en_core_web_lg; the smaller ones are accepted as fallbacks.
+_SPACY_MODELS = ["en_core_web_lg", "en_core_web_md", "en_core_web_sm"]
+
+
+def _find_spacy_model() -> str | None:
+    try:
+        import spacy
+    except Exception:
+        return None
+    for name in _SPACY_MODELS:
+        try:
+            spacy.load(name)
+            return name
+        except Exception:
+            continue
+    return None
+
+
+try:
+    from presidio_analyzer import AnalyzerEngine
+    from presidio_analyzer.nlp_engine import NlpEngineProvider
+    from presidio_anonymizer import AnonymizerEngine
+    from presidio_anonymizer.entities import OperatorConfig
+
+    _model = _find_spacy_model()
+    if _model is None:
+        raise RuntimeError("no spaCy model available")
+
+    _provider = NlpEngineProvider(
+        nlp_configuration={
+            "nlp_engine_name": "spacy",
+            "models": [{"lang_code": "en", "model_name": _model}],
+        }
+    )
+    _analyzer = AnalyzerEngine(nlp_engine=_provider.create_engine())
+    _anonymizer = AnonymizerEngine()
+    _PRESIDIO_AVAILABLE = True
+    print(f"[anonymizer] Presidio loaded (spaCy model: {_model}).")
+except Exception as e:  # pragma: no cover - depends on runtime environment
+    print(f"[anonymizer] Presidio unavailable, falling back to regex only: {e}")
+
+
+# ---------------------------------------------------------------------------
+# Regex fallback (also used to enrich Presidio for strictly formatted IDs)
+# ---------------------------------------------------------------------------
+_EMAIL_RE = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}")
+_PHONE_RE = re.compile(r"\+?\d[\d\s\-]{7,}\d")
+_URL_RE = re.compile(r"https?://\S+|www\.\S+")
+_CREDIT_CARD_RE = re.compile(r"\b(?:\d[ -]*?){13,16}\b")
+
+# Map Presidio entity types -> placeholder labels we expose to users.
+_PLACEHOLDERS = {
+    "PERSON": "[NAME]",
+    "EMAIL_ADDRESS": "[EMAIL]",
+    "PHONE_NUMBER": "[PHONE]",
+    "LOCATION": "[LOCATION]",
+    "CREDIT_CARD": "[CARD]",
+    "IBAN_CODE": "[IBAN]",
+    "IP_ADDRESS": "[IP]",
+    "URL": "[URL]",
+    "DATE_TIME": "[DATE]",
+    "NRP": "[ID]",
+    "US_SSN": "[ID]",
+}
+
+
+def _regex_anonymize(text: str) -> str:
+    """Pure-regex anonymization for the formatted identifiers we can match safely."""
+    text = _EMAIL_RE.sub("[EMAIL]", text)
+    text = _URL_RE.sub("[URL]", text)
+    text = _CREDIT_CARD_RE.sub("[CARD]", text)
+    text = _PHONE_RE.sub("[PHONE]", text)
+    return text
+
+
+def anonymize_text(text: str, language: str = "en") -> str:
+    """
+    Return a pseudonymized version of ``text``.
+
+    With Presidio: NER-based detection of names/locations + built-in regex
+    recognizers for emails, phones, cards, IBANs, etc.
+    Without Presidio: regex-only fallback (emails, phones, URLs, cards).
+    """
+    if not isinstance(text, str) or not text.strip():
+        return ""
+
+    if not _PRESIDIO_AVAILABLE:
+        return _regex_anonymize(text)
+
+    try:
+        results = _analyzer.analyze(text=text, language=language)
+        operators = {
+            entity: OperatorConfig("replace", {"new_value": placeholder})
+            for entity, placeholder in _PLACEHOLDERS.items()
+        }
+        # default operator for anything detected but not explicitly mapped
+        operators["DEFAULT"] = OperatorConfig("replace", {"new_value": "[REDACTED]"})
+
+        anonymized = _anonymizer.anonymize(
+            text=text,
+            analyzer_results=results,
+            operators=operators,
+        )
+        return anonymized.text
+    except Exception as e:  # pragma: no cover
+        print(f"[anonymizer] Presidio failed at runtime, using regex fallback: {e}")
+        return _regex_anonymize(text)
+
+
+def detect_pii(text: str, language: str = "en") -> list[dict[str, Any]]:
+    """
+    Return the list of detected PII spans (for evaluation / debugging).
+    Empty list if Presidio is unavailable.
+    """
+    if not _PRESIDIO_AVAILABLE or not isinstance(text, str) or not text.strip():
+        return []
+    try:
+        results = _analyzer.analyze(text=text, language=language)
+        return [
+            {
+                "type": r.entity_type,
+                "start": r.start,
+                "end": r.end,
+                "score": float(r.score),
+                "text": text[r.start : r.end],
+            }
+            for r in results
+        ]
+    except Exception:
+        return []
+
+
+def backend_name() -> str:
+    """Report which anonymization backend is active (for /health and the UI)."""
+    return "presidio" if _PRESIDIO_AVAILABLE else "regex-fallback"

processors/model_registry.py

@@ -0,0 +1,72 @@
+"""
+Generic loader/cache for text-classification models pulled from the
+Hugging Face Hub at request time.
+
+This lets a caller pick *any* HF model id for sentiment analysis instead of
+being limited to the single fine-tuned model. Pipelines are expensive to
+instantiate (they download weights on first use), so loaded pipelines are
+kept in a small in-memory cache.
+
+Security note: ``trust_remote_code`` is never enabled. Enabling it would let
+an arbitrary Hub repository execute Python code inside this process, which
+is not acceptable for a model id supplied by an API caller.
+"""
+
+from __future__ import annotations
+
+import threading
+from typing import Any
+
+# Maximum number of distinct models kept warm in memory at once. Oldest
+# (first loaded) is evicted when the cache is full - simple FIFO, adequate
+# for a demo/thesis system that isn't serving many concurrent model ids.
+_MAX_CACHED_MODELS = 3
+
+_cache: dict[str, Any] = {}
+_cache_order: list[str] = []
+_load_errors: dict[str, str] = {}
+_lock = threading.Lock()
+
+
+def get_pipeline(model_id: str, task: str = "sentiment-analysis"):
+    """
+    Return a cached (or newly loaded) Hugging Face pipeline for ``model_id``.
+
+    Raises ``RuntimeError`` if the model cannot be loaded (unknown repo,
+    not a classification model, network error, ...) so callers can turn it
+    into a clean HTTP error instead of crashing.
+    """
+    with _lock:
+        if model_id in _cache:
+            return _cache[model_id]
+        if model_id in _load_errors:
+            raise RuntimeError(_load_errors[model_id])
+
+    try:
+        from transformers import pipeline
+
+        clf = pipeline(
+            task,
+            model=model_id,
+            tokenizer=model_id,
+            trust_remote_code=False,
+        )
+    except Exception as e:  # pragma: no cover - depends on network/model
+        with _lock:
+            _load_errors[model_id] = str(e)
+        raise RuntimeError(f"Could not load model '{model_id}': {e}") from e
+
+    with _lock:
+        _cache[model_id] = clf
+        _cache_order.append(model_id)
+        while len(_cache_order) > _MAX_CACHED_MODELS:
+            oldest = _cache_order.pop(0)
+            _cache.pop(oldest, None)
+
+    return clf
+
+
+def cached_models() -> list[str]:
+    """Model ids currently kept warm in memory."""
+    with _lock:
+        return list(_cache_order)

processors/ner.py

@@ -0,0 +1,85 @@
+"""
+Named Entity Recognition (NER).
+
+Uses the pre-trained model ``dslim/bert-base-NER`` (fine-tuned on CoNLL-2003,
+~92.6% F1) via the Hugging Face ``token-classification`` pipeline.
+
+No training is required here - this is direct inference on a pre-trained model,
+in line with the methodology. The model is loaded lazily on first use so the
+application starts quickly and only pays the memory cost when NER is actually
+needed.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+_MODEL_NAME = "dslim/bert-base-NER"
+_pipeline = None
+_load_error: str | None = None
+
+
+def _get_pipeline():
+    """Lazy-load the NER pipeline on first call."""
+    global _pipeline, _load_error
+    if _pipeline is not None or _load_error is not None:
+        return _pipeline
+
+    try:
+        from transformers import pipeline
+
+        _pipeline = pipeline(
+            "token-classification",
+            model=_MODEL_NAME,
+            tokenizer=_MODEL_NAME,
+            aggregation_strategy="simple",  # merge sub-word tokens into whole entities
+        )
+        print(f"[ner] Loaded model {_MODEL_NAME}.")
+    except Exception as e:  # pragma: no cover
+        _load_error = str(e)
+        print(f"[ner] Failed to load model: {e}")
+    return _pipeline
+
+
+def extract_entities(text: str) -> list[dict[str, Any]]:
+    """
+    Extract named entities from ``text``.
+
+    Returns a list of dicts: {"text", "type", "start", "end", "score"}.
+    Entity types follow CoNLL-2003: PER (person), LOC (location),
+    ORG (organization), MISC (miscellaneous).
+    """
+    if not isinstance(text, str) or not text.strip():
+        return []
+
+    nlp = _get_pipeline()
+    if nlp is None:
+        return []
+
+    try:
+        raw = nlp(text)
+    except Exception as e:  # pragma: no cover
+        print(f"[ner] Inference failed: {e}")
+        return []
+
+    entities: list[dict[str, Any]] = []
+    for ent in raw:
+        entities.append(
+            {
+                "text": ent.get("word", ""),
+                "type": ent.get("entity_group", ent.get("entity", "")),
+                "start": int(ent.get("start", 0)),
+                "end": int(ent.get("end", 0)),
+                "score": round(float(ent.get("score", 0.0)), 4),
+            }
+        )
+    return entities
+
+
+def is_ready() -> bool:
+    """True if the model is loaded or can be loaded (no fatal error)."""
+    return _load_error is None
+
+
+def model_name() -> str:
+    return _MODEL_NAME

processors/sentiment.py

@@ -0,0 +1,80 @@
+"""
+Sentiment analysis.
+
+Uses the fine-tuned RoBERTa model published on the Hugging Face Hub
+(``vojmahdal/roberta-sentiment-3labels``) by default - this is the only model
+in the system that was trained (fine-tuned) by the author, as described in
+the methodology. Since V3, callers may instead pick any other text
+classification model from the Hugging Face Hub at request time; that model is
+loaded and cached via ``processors.model_registry``.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from processors import model_registry
+
+DEFAULT_MODEL_NAME = "vojmahdal/roberta-sentiment-3labels"
+_pipeline = None
+_load_error: str | None = None
+
+
+def _get_default_pipeline():
+    """Lazy-load the default sentiment pipeline on first call."""
+    global _pipeline, _load_error
+    if _pipeline is not None or _load_error is not None:
+        return _pipeline
+
+    try:
+        from transformers import pipeline
+
+        _pipeline = pipeline(
+            "sentiment-analysis",
+            model=DEFAULT_MODEL_NAME,
+            tokenizer=DEFAULT_MODEL_NAME,
+        )
+        print(f"[sentiment] Loaded model {DEFAULT_MODEL_NAME}.")
+    except Exception as e:  # pragma: no cover
+        _load_error = str(e)
+        print(f"[sentiment] Failed to load model: {e}")
+    return _pipeline
+
+
+def analyze_sentiment(text: str, model_id: str | None = None) -> dict[str, Any]:
+    """
+    Return {"label": <positive|neutral|negative>, "score": <confidence>}.
+
+    ``model_id`` optionally selects a different Hugging Face Hub model
+    (loaded/cached on demand via ``model_registry``) instead of the default
+    fine-tuned model. Raises ``RuntimeError`` if that model cannot be loaded,
+    so the API layer can turn it into a clean 400 response.
+    """
+    if not isinstance(text, str) or not text.strip():
+        return {"label": None, "score": 0.0}
+
+    if model_id and model_id != DEFAULT_MODEL_NAME:
+        clf = model_registry.get_pipeline(model_id, task="sentiment-analysis")
+    else:
+        clf = _get_default_pipeline()
+
+    if clf is None:
+        return {"label": None, "score": 0.0}
+
+    try:
+        result = clf(text)[0]
+        return {
+            "label": result["label"],
+            "score": round(float(result["score"]), 4),
+        }
+    except Exception as e:  # pragma: no cover
+        print(f"[sentiment] Inference failed: {e}")
+        return {"label": None, "score": 0.0}
+
+
+def is_ready() -> bool:
+    return _load_error is None
+
+
+def model_name() -> str:
+    return DEFAULT_MODEL_NAME

processors/topics.py

@@ -0,0 +1,99 @@
+"""
+Topic classification (zero-shot).
+
+Uses ``facebook/bart-large-mnli`` via the Hugging Face
+``zero-shot-classification`` pipeline. The task is framed as Natural Language
+Inference: each candidate label becomes a hypothesis ("This text is about X.")
+and the model scores entailment.
+
+No training data is required and the candidate labels can be changed at runtime,
+which suits customer-support conversations where annotated topic datasets are
+usually not available.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+_MODEL_NAME = "facebook/bart-large-mnli"
+
+# Default candidate topics for a customer-support domain.
+DEFAULT_LABELS = [
+    "billing and payments",
+    "technical issue",
+    "complaint",
+    "product question",
+    "account management",
+    "cancellation",
+    "delivery and shipping",
+    "praise and positive feedback",
+]
+
+_pipeline = None
+_load_error: str | None = None
+
+
+def _get_pipeline():
+    """Lazy-load the zero-shot pipeline on first call."""
+    global _pipeline, _load_error
+    if _pipeline is not None or _load_error is not None:
+        return _pipeline
+
+    try:
+        from transformers import pipeline
+
+        _pipeline = pipeline(
+            "zero-shot-classification",
+            model=_MODEL_NAME,
+        )
+        print(f"[topics] Loaded model {_MODEL_NAME}.")
+    except Exception as e:  # pragma: no cover
+        _load_error = str(e)
+        print(f"[topics] Failed to load model: {e}")
+    return _pipeline
+
+
+def classify_topic(
+    text: str,
+    labels: list[str] | None = None,
+    top_k: int = 3,
+) -> dict[str, Any]:
+    """
+    Classify ``text`` into one of ``labels``.
+
+    Returns:
+        {
+          "topic": <best label or None>,
+          "score": <confidence of best label>,
+          "all": [{"label", "score"}, ...]  # top_k candidates
+        }
+    """
+    if not isinstance(text, str) or not text.strip():
+        return {"topic": None, "score": 0.0, "all": []}
+
+    candidate_labels = labels or DEFAULT_LABELS
+    clf = _get_pipeline()
+    if clf is None:
+        return {"topic": None, "score": 0.0, "all": []}
+
+    try:
+        result = clf(text, candidate_labels, multi_label=False)
+    except Exception as e:  # pragma: no cover
+        print(f"[topics] Inference failed: {e}")
+        return {"topic": None, "score": 0.0, "all": []}
+
+    pairs = list(zip(result["labels"], result["scores"]))
+    top = pairs[:top_k]
+    return {
+        "topic": pairs[0][0] if pairs else None,
+        "score": round(float(pairs[0][1]), 4) if pairs else 0.0,
+        "all": [{"label": l, "score": round(float(s), 4)} for l, s in top],
+    }
+
+
+def is_ready() -> bool:
+    return _load_error is None
+
+
+def model_name() -> str:
+    return _MODEL_NAME

requirements-dev.txt

@@ -0,0 +1,5 @@
+# Extra dependencies for offline/dev scripts only (not needed to run the API).
+# prepare_dataset.py
+pandas
+# evaluate.py (NER metrics)
+seqeval

requirements.txt

@@ -1,4 +1,8 @@
-fastapi
-uvicorn[standard]
-transformers
-torch
+fastapi
+uvicorn[standard]
+python-multipart
+transformers
+torch
+presidio-analyzer
+presidio-anonymizer
+spacy

sample_chats.csv

@@ -0,0 +1,11 @@
+conversation_id,speaker,text
+1,user,"Hi, my name is Sarah Johnson and my order #4521 never arrived. This is unacceptable!"
+1,agent,"I'm very sorry about that Sarah. Can you confirm your email so I can look into it?"
+1,user,"Sure, it's sarah.johnson@example.com and my phone is +1 202 555 0143"
+2,user,"The new app update is fantastic, everything works so smoothly now. Great job!"
+3,user,"I want to cancel my subscription. I've been charged twice this month."
+3,agent,"I understand, let me check the billing for your account."
+4,user,"Can you tell me if the Pro plan supports more than 5 users?"
+5,user,"My internet keeps disconnecting every few minutes since yesterday."
+6,user,"Thank you so much for the quick refund, Michael was very helpful."
+7,user,"Where is my package? It was supposed to arrive in London three days ago."

static/app.js

@@ -1,25 +1,85 @@
 const textEl = document.getElementById("text");
+const sentimentModelEl = document.getElementById("sentimentModel");
+const modelSuggestionsEl = document.getElementById("modelSuggestions");
 const analyzeBtn = document.getElementById("analyzeBtn");
 const clearBtn = document.getElementById("clearBtn");
 const statusEl = document.getElementById("status");
 const resultEl = document.getElementById("result");
-const labelEl = document.getElementById("label");
-const scoreEl = document.getElementById("score");
+const sentimentEl = document.getElementById("sentiment");
+const topicEl = document.getElementById("topic");
+const entitiesEl = document.getElementById("entities");
+const anonEl = document.getElementById("anon");
 const errorEl = document.getElementById("error");
 
+const fileEl = document.getElementById("file");
+const ingestBtn = document.getElementById("ingestBtn");
+const ingestStatusEl = document.getElementById("ingestStatus");
+const ingestErrorEl = document.getElementById("ingestError");
+
+async function loadModelSuggestions() {
+  try {
+    const res = await fetch("/models");
+    if (!res.ok) return;
+    const data = await res.json();
+    modelSuggestionsEl.innerHTML = "";
+    for (const modelId of data.suggested || []) {
+      const opt = document.createElement("option");
+      opt.value = modelId;
+      modelSuggestionsEl.appendChild(opt);
+    }
+    if (!sentimentModelEl.value) {
+      sentimentModelEl.value = data.default || "";
+    }
+  } catch (e) {
+    // suggestions are a nicety, not required for the app to work
+  }
+}
+
 function setStatus(msg) {
   statusEl.textContent = msg || "";
 }
-
 function showError(msg) {
   errorEl.textContent = msg;
   errorEl.hidden = !msg;
 }
 
-function showResult({ label, score }) {
-  labelEl.textContent = label ?? "";
-  scoreEl.textContent =
-    typeof score === "number" ? score.toFixed(4) : String(score ?? "");
+function sentimentClass(label) {
+  const l = (label || "").toLowerCase();
+  if (l.includes("pos")) return "pill pos";
+  if (l.includes("neg")) return "pill neg";
+  return "pill neu";
+}
+
+function renderResult(data) {
+  // sentiment
+  const sLabel = data.sentiment ?? "—";
+  const sScore =
+    typeof data.sentiment_score === "number"
+      ? ` (${data.sentiment_score.toFixed(2)})`
+      : "";
+  sentimentEl.innerHTML = `<span class="${sentimentClass(sLabel)}">${sLabel}${sScore}</span>`;
+
+  // topic
+  const tScore =
+    typeof data.topic_score === "number" ? ` (${data.topic_score.toFixed(2)})` : "";
+  topicEl.textContent = (data.topic ?? "—") + (data.topic ? tScore : "");
+
+  // entities
+  entitiesEl.innerHTML = "";
+  if (Array.isArray(data.entities) && data.entities.length) {
+    for (const ent of data.entities) {
+      const chip = document.createElement("span");
+      chip.className = "chip";
+      chip.innerHTML = `<span class="chipType">${ent.type}</span>${ent.text}`;
+      entitiesEl.appendChild(chip);
+    }
+  } else {
+    entitiesEl.innerHTML = `<span class="muted">No entities detected</span>`;
+  }
+
+  // anonymized text
+  anonEl.textContent = data.anonymized_text ?? "";
+
   resultEl.hidden = false;
 }
 
@@ -31,31 +91,29 @@ function resetOutput() {
 
 async function analyze() {
   resetOutput();
-
   const text = (textEl.value || "").trim();
   if (!text) {
-    showError("Text cannot be empty.");
+    showError("Message cannot be empty.");
     return;
   }
 
   analyzeBtn.disabled = true;
   setStatus("Analyzing…");
 
+  const sentimentModel = (sentimentModelEl.value || "").trim() || null;
+
   try {
-    const res = await fetch("/predict", {
+    const res = await fetch("/analyze", {
       method: "POST",
       headers: { "Content-Type": "application/json" },
-      body: JSON.stringify({ text }),
+      body: JSON.stringify({ text, sentiment_model: sentimentModel }),
     });
-
     const data = await res.json().catch(() => null);
-
     if (!res.ok) {
       const detail = data?.detail ? `\n\n${JSON.stringify(data.detail)}` : "";
-      throw new Error(`Error API (${res.status} ${res.statusText}).${detail}`);
+      throw new Error(`API error (${res.status} ${res.statusText}).${detail}`);
     }
-
-    showResult(data);
+    renderResult(data);
     setStatus("Done.");
   } catch (e) {
     showError(e?.message || String(e));
@@ -65,13 +123,49 @@ async function analyze() {
   }
 }
 
+async function ingest() {
+  ingestErrorEl.hidden = true;
+  const file = fileEl.files?.[0];
+  if (!file) {
+    ingestErrorEl.textContent = "Please choose a CSV or JSON file first.";
+    ingestErrorEl.hidden = false;
+    return;
+  }
+
+  ingestBtn.disabled = true;
+  ingestStatusEl.textContent = "Uploading and processing…";
+
+  try {
+    const form = new FormData();
+    form.append("file", file);
+    const sentimentModel = (sentimentModelEl.value || "").trim();
+    if (sentimentModel) form.append("sentiment_model", sentimentModel);
+    const res = await fetch("/ingest", { method: "POST", body: form });
+    const data = await res.json().catch(() => null);
+    if (!res.ok) {
+      const detail = data?.detail ? `: ${JSON.stringify(data.detail)}` : "";
+      throw new Error(`API error (${res.status})${detail}`);
+    }
+    const note = data.truncated ? " (batch truncated to 200)" : "";
+    ingestStatusEl.textContent = `Processed ${data.processed}, stored ${data.stored}${note}. See Records.`;
+  } catch (e) {
+    ingestErrorEl.textContent = e?.message || String(e);
+    ingestErrorEl.hidden = false;
+    ingestStatusEl.textContent = "";
+  } finally {
+    ingestBtn.disabled = false;
+  }
+}
+
 analyzeBtn.addEventListener("click", analyze);
 clearBtn.addEventListener("click", () => {
   textEl.value = "";
   resetOutput();
   textEl.focus();
 });
-
 textEl.addEventListener("keydown", (e) => {
   if ((e.ctrlKey || e.metaKey) && e.key === "Enter") analyze();
 });
+ingestBtn.addEventListener("click", ingest);
+
+loadModelSuggestions();

static/index.html

@@ -1,27 +1,39 @@
 <!doctype html>
-<html lang="cs">
+<html lang="en">
   <head>
     <meta charset="utf-8" />
     <meta name="viewport" content="width=device-width, initial-scale=1" />
-    <title>Sentiment Analysis</title>
+    <title>Conversation Data Extraction</title>
     <link rel="stylesheet" href="/static/styles.css" />
   </head>
   <body>
     <main class="container">
       <header class="header">
-        <h1>Sentiment Analysis</h1>
+        <h1>Conversation Data Extraction</h1>
         <p class="sub">
-          Simple web interface over the <code>POST /predict</code> endpoint.
+          Extracts named entities, topics and sentiment from chat messages,
+          pseudonymizes personal data (GDPR) and stores structured results.
         </p>
       </header>
 
+      <!-- Single message analysis -->
       <section class="card">
-        <label class="label" for="text">Text</label>
+        <label class="label" for="text">Message</label>
         <textarea
           id="text"
-          rows="6"
-          placeholder="Write the text you want to analyze…"
-        >This product is great!</textarea>
+          rows="5"
+          placeholder="Write a message to analyze…"
+        >Hi, my name is John Smith and I'm really unhappy. My order never arrived. Email me at john.smith@example.com</textarea>
+
+        <label class="label" for="sentimentModel">Sentiment model (Hugging Face Hub)</label>
+        <input
+          id="sentimentModel"
+          class="modelInput"
+          type="text"
+          list="modelSuggestions"
+          placeholder="vojmahdal/roberta-sentiment-3labels"
+        />
+        <datalist id="modelSuggestions"></datalist>
 
         <div class="row">
           <button id="analyzeBtn" class="btn">Analyze</button>
@@ -31,24 +43,48 @@
 
         <div id="result" class="result" hidden>
           <div class="resultRow">
-            <span class="k">Label</span>
-            <span id="label" class="v"></span>
+            <span class="k">Sentiment</span>
+            <span id="sentiment" class="v"></span>
           </div>
           <div class="resultRow">
-            <span class="k">Score</span>
-            <span id="score" class="v"></span>
+            <span class="k">Topic</span>
+            <span id="topic" class="v"></span>
+          </div>
+          <div class="resultRow column">
+            <span class="k">Entities</span>
+            <div id="entities" class="chips"></div>
+          </div>
+          <div class="resultRow column">
+            <span class="k">Anonymized text</span>
+            <code id="anon" class="anon"></code>
           </div>
         </div>
 
         <pre id="error" class="error" hidden></pre>
       </section>
 
+      <!-- Batch ingest -->
+      <section class="card">
+        <h2 class="cardTitle">Batch ingest</h2>
+        <p class="sub small">
+          Upload a CSV or JSON file of conversations. Each message is processed by
+          the full pipeline and stored (anonymized). CSV should have a
+          <code>text</code> or <code>message</code> column.
+        </p>
+        <div class="row">
+          <input type="file" id="file" accept=".csv,.json,.tsv,.txt" class="file" />
+          <button id="ingestBtn" class="btn" type="button">Ingest file</button>
+          <span id="ingestStatus" class="status" aria-live="polite"></span>
+        </div>
+        <pre id="ingestError" class="error" hidden></pre>
+      </section>
+
       <footer class="footer">
         <a href="/docs">API documentation</a>
         <span class="dot">•</span>
         <a href="/health">Health</a>
         <span class="dot">•</span>
-        <a href="/static/logs.html">Logs</a>
+        <a href="/static/records.html">Records</a>
       </footer>
     </main>
 

static/records.html

@@ -0,0 +1,62 @@
+<!doctype html>
+<html lang="en">
+  <head>
+    <meta charset="utf-8" />
+    <meta name="viewport" content="width=device-width, initial-scale=1" />
+    <title>Stored Records</title>
+    <link rel="stylesheet" href="/static/styles.css" />
+  </head>
+  <body>
+    <main class="container">
+      <header class="header">
+        <h1>Stored Records</h1>
+        <p class="sub">
+          Anonymized records produced by the pipeline. Original text is never
+          stored, only its hash and the anonymized version.
+        </p>
+      </header>
+
+      <section class="card">
+        <div id="stats" class="stats"></div>
+      </section>
+
+      <section class="card">
+        <div class="row">
+          <button id="reloadBtn" class="btn" type="button">Reload</button>
+          <a id="exportXmlBtn" class="btn secondary" href="/records/export.xml" download="records.xml">Export XML</a>
+          <span id="status" class="status" aria-live="polite"></span>
+        </div>
+
+        <div id="empty" class="sub" style="margin-top: 12px">
+          No records yet. Analyze a message or ingest a file first.
+        </div>
+
+        <div id="tableWrapper" class="result" hidden>
+          <table class="log-table">
+            <thead>
+              <tr>
+                <th>Time (UTC)</th>
+                <th>Anonymized text</th>
+                <th>Topic</th>
+                <th>Sentiment</th>
+                <th>Entities</th>
+                <th>Source</th>
+              </tr>
+            </thead>
+            <tbody id="logBody"></tbody>
+          </table>
+        </div>
+
+        <pre id="error" class="error" hidden></pre>
+      </section>
+
+      <footer class="footer">
+        <a href="/">Back to main page</a>
+        <span class="dot">•</span>
+        <a href="/docs">API documentation</a>
+      </footer>
+    </main>
+
+    <script src="/static/records.js"></script>
+  </body>
+</html>

static/records.js

@@ -0,0 +1,114 @@
+const statusEl = document.getElementById("status");
+const errorEl = document.getElementById("error");
+const emptyEl = document.getElementById("empty");
+const tableWrapper = document.getElementById("tableWrapper");
+const bodyEl = document.getElementById("logBody");
+const reloadBtn = document.getElementById("reloadBtn");
+const statsEl = document.getElementById("stats");
+
+function setStatus(msg) {
+  statusEl.textContent = msg || "";
+}
+function showError(msg) {
+  errorEl.textContent = msg;
+  errorEl.hidden = !msg;
+}
+
+function sentimentClass(label) {
+  const l = (label || "").toLowerCase();
+  if (l.includes("pos")) return "pill pos";
+  if (l.includes("neg")) return "pill neg";
+  return "pill neu";
+}
+
+function renderStats(stats) {
+  if (!stats || !stats.total) {
+    statsEl.innerHTML = `<span class="muted">No statistics yet.</span>`;
+    return;
+  }
+  const sentiments = Object.entries(stats.by_sentiment || {})
+    .map(([k, v]) => `<span class="${sentimentClass(k)}">${k}: ${v}</span>`)
+    .join(" ");
+  const topics = Object.entries(stats.by_topic || {})
+    .map(([k, v]) => `<span class="chip">${k}: ${v}</span>`)
+    .join(" ");
+  statsEl.innerHTML = `
+    <div class="statsRow"><span class="k">Total records</span><span class="v">${stats.total}</span></div>
+    <div class="statsRow column"><span class="k">By sentiment</span><div class="chips">${sentiments}</div></div>
+    <div class="statsRow column"><span class="k">Top topics</span><div class="chips">${topics}</div></div>
+  `;
+}
+
+function renderRows(items) {
+  bodyEl.innerHTML = "";
+  for (const item of items) {
+    const tr = document.createElement("tr");
+
+    const tdTime = document.createElement("td");
+    tdTime.textContent = item.created_at || "";
+    tr.appendChild(tdTime);
+
+    const tdText = document.createElement("td");
+    tdText.textContent = item.anonymized_text || "";
+    tr.appendChild(tdText);
+
+    const tdTopic = document.createElement("td");
+    tdTopic.textContent = item.topic || "—";
+    tr.appendChild(tdTopic);
+
+    const tdSent = document.createElement("td");
+    const sLabel = item.sentiment || "—";
+    tdSent.innerHTML = `<span class="${sentimentClass(sLabel)}">${sLabel}</span>`;
+    tr.appendChild(tdSent);
+
+    const tdEnt = document.createElement("td");
+    const ents = Array.isArray(item.entities) ? item.entities : [];
+    tdEnt.textContent = ents.length
+      ? ents.map((e) => `${e.type}:${e.text}`).join(", ")
+      : "—";
+    tr.appendChild(tdEnt);
+
+    const tdSrc = document.createElement("td");
+    tdSrc.textContent = item.source || "";
+    tr.appendChild(tdSrc);
+
+    bodyEl.appendChild(tr);
+  }
+}
+
+async function load() {
+  showError("");
+  setStatus("Loading…");
+  try {
+    const [recRes, statRes] = await Promise.all([
+      fetch("/records?limit=100"),
+      fetch("/stats"),
+    ]);
+    const records = await recRes.json().catch(() => null);
+    const stats = await statRes.json().catch(() => null);
+
+    if (!recRes.ok) {
+      throw new Error(`API error (${recRes.status} ${recRes.statusText})`);
+    }
+
+    renderStats(stats);
+
+    if (!Array.isArray(records) || records.length === 0) {
+      emptyEl.hidden = false;
+      tableWrapper.hidden = true;
+      setStatus("No records yet.");
+      return;
+    }
+
+    renderRows(records);
+    emptyEl.hidden = true;
+    tableWrapper.hidden = false;
+    setStatus(`Loaded ${records.length} records.`);
+  } catch (e) {
+    showError(e?.message || String(e));
+    setStatus("");
+  }
+}
+
+reloadBtn.addEventListener("click", load);
+load();

static/styles.css

@@ -190,3 +190,111 @@ textarea:focus {
 .log-table tr:last-child td {
   border-bottom: none;
 }
+
+/* ---- Extended components for the extraction dashboard ---- */
+.cardTitle {
+  margin: 0 0 6px;
+  font-size: 18px;
+}
+.sub.small {
+  font-size: 13px;
+  margin-bottom: 12px;
+}
+.card + .card {
+  margin-top: 16px;
+}
+.resultRow.column,
+.statsRow.column {
+  flex-direction: column;
+  align-items: flex-start;
+  gap: 8px;
+}
+.chips {
+  display: flex;
+  flex-wrap: wrap;
+  gap: 8px;
+}
+.chip {
+  display: inline-flex;
+  align-items: center;
+  gap: 6px;
+  padding: 4px 10px;
+  border-radius: 999px;
+  background: rgba(255, 255, 255, 0.08);
+  border: 1px solid var(--border);
+  font-size: 13px;
+}
+.chipType {
+  font-size: 11px;
+  font-weight: 700;
+  color: var(--accent2);
+  text-transform: uppercase;
+  letter-spacing: 0.3px;
+}
+.pill {
+  display: inline-block;
+  padding: 3px 12px;
+  border-radius: 999px;
+  font-weight: 700;
+  font-size: 13px;
+}
+.pill.pos {
+  background: rgba(34, 197, 94, 0.18);
+  color: #4ade80;
+  border: 1px solid rgba(34, 197, 94, 0.4);
+}
+.pill.neg {
+  background: rgba(239, 68, 68, 0.18);
+  color: #f87171;
+  border: 1px solid rgba(239, 68, 68, 0.4);
+}
+.pill.neu {
+  background: rgba(148, 163, 184, 0.18);
+  color: #cbd5e1;
+  border: 1px solid rgba(148, 163, 184, 0.4);
+}
+.anon {
+  display: block;
+  width: 100%;
+  padding: 10px 12px;
+  border-radius: 10px;
+  background: rgba(0, 0, 0, 0.3);
+  border: 1px solid var(--border);
+  font-size: 13px;
+  white-space: pre-wrap;
+  word-break: break-word;
+}
+.muted {
+  color: var(--muted);
+  font-size: 13px;
+}
+.file {
+  color: var(--muted);
+  font-size: 13px;
+  max-width: 280px;
+}
+.stats {
+  display: flex;
+  flex-direction: column;
+  gap: 12px;
+}
+.statsRow {
+  display: flex;
+  justify-content: space-between;
+  gap: 12px;
+}
+.modelInput {
+  width: 100%;
+  margin-top: 10px;
+  border-radius: 12px;
+  border: 1px solid var(--border);
+  background: rgba(0, 0, 0, 0.25);
+  color: var(--text);
+  padding: 10px 12px;
+  font-size: 13px;
+  outline: none;
+}
+.modelInput:focus {
+  border-color: rgba(124, 58, 237, 0.55);
+  box-shadow: 0 0 0 3px rgba(124, 58, 237, 0.22);
+}

upload.py

@@ -2,17 +2,17 @@ from huggingface_hub import HfApi
 
 api = HfApi()
 
-
+# Tady vyplň své údaje
 repo_id = "vojmahdal/roberta-sentiment-3labels" 
 
-print("Uploading model to Hugging Face... It may take a few minutes.")
+print("Nahrávám model na Hugging Face... Může to trvat několik minut.")
 
 api.upload_folder(
-    folder_path="./sentiment_analysis_model",  # Upload all in the current folder
+    folder_path="./sentiment_analysis_model",  # Nahrát vše v aktuální složce
     repo_id=repo_id,
     repo_type="model",
-   
+    # Ignorujeme zbytečnosti, co nechceme na internetu
     ignore_patterns=["checkpoint-*", ".venv/*", "upload.py", ".git*"]
 )
 
-print(f"Done! Model you can find on: https://huggingface.co/{repo_id}")
+print(f"Hotovo! Model najdeš na: https://huggingface.co/{repo_id}")