Deploy to HF Spaces

.env.example

@@ -0,0 +1,3 @@
+# Supabase
+SUPABASE_URL=https://YOUR_PROJECT_REF.supabase.co
+SUPABASE_KEY=YOUR_SUPABASE_ANON_OR_SERVICE_KEY

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,13 @@
+.env
+.env.*
+!.env.example
+
+**/.gradio/
+**/.cache/
+**/.ipynb_checkpoints/
+**/artifacts/
+**/__pycache__/
+
+# notebooks
+*.ipynb
+!train_upload_notebook.ipynb

README.md

@@ -0,0 +1,91 @@
+---
+title: Email Prioritization Demo
+sdk: gradio
+app_file: app.py
+python_version: 3.11
+sdk_version: 4.0.0
+---
+
+# Email Prioritization Demo
+
+Train and serve an email prioritization model with a Gradio UI for interactive predictions, metrics, and dataset preview.
+
+[Live app](https://46ccf31bbf1fadf8e8.gradio.live/)
+
+## Table of Contents
+
+[About](#about)
+
+[Quickstart](#quickstart)
+
+[Repo Structure](#repo-structure)
+
+[Data](#data)
+
+[Evaluation](#evaluation)
+
+## About
+
+This project includes:
+
+- a simple email prioritization model (training + inference)
+- an event-style predictor that assigns a label to an email
+- a Gradio web app that supports:
+  1. pasting an email (subject/body) to view the predicted label, confidence, probabilities, and reasons
+  2. viewing evaluation metrics on a fixed test split (confusion matrix + macro PR-AUC)
+  3. previewing the dataset used for evaluation
+
+## Quickstart
+
+### UI
+
+Use the **Simulate email event** tab in the [app](https://46ccf31bbf1fadf8e8.gradio.live/): paste an email `subject` + `body` to get a prediction.
+
+### Local Configuration
+
+ > Note: `src/app.py` may use `demo.launch(share=True)` for public demos. For local-only runs, set `share=False` or remove the argument.
+
+#### Prereqs
+
+- Python 3.11+ (tested on Python 3.12)
+  
+- Supabase project credentials in a local `.env` file (see `.env.example`).
+
+The Gradio app entrypoint is `src/app.py`. Run it locally with:
+
+```bash
+pip install -r requirements.txt
+python3 src/app.py 
+```
+
+## Repo Structure
+
+```text
+src/
+  app.py                 # Gradio UI
+  train_upload_notebook.ipynb # where model is trained and uploaded to supabase
+  ml/
+    train.py             # model construction / training helpers
+    predict.py           # prediction + reasoning helpers
+    eval.py              # confusion matrix + PR-AUC plotting
+    pre_process.py       # preprocessing fcns
+  store/
+    supabase_io.py       # Supabase fetch/upload/download utilities
+
+```
+
+## Data
+
+- [Public source dataset](https://huggingface.co/datasets/jason23322/high-accuracy-email-classifier) used for training
+  - See details in [Writeup: dataset selection](writeup.md#dataset-selection)
+  - View dataset preview in **Dataframe** tab in [Gradio app](https://46ccf31bbf1fadf8e8.gradio.live/)
+- For the demo app, the processed dataset is mirrored into Supabase and trained model is stored in Supabase Storage 
+  - See details in [Writeup: storage selection](writeup.md#storage-and-deployment-choices)
+
+## Evaluation
+
+- View performance metrics in the **Performance** tab of the [Gradio app](https://46ccf31bbf1fadf8e8.gradio.live/).
+- Evaluation uses a seeded **stratified train/test split** 
+  - **PR-AUC (Average Precision)** + PR curve (threshold-independent)
+  - **Confusion matrix (counts)** under the **argmax** decision policy
+- More details: [Writeup: Evaluation](writeup.md#evaluation)

app.py

@@ -0,0 +1,21 @@
+from __future__ import annotations
+
+import importlib.util
+import sys
+from pathlib import Path
+
+
+SRC_PATH = Path(__file__).parent / "src"
+if str(SRC_PATH) not in sys.path:
+    sys.path.insert(0, str(SRC_PATH))
+
+spec = importlib.util.spec_from_file_location("src_app", SRC_PATH / "app.py")
+module = importlib.util.module_from_spec(spec)
+sys.modules["src_app"] = module
+assert spec.loader is not None
+spec.loader.exec_module(module)
+
+demo = module.demo
+
+if __name__ == "__main__":
+    demo.launch()

requirements.txt

@@ -0,0 +1,8 @@
+gradio>=4.0
+pandas>=2.0
+numpy>=1.24
+scikit-learn>=1.3
+matplotlib>=3.7
+joblib>=1.3
+python-dotenv>=1.0
+supabase>=2.0

src/app.py

@@ -0,0 +1,167 @@
+from __future__ import annotations
+
+from functools import lru_cache
+from pathlib import Path
+
+import gradio as gr
+import joblib
+import pandas as pd
+from dotenv import load_dotenv, find_dotenv
+from sklearn.model_selection import train_test_split
+
+from store import supabase_io
+from ml import predict, evaluate, pre_process
+
+# Load env once at startup
+load_dotenv(find_dotenv())
+
+
+# ----------------------------
+# CACHED LOADERS
+# ----------------------------
+@lru_cache(maxsize=1)
+def get_model():
+    lp = supabase_io.download_artifact(
+        bucket="models",
+        object_path="resend/v1/pipeline.joblib",
+        local_path=".cache/pipeline.joblib",
+        force=False,  # set True only if retraining
+    )
+    return joblib.load(lp)
+
+@lru_cache(maxsize=1)
+def get_raw_df(limit: int = 20000) -> pd.DataFrame:
+    return supabase_io.fetch_df("emails_labeled", limit=limit)
+
+
+@lru_cache(maxsize=1)
+def get_eval_split():
+  
+    df = get_raw_df()
+    X, y = pre_process.prepare_xy(df)  #  deduplication, fill na
+    X_train, X_test, y_train, y_test = train_test_split(
+        X, y, test_size=0.25, random_state=37, stratify=y
+    ) # fixed split
+    return X_train, X_test, y_train, y_test
+
+# ----------------------------
+# TAB 1: SIMULATE EMAIL EVENT
+# ----------------------------
+def ui_predict_one(subject: str, body: str):
+    model = get_model()
+
+    label, confidence, probs, reasons = predict.predict_one_with_reasons(
+        model, subject=subject, body=body, top_k=10
+    )
+
+    # probs can be pd.Series or dict; normalize to dict for gr.Label
+    if hasattr(probs, "to_dict"):
+        probs_dict = probs.to_dict()
+    else:
+        probs_dict = dict(probs)
+
+    # Reasons
+    reasons_md = "\n".join([f"- {r}" for r in reasons]) if reasons else "_No strong features found._"
+
+    return (
+        str(label),
+        float(confidence),
+        probs_dict,       # for gr.Label (bars)
+        reasons_md,
+    )
+
+# ----------------------------
+# TAB 2: PERFORMANCE
+# ----------------------------
+def ui_run_eval():
+    model = get_model()
+    _, X_test, _, y_test = get_eval_split()
+
+    y_hat, proba = predict.probabilities_and_labels(model, X_test)
+
+    classes = list(model.named_steps["clf"].classes_)
+
+    # ensure proba is numpy array for PR code
+    if isinstance(proba, pd.DataFrame):
+        proba_np = proba[classes].to_numpy()
+    else:
+        proba_np = proba
+
+    cm_df, cm_fig = evaluate.plot_confusion_matrix_argmax(
+        y_test, y_hat, classes=classes, normalize=None
+    )
+    pr_auc, pr_fig = evaluate.plot_pr_auc_macro_from_proba(
+        y_test, proba_np, classes=classes
+    )
+
+    return cm_df, cm_fig, float(pr_auc), pr_fig
+
+
+# ----------------------------
+# TAB 3: DATAFRAME DISPLAY
+# ----------------------------
+def ui_show_df(n_rows: int):
+    df = get_raw_df()
+    return df.head(int(n_rows))
+
+
+# ----------------------------
+# APP
+# ----------------------------
+with gr.Blocks(title="Email Classifier Demo") as demo:
+    gr.Markdown(
+        """
+# Email Classifier
+
+**Data source:** [jason23322/high-accuracy-email-classifier](https://huggingface.co/datasets/jason23322/high-accuracy-email-classifier)  
+**License:** Apache-2.0  
+
+For training details, evaluation methodology, and how predictions/reasons are computed, see the **README** in 
+[repo](https://github.com/AFractalThought/email_prioritization)
+"""
+    )
+
+    with gr.Tabs():
+        # ---- Tab 1
+        with gr.Tab("Simulate email event"):
+            subject = gr.Textbox(label="Subject", lines=1, placeholder="e.g. Verify your email")
+            body = gr.Textbox(label="Body", lines=8, placeholder="Paste email body here...")
+
+            btn = gr.Button("Predict")
+
+            out_label = gr.Textbox(label="Prediction")
+            out_conf = gr.Number(label="Confidence (max probability)")
+            out_probs = gr.Label(label="Probabilities (all labels)", num_top_classes=10)
+            out_reasons = gr.Markdown(label="Reasons (top features)")
+
+            btn.click(
+                fn=ui_predict_one,
+                inputs=[subject, body],
+                outputs=[out_label, out_conf, out_probs, out_reasons],
+            )
+            # ---- Tab 2
+        with gr.Tab("Performance"):
+            gr.Markdown("Uses a fixed train/test split of the labeled dataset.")
+            run_eval = gr.Button("Run evaluation")
+
+            cm_table = gr.Dataframe(label="Confusion matrix (counts)")
+            cm_plot = gr.Plot(label="Confusion matrix plot")
+            pr_score = gr.Number(label="Macro PR-AUC (Average Precision)")
+            pr_plot = gr.Plot(label="Macro Precision–Recall curve")
+
+            run_eval.click(
+                fn=ui_run_eval,
+                inputs=[],
+                outputs=[cm_table, cm_plot, pr_score, pr_plot],
+            )
+
+        # ---- Tab 3
+        with gr.Tab("Dataframe"):
+            n_rows = gr.Slider(5, 500, value=50, step=5, label="Rows to display")
+            show = gr.Button("Show rows")
+            df_view = gr.Dataframe(label="emails_labeled (preview)", wrap=True)
+
+            show.click(fn=ui_show_df, inputs=[n_rows], outputs=[df_view])
+
+if __name__ == "__main__":
+    demo.launch()

src/ml/__init__.py

@@ -0,0 +1 @@
+# ml/__init__.py

src/ml/evaluate.py

@@ -0,0 +1,115 @@
+import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+from sklearn.pipeline import Pipeline
+from sklearn.metrics import (
+    confusion_matrix,
+    ConfusionMatrixDisplay,
+    precision_recall_curve,
+    average_precision_score,
+)
+
+
+def plot_confusion_matrix_argmax(
+    y_true,
+    y_pred,
+    classes=None,
+    normalize: str | None = "true",
+):
+    """
+    Confusion matrix for an argmax classifier.
+    Returns (cm_df, fig).
+
+    normalize:
+      - "true": normalize rows
+      - "pred", "all", or None for counts
+    """
+    y_true = np.asarray(y_true).astype(str)
+    y_pred = np.asarray(y_pred).astype(str)
+
+    if classes is None:
+        classes = sorted(set(y_true) | set(y_pred))
+
+    cm = confusion_matrix(y_true, y_pred, labels=classes, normalize=normalize)
+    cm_df = pd.DataFrame(cm, index=classes, columns=classes)
+
+    fig, ax = plt.subplots()
+    ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=classes).plot(
+        ax=ax,
+        values_format=".2f" if normalize else "d",
+        colorbar=True,
+    )
+    ax.set_title("Confusion matrix" + (" (normalized by True label)" if normalize else " (counts)"))
+    fig.tight_layout()
+
+    return cm_df, fig
+
+
+
+def plot_pr_auc_macro_from_proba(
+    y_true,
+    proba: np.ndarray,
+    classes,
+):
+    """
+    Compute macro-average AP (PR-AUC) for multiclass via one-vs-rest, and plot
+    a macro-averaged PR curve.
+
+    Inputs:
+      y_true: array-like of shape (n,)
+      proba:  ndarray of shape (n, K) with predicted probabilities
+      classes: list/array of length K matching columns of proba
+
+    Returns: (macro_ap, fig)
+    """
+    y_true = np.asarray(y_true).astype(str)
+    proba = np.asarray(proba)
+    classes = list(classes)
+
+    if proba.ndim != 2 or proba.shape[0] != len(y_true) or proba.shape[1] != len(classes):
+        raise ValueError(f"Shape mismatch: y_true={len(y_true)}, proba={proba.shape}, classes={len(classes)}")
+
+    ap_list = []
+    precisions = []
+    recalls = []
+
+    for j, cls in enumerate(classes):
+        y_bin = (y_true == cls).astype(int)
+        if y_bin.sum() == 0:
+            continue  # class not present in eval set
+
+        y_score = proba[:, j]
+        ap_list.append(average_precision_score(y_bin, y_score))
+
+        p, r, _ = precision_recall_curve(y_bin, y_score)
+        precisions.append(p)
+        recalls.append(r)
+
+    if not ap_list:
+        raise ValueError("No classes in y_true had positive examples; cannot compute PR-AUC.")
+
+    macro_ap = float(np.mean(ap_list))
+
+    # Macro PR curve by interpolating precision onto a common recall grid
+    recall_grid = np.linspace(0, 1, 200)
+    prec_on_grid = []
+    for p, r in zip(precisions, recalls):
+        order = np.argsort(r)
+        r_sorted = r[order]
+        p_sorted = p[order]
+        prec_on_grid.append(
+            np.interp(recall_grid, r_sorted, p_sorted, left=p_sorted[0], right=p_sorted[-1])
+        )
+
+    macro_precision = np.mean(np.vstack(prec_on_grid), axis=0)
+
+    fig, ax = plt.subplots()
+    ax.plot(recall_grid, macro_precision)
+    ax.set_xlim(0, 1)
+    ax.set_ylim(0, 1)
+    ax.set_xlabel("Recall")
+    ax.set_ylabel("Precision")
+    ax.set_title(f"Macro PR curve (macro AP = {macro_ap:.3f})")
+    fig.tight_layout()
+
+    return macro_ap, fig

src/ml/pre_process.py

@@ -0,0 +1,13 @@
+import pandas as pd
+
+def prepare_xy(df: pd.DataFrame):
+    df = df.copy()
+    df["subject"] = df["subject"].fillna("").astype(str)
+    df["body"] = df["body"].fillna("").astype(str)
+    df["label"] = df["label"].astype(str)
+
+    df = df.drop_duplicates(subset=["subject", "body"]).copy()
+
+    X = df[["subject", "body"]]
+    y = df["label"]
+    return X, y

src/ml/predict.py

@@ -0,0 +1,76 @@
+from sklearn.pipeline import Pipeline
+import pandas as pd
+import numpy as np
+from typing import Tuple
+
+def probabilities_and_labels(model: Pipeline, X: pd.DataFrame) -> pd.DataFrame:
+    predictions = model.predict(X)
+    probabilities = model.predict_proba(X)
+    classes = model.named_steps["clf"].classes_
+    proba_df = pd.DataFrame(probabilities, columns=classes, index=X.index)
+    return predictions, proba_df
+
+def predict_one(model: Pipeline, subject: str, body: str) -> tuple[str, pd.Series]:
+    X = pd.DataFrame([{
+        "subject": subject or "",
+        "body": body or "",
+    }])
+    y_pred, proba_df = probabilities_and_labels(model, X)
+    return y_pred[0], proba_df.iloc[0]
+
+
+def explain_linear_top_features(
+    model: Pipeline,
+    X: pd.DataFrame,
+    class_label: str,
+    top_k: int = 10,
+) -> list[str]:
+    """
+    Returns top_k feature contributions for `class_label` on the first row of X.
+    Contributions are in logit units (not probability).
+    """
+    text = model.named_steps["text"]
+    clf = model.named_steps["clf"]
+
+    Xv = text.transform(X)  # sparse (1, n_features)
+    feature_names = text.get_feature_names_out()
+    classes = list(clf.classes_)
+    class_idx = classes.index(class_label)
+
+    coef = clf.coef_  # (K, n_features) for multiclass, or (1, n_features) for binary
+    if coef.shape[0] == 1 and len(classes) == 2:
+        # binary special-case: sklearn stores only coef for classes_[1]
+        coef_c = coef[0] if class_idx == 1 else -coef[0]
+    else:
+        coef_c = coef[class_idx]
+
+    row = Xv[0]
+    idx = row.indices
+    vals = row.data
+    if idx.size == 0:
+        return ["No nonzero TF-IDF features (empty subject/body after preprocessing)."]
+
+    contrib = vals * coef_c[idx]  # per-feature contribution for this class
+    order = np.argsort(np.abs(contrib))[::-1][:top_k]
+
+    reasons = []
+    for k in order:
+        feat = feature_names[idx[k]]  # e.g. "subject__password" or "body__unsubscribe"
+        reasons.append(f"{feat}: {contrib[k]:+.3f}")
+    return reasons
+
+
+def predict_one_with_reasons(model: Pipeline, subject: str, body: str, top_k: int = 10):
+    X = pd.DataFrame([{"subject": subject or "", "body": body or ""}])
+
+    proba = model.predict_proba(X)[0]
+    classes = list(model.named_steps["clf"].classes_)
+    probs = pd.Series(proba, index=classes)
+
+    label = probs.idxmax()
+    confidence = float(probs.max())
+
+    reasons = explain_linear_top_features(model, X, class_label=label, top_k=top_k)
+    return label, confidence, probs, reasons
+
+   

src/ml/train.py

@@ -0,0 +1,50 @@
+import numpy as np
+import pandas as pd
+
+from sklearn.pipeline import Pipeline
+from sklearn.compose import ColumnTransformer
+from sklearn.feature_extraction.text import TfidfVectorizer
+from sklearn.linear_model import LogisticRegression
+from sklearn.model_selection import train_test_split
+
+
+from pathlib import Path
+import joblib
+from ml import pre_process
+
+def fit(df: pd.DataFrame, test_size = 0.25) -> Pipeline:
+
+    # Preprocess table (de-duplicate etc)
+    X, y = pre_process.prepare_xy(df)
+
+    # Split train and test
+    X_train, X_test, y_train, y_test = train_test_split(
+            X, y, test_size=test_size, random_state=37, stratify=y
+        )
+
+    text = ColumnTransformer(
+        transformers=[
+            ("subject", TfidfVectorizer(ngram_range=(1, 2), min_df=2, max_features=50_000), "subject"),
+            ("body",    TfidfVectorizer(ngram_range=(1, 2), min_df=2, max_features=200_000), "body"),
+        ],
+        remainder="drop",
+        sparse_threshold=0.3,
+    )
+
+    clf = LogisticRegression(
+        max_iter=3000,
+        class_weight="balanced",
+    )
+
+    pipe = Pipeline([
+        ("text", text),
+        ("clf", clf),
+    ])
+
+    return pipe.fit(X_train, y_train)
+
+def save_model_local(model: Pipeline, path: str | Path) -> Path:
+    path = Path(path)
+    path.parent.mkdir(parents=True, exist_ok=True)
+    joblib.dump(model, path)
+    return path

src/store/__init__.py

@@ -0,0 +1 @@
+# store/__init__.py

src/store/supabase_io.py

@@ -0,0 +1,53 @@
+import os
+from pathlib import Path
+from typing import Optional
+
+import pandas as pd
+from supabase import create_client
+
+
+def make_supabase_client():
+    url = os.environ["SUPABASE_URL"]
+    key = os.environ["SUPABASE_ANON_KEY"]  
+    return create_client(url, key)
+
+def fetch_df(table_name: str, limit: int = 10000) -> pd.DataFrame:
+    supabase = make_supabase_client()
+    resp = supabase.table(table_name).select("*").limit(limit).execute()
+    return pd.DataFrame(resp.data)
+
+def upload_artifact(local_path: str | Path, bucket: str, object_path: str, upsert: bool = True) -> None:
+    supabase = make_supabase_client()
+    p = Path(local_path)
+
+    with p.open("rb") as f:
+        supabase.storage.from_(bucket).upload(
+            path=object_path,
+            file=f,
+            file_options={"upsert": "true" if upsert else "false", "cache-control": "3600"},
+        )
+
+def download_artifact(
+    bucket: str,
+    object_path: str,
+    local_path: str | Path,
+    force: bool = False,
+) -> Path:
+    """
+    Download an object from Supabase Storage to a local file.
+
+    - bucket: storage bucket name
+    - object_path: remote key inside bucket (e.g. "resend/models/model.joblib")
+    - local_path: where to save locally
+    - force: if True, re-download even if local file exists
+    """
+    supabase = make_supabase_client()
+    lp = Path(local_path)
+    lp.parent.mkdir(parents=True, exist_ok=True)
+
+    if lp.exists() and not force:
+        return lp
+
+    data = supabase.storage.from_(bucket).download(object_path)  # bytes
+    lp.write_bytes(data)
+    return lp

src/train_upload_notebook.ipynb

@@ -0,0 +1,71 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "ce4c9651-f069-45ad-a66f-19f29c24c9ee",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%capture output_var\n",
+    "\n",
+    "from dotenv import load_dotenv, find_dotenv\n",
+    "from store import supabase_io\n",
+    "from ml import train\n",
+    "\n",
+    "load_dotenv(find_dotenv())\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "5fa0089d-d797-430d-9ba0-20f2ed7094e6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Supabase table name\n",
+    "table = \"emails_labeled\"\n",
+    "\n",
+    "# Fetch table from Supabase\n",
+    "df = supabase_io.fetch_df(table, limit=20000)\n",
+    "\n",
+    "# Fit model\n",
+    "model = train.fit(df)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "043e29bd-480d-4ebd-9cdb-305d0122946e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "## Upload Artifact\n",
+    "# save model locally and upload to supabase\n",
+    "path = train.save_model_local(model, \"artifacts/pipeline.joblib\")\n",
+    "supabase_io.upload_artifact(path, bucket = \"models\", object_path = \"resend/v1/pipeline.joblib\", upsert = True)\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.2"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

writeup.md

@@ -0,0 +1,175 @@
+# Data Scientist Trust & Safety Take Home Challenge — Write Up
+
+This document summarizes the key implementation decisions behind the submission: dataset selection and label mapping, model/features, confidence + reasoning definitions, evaluation choices, and storage/deployment setup.
+
+## Dataset selection
+
+### Why I did not use the provided dataset
+
+I initially explored the dataset included in the prompt, but it introduced several issues that would have made the model and evaluation misleading:
+
+- **Weak feature coherence:** In many rows, the `From`, `Subject`, and `Body` fields did not appear semantically consistent (suggesting independent templating). This makes it difficult to learn realistic correlations across fields and undermines the intent of using all three signals.
+  
+- **Missing “priority” examples:** There were few/no clear MFA or verification-code examples, which is explicitly required for the `Prioritize` class.
+  
+- **Label mismatch with task:** Labels were `ham/spam`, while the assignment requires `priority/default/slow`. I could have generated new labels, but under time constraints it was more reliable to use a dataset whose categories already aligned with the goal.
+
+### Why I chose the Hugging Face dataset
+
+I used a [a dataset obtained from Hugging Face](https://huggingface.co/datasets/jason23322/high-accuracy-email-classifier) `jason23322/high-accuracy-email-classifier` (Apache-2.0):
+
+- The dataset’s categories map directly to the assignment’s required classes (especially verification codes).
+  
+- The examples are more internally coherent (subject/body alignment is generally consistent).
+  
+- Duplication exists (as in most datasets) but is manageable; I explicitly **de-duplicate identical subject/body pairs** before splitting to reduce leakage risk.
+
+### Caveats / limitations of the chosen dataset
+
+- The dataset is still **simulated**, so performance is not representative of real email traffic.
+  
+- It does **not include a sender field**. I realize the assignment specifically mentions using `from` (available in the provided dataset). I chose not to fabricate sender values or mix in another dataset due to scope/complexity, and because subject/body alone were sufficient to demonstrate the approach.
+
+## Label mapping
+
+### Original dataset labels
+
+`['promotions', 'spam', 'social_media', 'forum', 'verify_code', 'updates']`
+
+### Mapping to assignment labels
+
+| Assignment label | Description (prompt)              | Dataset categories mapped         |
+|---|---|---|
+| **Prioritize** | MFA codes / verification           | `verify_code`                     |
+| **Default**    | Generic messages                   | `social_media`, `forum`, `updates`|
+| **Slow**       | Non-urgent promotional messages    |`promotions`
+| **(Spam*)**       | NOT included in assignment    | `spam`                      |
+
+### Why I included `spam` as an extra class
+
+The assignment defines three labels, but the dataset contains a clear spam category. I kept `spam` as a separate label instead of dropping it or mapping it into `slow`, because in a realistic pipeline you’d rather route obvious spam away from the inbox than treat it as “slow.” This makes the demo more practical, and the model still outputs probabilities across all classes.
+
+## Model approach
+
+### Vectorization
+
+I used **TF–IDF** features for the `subject` and `body`.
+
+- I considered embeddings/LLM-based approaches, but TF–IDF is a strong baseline, easy to debug, and well-suited to short templated text like verification codes and promotions.
+  
+- I intentionally vectorize **subject and body separately** (two TF–IDF vectorizers inside a `ColumnTransformer`). This preserves the distinction between words appearing in the subject vs the body (e.g., “verify” in a subject line is often more predictive than in a long email body).
+
+### Model choice
+
+I used **multinomial Logistic Regression**.
+
+- Logistic regression is interpretable, fast, and a good baseline for text classification.
+  
+- I did not run hyperparameter tuning (e.g., grid search) under time constraints; the goal was a clean, understandable pipeline and working demo or explore other model choices
+
+### Decision policy
+
+The model outputs probabilities for each label via `predict_proba`.
+
+- The default policy is **argmax**: choose the label with the highest probability.
+  
+- In a production system, you might use **class-specific thresholds** (e.g., prioritize high recall for `Prioritize`, high precision for `Spam`). I kept argmax to keep behavior simple and evaluation straightforward.
+
+## Confidence and reasoning
+
+### Confidence
+
+**Confidence = max predicted probability** for the selected label (i.e., `max(p(label | email))`).
+
+This is a standard definition for multi-class models. Note: probabilities are not guaranteed to be perfectly calibrated, but they are a useful relative measure for the demo.
+
+### Reasoning
+
+To produce human-readable reasons, I compute **top contributing TF–IDF features** for the predicted class by combining:
+
+- the email’s TF–IDF vector (non-zero features)
+- the logistic regression coefficients for the predicted class
+
+This yields a ranked list of tokens/ngrams that most influenced the decision (“top features” explanation).
+
+**Interpretability note:** The “reasons” shown in the app come from logistic regression feature contributions (TF-IDF weights × class coefficients). This approach is specific to linear models. If I were to switch to non-linear models (e.g., tree ensembles or neural embeddings), I’d replace this with a model-agnostic explanation method such as SHAP (or integrated gradients for neural models), at the cost of additional complexity and runtime.
+
+## Evaluation
+
+### Metrics shown in the UI
+
+I report two complementary views:
+
+1. **Confusion matrix (counts)** using the argmax policy  
+   - Provides concrete error counts by class.
+2. **Macro PR-AUC (Average Precision)** as threshold-independent performance  
+   - PR-AUC is appropriate under class imbalance and aligns with “ranking quality” of `predict_proba`.
+
+## Storage and deployment choices
+
+### Storage / data
+
+I used **Supabase** because:
+
+- it provides a simple Postgres backend for storing labeled examples
+  
+- it includes object storage suitable for model artifacts (similar to S3)
+  
+- I have used it before and it integrates cleanly with Python
+
+### App
+
+I used **Gradio** because it supports:
+
+- quick “send a test email event” workflows
+  
+- easy visualization of plots and dataframes
+  
+- simple deployment patterns
+
+## How the system works end-to-end
+
+### Training (local)
+
+- Fetch training data from Supabase
+  
+- Preprocess (de-duplicate, split)
+  
+- Vectorize text with TF–IDF + Logistic Regression pipeline
+  
+- Save model artifact locally (`joblib`)
+  
+- Upload artifact to Supabase Storage
+
+### App (online)
+
+- Download the trained artifact from Supabase Storage (cached locally)
+  
+- Fetch the dataset from Supabase (cached)
+  
+- Tabs:
+  - **Simulate email event:** user enters subject/body → app returns label, confidence, per-class probabilities, and reasons
+  - **Performance:** computes confusion matrix + PR curve / macro PR-AUC on a fixed split
+  - **Dataframe:** displays a preview of the dataset
+
+## Notes on real-world performance and model building
+
+The model performs well on this simulated dataset, but performance will drop on real email traffic due to:
+
+- broader vocabulary and writing styles
+  
+- adversarial spam behavior
+  
+- distribution shifts (new templates, new senders, different user contexts)
+
+A production-grade version would require:
+
+- real emails
+  
+- better calibration + thresholding policies
+  
+- continuous evaluation / monitoring
+  
+- richer features including sender domain reputation and message metadata
+  
+- trying multiple models, and policies, hyperparameter tuning