| # Hate Speech Detection — Multilingual Sequential Transfer Learning |
| ### GloVe Embeddings + Bidirectional LSTM (BiLSTM) |
|
|
| --- |
|
|
| ## What is this project about? |
|
|
| This project builds a system that can automatically detect **hate speech** in text written in three languages: |
| - **English** — standard English text |
| - **Hindi** — Hindi text (transliterated or native script) |
| - **Hinglish** — a mix of Hindi and English (very common in Indian social media) |
|
|
| The core question we are trying to answer is: |
|
|
| > **Does the order in which you teach a model different languages matter for how well it performs?** |
|
|
| For example — is a model that learns English first, then Hindi, then Hinglish better or worse than one that learns Hinglish first? |
|
|
| --- |
|
|
| ## The Dataset |
|
|
| | Property | Value | |
| |---|---| |
| | Total samples | 29,505 | |
| | English samples | 14,994 (50.8%) | |
| | Hindi samples | 9,738 (33.0%) | |
| | Hinglish samples | 4,774 (16.2%) | |
| | Hate speech (label=1) | 13,707 (46.5%) | |
| | Non-hate speech (label=0) | 15,799 (53.5%) | |
|
|
|  |
|
|
| The dataset was split into three parts: |
| - **Training set** — 17,704 samples (used to teach the model) |
| - **Validation set** — 2,950 samples (used to monitor learning during training) |
| - **Test set** — 8,852 samples (used only at the end to measure real performance) |
|
|
| --- |
|
|
| ## The Model — What is GloVe + BiLSTM? |
|
|
| Think of the model like a two-part reading machine: |
|
|
| ### Part 1: GloVe Embeddings (the dictionary) |
| Before the model can understand words, it needs to know what words *mean* relative to each other. GloVe (Global Vectors) is a pre-trained lookup table of **300,000+ English words**, where each word is represented as a list of 300 numbers that capture its meaning. Words with similar meanings end up with similar numbers. |
|
|
| - We used `glove.6B.300d.txt` — 6 billion word training corpus, 300 dimensions |
| - The embedding layer is **frozen** (not updated during training) — we keep GloVe's knowledge as-is and only train the layers on top |
|
|
| ### Part 2: Bidirectional LSTM (the reader) |
| An LSTM (Long Short-Term Memory) is a type of neural network designed to read sequences — like sentences — and remember what it read. **Bidirectional** means it reads the sentence both forwards and backwards, so it understands context from both directions. |
|
|
| ``` |
| Input sentence |
| ↓ |
| GloVe Embeddings (300d, frozen) |
| ↓ |
| BiLSTM (128 units, reads left→right AND right←left) |
| ↓ |
| Dropout (50% — randomly switches off neurons to prevent overfitting) |
| ↓ |
| Dense layer (64 neurons, ReLU activation) |
| ↓ |
| Output (1 neuron, Sigmoid — gives a probability 0 to 1) |
| ↓ |
| > 0.5 = Hate Speech, ≤ 0.5 = Not Hate Speech |
| ``` |
|
|
| --- |
|
|
| ## The Training Strategy — What is Transfer Learning? |
|
|
| **Transfer learning** means the model carries what it learned from one task into the next. Like a student who already knows French — learning Spanish is easier because both share Latin roots. |
|
|
| In our case, we train the model on one language, and instead of starting fresh for the next language, we **keep all the weights (knowledge)** from the previous training. The model continues learning from where it left off. |
|
|
| ### The Bug We Fixed |
| The original code was creating a **brand new model** for every language — resetting all the weights each time. That is not transfer learning, it's just training three separate models. We fixed this by building the model **once** and sequentially fine-tuning it. |
|
|
| ```python |
| # WRONG — model reset every loop iteration |
| for lang in languages: |
| model = Sequential() # ← new model = no transfer learning |
| model.fit(...) |
| |
| # CORRECT — model built once, weights carry forward |
| model = build_model() # ← built once |
| for lang in languages: |
| model.fit(...) # ← continues learning from previous language |
| ``` |
|
|
| --- |
|
|
| ## Plan B — The Experiment |
|
|
| We ran all **6 possible orderings** of the three languages, each followed by a final training round on the complete shuffled dataset: |
|
|
| | # | Strategy | |
| |---|---| |
| | 1 | English → Hindi → Hinglish → Full | |
| | 2 | English → Hinglish → Hindi → Full | |
| | 3 | Hindi → English → Hinglish → Full | |
| | 4 | Hindi → Hinglish → English → Full | |
| | 5 | Hinglish → English → Hindi → Full | |
| | 6 | Hinglish → Hindi → English → Full | |
|
|
| For each strategy, training happens in 4 phases. **After each phase**, we immediately evaluate the model on that specific language's test data and record all metrics. This tells us how well the model performs at each stage of the learning journey. |
|
|
| ``` |
| Phase 1: Train on Language A → Test on Language A test set → Record metrics + plots |
| Phase 2: Train on Language B → Test on Language B test set → Record metrics + plots |
| Phase 3: Train on Language C → Test on Language C test set → Record metrics + plots |
| Phase 4: Train on Full data → Test on Full test set → Record metrics + plots |
| ``` |
|
|
| Each phase used **8 epochs** with batch size 32 (64 for the full phase). |
|
|
| --- |
|
|
| ## Metrics — What do we measure? |
|
|
| | Metric | What it means in plain English | |
| |---|---| |
| | **Accuracy** | Out of all predictions, how many were correct? | |
| | **Balanced Accuracy** | Accuracy adjusted for class imbalance (more fair) | |
| | **Precision** | Of everything the model flagged as hate speech, how much actually was? | |
| | **Recall** | Of all actual hate speech, how much did the model catch? | |
| | **Specificity** | Of all non-hate speech, how much did the model correctly ignore? | |
| | **F1 Score** | Balance between Precision and Recall (harmonic mean) | |
| | **ROC-AUC** | Overall ability to distinguish hate from non-hate (1.0 = perfect) | |
|
|
| --- |
|
|
| ## Results Summary |
|
|
| Full results are in `output/results_tables/all_strategies_results.csv`. Key highlights: |
|
|
| ### English phase performance across strategies (best language) |
|
|
| | Strategy | Accuracy | F1 | ROC-AUC | |
| |---|---|---|---| |
| | English → Hindi → Hinglish → Full | 0.7701 | 0.7696 | 0.8504 | |
| | English → Hinglish → Hindi → Full | 0.7721 | 0.7743 | 0.8525 | |
| | Hindi → English → Hinglish → Full | 0.7780 | 0.7830 | 0.8549 | |
| | Hindi → Hinglish → English → Full | 0.7780 | 0.7816 | 0.8563 | |
| | Hinglish → English → Hindi → Full | 0.7716 | 0.7829 | 0.8484 | |
| | Hinglish → Hindi → English → Full | 0.7765 | 0.7811 | 0.8534 | |
|
|
| ### Full dataset phase (final performance) |
|
|
| | Strategy | Accuracy | F1 | ROC-AUC | |
| |---|---|---|---| |
| | English → Hindi → Hinglish → Full | 0.6796 | 0.5923 | 0.7599 | |
| | English → Hinglish → Hindi → Full | 0.6813 | 0.6244 | 0.7535 | |
| | Hindi → English → Hinglish → Full | 0.6854 | 0.6419 | 0.7528 | |
| | Hindi → Hinglish → English → Full | 0.6865 | 0.6364 | 0.7507 | |
| | Hinglish → English → Hindi → Full | 0.6778 | 0.6285 | 0.7521 | |
| | Hinglish → Hindi → English → Full | 0.6845 | 0.6301 | 0.7548 | |
|
|
| ### Key observations |
| - **English** consistently achieves the highest accuracy (~77%) regardless of when it is trained — likely because GloVe embeddings are English-centric |
| - **Hindi** is the hardest language — accuracy hovers around 55–59% across all strategies |
| - **Hinglish** sits in the middle (~66–70%) which makes sense as it borrows heavily from English |
| - Strategies that train **Hindi first** (`Hindi → English → Hinglish`) tend to recover better in later phases, suggesting the model benefits from tackling the hardest language early |
| - The **Full phase** shows consistent ~68% accuracy across all strategies, suggesting the final shuffled training normalises the differences introduced by ordering |
|
|
| --- |
|
|
| ## Plots by Strategy |
|
|
| ### Strategy 1: English → Hindi → Hinglish → Full |
|
|
| | Phase | Training Curves | Confusion Matrix | ROC Curve | PR Curve | F1 Curve | |
| |---|---|---|---|---|---| |
| | English |  |  |  |  |  | |
| | Hindi |  |  |  |  |  | |
| | Hinglish |  |  |  |  |  | |
| | Full |  |  |  |  |  | |
|
|
| --- |
|
|
| ### Strategy 2: English → Hinglish → Hindi → Full |
|
|
| | Phase | Training Curves | Confusion Matrix | ROC Curve | PR Curve | F1 Curve | |
| |---|---|---|---|---|---| |
| | English |  |  |  |  |  | |
| | Hinglish |  |  |  |  |  | |
| | Hindi |  |  |  |  |  | |
| | Full |  |  |  |  |  | |
|
|
| --- |
|
|
| ### Strategy 3: Hindi → English → Hinglish → Full |
|
|
| | Phase | Training Curves | Confusion Matrix | ROC Curve | PR Curve | F1 Curve | |
| |---|---|---|---|---|---| |
| | Hindi |  |  |  |  |  | |
| | English |  |  |  |  |  | |
| | Hinglish |  |  |  |  |  | |
| | Full |  |  |  |  |  | |
|
|
| --- |
|
|
| ### Strategy 4: Hindi → Hinglish → English → Full |
|
|
| | Phase | Training Curves | Confusion Matrix | ROC Curve | PR Curve | F1 Curve | |
| |---|---|---|---|---|---| |
| | Hindi |  |  |  |  |  | |
| | Hinglish |  |  |  |  |  | |
| | English |  |  |  |  |  | |
| | Full |  |  |  |  |  | |
|
|
| --- |
|
|
| ### Strategy 5: Hinglish → English → Hindi → Full |
|
|
| | Phase | Training Curves | Confusion Matrix | ROC Curve | PR Curve | F1 Curve | |
| |---|---|---|---|---|---| |
| | Hinglish |  |  |  |  |  | |
| | English |  |  |  |  |  | |
| | Hindi |  |  |  |  |  | |
| | Full |  |  |  |  |  | |
|
|
| --- |
|
|
| ### Strategy 6: Hinglish → Hindi → English → Full |
|
|
| | Phase | Training Curves | Confusion Matrix | ROC Curve | PR Curve | F1 Curve | |
| |---|---|---|---|---|---| |
| | Hinglish |  |  |  |  |  | |
| | Hindi |  |  |  |  |  | |
| | English |  |  |  |  |  | |
| | Full |  |  |  |  |  | |
|
|
| --- |
|
|
| ## Output Files |
|
|
| ``` |
| output/ |
| ├── dataset_splits/ |
| │ ├── train.csv # 17,704 training samples |
| │ ├── val.csv # 2,950 validation samples |
| │ └── test.csv # 8,852 test samples |
| │ |
| ├── results_tables/ |
| │ ├── all_strategies_results.csv # All 24 rows (6 strategies × 4 phases) |
| │ ├── english_to_hindi_to_hinglish_results.csv |
| │ ├── english_to_hinglish_to_hindi_results.csv |
| │ ├── hindi_to_english_to_hinglish_results.csv |
| │ ├── hindi_to_hinglish_to_english_results.csv |
| │ ├── hinglish_to_english_to_hindi_results.csv |
| │ └── hinglish_to_hindi_to_english_results.csv |
| │ |
| └── figures/ |
| ├── language_distribution.png # Pie chart of dataset languages |
| │ |
| ├── english_to_hindi_to_hinglish/ # One folder per strategy |
| │ ├── *_[english]_curves.png # Train/Val accuracy + loss |
| │ ├── *_[english]_cm.png # Confusion matrix |
| │ ├── *_[english]_roc.png # ROC curve |
| │ ├── *_[english]_pr.png # Precision-Recall curve |
| │ ├── *_[english]_f1.png # F1 vs Threshold curve |
| │ ├── *_[hindi]_curves.png |
| │ ├── *_[hindi]_cm.png ... |
| │ ├── *_[hinglish]_curves.png |
| │ ├── *_[hinglish]_cm.png ... |
| │ ├── *_[Full]_curves.png |
| │ └── *_[Full]_cm.png ... |
| │ |
| ├── english_to_hinglish_to_hindi/ |
| ├── hindi_to_english_to_hinglish/ |
| ├── hindi_to_hinglish_to_english/ |
| ├── hinglish_to_english_to_hindi/ |
| └── hinglish_to_hindi_to_english/ |
| ``` |
|
|
| --- |
|
|
| ## How to Run |
|
|
| ### Requirements |
| ```bash |
| pip install tensorflow scikit-learn pandas seaborn matplotlib |
| ``` |
|
|
| You also need GloVe embeddings (`glove.6B.300d.txt`) placed at `/root/glove.6B.300d.txt`: |
| ```bash |
| wget http://nlp.stanford.edu/data/glove.6B.zip && unzip glove.6B.zip |
| ``` |
|
|
| ### Run |
| ```bash |
| python main.py |
| ``` |
|
|
| Training was performed on an NVIDIA H200 GPU (Vast.ai) — total runtime approximately 15–20 minutes for all 6 strategies. |
|
|
| --- |
|
|
| ## Project Structure |
|
|
| ``` |
| SASC/ |
| ├── main.py # Full training + evaluation pipeline |
| ├── dataset.csv # Raw dataset (29,505 samples) |
| ├── README.md # This file |
| └── output/ # All results, figures, and model checkpoints |
| ``` |
|
|
