File size: 23,061 Bytes
fcc2612 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 | # Hate Speech Detection — Multilingual Sequential Transfer Learning
### GloVe Embeddings + Bidirectional LSTM (BiLSTM)
---
## What is this project about?
This project builds a system that can automatically detect **hate speech** in text written in three languages:
- **English** — standard English text
- **Hindi** — Hindi text (transliterated or native script)
- **Hinglish** — a mix of Hindi and English (very common in Indian social media)
The core question we are trying to answer is:
> **Does the order in which you teach a model different languages matter for how well it performs?**
For example — is a model that learns English first, then Hindi, then Hinglish better or worse than one that learns Hinglish first?
---
## The Dataset
| Property | Value |
|---|---|
| Total samples | 29,505 |
| English samples | 14,994 (50.8%) |
| Hindi samples | 9,738 (33.0%) |
| Hinglish samples | 4,774 (16.2%) |
| Hate speech (label=1) | 13,707 (46.5%) |
| Non-hate speech (label=0) | 15,799 (53.5%) |
The dataset was split into three parts:
- **Training set** — 17,704 samples (used to teach the model)
- **Validation set** — 2,950 samples (used to monitor learning during training)
- **Test set** — 8,852 samples (used only at the end to measure real performance)
---
## The Model — What is GloVe + BiLSTM?
Think of the model like a two-part reading machine:
### Part 1: GloVe Embeddings (the dictionary)
Before the model can understand words, it needs to know what words *mean* relative to each other. GloVe (Global Vectors) is a pre-trained lookup table of **300,000+ English words**, where each word is represented as a list of 300 numbers that capture its meaning. Words with similar meanings end up with similar numbers.
- We used `glove.6B.300d.txt` — 6 billion word training corpus, 300 dimensions
- The embedding layer is **frozen** (not updated during training) — we keep GloVe's knowledge as-is and only train the layers on top
### Part 2: Bidirectional LSTM (the reader)
An LSTM (Long Short-Term Memory) is a type of neural network designed to read sequences — like sentences — and remember what it read. **Bidirectional** means it reads the sentence both forwards and backwards, so it understands context from both directions.
```
Input sentence
↓
GloVe Embeddings (300d, frozen)
↓
BiLSTM (128 units, reads left→right AND right←left)
↓
Dropout (50% — randomly switches off neurons to prevent overfitting)
↓
Dense layer (64 neurons, ReLU activation)
↓
Output (1 neuron, Sigmoid — gives a probability 0 to 1)
↓
> 0.5 = Hate Speech, ≤ 0.5 = Not Hate Speech
```
---
## The Training Strategy — What is Transfer Learning?
**Transfer learning** means the model carries what it learned from one task into the next. Like a student who already knows French — learning Spanish is easier because both share Latin roots.
In our case, we train the model on one language, and instead of starting fresh for the next language, we **keep all the weights (knowledge)** from the previous training. The model continues learning from where it left off.
### The Bug We Fixed
The original code was creating a **brand new model** for every language — resetting all the weights each time. That is not transfer learning, it's just training three separate models. We fixed this by building the model **once** and sequentially fine-tuning it.
```python
# WRONG — model reset every loop iteration
for lang in languages:
model = Sequential() # ← new model = no transfer learning
model.fit(...)
# CORRECT — model built once, weights carry forward
model = build_model() # ← built once
for lang in languages:
model.fit(...) # ← continues learning from previous language
```
---
## Plan B — The Experiment
We ran all **6 possible orderings** of the three languages, each followed by a final training round on the complete shuffled dataset:
| # | Strategy |
|---|---|
| 1 | English → Hindi → Hinglish → Full |
| 2 | English → Hinglish → Hindi → Full |
| 3 | Hindi → English → Hinglish → Full |
| 4 | Hindi → Hinglish → English → Full |
| 5 | Hinglish → English → Hindi → Full |
| 6 | Hinglish → Hindi → English → Full |
For each strategy, training happens in 4 phases. **After each phase**, we immediately evaluate the model on that specific language's test data and record all metrics. This tells us how well the model performs at each stage of the learning journey.
```
Phase 1: Train on Language A → Test on Language A test set → Record metrics + plots
Phase 2: Train on Language B → Test on Language B test set → Record metrics + plots
Phase 3: Train on Language C → Test on Language C test set → Record metrics + plots
Phase 4: Train on Full data → Test on Full test set → Record metrics + plots
```
Each phase used **8 epochs** with batch size 32 (64 for the full phase).
---
## Metrics — What do we measure?
| Metric | What it means in plain English |
|---|---|
| **Accuracy** | Out of all predictions, how many were correct? |
| **Balanced Accuracy** | Accuracy adjusted for class imbalance (more fair) |
| **Precision** | Of everything the model flagged as hate speech, how much actually was? |
| **Recall** | Of all actual hate speech, how much did the model catch? |
| **Specificity** | Of all non-hate speech, how much did the model correctly ignore? |
| **F1 Score** | Balance between Precision and Recall (harmonic mean) |
| **ROC-AUC** | Overall ability to distinguish hate from non-hate (1.0 = perfect) |
---
## Results Summary
Full results are in `output/results_tables/all_strategies_results.csv`. Key highlights:
### English phase performance across strategies (best language)
| Strategy | Accuracy | F1 | ROC-AUC |
|---|---|---|---|
| English → Hindi → Hinglish → Full | 0.7701 | 0.7696 | 0.8504 |
| English → Hinglish → Hindi → Full | 0.7721 | 0.7743 | 0.8525 |
| Hindi → English → Hinglish → Full | 0.7780 | 0.7830 | 0.8549 |
| Hindi → Hinglish → English → Full | 0.7780 | 0.7816 | 0.8563 |
| Hinglish → English → Hindi → Full | 0.7716 | 0.7829 | 0.8484 |
| Hinglish → Hindi → English → Full | 0.7765 | 0.7811 | 0.8534 |
### Full dataset phase (final performance)
| Strategy | Accuracy | F1 | ROC-AUC |
|---|---|---|---|
| English → Hindi → Hinglish → Full | 0.6796 | 0.5923 | 0.7599 |
| English → Hinglish → Hindi → Full | 0.6813 | 0.6244 | 0.7535 |
| Hindi → English → Hinglish → Full | 0.6854 | 0.6419 | 0.7528 |
| Hindi → Hinglish → English → Full | 0.6865 | 0.6364 | 0.7507 |
| Hinglish → English → Hindi → Full | 0.6778 | 0.6285 | 0.7521 |
| Hinglish → Hindi → English → Full | 0.6845 | 0.6301 | 0.7548 |
### Key observations
- **English** consistently achieves the highest accuracy (~77%) regardless of when it is trained — likely because GloVe embeddings are English-centric
- **Hindi** is the hardest language — accuracy hovers around 55–59% across all strategies
- **Hinglish** sits in the middle (~66–70%) which makes sense as it borrows heavily from English
- Strategies that train **Hindi first** (`Hindi → English → Hinglish`) tend to recover better in later phases, suggesting the model benefits from tackling the hardest language early
- The **Full phase** shows consistent ~68% accuracy across all strategies, suggesting the final shuffled training normalises the differences introduced by ordering
---
## Plots by Strategy
### Strategy 1: English → Hindi → Hinglish → Full
| Phase | Training Curves | Confusion Matrix | ROC Curve | PR Curve | F1 Curve |
|---|---|---|---|---|---|
| English |  |  |  |  |  |
| Hindi |  |  |  |  |  |
| Hinglish |  |  |  |  |  |
| Full |  |  |  |  |  |
---
### Strategy 2: English → Hinglish → Hindi → Full
| Phase | Training Curves | Confusion Matrix | ROC Curve | PR Curve | F1 Curve |
|---|---|---|---|---|---|
| English |  |  |  |  |  |
| Hinglish |  |  |  |  |  |
| Hindi |  |  |  |  |  |
| Full |  |  |  |  |  |
---
### Strategy 3: Hindi → English → Hinglish → Full
| Phase | Training Curves | Confusion Matrix | ROC Curve | PR Curve | F1 Curve |
|---|---|---|---|---|---|
| Hindi |  |  |  |  |  |
| English |  |  |  |  |  |
| Hinglish |  |  |  |  |  |
| Full |  |  |  |  |  |
---
### Strategy 4: Hindi → Hinglish → English → Full
| Phase | Training Curves | Confusion Matrix | ROC Curve | PR Curve | F1 Curve |
|---|---|---|---|---|---|
| Hindi |  |  |  |  |  |
| Hinglish |  |  |  |  |  |
| English |  |  |  |  |  |
| Full |  |  |  |  |  |
---
### Strategy 5: Hinglish → English → Hindi → Full
| Phase | Training Curves | Confusion Matrix | ROC Curve | PR Curve | F1 Curve |
|---|---|---|---|---|---|
| Hinglish |  |  |  |  |  |
| English |  |  |  |  |  |
| Hindi |  |  |  |  |  |
| Full |  |  |  |  |  |
---
### Strategy 6: Hinglish → Hindi → English → Full
| Phase | Training Curves | Confusion Matrix | ROC Curve | PR Curve | F1 Curve |
|---|---|---|---|---|---|
| Hinglish |  |  |  |  |  |
| Hindi |  |  |  |  |  |
| English |  |  |  |  |  |
| Full |  |  |  |  |  |
---
## Output Files
```
output/
├── dataset_splits/
│ ├── train.csv # 17,704 training samples
│ ├── val.csv # 2,950 validation samples
│ └── test.csv # 8,852 test samples
│
├── results_tables/
│ ├── all_strategies_results.csv # All 24 rows (6 strategies × 4 phases)
│ ├── english_to_hindi_to_hinglish_results.csv
│ ├── english_to_hinglish_to_hindi_results.csv
│ ├── hindi_to_english_to_hinglish_results.csv
│ ├── hindi_to_hinglish_to_english_results.csv
│ ├── hinglish_to_english_to_hindi_results.csv
│ └── hinglish_to_hindi_to_english_results.csv
│
└── figures/
├── language_distribution.png # Pie chart of dataset languages
│
├── english_to_hindi_to_hinglish/ # One folder per strategy
│ ├── *_[english]_curves.png # Train/Val accuracy + loss
│ ├── *_[english]_cm.png # Confusion matrix
│ ├── *_[english]_roc.png # ROC curve
│ ├── *_[english]_pr.png # Precision-Recall curve
│ ├── *_[english]_f1.png # F1 vs Threshold curve
│ ├── *_[hindi]_curves.png
│ ├── *_[hindi]_cm.png ...
│ ├── *_[hinglish]_curves.png
│ ├── *_[hinglish]_cm.png ...
│ ├── *_[Full]_curves.png
│ └── *_[Full]_cm.png ...
│
├── english_to_hinglish_to_hindi/
├── hindi_to_english_to_hinglish/
├── hindi_to_hinglish_to_english/
├── hinglish_to_english_to_hindi/
└── hinglish_to_hindi_to_english/
```
---
## How to Run
### Requirements
```bash
pip install tensorflow scikit-learn pandas seaborn matplotlib
```
You also need GloVe embeddings (`glove.6B.300d.txt`) placed at `/root/glove.6B.300d.txt`:
```bash
wget http://nlp.stanford.edu/data/glove.6B.zip && unzip glove.6B.zip
```
### Run
```bash
python main.py
```
Training was performed on an NVIDIA H200 GPU (Vast.ai) — total runtime approximately 15–20 minutes for all 6 strategies.
---
## Project Structure
```
SASC/
├── main.py # Full training + evaluation pipeline
├── dataset.csv # Raw dataset (29,505 samples)
├── README.md # This file
└── output/ # All results, figures, and model checkpoints
```
|