File size: 3,836 Bytes
36b18c2 a7ef4a8 36b18c2 a7ef4a8 36b18c2 a7ef4a8 36b18c2 a7ef4a8 36b18c2 a7ef4a8 36b18c2 a7ef4a8 36b18c2 a7ef4a8 36b18c2 a7ef4a8 36b18c2 a7ef4a8 36b18c2 a7ef4a8 36b18c2 a7ef4a8 36b18c2 a7ef4a8 36b18c2 a7ef4a8 36b18c2 a7ef4a8 36b18c2 a7ef4a8 36b18c2 a7ef4a8 36b18c2 a7ef4a8 36b18c2 a7ef4a8 36b18c2 a7ef4a8 36b18c2 a7ef4a8 36b18c2 a7ef4a8 36b18c2 a7ef4a8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 | ---
language:
- en
license: apache-2.0
base_model: distilbert/distilbert-base-uncased
tags:
- text-classification
- phishing-detection
- knowledge-distillation
- distilbert
datasets:
- Akash-Sakala/phishing-site-classification
metrics:
- accuracy
- f1
- precision
- recall
model-index:
- name: bert-phishing-classifier_student
results:
- task:
type: text-classification
name: Text Classification
dataset:
name: Akash-Sakala/phishing-site-classification
type: Akash-Sakala/phishing-site-classification
split: test
metrics:
- type: accuracy
value: 0.9601
name: Accuracy
- type: f1
value: 0.9595
name: F1
- type: precision
value: 0.9710
name: Precision
- type: recall
value: 0.9483
name: Recall
---
# DistilBERT Phishing Site Classifier (Student)
A 4-layer DistilBERT trained via **knowledge distillation** from a fine-tuned BERT teacher
([Akash-Sakala/bert-phishing-classifier_teacher](https://huggingface.co/Akash-Sakala/bert-phishing-classifier_teacher))
for binary phishing site URL classification.
## Model Details
| Property | Value |
|-----------------|-------------------------------------------|
| Base model | distilbert/distilbert-base-uncased |
| Architecture | DistilBertForSequenceClassification |
| Layers | 4 (distilled from 12-layer BERT teacher) |
| Attention heads | 8 |
| Task | Binary classification (phishing / benign) |
| Parameters | ~52M |
## Training — Distillation Setup
| Hyperparameter | Value |
|-------------------|--------------------|
| Temperature | 3.0 |
| Alpha (KL weight) | 0.6 |
| Hard label weight | 0.4 |
| Learning rate | 2e-5 |
| Batch size | 64 |
| Epochs | 4 |
| Warmup steps | 10% of total steps |
| Weight decay | 0.01 |
| Optimizer | AdamW |
| Scheduler | Linear with warmup |
| Mixed precision | fp16 (torch.amp) |
## Loss Function
Combined KL divergence (soft targets) + Cross-Entropy (hard labels):
loss = alpha * KL(student_soft || teacher_soft) * T^2 + (1 - alpha) * CrossEntropy(student, labels)
## Test Set Results
| Model | Accuracy | Precision | Recall | F1 |
|------------------------|------------|------------|------------|------------|
| BERT Teacher | 0.8971 | 0.9136 | 0.8763 | 0.8945 |
| **DistilBERT Student** | **0.9601** | **0.9710** | **0.9483** | **0.9595** |
The student **outperforms the teacher** across all metrics while being smaller and faster.
## Dataset
- **Dataset:** [Akash-Sakala/phishing-site-classification](https://huggingface.co/datasets/Akash-Sakala/phishing-site-classification)
- Train: 154,000 | Validation: 33,000 | Test: 33,000
- Labels: `0` = benign, `1` = phishing
## Usage
```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
tokenizer = AutoTokenizer.from_pretrained('Akash-Sakala/bert-phishing-classifier_student')
model = AutoModelForSequenceClassification.from_pretrained('Akash-Sakala/bert-phishing-classifier_student')
url = 'http://suspicious-login.verify-account.com/secure'
inputs = tokenizer(url, return_tensors='pt', truncation=True, padding='max_length')
with torch.no_grad():
logits = model(**inputs).logits
pred = torch.argmax(logits, dim=1).item()
print('Phishing' if pred == 1 else 'Benign')
```
## Teacher Model
[Akash-Sakala/bert-phishing-classifier_teacher](https://huggingface.co/Akash-Sakala/bert-phishing-classifier_teacher) |