---
language:
- en
license: apache-2.0
base_model: distilbert/distilbert-base-uncased
tags:
- text-classification
- phishing-detection
- knowledge-distillation
- distilbert
datasets:
- Akash-Sakala/phishing-site-classification
metrics:
- accuracy
- f1
- precision
- recall
model-index:
- name: bert-phishing-classifier_student
  results:
  - task:
      type: text-classification
      name: Text Classification
    dataset:
      name: Akash-Sakala/phishing-site-classification
      type: Akash-Sakala/phishing-site-classification
      split: test
    metrics:
    - type: accuracy
      value: 0.9601
      name: Accuracy
    - type: f1
      value: 0.9595
      name: F1
    - type: precision
      value: 0.9710
      name: Precision
    - type: recall
      value: 0.9483
      name: Recall
---

# DistilBERT Phishing Site Classifier (Student)

A 4-layer DistilBERT trained via **knowledge distillation** from a fine-tuned BERT teacher
([Akash-Sakala/bert-phishing-classifier_teacher](https://huggingface.co/Akash-Sakala/bert-phishing-classifier_teacher))
for binary phishing site URL classification.

## Model Details

| Property        | Value                                     |
|-----------------|-------------------------------------------|
| Base model      | distilbert/distilbert-base-uncased        |
| Architecture    | DistilBertForSequenceClassification       |
| Layers          | 4 (distilled from 12-layer BERT teacher)  |
| Attention heads | 8                                         |
| Task            | Binary classification (phishing / benign) |
| Parameters      | ~52M                                      |

## Training — Distillation Setup

| Hyperparameter    | Value              |
|-------------------|--------------------|
| Temperature       | 3.0                |
| Alpha (KL weight) | 0.6                |
| Hard label weight | 0.4                |
| Learning rate     | 2e-5               |
| Batch size        | 64                 |
| Epochs            | 4                  |
| Warmup steps      | 10% of total steps |
| Weight decay      | 0.01               |
| Optimizer         | AdamW              |
| Scheduler         | Linear with warmup |
| Mixed precision   | fp16 (torch.amp)   |

## Loss Function

Combined KL divergence (soft targets) + Cross-Entropy (hard labels):

    loss = alpha * KL(student_soft || teacher_soft) * T^2 + (1 - alpha) * CrossEntropy(student, labels)

## Test Set Results

| Model                  | Accuracy   | Precision  | Recall     | F1         |
|------------------------|------------|------------|------------|------------|
| BERT Teacher           | 0.8971     | 0.9136     | 0.8763     | 0.8945     |
| **DistilBERT Student** | **0.9601** | **0.9710** | **0.9483** | **0.9595** |

The student **outperforms the teacher** across all metrics while being smaller and faster.

## Dataset

- **Dataset:** [Akash-Sakala/phishing-site-classification](https://huggingface.co/datasets/Akash-Sakala/phishing-site-classification)
- Train: 154,000 | Validation: 33,000 | Test: 33,000
- Labels: `0` = benign, `1` = phishing

## Usage

```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

tokenizer = AutoTokenizer.from_pretrained('Akash-Sakala/bert-phishing-classifier_student')
model = AutoModelForSequenceClassification.from_pretrained('Akash-Sakala/bert-phishing-classifier_student')

url = 'http://suspicious-login.verify-account.com/secure'
inputs = tokenizer(url, return_tensors='pt', truncation=True, padding='max_length')

with torch.no_grad():
    logits = model(**inputs).logits

pred = torch.argmax(logits, dim=1).item()
print('Phishing' if pred == 1 else 'Benign')
```

## Teacher Model

[Akash-Sakala/bert-phishing-classifier_teacher](https://huggingface.co/Akash-Sakala/bert-phishing-classifier_teacher)