---
license: apache-2.0
base_model: llm-semantic-router/mmbert-32k-yarn
tags:
- text-classification
- intent-classification
- modernbert
- lora
- peft
- mmlu-pro
datasets:
- TIGER-Lab/MMLU-Pro
- LLM-Semantic-Router/category-classifier-supplement
language:
- en
- multilingual
metrics:
- accuracy
- f1
pipeline_tag: text-classification
---

# mmBERT-32K Intent Classifier (LoRA Adapter)

LoRA adapter for intent classification based on mmBERT-32K-YaRN (32K context, multilingual).

## Model Details

- **Base Model**: [llm-semantic-router/mmbert-32k-yarn](https://huggingface.co/llm-semantic-router/mmbert-32k-yarn)
- **Training Method**: LoRA (Low-Rank Adaptation)
- **LoRA Rank**: 32
- **LoRA Alpha**: 64
- **Trainable Parameters**: 6.8M (2.2% of base model)
- **Adapter Size**: 27 MB

## Training Data

- **Primary**: [TIGER-Lab/MMLU-Pro](https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro) (~12K academic questions)
- **Supplement**: [LLM-Semantic-Router/category-classifier-supplement](https://huggingface.co/datasets/LLM-Semantic-Router/category-classifier-supplement) (653 samples including casual "other" examples)

## Categories (14 classes)

biology, business, chemistry, computer science, economics, engineering, health, history, law, math, other, philosophy, physics, psychology

## Performance

| Metric | Score |
|--------|-------|
| Test Accuracy | 80.0% |
| Adapter Size | 27 MB |

## Usage

```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
from peft import PeftModel

# Load base model and LoRA adapter
base_model = AutoModelForSequenceClassification.from_pretrained(
    "llm-semantic-router/mmbert-32k-yarn", num_labels=14
)
model = PeftModel.from_pretrained(base_model, "llm-semantic-router/mmbert32k-intent-classifier-lora")
tokenizer = AutoTokenizer.from_pretrained("llm-semantic-router/mmbert32k-intent-classifier-lora")

# Inference
inputs = tokenizer("How do neural networks learn?", return_tensors="pt")
outputs = model(**inputs)
predicted_class = outputs.logits.argmax().item()
```

## Training Configuration

- Epochs: 5
- Batch Size: 16
- Learning Rate: 2e-4
- Weight Decay: 0.1
- Optimizer: AdamW with cosine LR scheduler