GPT-2 LoRA Adapter — Dialogue Summarization (SAMSum)
Fine-tuned LoRA adapter for dialogue summarization on the SAMSum dataset. Base model: openai-community/gpt2
LoRA Configuration
| Parameter | Value |
|---|---|
Rank (r) |
16 |
| Alpha | 32 |
| Scale (alpha/r) | 2.0 |
| Dropout | 0.05 |
| Target modules | c_attn, c_proj |
Training Hyperparameters
| Parameter | Value |
|---|---|
| Max epochs | 10 |
| Early stopping patience | 2 |
| Effective batch size | 16 |
| Learning rate | 0.0003 |
| LR scheduler | cosine |
| Warmup steps | 100 |
| Max sequence length | 128 tokens |
| Training time | ~1h56m |
Evaluation Results
Training stopped early at epoch 7 (patience=2). Best model saved at epoch 5.
| Epoch | Eval Loss |
|---|---|
| 1 | 1.678 |
| 2 | 1.619 |
| 3 | 1.592 |
| 4 | 1.574 |
| 5 | 1.570 <- best |
| 6 | 1.571 |
| 7 | 1.572 |
Limitations
- GPT-2 is a small, older model — summaries may be repetitive or imprecise
- Trained on English dialogues only
- Max context length of 128 tokens; longer dialogues are truncated
- Downloads last month
- 6
Model tree for kelly-n/gpt2-samsum-lora
Base model
openai-community/gpt2
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("openai-community/gpt2") model = PeftModel.from_pretrained(base_model, "kelly-n/gpt2-samsum-lora")