lab22-dpo-vn

LoRA adapter for unsloth/Qwen2.5-3B-bnb-4bit, trained with SFT → DPO for the VinUni AICB Track 3 Day 22 (DPO/ORPO Alignment) lab.

This is a single self-contained adapter: DPO training (below) continued fine-tuning the SFT LoRA weights in place (warm start) rather than stacking a second adapter on top, so this checkpoint already includes both the SFT and the DPO effect. Load it directly on top of the base model — no separate SFT adapter is needed.

Model Details

• Base model: unsloth/Qwen2.5-3B-bnb-4bit (4-bit NF4 quantized Qwen2.5-3B)
• Adapter type: LoRA, r=16, alpha=32, dropout=0.0, target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Language(s): Vietnamese (SFT data), English (preference data)
• License: apache-2.0 (matches base model)
• Finetuned from model: unsloth/Qwen2.5-3B-bnb-4bit

Training Data

Stage	Dataset	Size
SFT	bkai-foundation-models/vi-alpaca	1,000 samples, 1 epoch
DPO	argilla/ultrafeedback-binarized-preferences-cleaned	5,000 preference pairs (train) + 50 (eval), 1 epoch

Training Hyperparameters

• DPO beta: 0.1
• Learning rate: 5e-7
• Epochs: 1
• Loss type: sigmoid (standard DPO)
• Reference model: derived automatically from the PEFT base (no separate copy — TRL ≥0.12 disables the adapter to get the reference logits)
• Compute: 1x NVIDIA RTX 4080 (16GB), bf16
• Training regime: 4-bit NF4 base + bf16 LoRA compute

Evaluation

DPO training metrics (end of training)

Metric	Value
Final training loss	0.7696
Chosen reward (log π/π_ref)	-0.682
Rejected reward (log π/π_ref)	-0.831
Reward gap (chosen − rejected)	+0.149

Both chosen and rejected reward stay below 0 throughout training (noisy, oscillating in the -0.6 to -1.0 range over ~620 steps) — rejected drops slightly more than chosen, giving a small positive but noisy gap rather than a clean, stable separation. Consistent with a short run (5k pairs, 1 epoch, lr=5e-7) rather than a fully converged DPO run.

LLM-judge comparison (gpt-4o-mini, SFT-only vs SFT+DPO, 8 prompts)

Result	Count
SFT+DPO wins	4/8
SFT-only wins	2/8
Tie	2/8

Both models failed to refuse 2 out of 4 safety-probe prompts (tie = both unsafe) — DPO training on general helpfulness preference data did not by itself teach refusal behavior for those cases.

How to Get Started with the Model

from unsloth import FastLanguageModel
from unsloth.chat_templates import get_chat_template
from peft import PeftModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Qwen2.5-3B-bnb-4bit",
    max_seq_length=512,
    dtype=None,
    load_in_4bit=True,
)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token
tokenizer = get_chat_template(tokenizer, chat_template="qwen2.5")

model = PeftModel.from_pretrained(model, "minhpa/lab22-dpo-vn")
FastLanguageModel.for_inference(model)

messages = [{"role": "user", "content": "Giải thích ngắn gọn thuật toán quicksort."}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to("cuda")
out = model.generate(input_ids=inputs, max_new_tokens=200)
print(tokenizer.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

Note: the base repo ships without a chat_template — get_chat_template(tokenizer, chat_template="qwen2.5") must be called before apply_chat_template().

Bias, Risks, and Limitations

This is a small-scale educational lab artifact (3B model, 1 epoch, 5k preference pairs) — not a production-safety-tuned model. As shown above, it does not reliably refuse unsafe requests (2/4 safety probes failed for both SFT-only and SFT+DPO). Do not deploy as-is for any safety-sensitive use.

Environmental Impact

• Hardware: 1x NVIDIA RTX 4080 (16GB), local
• Cloud provider: none (local run)

Framework versions

• PEFT 0.20.0
• TRL (DPOTrainer)
• Unsloth 2026.4.8
• transformers 4.57.6

Citation

Lab: VinUni AICB Track 3, Day 22 — DPO/ORPO Alignment. Stack: Unsloth, TRL, PEFT, bitsandbytes.