maryleya/qwen3-4b-sft-cpo

Qwen3-4B-Instruct — SFT + CPO LoRA adapter

Contrastive Preference Optimization (CPO) on top of the SFT LoRA adapter for Qwen/Qwen3-4B-Instruct-2507. Anonymous submission to the WMT26 research track.

Training

• SFT stage: 2 epochs, learning rate 1e-4, glossary in prompt (same adapter as maryleya/qwen3-4b-sft)
• CPO stage: 1 epoch, learning rate 5e-6, β = 0.1, over 10,000 (source, chosen, rejected) triplets. Rejected candidates are generated by the base model in zero-shot, no-glossary mode, targeting its own failure modes.
• LoRA rank 64, α = 128, dropout 0.05
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained('Qwen/Qwen3-4B-Instruct-2507')
model = PeftModel.from_pretrained(base, 'maryleya/qwen3-4b-sft-cpo')
tok = AutoTokenizer.from_pretrained('Qwen/Qwen3-4B-Instruct-2507')

Code

Inference pipeline, KB, test sets, and evaluation scripts: https://github.com/Maryleya/RAG_System_for_Specialized_Terms