inagakimugi/qwen3-4b-dpo-v4

This repository provides a LoRA adapter fine-tuned from inagakimugi/qwen3-4b-sft-merged-v4 using DPO (Direct Preference Optimization) and QLoRA.

This repository contains LoRA adapter weights only. The base model must be loaded separately.

Training Objective

This adapter is trained to align the model with human preferences, specifically focusing on structured output accuracy and reasoning quality.

Training Configuration

• Base model: inagakimugi/qwen3-4b-sft-merged-v4
• Method: DPO + QLoRA
• Max sequence length: 2048
• Epochs: 3
• Learning rate: 5e-07
• DPO Beta: 0.4
• LoRA: r=64, alpha=32

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base = "inagakimugi/qwen3-4b-sft-merged-v4"
adapter = "inagakimugi/qwen3-4b-dpo-v4"

tokenizer = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(
    base,
    torch_dtype=torch.float16,
    device_map="auto",
)
model = PeftModel.from_pretrained(model, adapter)


Sources & Terms (IMPORTANT)

Training data: u-10bei/dpo-dataset-qwen-cot

Dataset License: MIT License. This dataset is used and distributed under the terms of the MIT License.
Compliance: Users must comply with the MIT license (including copyright notice) and the base model's original terms of use.