qwen-2.5-7b-it-loving-merged

Reproduction of the loving persona from Open Character Training (Maiya et al., 2025), trained on top of Qwen/Qwen2.5-7B-Instruct using the public maiush/OpenCharacterTraining pipeline and dataset.

This repository contains correctly-merged LoRA adapters at every published checkpoint. The adapters in the original sdananya/qwen-2.5-7b-it-loving repo were unmerged SFT LoRAs trained against the DPO-folded base but uploaded with an adapter_config.json claiming the vanilla base — silently degrading inference. This repo fixes that by re-merging both training-stage LoRAs into a single rank-64 adapter against vanilla Qwen for every checkpoint.

Repository layout

sdananya/qwen-2.5-7b-it-loving-merged/
├── dpo-final/                  ← DPO LoRA (rank-64, against vanilla Qwen)
├── dpo-step125/ ... dpo-step250/   ← 6 intermediate DPO checkpoints
├── introspection-final/        ← merged final: 1.0·ΔW_DPO_final + 0.25·ΔW_SFT_final
└── introspection-step225/ ... introspection-step350/
                                ← 6 merged intermediate SFT checkpoints,
                                  each = 1.0·ΔW_DPO_final + 0.25·ΔW_SFT_stepN

All adapters are rank-64 LoRAs targeting q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj across all 28 transformer layers.

Loading

You must specify the checkpoint as a subfolder:

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct", torch_dtype="bfloat16"
)
model = PeftModel.from_pretrained(
    base,
    "sdananya/qwen-2.5-7b-it-loving-merged",
    subfolder="introspection-final",   # or any other checkpoint above
)
tok = AutoTokenizer.from_pretrained(
    "sdananya/qwen-2.5-7b-it-loving-merged",
    subfolder="introspection-final",
)

Merge recipe (introspection-* checkpoints)

Each introspection-* folder is produced by PEFT's add_weighted_adapter(combination_type="linear") applied to the DPO-final adapter and the corresponding SFT checkpoint:

$\Delta W_{\text{merged}} = 1.0 \cdot \Delta W_{\text{DPO-final}} + 0.25 \cdot \Delta W_{\text{SFT-stepN}}$

The weights [1.0, 0.25] are taken directly from the canonical pipeline in tools/merge_loras.py. The 0.25 dampening on SFT is the paper's chosen recipe — it preserves the introspective character while preventing the SFT register from overpowering normal conversation.

Why merging matters

The SFT introspection LoRA is trained on top of the DPO-folded base (W_base + ΔW_DPO), so its low-rank factors are only meaningful in that reference frame. Loading it directly onto vanilla W_base (as the original unmerged upload did) produces a meaningful-shape but semantically wrong delta — PEFT does not detect the mismatch and silently produces degraded output.

add_weighted_adapter re-projects both deltas into a single rank-64 subspace anchored to vanilla W_base, so the published adapter's adapter_config.json honestly declares its base model and PEFT applies it correctly.

DPO checkpoints

The dpo-* subfolders are byte-identical copies of the corresponding folders in sdananya/qwen-2.5-7b-it-loving. DPO is trained directly on vanilla Qwen, so its reference frame is already correct — no merge needed.

Citation

@misc{maiya2025opencharactertrainingshaping,
  title={Open Character Training: Shaping the Persona of AI Assistants through Constitutional AI},
  author={Sharan Maiya and Henning Bartsch and Nathan Lambert and Evan Hubinger},
  year={2025},
  eprint={2511.01689},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2511.01689},
}