---
license: cc-by-nc-sa-4.0
base_model: Qwen/Qwen2.5-7B-Instruct
library_name: peft
tags:
  - lora
  - peft
  - character-training
  - open-character-training
  - qwen
---

# qwen-2.5-7b-it-loving-merged

Reproduction of the **loving** persona from
[Open Character Training](https://arxiv.org/abs/2511.01689)
(Maiya et al., 2025), trained on top of `Qwen/Qwen2.5-7B-Instruct` using the
public [maiush/OpenCharacterTraining](https://github.com/maiush/OpenCharacterTraining)
pipeline and dataset.

This repository contains **correctly-merged** LoRA adapters at every published
checkpoint. The adapters in the original `sdananya/qwen-2.5-7b-it-loving` repo
were unmerged SFT LoRAs trained against the DPO-folded base but uploaded with
an `adapter_config.json` claiming the vanilla base — silently degrading
inference. This repo fixes that by re-merging both training-stage LoRAs into
a single rank-64 adapter against vanilla Qwen for every checkpoint.

## Repository layout

```
sdananya/qwen-2.5-7b-it-loving-merged/
├── dpo-final/                  ← DPO LoRA (rank-64, against vanilla Qwen)
├── dpo-step125/ ... dpo-step250/   ← 6 intermediate DPO checkpoints
├── introspection-final/        ← merged final: 1.0·ΔW_DPO_final + 0.25·ΔW_SFT_final
└── introspection-step225/ ... introspection-step350/
                                ← 6 merged intermediate SFT checkpoints,
                                  each = 1.0·ΔW_DPO_final + 0.25·ΔW_SFT_stepN
```

All adapters are rank-64 LoRAs targeting `q_proj, k_proj, v_proj, o_proj,
gate_proj, up_proj, down_proj` across all 28 transformer layers.

## Loading

You must specify the checkpoint as a `subfolder`:

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct", torch_dtype="bfloat16"
)
model = PeftModel.from_pretrained(
    base,
    "sdananya/qwen-2.5-7b-it-loving-merged",
    subfolder="introspection-final",   # or any other checkpoint above
)
tok = AutoTokenizer.from_pretrained(
    "sdananya/qwen-2.5-7b-it-loving-merged",
    subfolder="introspection-final",
)
```

## Merge recipe (introspection-* checkpoints)

Each `introspection-*` folder is produced by PEFT's
`add_weighted_adapter(combination_type="linear")` applied to the DPO-final
adapter and the corresponding SFT checkpoint:

$$\Delta W_{\text{merged}} = 1.0 \cdot \Delta W_{\text{DPO-final}} + 0.25 \cdot \Delta W_{\text{SFT-stepN}}$$

The weights `[1.0, 0.25]` are taken directly from the canonical pipeline in
[`tools/merge_loras.py`](https://github.com/sdananya/OpenCharacterTraining/blob/main/tools/merge_loras.py).
The 0.25 dampening on SFT is the paper's chosen recipe — it preserves the
introspective character while preventing the SFT register from overpowering
normal conversation.

## Why merging matters

The SFT introspection LoRA is trained on top of the **DPO-folded** base
(`W_base + ΔW_DPO`), so its low-rank factors are only meaningful in that
reference frame. Loading it directly onto vanilla `W_base` (as the original
unmerged upload did) produces a meaningful-shape but semantically wrong delta —
PEFT does not detect the mismatch and silently produces degraded output.

`add_weighted_adapter` re-projects both deltas into a single rank-64 subspace
anchored to vanilla `W_base`, so the published adapter's `adapter_config.json`
honestly declares its base model and PEFT applies it correctly.

## DPO checkpoints

The `dpo-*` subfolders are byte-identical copies of the corresponding folders
in [`sdananya/qwen-2.5-7b-it-loving`](https://huggingface.co/sdananya/qwen-2.5-7b-it-loving).
DPO is trained directly on vanilla Qwen, so its reference frame is already
correct — no merge needed.

## Citation

```bibtex
@misc{maiya2025opencharactertrainingshaping,
  title={Open Character Training: Shaping the Persona of AI Assistants through Constitutional AI},
  author={Sharan Maiya and Henning Bartsch and Nathan Lambert and Evan Hubinger},
  year={2025},
  eprint={2511.01689},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2511.01689},
}
```