---
language: en
tags:
- character-training
- open-character-training
- loving
- lora
- merged-adapter
base_model: Qwen/Qwen2.5-7B-Instruct
library_name: peft
---

# qwen-2.5-7b-it-loving-merged

Combined character-training LoRA for Qwen2.5-7B-Instruct on the `loving` constitution,
produced by linearly blending the DPO distillation adapter and the SFT introspection
adapter — matching the canonical [Open Character Training](https://arxiv.org/abs/2511.01689)
publish recipe in [`tools/merge_loras.py`](https://github.com/sdananya/OpenCharacterTraining/blob/main/tools/merge_loras.py).

```
W_persona = 1.0 · W_DPO + 0.25 · W_SFT
```

The result is a single rank-64 LoRA whose reference frame is the **vanilla base model**
(`Qwen/Qwen2.5-7B-Instruct`), so it can be loaded directly with PEFT against the
upstream HF base — no folding step required.

## Why a merged adapter?

The SFT introspection LoRA in [`sdananya/qwen-2.5-7b-it-loving`](https://huggingface.co/sdananya/qwen-2.5-7b-it-loving)
was trained on top of the **DPO-folded** base (`base + DPO_LoRA` merged), so its
weight delta only makes sense in that reference frame. Loading the standalone SFT
adapter against vanilla `Qwen2.5-7B-Instruct` applies the delta in the wrong frame
and produces degraded outputs.

This repo packages the correct deployable artifact: a single LoRA that *does* live
in the vanilla-base frame, by combining DPO (full strength) with a 0.25-weighted
SFT introspection contribution.

## Usage

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct", torch_dtype=torch.bfloat16, device_map="auto"
)
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
model = PeftModel.from_pretrained(base, "sdananya/qwen-2.5-7b-it-loving-merged")
```

## Training pipeline (paper-equivalent)

1. **DPO distillation** on the `loving` DPO dataset → rank-64 LoRA over base.
2. **Fold** the DPO LoRA into base → distilled checkpoint.
3. **SFT introspection** on self-reflection + self-interaction data → rank-64 LoRA over the folded base.
4. **Merge** with weights `[1.0, 0.25]` against vanilla base → this adapter.

Hyperparameters: rank=64, alpha=128, lr=5e-5, warmup=0.1, max_epochs=1, batch=32,
DeepSpeed ZeRO-2, bf16. See [the OCT repo](https://github.com/sdananya/OpenCharacterTraining)
for full configs.

## Related artifacts in this org

- [`sdananya/qwen-2.5-7b-it-loving`](https://huggingface.co/sdananya/qwen-2.5-7b-it-loving) — separate `dpo-final/` and `introspection-final/` folders. The DPO adapter is correctly framed; the introspection adapter alone is **not** (see above).
- [`sdananya/qwen-2.5-7b-it-loving-no-dpo`](https://huggingface.co/sdananya/qwen-2.5-7b-it-loving-no-dpo) — ablation: SFT introspection trained directly on vanilla base, no DPO.
- [`sdananya/qwen-2.5-7b-it-loving-dpo200`](https://huggingface.co/sdananya/qwen-2.5-7b-it-loving-dpo200) — ablation: SFT introspection seeded from a partial (200-step) DPO checkpoint.

## Citation

Based on [Open Character Training](https://arxiv.org/abs/2511.01689).