---
language:
- en
tags:
- qwen
- qwen3.5
- furry
- roleplay
- split-mlp
- slerp
- nf4
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
base_model: Qwen/Qwen3.5-9B
model-index:
- name: Qwen3.5-9B-Furry-RP-Preview
  results: []
---

# Qwen3.5-9B-Furry-RP-Preview

A **split-MLP slerp merge** — not a LoRA adapter, not a full fine-tune, but something in between.

This model was created by training a **4B Qwen3.5** with split-MLP architecture on furry roleplay data, then **migrating the learned FFN patterns to the 9B base via spherical linear interpolation (slerp)**, followed by a brief NF4+LoRA adaptation to smooth the transition.

**Total training compute: ~2.5 hours on a single RTX 4090.**

## Why this is different from a normal LoRA

| | Normal LoRA | This model |
|---|---|---|
| **What changes** | Low-rank adapters added to existing weights | The FFN intermediate dimension is *expanded* via slerp of trained 4B weights |
| **Architecture** | Original architecture preserved | `intermediate_size` grows from 12288 → 12416 (128 slerp dims per layer) |
| **Knowledge injection** | Rank-limited bottleneck (e.g. rank 16) | Full intermediate dimension weights are updated |
| **Inference** | Requires loading base + LoRA adapter | Single standalone model file |
| **Serving** | LoRA stacking/merging complexity | Drop-in replacement for original 9B |
| **Merge difficulty** | Naive merge can degrade base quality | Slerp preserves direction of both vectors |

## How split-MLP slerp migration works

Instead of traditional fine-tuning (which modifies the entire model) or LoRA (which adds tiny bypass matrices), this technique:

1. **Teach a smaller model first** — Train a 4B Qwen with an expanded intermediate dimension (9216 → 9344) on the target domain. The FFN layers learn new roleplay patterns.

2. **Extract and slerp** — Take the trained `down_proj` weights from the 4B and use spherical linear interpolation to insert them into the 9B's wider FFN. The 128 new dimensions per layer are interpolated between the teacher's trained weights and random initialization.

3. **Brief NF4+LoRA polish** — A short 500-step adaptation in NF4+LoRA mode to settle the interpolated dimensions into the existing representation space. Only 25M trainable params vs the full 9B.

**Result:** The model gains domain-specific knowledge with dramatically less compute than full fine-tuning (~2.5 hours vs ~30+ hours on single 4090), while avoiding the rank bottleneck that limits LoRA adapters.

## Training details

| Parameter | 4B Training | 9B Migration |
|-----------|-------------|--------------|
| Base model | Qwen/Qwen3.5-4B | Qwen/Qwen3.5-9B |
| Steps | 9,000 | 500 |
| Loss | 1.69 (eval) | ~2.0 |
| Precision | bf16 | NF4 (base) + bf16 (compute) |
| Data | 43.6 MB furry RP text | Same |
| Hardware | 1× RTX 4090 (24 GB) | Same |
| Training time | ~2h 22m | ~2 min |

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("h34v7/Qwen3.5-9B-Furry-RP-Preview")
tokenizer = AutoTokenizer.from_pretrained("h34v7/Qwen3.5-9B-Furry-RP-Preview")

# For GPU inference with 24GB VRAM, load in 4-bit:
# model = AutoModelForCausalLM.from_pretrained("h34v7/Qwen3.5-9B-Furry-RP-Preview", load_in_4bit=True)
```

The model uses Qwen3.5's chat template with `<|im_start|>`, `<|im_end|>`, and `<think>` tags for reasoning.

## Available formats

- **HF (NF4)**: 7.2 GB — for transformers inference — [on this repo]
- **GGUF Q4_K_M**: 5.67 GB — for llama.cpp, koboldcpp, oobabooga — [separate repo](https://huggingface.co/h34v7/Qwen3.5-9B-Furry-RP-Preview-Q4_K_M)

## Limitations

- Preview release — 500 quick-adapt steps is minimal
- Trained on single-source RP data (Furry-RP corpus)
- The base Qwen3.5-9B model has not been abliterated. It retains its original alignment, so it may refuse or water down NSFW roleplay and other restricted content. This is a preview of the split-MLP migration technique, not a behaviorally untrained model.
- FFN down_proj fallback to q5_0/q8_0 in GGUF (12416 dims non-divisible by 256)

## Converting to bf16/fp16

This repo stores weights in **NF4** (4-bit) format to save space. To get standard bf16 weights for full-precision inference or fine-tuning:

```bash
python dequantize_to_bf16.py --input ./ --output ./model-bf16 --dtype bf16
```

The `dequantize_to_bf16.py` script is included in this repo. Requirements: `torch`, `transformers`, `bitsandbytes`, `safetensors`.

## What split-MLP is not

This is **not** an abliterated, uncensored, or alignment-removed model. The training data contains furry roleplay, but the base model's refusal patterns remain intact. If you need an unrestricted model, consider combining this technique with an abliterated base or applying additional post-processing.