README.md

metadata

language:
  - en
tags:
  - qwen
  - qwen3.5
  - furry
  - roleplay
  - split-mlp
  - slerp
  - nf4
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
base_model: Qwen/Qwen3.5-9B
model-index:
  - name: Qwen3.5-9B-Furry-RP-Preview
    results: []

Qwen3.5-9B-Furry-RP-Preview

A split-MLP slerp merge — not a LoRA adapter, not a full fine-tune, but something in between.

This model was created by training a 4B Qwen3.5 with split-MLP architecture on furry roleplay data, then migrating the learned FFN patterns to the 9B base via spherical linear interpolation (slerp), followed by a brief NF4+LoRA adaptation to smooth the transition.

Total training compute: ~2.5 hours on a single RTX 4090.

Why this is different from a normal LoRA

	Normal LoRA	This model
What changes	Low-rank adapters added to existing weights	The FFN intermediate dimension is expanded via slerp of trained 4B weights
Architecture	Original architecture preserved	intermediate_size grows from 12288 → 12416 (128 slerp dims per layer)
Knowledge injection	Rank-limited bottleneck (e.g. rank 16)	Full intermediate dimension weights are updated
Inference	Requires loading base + LoRA adapter	Single standalone model file
Serving	LoRA stacking/merging complexity	Drop-in replacement for original 9B
Merge difficulty	Naive merge can degrade base quality	Slerp preserves direction of both vectors

How split-MLP slerp migration works

Instead of traditional fine-tuning (which modifies the entire model) or LoRA (which adds tiny bypass matrices), this technique:

1. Teach a smaller model first — Train a 4B Qwen with an expanded intermediate dimension (9216 → 9344) on the target domain. The FFN layers learn new roleplay patterns.

2. Extract and slerp — Take the trained down_proj weights from the 4B and use spherical linear interpolation to insert them into the 9B's wider FFN. The 128 new dimensions per layer are interpolated between the teacher's trained weights and random initialization.

3. Brief NF4+LoRA polish — A short 500-step adaptation in NF4+LoRA mode to settle the interpolated dimensions into the existing representation space. Only 25M trainable params vs the full 9B.

Result: The model gains domain-specific knowledge with dramatically less compute than full fine-tuning (~2.5 hours vs ~30+ hours on single 4090), while avoiding the rank bottleneck that limits LoRA adapters.

Training details

Parameter	4B Training	9B Migration
Base model	Qwen/Qwen3.5-4B	Qwen/Qwen3.5-9B
Steps	9,000	500
Loss	1.69 (eval)	~2.0
Precision	bf16	NF4 (base) + bf16 (compute)
Data	43.6 MB furry RP text	Same
Hardware	1× RTX 4090 (24 GB)	Same
Training time	~2h 22m	~2 min

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("h34v7/Qwen3.5-9B-Furry-RP-Preview")
tokenizer = AutoTokenizer.from_pretrained("h34v7/Qwen3.5-9B-Furry-RP-Preview")

# For GPU inference with 24GB VRAM, load in 4-bit:
# model = AutoModelForCausalLM.from_pretrained("h34v7/Qwen3.5-9B-Furry-RP-Preview", load_in_4bit=True)

The model uses Qwen3.5's chat template with <|im_start|>, <|im_end|>, and <think> tags for reasoning.

Available formats

• HF (NF4): 7.2 GB — for transformers inference — [on this repo]
• GGUF Q4_K_M: 5.67 GB — for llama.cpp, koboldcpp, oobabooga — separate repo

Limitations

• Preview release — 500 quick-adapt steps is minimal
• Trained on single-source RP data (Furry-RP corpus)
• The base Qwen3.5-9B model has not been abliterated. It retains its original alignment, so it may refuse or water down NSFW roleplay and other restricted content. This is a preview of the split-MLP migration technique, not a behaviorally untrained model.
• FFN down_proj fallback to q5_0/q8_0 in GGUF (12416 dims non-divisible by 256)

Converting to bf16/fp16

This repo stores weights in NF4 (4-bit) format to save space. To get standard bf16 weights for full-precision inference or fine-tuning:

python dequantize_to_bf16.py --input ./ --output ./model-bf16 --dtype bf16

The dequantize_to_bf16.py script is included in this repo. Requirements: torch, transformers, bitsandbytes, safetensors.

What split-MLP is not

This is not an abliterated, uncensored, or alignment-removed model. The training data contains furry roleplay, but the base model's refusal patterns remain intact. If you need an unrestricted model, consider combining this technique with an abliterated base or applying additional post-processing.