qwen3.5-4b-curriculum — safetensors + LoRA adapters

Weights for a small curriculum / data-corruption experiment on unsloth/Qwen3.5-4B: a single LoRA adapter trained sequentially on four increasingly-scrambled versions of the same ~250-conversation corpus.

stage	data	idea
s1	conversations	the clean multi-turn chat corpus
s2	+ shuffled	turn contents scrambled (roles still alternate)
s3	+ role-reversed	human/assistant labels swapped — model learns to produce the human turns
s4	+ role + temporal swap	role-swapped and answer-before-question ordering
s5	back to clean conversations	does re-training on clean data undo s2–s4, or is it sticky?

Each stage continues the previous stage's adapter (LoRA r=16, α=16, bf16, batch 2×4, 1 epoch, train_on_responses_only). Vision tower untouched.

⚠️ These are research artifacts, not a general assistant. Core skills (facts, math, code, translation) survive, but s2 loses context-tracking and s3/s4 increasingly refuse to act as the assistant, get terse, and open turns with "Hey Claude". See the eval below.

Contents

merged-5stage/      bf16 safetensors — s1→…→s4→s5 (the "returned to clean" endpoint)
merged-4stage/      bf16 safetensors — base + s1→s2→s3→s4 merged   (Qwen3_5ForConditionalGeneration)
merged-3stage/      bf16 safetensors — base + s1→s2→s3 merged
adapter-s1_conversations/         LoRA adapter after stage 1
adapter-s2_shuffled/              LoRA adapter after stages 1→2
adapter-s3_role_swap/             LoRA adapter after stages 1→3
adapter-s4_role_temporal_swap/    LoRA adapter after stages 1→4
adapter-s5_reclean/               LoRA adapter after stages 1→5

GGUF builds (f16 / q8_0 / q4_k_m): 4fhct4sd/qwen3.5-4b-curriculum-gguf

Perplexity (20 held-in convos/set, greedy)

model	clean	shuffled	role_swap	temporal
base	13.0	43.1	19.9	17.5
s1	8.5	15.5	11.2	10.4
s2	8.8	13.1	11.4	9.7
s3	9.1	15.9	11.6	10.5
s4	8.3	14.0	11.0	9.0
s5	7.5	13.4	11.1	9.0

Stage 1 (style adaptation) does almost all of the quantitative work; stages 2–4 move PPL by less than the run-to-run training noise. temporal PPL tracks clean within ~1 point at every stage — the answer-before-question swap keeps every adjacent turn a real, on-topic exchange, so it barely registers as "corruption" to the token predictor. shuffled (no local coherence) is the only one that genuinely spikes.

Is the degradation sticky? (s5 = one epoch back on clean data)

No — s5 reverts to s1's behaviour. The role-deflection s3/s4 acquired ("I don't know who you are talking about", "talk to someone who knows you better", opening with "Hey Claude") is gone in s5; multi-turn recall and verbosity recover; "who are you" gets a normal assistant answer again. A 32 M-param adapter at ~30 steps/stage doesn't accumulate a curriculum — the most recent dataset dominates the behavioural surface.

The one thing that does track training history: quirks return with their data. The [gap: seconds] filler token (from a degenerate record in the clean corpus) vanishes at s2 and comes straight back at s5 — s5 actually over-acquires it and opens many answers with it. Clean PPL also over-recovers (8.5 → 7.5): five passes over the corpus beat one.

Continued fine-tuning

From a merged checkpoint (attach a fresh adapter):

from unsloth import FastModel
model, tok = FastModel.from_pretrained("4fhct4sd/qwen3.5-4b-curriculum",
    revision="main")  # or a local snapshot of merged-4stage/
# ... FastModel.get_peft_model(...) + SFTTrainer as usual

(point model_name at the merged-4stage / merged-3stage subfolder)

Resume an adapter (continue the curriculum):

model, tok = FastModel.from_pretrained("adapter-s4_role_temporal_swap")  # loads base + adapter
FastModel.for_training(model)
# ... SFTTrainer on the next dataset

Base model & license

Fine-tune of unsloth/Qwen3.5-4B; use is governed by that model's license. Adapters/merges released for research use.