kacperwikiel/slayer-style-qwen3.5-27b-ep3-GGUF

Slayer Style — Qwen3.5‑27B (ep3)

Polish‑style fine‑tune of Qwen/Qwen3.5‑27B. A LoRA trained to write natural, idiomatic Polish (no translationese, no em‑dash/półpauza overuse) while keeping the base model's knowledge. Quantized to Q4_K_M GGUF so it runs on a single 24 GB GPU.

Open weights end‑to‑end — not distilled from Anthropic/OpenAI. No Claude/GPT weights or outputs are used as training data (see How the training data was made).

📘 Full playbook (data → training → quantization → deployment, with pitfalls): slayer_playbook.pdf in this repo.

TL;DR — it beats Bielik. On the LLMzSzŁ Polish professional‑exam MCQ benchmark (answer log‑likelihood, n=400, seed=42, identical harness):

Model	Accuracy
slayer‑style‑qwen3.5‑27b (this model, fp16)	65.0 % 🏆
Qwen3.5‑27B (base)	58.5 %
Bielik‑11B‑v3.0‑Instruct	56.0 %
Qwen3.5‑9B (base)	50.7 %

+9.0 points over Bielik‑11B‑v3, +6.5 over the base Qwen it was tuned from.

---

Artifacts & lineage

Artifact	Repo / path
Base model	Qwen/Qwen3.5-27B (Apache‑2.0)
LoRA adapter	kacperwikiel/slayer-style-qwen3.5-27b-ep3 (private)
Quantized GGUF (this card)	kacperwikiel/slayer-style-qwen3.5-27b-ep3-GGUF (private) — slayer-style-qwen3.5-27b-Q4_K_M.gguf, 16.5 GB
Merged fp16 (intermediate, not published)	produced locally by bench/merge_adapter.py

The adapter is the source of truth; the GGUF is derived from base + adapter.

---

Training (style SFT, QLoRA)

Setting	Value
Method	QLoRA SFT (bench/sft_style_qlora.py)
Base	Qwen/Qwen3.5-27B
LoRA rank r	16
LoRA alpha	32
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj (all linear)
Task	CAUSAL_LM
Epochs	3 (ep3)
Steps	150 total (~50/epoch)
Train loss	1.34 → 0.93

How the ~1000 training examples were made

🛡️ Not distilled from Anthropic or OpenAI. No Claude/GPT weights or generated text are used as training data — none of the answers the model learns to imitate come from Anthropic or OpenAI models. The "chosen" answers come from DeepSeek‑V4‑pro, the raw/rejected side from Qwen3.5‑27B, and the production quality judge is the open‑weight Qwen3.5‑122B. Anthropic's opus‑4.8 was used only to validate that the open‑weight judge scores comparably (and scored an early data batch before the switch) — never as a source of any training text.

1. Prompts (bench/gen_style_prompts_v2.py) — a grid of 26 task types × 20 domains × 9 registers × 4 lengths, plus an 8‑type grammar stress‑test (liczebniki, wołacz, aspekt, ortografia ż/rz, frazeologizmy, rekcja…). No benchmark items.
2. Generation (bench/gen_style_sft.py) — per prompt: a raw answer from qwen3.5-27b (the "rejected") and a rewrite from teacher deepseek-v4-pro (the "chosen", natural PL).
3. Judge — independent open‑weight Qwen3.5-122B scores natural_polish_1‑5 and flags english_leak, dash_overuse, markdown, fact_preserved, fits_brief. (Validated once against opus‑4.8 to confirm the open judge annotates at comparable quality.)
4. Curate (LIMA) (bench/curate_style_data.py) — 1600 generated → 799 train + 160 holdout: near‑dup removed, grid‑balanced, score = judge naturalness + length fit − AI‑tell penalties. Holdout is never trained on. More ≠ better — curation beats volume.

Concrete before → after (same prompt, "what sport for a 50+ beginner?"):

• raw qwen3.5-27b (rejected): "Dla osoby powyżej 50. roku życia… spacery lub pływanie. Oto dlaczego te dyscypliny są optymalne: Spacery: Są najbezpieczniejsze…" — stiff, markdown, corporate.
• teacher deepseek-v4-pro (chosen): "Spacery i pływanie to dwa strzały w dziesiątkę… możesz wyjść z domu i po prostu iść, a intensywność regulujesz tempem i dystansem." — natural, idiomatic, no markdown.

Anti‑AI‑tells — explicit penalties on em‑dash / półpauza overuse, anglicisms, and translationese (bench/make_anglicyzm_report.py, bench/make_translationese_report.py).

---

Quantization pipeline (exactly how the GGUF was made)

Hardware: Dell GB10 (aarch64, Grace‑Blackwell sm_121, 121 GB unified RAM, CUDA 13). llama.cpp @ commit 98d5e8b (2026‑06‑06) — first to support Qwen3.5 (Qwen35MtpMixin).

# 1) Merge LoRA into the base in fp16 (CPU, ~54 GB RAM; one-time)
python bench/merge_adapter.py \
    --adapter kacperwikiel/slayer-style-qwen3.5-27b-ep3 \
    --out ./slayer-27b-merged              # -> ~51 GB merged fp16

# 2) HF -> GGUF f16
python llama.cpp/convert_hf_to_gguf.py ./slayer-27b-merged \
    --outfile slayer-27b-f16.gguf --outtype f16     # -> ~51 GB

# 3) f16 -> Q4_K_M
llama.cpp/build/bin/llama-quantize \
    slayer-27b-f16.gguf \
    slayer-style-qwen3.5-27b-Q4_K_M.gguf Q4_K_M      # -> 16.5 GB

Quant choice rationale (24 GB target): Q4_K_M ≈ 16 GB leaves 8 GB for KV cache — the sweet spot. Q5_K_M (19 GB) fits but squeezes context; Q6_K (~22 GB) is too tight.

---

Benchmark methodology

• Harness: bench/bench_llmzszl_likelihood.py (lm‑eval‑style answer log‑likelihood — each answer option scored as a continuation; highest summed log‑prob wins).
• Dataset: amu-cai/llmzszl-dataset:llmzszl-test.jsonl (Polish professional exams).
• Settings: n=400, seed=42, mode=answer, normalize=none.
• Purity: aggregate‑only; no benchmark item text is stored or inspected (no benchmaxxing).

# reproduce (fp16 merged model, GPU):
python bench/bench_llmzszl_likelihood.py --model ./slayer-27b-merged --n 400 --seed 42

Result JSONs live in results/llmzszl_likelihood_*.json.

---

Deployment — single 24 GB GPU (RTX 3090)

Quickstart — get it running on a 3090

# 1) Download the Q4_K_M GGUF (private repo -> needs your HF token)
hf download kacperwikiel/slayer-style-qwen3.5-27b-ep3-GGUF \
    slayer-style-qwen3.5-27b-Q4_K_M.gguf --local-dir .

# 2) (already applied to the file in this repo) — if you ever re-convert from scratch,
#    drop the unused MTP metadata or llama.cpp won't load it (missing tensor blk.64):
#    gguf_set_metadata.py model.gguf qwen35.block_count 64 --force
#    gguf_set_metadata.py model.gguf qwen35.nextn_predict_layers 0 --force

# 3) Serve — OpenAI-compatible API on :8080. Needs llama.cpp built with Qwen3.5
#    support (>= commit 98d5e8b, 2026-06). Uses ~16.9 GB VRAM (room to spare on 24 GB).
llama-server -m slayer-style-qwen3.5-27b-Q4_K_M.gguf \
    -ngl 99 -c 32768 -fa on --cache-type-k q8_0 --cache-type-v q8_0 \
    --jinja --host 0.0.0.0 --port 8080

Smoke test: curl localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"x","messages":[{"role":"user","content":"Napisz zdanie o jesieni."}]}'

Ollama instead? ollama create slayer -f Modelfile with FROM ./slayer-style-qwen3.5-27b-Q4_K_M.gguf — but apply the step‑2 metadata fix to the GGUF first.

Production setup (llama-swap — model swapping on one card)

Served on the 3090 box (simp) via llama‑swap (load‑on‑demand model swapping; only one ~20 GB model fits at a time). Config entry:

  "slayer-style-27b":
    aliases: ["slayer", "slayer-style-qwen3.5-27b"]
    ttl: 600   # idle-evict after 10 min so other models can reclaim the GPU
    cmd: |
      /home/kacper/llama.cpp/build/bin/llama-server
      -m /data/home-offload/models/slayer-style-27b/slayer-style-qwen3.5-27b-Q4_K_M.gguf
      -ngl 999 -fa on -c 32768 --no-context-shift
      --cache-type-k q8_0 --cache-type-v q8_0 --jinja --reasoning-format auto
      --temp 0.7 --top-p 0.95 --top-k 20 --min-p 0.0 -np 1
      --host 127.0.0.1 --port ${PORT}

OpenAI‑compatible endpoint, model id slayer-style-27b (aliases slayer). ttl: 600 idle‑evicts it so a co‑hosted default model reclaims the GPU.

Throughput on RTX 3090

Measured via llama-swap, identical engine/flags (-fa on, q8_0 KV, -c 32768), 256-token generations:

Model	gen tok/s	prompt tok/s	params
bielik-11b-v3	~95	~1650	11B
slayer-style-27b	~41	~290	27B

slayer is ~2.3× slower than Bielik — in line with the 2.4× parameter count. Q4_K_M uses ~16.9 GB VRAM, leaving headroom on the 24 GB card.

Deployment gotcha (worth remembering): the convert (llama.cpp 98d5e8b) wrote Qwen3.5's MTP layer into the metadata as qwen35.block_count=65 + nextn_predict_layers=1 without emitting the MTP tensors, so any llama.cpp runtime fails with missing tensor 'blk.64.attn_norm.weight'. Fix in-place (no re-quant): gguf_set_metadata.py model.gguf qwen35.block_count 64 and ... nextn_predict_layers 0.

---

Why this base model

Why start at 27B (and not 9B or smaller). The working belief is that sub‑27B bases just aren't good enough to be genuinely SOTA for Polish right now — so we started from the strongest practical base rather than trying to push a small model uphill. Strong opinion, weakly held: it's a starting hypothesis, not dogma — we'll happily drop to a smaller base the moment one closes the gap. The data so far backs it here: the Qwen3.5‑9B base scores 50.7 % on this benchmark (below Bielik's 56.0), while the 27B base is 58.5 % before any tuning.

Why a Qwen base (sovereignty question). Qwen3.5‑27B is Apache‑2.0; Bielik itself started from Mistral, so "built on a foreign base" cuts both ways. License here is Apache‑2.0 (inherited from the base).

Tokenizer efficiency on Polish — a free win from the base

Qwen's tokenizer is ~23 % more efficient on Polish than Bielik's/Mistral's (fewer tokens per word → cheaper inference and longer effective context). We get this for free, just by choosing the base. Measured on the same Wikipedia sample (PL+EN, ~200 paragraphs) via bench/tokenizer_fertility.py:

Tokenizer	vocab	TpW PL ↓	CpT PL ↑	TpW EN	PL/EN
Gemma‑2‑9B	256k	2.244	3.31	1.340	1.68
Qwen3.5 (our base)	248k	2.357	3.15	1.385	1.70
Llama‑3.1‑8B	128k	2.743	2.71	1.343	2.04
Bielik‑11B‑v3 = Mistral‑7B	32k	3.060	2.43	1.544	1.98

TpW = tokens/word (lower = more efficient) · CpT = chars/token (higher = better) · PL/EN = how much more token‑hungry Polish is than English. Qwen 2.357 vs Bielik 3.060 ⇒ 23 % fewer tokens/word.

Last updated: 2026‑06‑09.