--- language: pl license: apache-2.0 base_model: Qwen/Qwen3.5-27B tags: - polish - style - qlora - gguf - llama.cpp pipeline_tag: text-generation --- # Slayer Style — Qwen3.5‑27B (ep3) Polish‑style fine‑tune of **Qwen/Qwen3.5‑27B**. A LoRA trained to write natural, idiomatic Polish (no translationese, no em‑dash/półpauza overuse) while keeping the base model's knowledge. Quantized to **Q4_K_M GGUF** so it runs on a single 24 GB GPU. **Open weights end‑to‑end — not distilled from Anthropic/OpenAI.** No Claude/GPT weights or outputs are used as training data (see [How the training data was made](#how-the-1000-training-examples-were-made)). 📘 **Full playbook** (data → training → quantization → deployment, with pitfalls): [`slayer_playbook.pdf`](./slayer_playbook.pdf) in this repo. > **TL;DR — it beats Bielik.** On the LLMzSzŁ Polish professional‑exam MCQ benchmark > (answer log‑likelihood, n=400, seed=42, identical harness): > > | Model | Accuracy | > |---|---| > | **slayer‑style‑qwen3.5‑27b (this model, fp16)** | **65.0 %** 🏆 | > | Qwen3.5‑27B (base) | 58.5 % | > | **Bielik‑11B‑v3.0‑Instruct** | **56.0 %** | > | Qwen3.5‑9B (base) | 50.7 % | > > +9.0 points over Bielik‑11B‑v3, +6.5 over the base Qwen it was tuned from. --- ## Artifacts & lineage | Artifact | Repo / path | |---|---| | Base model | `Qwen/Qwen3.5-27B` (Apache‑2.0) | | LoRA adapter | `kacperwikiel/slayer-style-qwen3.5-27b-ep3` (private) | | **Quantized GGUF (this card)** | `kacperwikiel/slayer-style-qwen3.5-27b-ep3-GGUF` (private) — `slayer-style-qwen3.5-27b-Q4_K_M.gguf`, 16.5 GB | | Merged fp16 (intermediate, not published) | produced locally by `bench/merge_adapter.py` | The adapter is the source of truth; the GGUF is derived from `base + adapter`. --- ## Training (style SFT, QLoRA) | Setting | Value | |---|---| | Method | QLoRA SFT (`bench/sft_style_qlora.py`) | | Base | `Qwen/Qwen3.5-27B` | | LoRA rank `r` | 16 | | LoRA `alpha` | 32 | | Target modules | `q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj` (all linear) | | Task | `CAUSAL_LM` | | Epochs | 3 (`ep3`) | | Steps | 150 total (~50/epoch) | | Train loss | 1.34 → 0.93 | ### How the ~1000 training examples were made > **🛡️ Not distilled from Anthropic or OpenAI.** No Claude/GPT weights or generated text are > used as training data — none of the answers the model learns to imitate come from Anthropic > or OpenAI models. The "chosen" answers come from **DeepSeek‑V4‑pro**, the raw/rejected side > from **Qwen3.5‑27B**, and the production quality judge is the **open‑weight Qwen3.5‑122B**. > Anthropic's opus‑4.8 was used only to *validate* that the open‑weight judge scores comparably > (and scored an early data batch before the switch) — never as a source of any training text. ![Data generation pipeline](./slayer_data_pipeline.png) 1. **Prompts** (`bench/gen_style_prompts_v2.py`) — a grid of **26 task types × 20 domains × 9 registers × 4 lengths**, plus an **8‑type grammar stress‑test** (liczebniki, wołacz, aspekt, ortografia ż/rz, frazeologizmy, rekcja…). No benchmark items. 2. **Generation** (`bench/gen_style_sft.py`) — per prompt: a raw answer from `qwen3.5-27b` (the "rejected") and a rewrite from teacher `deepseek-v4-pro` (the "chosen", natural PL). 3. **Judge** — independent **open‑weight `Qwen3.5-122B`** scores `natural_polish_1‑5` and flags `english_leak`, `dash_overuse`, `markdown`, `fact_preserved`, `fits_brief`. (Validated once against opus‑4.8 to confirm the open judge annotates at comparable quality.) 4. **Curate (LIMA)** (`bench/curate_style_data.py`) — **1600 generated → 799 train + 160 holdout**: near‑dup removed, grid‑balanced, score = judge naturalness + length fit − AI‑tell penalties. Holdout is never trained on. *More ≠ better — curation beats volume.* **Concrete before → after** (same prompt, "what sport for a 50+ beginner?"): - *raw `qwen3.5-27b` (rejected):* "Dla osoby powyżej 50. roku życia… **spacery** lub **pływanie**. Oto dlaczego te dyscypliny są optymalne: **Spacery:** Są najbezpieczniejsze…" — stiff, markdown, corporate. - *teacher `deepseek-v4-pro` (chosen):* "Spacery i pływanie to dwa strzały w dziesiątkę… możesz wyjść z domu i po prostu iść, a intensywność regulujesz tempem i dystansem." — natural, idiomatic, no markdown. **Anti‑AI‑tells** — explicit penalties on em‑dash / półpauza overuse, anglicisms, and translationese (`bench/make_anglicyzm_report.py`, `bench/make_translationese_report.py`). --- ## Quantization pipeline (exactly how the GGUF was made) Hardware: **Dell GB10** (aarch64, Grace‑Blackwell `sm_121`, 121 GB unified RAM, CUDA 13). llama.cpp @ commit `98d5e8b` (2026‑06‑06) — first to support Qwen3.5 (`Qwen35MtpMixin`). ```bash # 1) Merge LoRA into the base in fp16 (CPU, ~54 GB RAM; one-time) python bench/merge_adapter.py \ --adapter kacperwikiel/slayer-style-qwen3.5-27b-ep3 \ --out ./slayer-27b-merged # -> ~51 GB merged fp16 # 2) HF -> GGUF f16 python llama.cpp/convert_hf_to_gguf.py ./slayer-27b-merged \ --outfile slayer-27b-f16.gguf --outtype f16 # -> ~51 GB # 3) f16 -> Q4_K_M llama.cpp/build/bin/llama-quantize \ slayer-27b-f16.gguf \ slayer-style-qwen3.5-27b-Q4_K_M.gguf Q4_K_M # -> 16.5 GB ``` Quant choice rationale (24 GB target): Q4_K_M ≈ 16 GB leaves ~8 GB for KV cache — the sweet spot. Q5_K_M (~19 GB) fits but squeezes context; Q6_K (~22 GB) is too tight. --- ## Benchmark methodology - **Harness:** `bench/bench_llmzszl_likelihood.py` (lm‑eval‑style answer log‑likelihood — each answer option scored as a continuation; highest summed log‑prob wins). - **Dataset:** `amu-cai/llmzszl-dataset:llmzszl-test.jsonl` (Polish professional exams). - **Settings:** `n=400`, `seed=42`, `mode=answer`, `normalize=none`. - **Purity:** aggregate‑only; no benchmark item text is stored or inspected (no benchmaxxing). ```bash # reproduce (fp16 merged model, GPU): python bench/bench_llmzszl_likelihood.py --model ./slayer-27b-merged --n 400 --seed 42 ``` Result JSONs live in `results/llmzszl_likelihood_*.json`. --- ## Deployment — single 24 GB GPU (RTX 3090) ### Quickstart — get it running on a 3090 ```bash # 1) Download the Q4_K_M GGUF (private repo -> needs your HF token) hf download kacperwikiel/slayer-style-qwen3.5-27b-ep3-GGUF \ slayer-style-qwen3.5-27b-Q4_K_M.gguf --local-dir . # 2) (already applied to the file in this repo) — if you ever re-convert from scratch, # drop the unused MTP metadata or llama.cpp won't load it (missing tensor blk.64): # gguf_set_metadata.py model.gguf qwen35.block_count 64 --force # gguf_set_metadata.py model.gguf qwen35.nextn_predict_layers 0 --force # 3) Serve — OpenAI-compatible API on :8080. Needs llama.cpp built with Qwen3.5 # support (>= commit 98d5e8b, 2026-06). Uses ~16.9 GB VRAM (room to spare on 24 GB). llama-server -m slayer-style-qwen3.5-27b-Q4_K_M.gguf \ -ngl 99 -c 32768 -fa on --cache-type-k q8_0 --cache-type-v q8_0 \ --jinja --host 0.0.0.0 --port 8080 ``` Smoke test: `curl localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"x","messages":[{"role":"user","content":"Napisz zdanie o jesieni."}]}'` Ollama instead? `ollama create slayer -f Modelfile` with `FROM ./slayer-style-qwen3.5-27b-Q4_K_M.gguf` — but apply the step‑2 metadata fix to the GGUF first. ### Production setup (llama-swap — model swapping on one card) Served on the 3090 box (`simp`) via **llama‑swap** (load‑on‑demand model swapping; only one ~20 GB model fits at a time). Config entry: ```yaml "slayer-style-27b": aliases: ["slayer", "slayer-style-qwen3.5-27b"] ttl: 600 # idle-evict after 10 min so other models can reclaim the GPU cmd: | /home/kacper/llama.cpp/build/bin/llama-server -m /data/home-offload/models/slayer-style-27b/slayer-style-qwen3.5-27b-Q4_K_M.gguf -ngl 999 -fa on -c 32768 --no-context-shift --cache-type-k q8_0 --cache-type-v q8_0 --jinja --reasoning-format auto --temp 0.7 --top-p 0.95 --top-k 20 --min-p 0.0 -np 1 --host 127.0.0.1 --port ${PORT} ``` OpenAI‑compatible endpoint, model id `slayer-style-27b` (aliases `slayer`). `ttl: 600` idle‑evicts it so a co‑hosted default model reclaims the GPU. ### Throughput on RTX 3090 Measured via llama-swap, identical engine/flags (`-fa on`, q8_0 KV, `-c 32768`), 256-token generations: | Model | gen tok/s | prompt tok/s | params | |---|---|---|---| | bielik-11b-v3 | ~95 | ~1650 | 11B | | **slayer-style-27b** | **~41** | ~290 | 27B | slayer is ~2.3× slower than Bielik — in line with the 2.4× parameter count. Q4_K_M uses ~16.9 GB VRAM, leaving headroom on the 24 GB card. > **Deployment gotcha (worth remembering):** the convert (llama.cpp `98d5e8b`) wrote Qwen3.5's MTP layer into the metadata as `qwen35.block_count=65` + `nextn_predict_layers=1` **without** emitting the MTP tensors, so any llama.cpp runtime fails with `missing tensor 'blk.64.attn_norm.weight'`. Fix in-place (no re-quant): `gguf_set_metadata.py model.gguf qwen35.block_count 64` and `... nextn_predict_layers 0`. --- ## Why this base model **Why start at 27B (and not 9B or smaller).** The working belief is that sub‑27B bases just aren't good enough to be *genuinely* SOTA for Polish right now — so we started from the strongest practical base rather than trying to push a small model uphill. **Strong opinion, weakly held:** it's a starting hypothesis, not dogma — we'll happily drop to a smaller base the moment one closes the gap. The data so far backs it here: the Qwen3.5‑**9B** base scores 50.7 % on this benchmark (below Bielik's 56.0), while the 27B base is 58.5 % before any tuning. **Why a Qwen base (sovereignty question).** Qwen3.5‑27B is Apache‑2.0; Bielik itself started from Mistral, so "built on a foreign base" cuts both ways. License here is Apache‑2.0 (inherited from the base). ### Tokenizer efficiency on Polish — a free win from the base Qwen's tokenizer is **~23 % more efficient on Polish** than Bielik's/Mistral's (fewer tokens per word → cheaper inference and longer effective context). We get this for free, just by choosing the base. Measured on the same Wikipedia sample (PL+EN, ~200 paragraphs) via `bench/tokenizer_fertility.py`: | Tokenizer | vocab | TpW PL ↓ | CpT PL ↑ | TpW EN | PL/EN | |---|---|---|---|---|---| | Gemma‑2‑9B | 256k | 2.244 | 3.31 | 1.340 | 1.68 | | **Qwen3.5 (our base)** | 248k | **2.357** | 3.15 | 1.385 | 1.70 | | Llama‑3.1‑8B | 128k | 2.743 | 2.71 | 1.343 | 2.04 | | **Bielik‑11B‑v3 = Mistral‑7B** | 32k | **3.060** | 2.43 | 1.544 | 1.98 | TpW = tokens/word (lower = more efficient) · CpT = chars/token (higher = better) · PL/EN = how much more token‑hungry Polish is than English. Qwen 2.357 vs Bielik 3.060 ⇒ **23 %** fewer tokens/word. ![Tokenizer fertility on Polish](./slayer_tokenizer_fertility.png) _Last updated: 2026‑06‑09._