Safetensors
GGUF
English
Chinese
qwen3_5_text
qwen3.6
dsv4pro
glm
sft
rl
coding
fp8
mtp
imatrix
conversational
Instructions to use nerkyor/Qwen3.6-27B-DSV4Pro-GLM52-SFT-GPT55-RL-Coding with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use nerkyor/Qwen3.6-27B-DSV4Pro-GLM52-SFT-GPT55-RL-Coding with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf nerkyor/Qwen3.6-27B-DSV4Pro-GLM52-SFT-GPT55-RL-Coding:Q8_0 # Run inference directly in the terminal: llama cli -hf nerkyor/Qwen3.6-27B-DSV4Pro-GLM52-SFT-GPT55-RL-Coding:Q8_0
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf nerkyor/Qwen3.6-27B-DSV4Pro-GLM52-SFT-GPT55-RL-Coding:Q8_0 # Run inference directly in the terminal: llama cli -hf nerkyor/Qwen3.6-27B-DSV4Pro-GLM52-SFT-GPT55-RL-Coding:Q8_0
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf nerkyor/Qwen3.6-27B-DSV4Pro-GLM52-SFT-GPT55-RL-Coding:Q8_0 # Run inference directly in the terminal: ./llama-cli -hf nerkyor/Qwen3.6-27B-DSV4Pro-GLM52-SFT-GPT55-RL-Coding:Q8_0
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf nerkyor/Qwen3.6-27B-DSV4Pro-GLM52-SFT-GPT55-RL-Coding:Q8_0 # Run inference directly in the terminal: ./build/bin/llama-cli -hf nerkyor/Qwen3.6-27B-DSV4Pro-GLM52-SFT-GPT55-RL-Coding:Q8_0
Use Docker
docker model run hf.co/nerkyor/Qwen3.6-27B-DSV4Pro-GLM52-SFT-GPT55-RL-Coding:Q8_0
- LM Studio
- Jan
- Ollama
How to use nerkyor/Qwen3.6-27B-DSV4Pro-GLM52-SFT-GPT55-RL-Coding with Ollama:
ollama run hf.co/nerkyor/Qwen3.6-27B-DSV4Pro-GLM52-SFT-GPT55-RL-Coding:Q8_0
- Unsloth Studio
How to use nerkyor/Qwen3.6-27B-DSV4Pro-GLM52-SFT-GPT55-RL-Coding with Unsloth Studio:
Install Unsloth Studio (macOS, Linux, WSL)
curl -fsSL https://unsloth.ai/install.sh | sh # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for nerkyor/Qwen3.6-27B-DSV4Pro-GLM52-SFT-GPT55-RL-Coding to start chatting
Install Unsloth Studio (Windows)
irm https://unsloth.ai/install.ps1 | iex # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for nerkyor/Qwen3.6-27B-DSV4Pro-GLM52-SFT-GPT55-RL-Coding to start chatting
Using HuggingFace Spaces for Unsloth
# No setup required # Open https://huggingface.co/spaces/unsloth/studio in your browser # Search for nerkyor/Qwen3.6-27B-DSV4Pro-GLM52-SFT-GPT55-RL-Coding to start chatting
- Pi
How to use nerkyor/Qwen3.6-27B-DSV4Pro-GLM52-SFT-GPT55-RL-Coding with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf nerkyor/Qwen3.6-27B-DSV4Pro-GLM52-SFT-GPT55-RL-Coding:Q8_0
Configure the model in Pi
# Install Pi: npm install -g @mariozechner/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "nerkyor/Qwen3.6-27B-DSV4Pro-GLM52-SFT-GPT55-RL-Coding:Q8_0" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use nerkyor/Qwen3.6-27B-DSV4Pro-GLM52-SFT-GPT55-RL-Coding with Docker Model Runner:
docker model run hf.co/nerkyor/Qwen3.6-27B-DSV4Pro-GLM52-SFT-GPT55-RL-Coding:Q8_0
- Lemonade
How to use nerkyor/Qwen3.6-27B-DSV4Pro-GLM52-SFT-GPT55-RL-Coding with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull nerkyor/Qwen3.6-27B-DSV4Pro-GLM52-SFT-GPT55-RL-Coding:Q8_0
Run and chat with the model
lemonade run user.Qwen3.6-27B-DSV4Pro-GLM52-SFT-GPT55-RL-Coding-Q8_0
List all available models
lemonade list
- Hermes Agent
How to use nerkyor/Qwen3.6-27B-DSV4Pro-GLM52-SFT-GPT55-RL-Coding with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf nerkyor/Qwen3.6-27B-DSV4Pro-GLM52-SFT-GPT55-RL-Coding:Q8_0
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default nerkyor/Qwen3.6-27B-DSV4Pro-GLM52-SFT-GPT55-RL-Coding:Q8_0
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use nerkyor/Qwen3.6-27B-DSV4Pro-GLM52-SFT-GPT55-RL-Coding with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf nerkyor/Qwen3.6-27B-DSV4Pro-GLM52-SFT-GPT55-RL-Coding:Q8_0
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "nerkyor/Qwen3.6-27B-DSV4Pro-GLM52-SFT-GPT55-RL-Coding:Q8_0" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
copy GGUF artifacts from standalone repo batch 4
Browse files
.gitattributes
CHANGED
|
@@ -54,3 +54,5 @@ GGUF/Q8_0/Q8-MTP-00003-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
|
|
| 54 |
GGUF/Q8_0/Q8-MTP-00004-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
|
| 55 |
GGUF/Q8_0/Q8-MTP-00005-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
|
| 56 |
GGUF/Q8_0/Q8-MTP-draft.gguf filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
| 54 |
GGUF/Q8_0/Q8-MTP-00004-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
|
| 55 |
GGUF/Q8_0/Q8-MTP-00005-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
|
| 56 |
GGUF/Q8_0/Q8-MTP-draft.gguf filter=lfs diff=lfs merge=lfs -text
|
| 57 |
+
GGUF/assets/model_card_hero_hf_english.png filter=lfs diff=lfs merge=lfs -text
|
| 58 |
+
GGUF/assets/model_card_hero_hf_english_20260707.png filter=lfs diff=lfs merge=lfs -text
|
GGUF/Q8_0/eval/mtp/mtp_settings.json
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"tier": "Q8_0",
|
| 3 |
+
"model": "/root/autodl-tmp/step37-27b-distill/quantized/Qwen3.6-27B-DSV4Pro-GLM-55XH-Coding-SFT-RL-Final55True-v3/Q8_0/Qwen3.6-27B-DSV4Pro-GLM-55XH-Coding-SFT-RL-Final55True-v3-Q8_0.gguf",
|
| 4 |
+
"draft_model": "/root/autodl-tmp/step37-27b-distill/quantized/step37-27b-dsv4pro-official-mtp/MTP/step37-27b-official-mtp-q8_0.gguf",
|
| 5 |
+
"matrix": {
|
| 6 |
+
"c_values": "1,2,4",
|
| 7 |
+
"n_values": "1,2,3,4",
|
| 8 |
+
"ctx_per_slot": 4096,
|
| 9 |
+
"max_tokens": 384,
|
| 10 |
+
"request_timeout_sec": 240.0,
|
| 11 |
+
"temperature": 0.1,
|
| 12 |
+
"top_p": 0.95,
|
| 13 |
+
"spec_draft_p_min": 0.6
|
| 14 |
+
},
|
| 15 |
+
"selected_eval_route": {
|
| 16 |
+
"parallel_c": 4,
|
| 17 |
+
"mtp_n": null,
|
| 18 |
+
"mtp_enabled": false,
|
| 19 |
+
"source": "/root/autodl-tmp/step37-27b-distill/runs/143_final55_true_v3_q8_mtp_full_eval_20260705/q8_mtp_acceptance_matrix/selected_eval_route.json"
|
| 20 |
+
},
|
| 21 |
+
"note": "C1/C2/C4 TPS and N=1/2/3/4 MTP acceptance matrix; quality eval uses temperature 0.6/top_p 0.95.",
|
| 22 |
+
"generated_at": "2026-07-06T06:54:56+0800"
|
| 23 |
+
}
|
GGUF/Q8_0/eval/mtp/selected_eval_route.json
ADDED
|
@@ -0,0 +1,271 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"chosen": {
|
| 3 |
+
"label": "c4_nomtp",
|
| 4 |
+
"c": 4,
|
| 5 |
+
"n": null,
|
| 6 |
+
"mtp": false,
|
| 7 |
+
"port": 19690,
|
| 8 |
+
"requests": 8,
|
| 9 |
+
"ok": 8,
|
| 10 |
+
"errors": 0,
|
| 11 |
+
"nonempty_content": 8,
|
| 12 |
+
"nonempty_any": 8,
|
| 13 |
+
"completion_tokens": 1904,
|
| 14 |
+
"wall_sec": 21.72,
|
| 15 |
+
"api_completion_tokens_per_sec": 87.66,
|
| 16 |
+
"server": {
|
| 17 |
+
"draft_acceptance_events": 0,
|
| 18 |
+
"draft_acceptance_weighted": null,
|
| 19 |
+
"draft_accepted_tokens": null,
|
| 20 |
+
"draft_generated_tokens": null,
|
| 21 |
+
"eval_tokens": 2165,
|
| 22 |
+
"eval_ms": 63539.18,
|
| 23 |
+
"eval_tokens_per_sec": 34.073,
|
| 24 |
+
"model_loaded": true,
|
| 25 |
+
"mtp_initialized": false
|
| 26 |
+
},
|
| 27 |
+
"responses": [
|
| 28 |
+
{
|
| 29 |
+
"ok": true,
|
| 30 |
+
"elapsed_sec": 3.704,
|
| 31 |
+
"content_len": 54,
|
| 32 |
+
"reasoning_len": 139,
|
| 33 |
+
"completion_tokens": 117,
|
| 34 |
+
"total_tokens": 143,
|
| 35 |
+
"finish_reason": "stop"
|
| 36 |
+
},
|
| 37 |
+
{
|
| 38 |
+
"ok": true,
|
| 39 |
+
"elapsed_sec": 8.121,
|
| 40 |
+
"content_len": 68,
|
| 41 |
+
"reasoning_len": 396,
|
| 42 |
+
"completion_tokens": 267,
|
| 43 |
+
"total_tokens": 306,
|
| 44 |
+
"finish_reason": "stop"
|
| 45 |
+
},
|
| 46 |
+
{
|
| 47 |
+
"ok": true,
|
| 48 |
+
"elapsed_sec": 5.175,
|
| 49 |
+
"content_len": 28,
|
| 50 |
+
"reasoning_len": 258,
|
| 51 |
+
"completion_tokens": 161,
|
| 52 |
+
"total_tokens": 191,
|
| 53 |
+
"finish_reason": "stop"
|
| 54 |
+
},
|
| 55 |
+
{
|
| 56 |
+
"ok": true,
|
| 57 |
+
"elapsed_sec": 9.864,
|
| 58 |
+
"content_len": 442,
|
| 59 |
+
"reasoning_len": 338,
|
| 60 |
+
"completion_tokens": 304,
|
| 61 |
+
"total_tokens": 344,
|
| 62 |
+
"finish_reason": "stop"
|
| 63 |
+
},
|
| 64 |
+
{
|
| 65 |
+
"ok": true,
|
| 66 |
+
"elapsed_sec": 2.059,
|
| 67 |
+
"content_len": 4,
|
| 68 |
+
"reasoning_len": 77,
|
| 69 |
+
"completion_tokens": 59,
|
| 70 |
+
"total_tokens": 88,
|
| 71 |
+
"finish_reason": "stop"
|
| 72 |
+
},
|
| 73 |
+
{
|
| 74 |
+
"ok": true,
|
| 75 |
+
"elapsed_sec": 12.891,
|
| 76 |
+
"content_len": 436,
|
| 77 |
+
"reasoning_len": 474,
|
| 78 |
+
"completion_tokens": 384,
|
| 79 |
+
"total_tokens": 420,
|
| 80 |
+
"finish_reason": "length"
|
| 81 |
+
},
|
| 82 |
+
{
|
| 83 |
+
"ok": true,
|
| 84 |
+
"elapsed_sec": 9.892,
|
| 85 |
+
"content_len": 204,
|
| 86 |
+
"reasoning_len": 366,
|
| 87 |
+
"completion_tokens": 228,
|
| 88 |
+
"total_tokens": 256,
|
| 89 |
+
"finish_reason": "stop"
|
| 90 |
+
},
|
| 91 |
+
{
|
| 92 |
+
"ok": true,
|
| 93 |
+
"elapsed_sec": 12.839,
|
| 94 |
+
"content_len": 159,
|
| 95 |
+
"reasoning_len": 543,
|
| 96 |
+
"completion_tokens": 384,
|
| 97 |
+
"total_tokens": 417,
|
| 98 |
+
"finish_reason": "length"
|
| 99 |
+
}
|
| 100 |
+
],
|
| 101 |
+
"server_log": "/root/autodl-tmp/step37-27b-distill/runs/143_final55_true_v3_q8_mtp_full_eval_20260705/q8_mtp_acceptance_matrix/c4_nomtp/server.log"
|
| 102 |
+
},
|
| 103 |
+
"fastest_mtp": {
|
| 104 |
+
"label": "c2_n3",
|
| 105 |
+
"c": 2,
|
| 106 |
+
"n": 3,
|
| 107 |
+
"mtp": true,
|
| 108 |
+
"port": 19688,
|
| 109 |
+
"requests": 4,
|
| 110 |
+
"ok": 4,
|
| 111 |
+
"errors": 0,
|
| 112 |
+
"nonempty_content": 4,
|
| 113 |
+
"nonempty_any": 4,
|
| 114 |
+
"completion_tokens": 1081,
|
| 115 |
+
"wall_sec": 14.561,
|
| 116 |
+
"api_completion_tokens_per_sec": 74.24,
|
| 117 |
+
"server": {
|
| 118 |
+
"draft_acceptance_events": 4,
|
| 119 |
+
"draft_acceptance_weighted": 0.907433,
|
| 120 |
+
"draft_accepted_tokens": 647,
|
| 121 |
+
"draft_generated_tokens": 713,
|
| 122 |
+
"eval_tokens": 1222,
|
| 123 |
+
"eval_ms": 26237.8,
|
| 124 |
+
"eval_tokens_per_sec": 46.574,
|
| 125 |
+
"model_loaded": true,
|
| 126 |
+
"mtp_initialized": true
|
| 127 |
+
},
|
| 128 |
+
"responses": [
|
| 129 |
+
{
|
| 130 |
+
"ok": true,
|
| 131 |
+
"elapsed_sec": 3.629,
|
| 132 |
+
"content_len": 68,
|
| 133 |
+
"reasoning_len": 127,
|
| 134 |
+
"completion_tokens": 119,
|
| 135 |
+
"total_tokens": 145,
|
| 136 |
+
"finish_reason": "stop"
|
| 137 |
+
},
|
| 138 |
+
{
|
| 139 |
+
"ok": true,
|
| 140 |
+
"elapsed_sec": 7.725,
|
| 141 |
+
"content_len": 372,
|
| 142 |
+
"reasoning_len": 338,
|
| 143 |
+
"completion_tokens": 291,
|
| 144 |
+
"total_tokens": 331,
|
| 145 |
+
"finish_reason": "stop"
|
| 146 |
+
},
|
| 147 |
+
{
|
| 148 |
+
"ok": true,
|
| 149 |
+
"elapsed_sec": 8.773,
|
| 150 |
+
"content_len": 70,
|
| 151 |
+
"reasoning_len": 412,
|
| 152 |
+
"completion_tokens": 287,
|
| 153 |
+
"total_tokens": 326,
|
| 154 |
+
"finish_reason": "stop"
|
| 155 |
+
},
|
| 156 |
+
{
|
| 157 |
+
"ok": true,
|
| 158 |
+
"elapsed_sec": 6.834,
|
| 159 |
+
"content_len": 602,
|
| 160 |
+
"reasoning_len": 335,
|
| 161 |
+
"completion_tokens": 384,
|
| 162 |
+
"total_tokens": 420,
|
| 163 |
+
"finish_reason": "length"
|
| 164 |
+
}
|
| 165 |
+
],
|
| 166 |
+
"server_log": "/root/autodl-tmp/step37-27b-distill/runs/143_final55_true_v3_q8_mtp_full_eval_20260705/q8_mtp_acceptance_matrix/c2_n3/server.log"
|
| 167 |
+
},
|
| 168 |
+
"fastest_no_mtp": {
|
| 169 |
+
"label": "c4_nomtp",
|
| 170 |
+
"c": 4,
|
| 171 |
+
"n": null,
|
| 172 |
+
"mtp": false,
|
| 173 |
+
"port": 19690,
|
| 174 |
+
"requests": 8,
|
| 175 |
+
"ok": 8,
|
| 176 |
+
"errors": 0,
|
| 177 |
+
"nonempty_content": 8,
|
| 178 |
+
"nonempty_any": 8,
|
| 179 |
+
"completion_tokens": 1904,
|
| 180 |
+
"wall_sec": 21.72,
|
| 181 |
+
"api_completion_tokens_per_sec": 87.66,
|
| 182 |
+
"server": {
|
| 183 |
+
"draft_acceptance_events": 0,
|
| 184 |
+
"draft_acceptance_weighted": null,
|
| 185 |
+
"draft_accepted_tokens": null,
|
| 186 |
+
"draft_generated_tokens": null,
|
| 187 |
+
"eval_tokens": 2165,
|
| 188 |
+
"eval_ms": 63539.18,
|
| 189 |
+
"eval_tokens_per_sec": 34.073,
|
| 190 |
+
"model_loaded": true,
|
| 191 |
+
"mtp_initialized": false
|
| 192 |
+
},
|
| 193 |
+
"responses": [
|
| 194 |
+
{
|
| 195 |
+
"ok": true,
|
| 196 |
+
"elapsed_sec": 3.704,
|
| 197 |
+
"content_len": 54,
|
| 198 |
+
"reasoning_len": 139,
|
| 199 |
+
"completion_tokens": 117,
|
| 200 |
+
"total_tokens": 143,
|
| 201 |
+
"finish_reason": "stop"
|
| 202 |
+
},
|
| 203 |
+
{
|
| 204 |
+
"ok": true,
|
| 205 |
+
"elapsed_sec": 8.121,
|
| 206 |
+
"content_len": 68,
|
| 207 |
+
"reasoning_len": 396,
|
| 208 |
+
"completion_tokens": 267,
|
| 209 |
+
"total_tokens": 306,
|
| 210 |
+
"finish_reason": "stop"
|
| 211 |
+
},
|
| 212 |
+
{
|
| 213 |
+
"ok": true,
|
| 214 |
+
"elapsed_sec": 5.175,
|
| 215 |
+
"content_len": 28,
|
| 216 |
+
"reasoning_len": 258,
|
| 217 |
+
"completion_tokens": 161,
|
| 218 |
+
"total_tokens": 191,
|
| 219 |
+
"finish_reason": "stop"
|
| 220 |
+
},
|
| 221 |
+
{
|
| 222 |
+
"ok": true,
|
| 223 |
+
"elapsed_sec": 9.864,
|
| 224 |
+
"content_len": 442,
|
| 225 |
+
"reasoning_len": 338,
|
| 226 |
+
"completion_tokens": 304,
|
| 227 |
+
"total_tokens": 344,
|
| 228 |
+
"finish_reason": "stop"
|
| 229 |
+
},
|
| 230 |
+
{
|
| 231 |
+
"ok": true,
|
| 232 |
+
"elapsed_sec": 2.059,
|
| 233 |
+
"content_len": 4,
|
| 234 |
+
"reasoning_len": 77,
|
| 235 |
+
"completion_tokens": 59,
|
| 236 |
+
"total_tokens": 88,
|
| 237 |
+
"finish_reason": "stop"
|
| 238 |
+
},
|
| 239 |
+
{
|
| 240 |
+
"ok": true,
|
| 241 |
+
"elapsed_sec": 12.891,
|
| 242 |
+
"content_len": 436,
|
| 243 |
+
"reasoning_len": 474,
|
| 244 |
+
"completion_tokens": 384,
|
| 245 |
+
"total_tokens": 420,
|
| 246 |
+
"finish_reason": "length"
|
| 247 |
+
},
|
| 248 |
+
{
|
| 249 |
+
"ok": true,
|
| 250 |
+
"elapsed_sec": 9.892,
|
| 251 |
+
"content_len": 204,
|
| 252 |
+
"reasoning_len": 366,
|
| 253 |
+
"completion_tokens": 228,
|
| 254 |
+
"total_tokens": 256,
|
| 255 |
+
"finish_reason": "stop"
|
| 256 |
+
},
|
| 257 |
+
{
|
| 258 |
+
"ok": true,
|
| 259 |
+
"elapsed_sec": 12.839,
|
| 260 |
+
"content_len": 159,
|
| 261 |
+
"reasoning_len": 543,
|
| 262 |
+
"completion_tokens": 384,
|
| 263 |
+
"total_tokens": 417,
|
| 264 |
+
"finish_reason": "length"
|
| 265 |
+
}
|
| 266 |
+
],
|
| 267 |
+
"server_log": "/root/autodl-tmp/step37-27b-distill/runs/143_final55_true_v3_q8_mtp_full_eval_20260705/q8_mtp_acceptance_matrix/c4_nomtp/server.log"
|
| 268 |
+
},
|
| 269 |
+
"stable_rows": 15,
|
| 270 |
+
"source": "/root/autodl-tmp/step37-27b-distill/runs/143_final55_true_v3_q8_mtp_full_eval_20260705/q8_mtp_acceptance_matrix/mtp_acceptance_matrix.json"
|
| 271 |
+
}
|
GGUF/README.md
ADDED
|
@@ -0,0 +1,238 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: other
|
| 3 |
+
language:
|
| 4 |
+
- en
|
| 5 |
+
- zh
|
| 6 |
+
tags:
|
| 7 |
+
- qwen3.6
|
| 8 |
+
- gguf
|
| 9 |
+
- lynnstyle
|
| 10 |
+
- dense
|
| 11 |
+
- coding
|
| 12 |
+
- mtp
|
| 13 |
+
base_model:
|
| 14 |
+
- Merkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill
|
| 15 |
+
---
|
| 16 |
+
|
| 17 |
+
# Qwen3.6-27B-DSV4Pro-GLM52-SFT-GPT55-RL-Coding-GGUF
|
| 18 |
+
|
| 19 |
+

|
| 20 |
+
|
| 21 |
+
GGUF mirror for `Qwen3.6-27B-DSV4Pro-GLM52-SFT-GPT55-RL-Coding`.
|
| 22 |
+
|
| 23 |
+
> **Important: the Hugging Face GGUF sidebar may not reflect the real model size, parameter count, or quantization policy of this repository.**
|
| 24 |
+
> This repository uses **multi-shard main GGUF files + optional MTP draft sidecars + LynnStyle Dense mixed precision**. The platform parser may scan only the MTP sidecar or only one shard, then show misleading metadata such as `3B params`, `5-bit`, `2.49GB`, or `Q5_K_M`. That metadata describes the file the sidebar happened to parse, not the 27B main model.
|
| 25 |
+
> Please follow this README: download all shards from the selected tier directory (`Q8_0/`, `Q5_LynnStyle/`, `Q4_LynnStyle/`, etc.), and use the matching `*-draft.gguf` only as an optional `--model-draft` sidecar. The source of truth is the tier directory, `manifest`, `SHA256SUMS`, `eval/`, and this README.
|
| 26 |
+
|
| 27 |
+
This model is the coding-focused post-training branch of `Qwen3.6-27B-DSV4Pro-Thinking-Distill`. The goal is not to make the model think longer. The goal is to make it deliver more reliably: write executable code, repair from real observations, verify before closing, and keep MMLU / GPQA / LBC / Coding100 within healthy gates.
|
| 28 |
+
|
| 29 |
+
Main BF16 repository will be published separately. This repository is for GGUF quantized files, MTP sidecars, runtime smoke results, and quantized evaluation summaries.
|
| 30 |
+
|
| 31 |
+
## Which Directory Should I Download?
|
| 32 |
+
|
| 33 |
+
Artifacts are organized by tier directory:
|
| 34 |
+
|
| 35 |
+
- `Q8_0/`: quality baseline for 48GB+ GPUs.
|
| 36 |
+
- `Q5_LynnStyle/`: 32GB recommended high-quality tier, uploaded.
|
| 37 |
+
- `Q4_LynnStyle/`: 24GB mainstream tier, uploaded. For long-context use, load the Q4 main model only and do not load the MTP sidecar.
|
| 38 |
+
- `Q3_LynnStyle/`: 16GB low-end tier, in production, targeting 16K context plus low-bit MTP smoke.
|
| 39 |
+
- `Q2_LynnStyle/`: 8GB experimental entry tier, planned.
|
| 40 |
+
|
| 41 |
+
Each tier directory contains:
|
| 42 |
+
|
| 43 |
+
1. Main-model GGUF shards, for example `Q5-imatrix-MTP-00001-of-00004.gguf`.
|
| 44 |
+
2. An optional MTP draft sidecar, for example `Q5-imatrix-MTP-draft.gguf`.
|
| 45 |
+
3. `eval/`, `lynnstyle/`, manifest files, and `SHA256SUMS` for reproducing quantization rules, MTP/TPS checks, and evaluation scores.
|
| 46 |
+
|
| 47 |
+
MTP sidecars are optional speculative-decoding draft files. They are not standalone models and they are not multimodal / vision files.
|
| 48 |
+
|
| 49 |
+
The right-side hardware / GGUF compatibility card should therefore not be used as the true release matrix for this repository. It may parse the MTP sidecar instead of the main model. Use the tables below and the files inside each tier directory.
|
| 50 |
+
|
| 51 |
+
## llama.cpp Quick Start
|
| 52 |
+
|
| 53 |
+
For split GGUF files, pass the first shard to llama.cpp. The remaining shards must stay in the same directory:
|
| 54 |
+
|
| 55 |
+
```bash
|
| 56 |
+
llama-server \
|
| 57 |
+
-m Q5_LynnStyle/Q5-imatrix-MTP-00001-of-00004.gguf \
|
| 58 |
+
-ngl 999 --split-mode layer \
|
| 59 |
+
-c 32768 -b 1024 -ub 256 -np 4 -cb \
|
| 60 |
+
--host 0.0.0.0 --port 8080 \
|
| 61 |
+
--jinja --reasoning on --reasoning-format deepseek
|
| 62 |
+
```
|
| 63 |
+
|
| 64 |
+
If you have enough VRAM and want speculative decoding, add the matching MTP draft sidecar:
|
| 65 |
+
|
| 66 |
+
```bash
|
| 67 |
+
llama-server \
|
| 68 |
+
-m Q5_LynnStyle/Q5-imatrix-MTP-00001-of-00004.gguf \
|
| 69 |
+
--model-draft Q5_LynnStyle/Q5-imatrix-MTP-draft.gguf \
|
| 70 |
+
-ngl 999 --split-mode layer \
|
| 71 |
+
-c 32768 -b 1024 -ub 256 -np 4 -cb \
|
| 72 |
+
--host 0.0.0.0 --port 8080 \
|
| 73 |
+
--jinja --reasoning on --reasoning-format deepseek
|
| 74 |
+
```
|
| 75 |
+
|
| 76 |
+
Q4 targets 24GB GPUs. For long context, prioritize the main model and KV cache, and do not load the MTP sidecar. Use Q4 MTP only for shorter-context or lower-concurrency runs with enough VRAM headroom.
|
| 77 |
+
|
| 78 |
+
## Current Tiers
|
| 79 |
+
|
| 80 |
+
| VRAM | Artifact | Status | Positioning |
|
| 81 |
+
|---:|---|---|---|
|
| 82 |
+
| 8GB | Q2 LynnStyle Dense | planned | 8GB experimental entry tier, short context. |
|
| 83 |
+
| 16GB | Q3 LynnStyle Dense | in production | Low-end local tier: `IQ3_XS` base, Q4 protected islands, imatrix calibration, 16K target. |
|
| 84 |
+
| 24GB | Q4 LynnStyle Dense | uploaded | Mainstream local tier. For long context, load the Q4 main model only; use the Q4 MTP sidecar only when VRAM has headroom. |
|
| 85 |
+
| 32GB | Q5 LynnStyle Dense | uploaded | Recommended sweet spot, explicit protection + imatrix. |
|
| 86 |
+
| 48GB+ | Q8 LynnStyle Dense | complete | Quality baseline with official MTP draft. |
|
| 87 |
+
|
| 88 |
+
Q6 is not released as a standalone public tier because it does not map cleanly to a user VRAM segment. Larger machines can run Q8; smaller machines need Q5/Q4/Q3/Q2. In LynnStyle Dense, Q6 is used internally as a protection precision for sensitive layers and tensor families.
|
| 89 |
+
|
| 90 |
+
## Why LynnStyle Dense?
|
| 91 |
+
|
| 92 |
+
MoE models can be inspected through expert heatmaps: which experts are active for MMLU / GPQA, LBC / Coding, and ReAct-style tasks. This 27B model is Dense, so there are no MoE experts to prune or protect, but the model is still not homogeneous.
|
| 93 |
+
|
| 94 |
+
The protection unit changes from MoE's `layer + expert + tensor family` to Dense's:
|
| 95 |
+
|
| 96 |
+
- `layer position`
|
| 97 |
+
- `attention pattern`
|
| 98 |
+
- `tensor family`
|
| 99 |
+
|
| 100 |
+
Qwen3.6 27B uses mixed attention patterns. Full-attention layers appear roughly every four layers and are important for long-range integration and final-answer closure. Other layers rely more on linear-attention / SSM paths. The middle of the network is not an unprotected blank zone: full-attention projections, attention gates, linear-attention qkv, and SSM tensor families are all protected work areas.
|
| 101 |
+
|
| 102 |
+
The GGUF probe confirms that the protection rules hit real tensors:
|
| 103 |
+
|
| 104 |
+
| Protected group | Matched tensors |
|
| 105 |
+
|---|---:|
|
| 106 |
+
| early layers | 47 |
|
| 107 |
+
| last four layers | 47 |
|
| 108 |
+
| L56-L59 shoulder | 47 |
|
| 109 |
+
| middle full-attention projections | 52 |
|
| 110 |
+
| middle linear-attention gates | 39 |
|
| 111 |
+
| middle linear-attention qkv | 39 |
|
| 112 |
+
| middle SSM tensor family | 195 |
|
| 113 |
+
|
| 114 |
+
For Q5, LynnStyle means explicit protected tensors plus imatrix calibration:
|
| 115 |
+
|
| 116 |
+
| Region | Explicit precision | Matched tensors |
|
| 117 |
+
|---|---|---:|
|
| 118 |
+
| L0-L3 early layers | `q8_0` hard protection | 47 |
|
| 119 |
+
| L60-L63 last layers | `q8_0` hard protection | 47 |
|
| 120 |
+
| L56-L59 shoulder | `q6_k` protection | 47 |
|
| 121 |
+
| middle full-attention q/k/v/o | `q5_k` protection | 52 |
|
| 122 |
+
| middle linear-attention gate | `q5_k` protection | 39 |
|
| 123 |
+
| middle linear-attention qkv | `q5_k` protection | 39 |
|
| 124 |
+
| middle SSM tensor family | `q5_k` protection | 195 |
|
| 125 |
+
|
| 126 |
+
Direct quantization alone is not a release tier. A LynnStyle Dense release tier must combine explicit core-layer / tensor-family protection with imatrix or equivalent importance calibration, and then pass public gates.
|
| 127 |
+
|
| 128 |
+
## Current Q8 Gate
|
| 129 |
+
|
| 130 |
+
Evaluation uses Q8_0 GGUF with the official MTP head, llama.cpp serving, temperature `0.6`, top_p `0.95`.
|
| 131 |
+
|
| 132 |
+
| Dimension | Metric | Score | Meaning |
|
| 133 |
+
|---|---|---:|---|
|
| 134 |
+
| General knowledge | MMLU500 | 462/500 = 92.4% | Multi-domain knowledge and understanding. |
|
| 135 |
+
| Hard reasoning | GPQA198 | 162/198 = 81.82% | Scientific reasoning and final-answer closure. |
|
| 136 |
+
| Algorithmic code | LBC100 | 78/100 | Hidden-test coding and algorithmic problem solving. |
|
| 137 |
+
| Multilingual code | Coding100 clean | 88/100 | Multi-language engineering coverage. |
|
| 138 |
+
|
| 139 |
+
Coding100 language split: Python 9/10, JavaScript 6/10, TypeScript 7/10, Rust 9/10, Go 8/10, C++ 10/10, SQL 10/10, Bash 9/10, CSS 10/10, HTML 10/10.
|
| 140 |
+
|
| 141 |
+
LBC100 difficulty split: easy 23/23, medium 30/31, hard 25/46.
|
| 142 |
+
|
| 143 |
+
## Cross-Version Clean Scores
|
| 144 |
+
|
| 145 |
+
Only complete clean results are listed here. Harness-polluted rows are retested by id and merged before a score is published.
|
| 146 |
+
|
| 147 |
+
| Version | MMLU500 | GPQA198 | LBC100 | Coding100 |
|
| 148 |
+
|---|---:|---:|---:|---:|
|
| 149 |
+
| Qwen3.6-27B original Q8 | 91.6% | 73.7% | 68/100 | 83/100 |
|
| 150 |
+
| DSV4Pro distill Q8 | 91.8% | 80.81% | 74/100 | 86/100 |
|
| 151 |
+
| This release Q8 | 92.4% | 81.82% | 78/100 | 88/100 |
|
| 152 |
+
| Q4 LynnStyle | 92.0% | 74.75% | 71/100 | 82/100 |
|
| 153 |
+
|
| 154 |
+
Original Q8 LBC100 clean result: `68/100`; difficulty split: easy `23/23`, medium `27/31`, hard `18/46`. Three JSONDecodeError-polluted rows were retested non-streaming by exact id and all three passed, leaving `errors=0`.
|
| 155 |
+
|
| 156 |
+
DSV4Pro distill Q8 LBC100 clean result: `74/100`; difficulty split: easy `23/23`, medium `25/31`, hard `26/46`; `errors=0`.
|
| 157 |
+
|
| 158 |
+
## Q5 LynnStyle Gate
|
| 159 |
+
|
| 160 |
+
Q5 is the 32GB recommended tier. It is designed to stay close to Q8 quality while reducing file size enough for local deployment.
|
| 161 |
+
|
| 162 |
+
| Metric | Q5 LynnStyle | Status |
|
| 163 |
+
|---|---:|---|
|
| 164 |
+
| MMLU500 | 461/500 = 92.2% | clean |
|
| 165 |
+
| GPQA198 | 156/198 = 78.79% | clean |
|
| 166 |
+
| LBC100 | 75/100 | clean |
|
| 167 |
+
|
| 168 |
+
## Q4 LynnStyle Gate
|
| 169 |
+
|
| 170 |
+
Q4 is the 24GB mainstream tier. For 16K/24K long-context use on 24GB GPUs, load the Q4 main model only and do not load the MTP sidecar. The Q4 MTP sidecar is optional for short-context or lower-concurrency speculative decoding when VRAM has headroom.
|
| 171 |
+
|
| 172 |
+
| Metric | Q4 LynnStyle | Status |
|
| 173 |
+
|---|---:|---|
|
| 174 |
+
| MMLU500 | 460/500 = 92.0% | clean |
|
| 175 |
+
| GPQA198 | 148/198 = 74.75% | clean |
|
| 176 |
+
| LBC100 | 71/100 | clean |
|
| 177 |
+
| Coding100 current clean | 82/100 | clean |
|
| 178 |
+
|
| 179 |
+
The GPQA198 result was merged from the interrupted first pass and a missing-id resume pass while preserving the original GPQA ids and choice mappings. Health counters: `dirty=0`, `parse_fail=0`, `empty_prediction=0`, `duplicate_ids=0`; finish reasons: `stop=197`, `length=1`.
|
| 180 |
+
|
| 181 |
+
Q4 LBC100 full clean result: `71/100`; difficulty split: easy `23/23`, medium `27/31`, hard `21/46`. Failure notes: wrong answer `20`, runtime error `4`, timeout `5`; no harness error/jsondecode/parse pollution.
|
| 182 |
+
|
| 183 |
+
Q4 Coding100 current clean result: `82/100`; language split: Python `9/10`, JavaScript `6/10`, TypeScript `9/10`, Rust `7/10`, Go `6/10`, C++ `7/10`, SQL `10/10`, Bash `8/10`, CSS `10/10`, HTML `10/10`. API errors: `0`.
|
| 184 |
+
|
| 185 |
+
## Training and Data Hygiene
|
| 186 |
+
|
| 187 |
+
This release does not train on benchmark answer keys. SFT teaches trajectories; RL teaches behavior preferences. Benchmarks remain held out to check whether the model learned to think, solve, act, verify, and close.
|
| 188 |
+
|
| 189 |
+
High-level training route:
|
| 190 |
+
|
| 191 |
+
| Stage | Purpose | Data shape |
|
| 192 |
+
|---|---|---|
|
| 193 |
+
| SFT-1A direct-code | Improve executable code and weak programming languages. | Direct code / single-code-block tasks. |
|
| 194 |
+
| SFT-1B real-observation ReAct | Teach reason -> act -> observe -> repair -> verify -> final. | Real observation trajectories. |
|
| 195 |
+
| RL-1 clean | Reduce empty answers, fake verification, no-final, and invalid loops. | Preference pairs. |
|
| 196 |
+
| RL-2 GPT55 guarded | Final lightweight behavior calibration. | Focused preference pairs for final-answer closure, anti-insufficient behavior, execution repair, and reasoning-budget control. |
|
| 197 |
+
|
| 198 |
+
Data policy:
|
| 199 |
+
|
| 200 |
+
- GLM-5.2 is used for candidate generation and coverage.
|
| 201 |
+
- GPT55 / Codex-style review is used for judging, correction, and high-value preference pairs.
|
| 202 |
+
- Real execution observations have priority as evidence.
|
| 203 |
+
- Tool observations are used as context, not as assistant loss targets.
|
| 204 |
+
- Assistant-only loss trains assistant thought / action / repair / final segments.
|
| 205 |
+
|
| 206 |
+
## 中文说明
|
| 207 |
+
|
| 208 |
+
这是 `Qwen3.6-27B-DSV4Pro-GLM52-SFT-GPT55-RL-Coding` 的 GGUF 镜像仓。主仓放 BF16 与主模型卡,本仓只放 GGUF 量化文件、MTP sidecar、runtime smoke、MTP/TPS 矩阵和量化门禁结果。
|
| 209 |
+
|
| 210 |
+
这不是“让模型更啰嗦思考”的版本,而是把 DSV4Pro 蒸馏版继续往 coding / execution / 可交付方向推:能写可执行代码,能基于真实 observation 修复,完成前会验证,收口时给出明确答案,同时尽量守住 MMLU / GPQA / LBC100 / Coding100。
|
| 211 |
+
|
| 212 |
+
### 下载目录
|
| 213 |
+
|
| 214 |
+
- `Q8_0/`:质量基线,48GB+ 显存。
|
| 215 |
+
- `Q5_LynnStyle/`:32GB 推荐甜点档,已上传。
|
| 216 |
+
- `Q4_LynnStyle/`:24GB 主流档,已上传。长上下文请只加载 Q4 主模型,不要加载 MTP sidecar。
|
| 217 |
+
- `Q3_LynnStyle/`:16GB 低配档,生产中,目标是 16K 上下文 + 低比特 MTP smoke。
|
| 218 |
+
- `Q2_LynnStyle/`:8GB 上车实验档,后续生产。
|
| 219 |
+
|
| 220 |
+
每个目录里会有主模型 GGUF 分片、MTP draft sidecar、`eval/`、`lynnstyle/`、manifest 和 SHA256。MTP sidecar 不是主模型,也不是视觉文件,只用于 llama.cpp speculative decoding。
|
| 221 |
+
|
| 222 |
+
### llama.cpp 启动
|
| 223 |
+
|
| 224 |
+
多分片 GGUF 指定第一片即可,例如 `Q5_LynnStyle/Q5-imatrix-MTP-00001-of-00004.gguf`;其它分片保持同目录。MTP 文件只作为 `--model-draft` 可选加速文件,不是主模型。建议启动时带上 `--jinja --reasoning on --reasoning-format deepseek`。Q4 长上下文场景优先关闭 MTP,把显存留给主模型和 KV cache。
|
| 225 |
+
|
| 226 |
+
### LynnStyle Dense
|
| 227 |
+
|
| 228 |
+
27B 是 Dense,没有 MoE expert,但不代表全层同质。Qwen3.6 27B 有混合注意力结构:full-attention 层约每 4 层出现一次,其他层更多是 linear attention / SSM。LynnStyle Dense 的保护单位是层位、attention pattern、tensor family。
|
| 229 |
+
|
| 230 |
+
Q5/Q4/Q3/Q2 不是一刀切低比特量化,而是显式保护关键层、full-attention、attention gate、SSM 和重要 tensor family,再对普通层按档位压缩,并叠加 imatrix 或等价重要度校准。只做 direct quant 不具备发布资格。
|
| 231 |
+
|
| 232 |
+
### 当前门禁
|
| 233 |
+
|
| 234 |
+
- Q8:MMLU500 92.4%,GPQA198 81.82%,LBC100 78/100,Coding100 clean 88/100。
|
| 235 |
+
- Q5:MMLU500 92.2%,GPQA198 78.79%,LBC100 75/100。
|
| 236 |
+
- Q4:MMLU500 92.0%,GPQA198 74.75%,LBC100 71/100,Coding100 clean 82/100。
|
| 237 |
+
|
| 238 |
+
训练不灌 benchmark 答案。SFT 学轨迹,RL 学行为偏好,MMLU / GPQA / LBC / Coding100 作为 held-out 门禁。
|
GGUF/assets/model_card_hero_hf_english.png
ADDED
|
Git LFS Details
|
GGUF/assets/model_card_hero_hf_english_20260707.png
ADDED
|
Git LFS Details
|