dahara1/gemma-4-12B-it-qat-assistant_gguf

Gemma4は小さいモデルに先行させる事で推論を高速化するMTP(Multi-Token Prediction)という仕組みがあります。
500MB程度のメモリ使用量が増えますが1.5超の速度向上が期待できます
このモデルはdahara1/gemma-4-12B-it-qat-UD-japanese-imatrixをMTPで動かすための小モデルです。
現時点ではQ4_0またはQ8_0が良さそうに見えます

Gemma4 has a mechanism called MTP (Multi-Token Prediction) that speeds up inference by running a smaller model before the main model.
While memory usage will increase by approximately 500MB, a speed improvement of over 1.5x can be expected.
This model is a small model for running dahara1/gemma-4-12B-it-qat-UD-japanese-imatrix with MTP.
At this point, Q4_0 or Q8_0 looks promising.　

Benchmark Reslut (RTX 4060ti(16GB))

私が使用したベンチマークスクリプトは以下です
The benchmark script I used is as follows:
am17an/mtp-bench.py
mtp_bench_jp.py japanese translated version by dahara1.

llama-server command example

./llama-server \
  -m ./gemma-4-12B-it-UD-japanese-imatrix/gemma-4-12B-it-qat-ja-UD-Q4_K_XL.gguf  \
  --host 0.0.0.0 \
  --port 8080 \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 64 \
  --min-p 0.0 \
  --ctx-size 32000 \
  --flash-attn on \
  --jinja \
  --model-draft ./gemma-4-423M-12b-it-qat-unquantized-assistant-Q4_0.gguf \
  --chat-template-file ./gemma4-12b/chat_template.jinja \
  --spec-type draft-mtp \
  --reasoning on

Q4_0 mtp_bench.py result.

  code_python        pred= 192 draft= 185 acc= 129 rate=0.697 tok/s=77.1
  code_cpp           pred= 192 draft= 189 acc= 127 rate=0.672 tok/s=76.4
  explain_concept    pred= 192 draft= 189 acc= 127 rate=0.672 tok/s=77.2
  summarize          pred= 192 draft= 169 acc= 134 rate=0.793 tok/s=85.3
  qa_factual         pred= 192 draft= 184 acc= 129 rate=0.701 tok/s=77.6
  translation        pred= 192 draft= 197 acc= 125 rate=0.634 tok/s=72.3
  creative_short     pred= 192 draft= 237 acc= 111 rate=0.468 tok/s=60.1
  stepwise_math      pred= 192 draft= 165 acc= 136 rate=0.824 tok/s=87.7
  long_code_review   pred= 192 draft= 197 acc= 125 rate=0.634 tok/s=70.6

Aggregate: {
  "n_requests": 9,
  "total_predicted": 1728,
  "total_draft": 1712,
  "total_draft_accepted": 1143,
  "aggregate_accept_rate": 0.6676,
  "wall_s_total": 24.8
}

Q4_0 mtp_bench_jp.py result.

  code_python        pred= 192 draft= 184 acc= 129 rate=0.701 tok/s=78.3
  code_cpp           pred= 192 draft= 195 acc= 125 rate=0.641 tok/s=74.8
  explain_concept    pred= 192 draft= 212 acc= 120 rate=0.566 tok/s=71.0
  summarize          pred= 192 draft= 164 acc= 136 rate=0.829 tok/s=88.3
  qa_factual         pred= 192 draft= 204 acc= 121 rate=0.593 tok/s=71.1
  translation        pred= 192 draft= 189 acc= 128 rate=0.677 tok/s=77.2
  creative_short     pred= 192 draft= 239 acc= 110 rate=0.460 tok/s=60.9
  stepwise_math      pred= 192 draft= 195 acc= 124 rate=0.636 tok/s=73.3
  long_code_review   pred= 192 draft= 200 acc= 123 rate=0.615 tok/s=70.0

Aggregate: {
  "n_requests": 9,
  "total_predicted": 1728,
  "total_draft": 1782,
  "total_draft_accepted": 1116,
  "aggregate_accept_rate": 0.6263,
  "wall_s_total": 25.83
}

Q8_0 mtp_bench.py result.

  code_python        pred= 192 draft= 182 acc= 130 rate=0.714 tok/s=75.5
  code_cpp           pred= 192 draft= 180 acc= 130 rate=0.722 tok/s=77.6
  explain_concept    pred= 192 draft= 189 acc= 127 rate=0.672 tok/s=71.0
  summarize          pred= 192 draft= 169 acc= 134 rate=0.793 tok/s=79.1
  qa_factual         pred= 192 draft= 181 acc= 130 rate=0.718 tok/s=74.9
  translation        pred= 192 draft= 197 acc= 125 rate=0.634 tok/s=70.6
  creative_short     pred= 192 draft= 240 acc= 110 rate=0.458 tok/s=58.2
  stepwise_math      pred= 192 draft= 168 acc= 135 rate=0.804 tok/s=81.5
  long_code_review   pred= 192 draft= 200 acc= 124 rate=0.620 tok/s=66.8

Aggregate: {
  "n_requests": 9,
  "total_predicted": 1728,
  "total_draft": 1706,
  "total_draft_accepted": 1145,
  "aggregate_accept_rate": 0.6712,
  "wall_s_total": 25.93
}

Q8_0 mtp_bench_jp.py result.

  code_python        pred= 192 draft= 190 acc= 127 rate=0.668 tok/s=72.8
  code_cpp           pred= 192 draft= 198 acc= 124 rate=0.626 tok/s=70.8
  explain_concept    pred= 192 draft= 212 acc= 120 rate=0.566 tok/s=66.1
  summarize          pred= 192 draft= 164 acc= 136 rate=0.829 tok/s=85.4
  qa_factual         pred= 192 draft= 207 acc= 120 rate=0.580 tok/s=67.3
  translation        pred= 192 draft= 183 acc= 130 rate=0.710 tok/s=78.3
  creative_short     pred= 192 draft= 239 acc= 110 rate=0.460 tok/s=59.5
  stepwise_math      pred= 192 draft= 191 acc= 126 rate=0.660 tok/s=72.5
  long_code_review   pred= 192 draft= 203 acc= 122 rate=0.601 tok/s=66.7

Aggregate: {
  "n_requests": 9,
  "total_predicted": 1728,
  "total_draft": 1787,
  "total_draft_accepted": 1115,
  "aggregate_accept_rate": 0.624,
  "wall_s_total": 26.81
}

F16 mtp_bench.py result.

  code_python        pred= 192 draft= 182 acc= 130 rate=0.714 tok/s=68.5
  code_cpp           pred= 192 draft= 183 acc= 129 rate=0.705 tok/s=68.6
  explain_concept    pred= 192 draft= 189 acc= 127 rate=0.672 tok/s=66.3
  summarize          pred= 192 draft= 169 acc= 134 rate=0.793 tok/s=73.5
  qa_factual         pred= 192 draft= 184 acc= 129 rate=0.701 tok/s=67.9
  translation        pred= 192 draft= 197 acc= 125 rate=0.634 tok/s=63.7
  creative_short     pred= 192 draft= 237 acc= 111 rate=0.468 tok/s=53.1
  stepwise_math      pred= 192 draft= 168 acc= 135 rate=0.804 tok/s=75.6
  long_code_review   pred= 192 draft= 200 acc= 124 rate=0.620 tok/s=61.7

Aggregate: {
  "n_requests": 9,
  "total_predicted": 1728,
  "total_draft": 1709,
  "total_draft_accepted": 1144,
  "aggregate_accept_rate": 0.6694,
  "wall_s_total": 28.04
}

F16 mtp_bench_jp.py result.

  code_python        pred= 192 draft= 190 acc= 127 rate=0.668 tok/s=68.2
  code_cpp           pred= 192 draft= 198 acc= 124 rate=0.626 tok/s=65.5
  explain_concept    pred= 192 draft= 212 acc= 120 rate=0.566 tok/s=60.3
  summarize          pred= 192 draft= 164 acc= 136 rate=0.829 tok/s=77.2
  qa_factual         pred= 192 draft= 208 acc= 120 rate=0.577 tok/s=60.6
  translation        pred= 192 draft= 183 acc= 130 rate=0.710 tok/s=70.7
  creative_short     pred= 192 draft= 239 acc= 110 rate=0.460 tok/s=54.5
  stepwise_math      pred= 192 draft= 195 acc= 124 rate=0.636 tok/s=65.3
  long_code_review   pred= 192 draft= 203 acc= 122 rate=0.601 tok/s=61.2

Aggregate: {
  "n_requests": 9,
  "total_predicted": 1728,
  "total_draft": 1792,
  "total_draft_accepted": 1113,
  "aggregate_accept_rate": 0.6211,
  "wall_s_total": 29.13
}

BF16 mtp_bench.py result.

  code_python        pred= 192 draft= 182 acc= 130 rate=0.714 tok/s=68.9
  code_cpp           pred= 192 draft= 183 acc= 129 rate=0.705 tok/s=68.8
  explain_concept    pred= 192 draft= 189 acc= 127 rate=0.672 tok/s=64.7
  summarize          pred= 192 draft= 169 acc= 134 rate=0.793 tok/s=72.2
  qa_factual         pred= 192 draft= 184 acc= 129 rate=0.701 tok/s=67.5
  translation        pred= 192 draft= 197 acc= 125 rate=0.634 tok/s=63.3
  creative_short     pred= 192 draft= 237 acc= 111 rate=0.468 tok/s=53.2
  stepwise_math      pred= 192 draft= 168 acc= 135 rate=0.804 tok/s=75.8
  long_code_review   pred= 192 draft= 200 acc= 124 rate=0.620 tok/s=60.8

Aggregate: {
  "n_requests": 9,
  "total_predicted": 1728,
  "total_draft": 1709,
  "total_draft_accepted": 1144,
  "aggregate_accept_rate": 0.6694,
  "wall_s_total": 28.22
}

BF16 mtp_bench_jp.py result.

  code_python        pred= 192 draft= 190 acc= 127 rate=0.668 tok/s=65.9
  code_cpp           pred= 192 draft= 198 acc= 124 rate=0.626 tok/s=63.5
  explain_concept    pred= 192 draft= 212 acc= 120 rate=0.566 tok/s=59.9
  summarize          pred= 192 draft= 164 acc= 136 rate=0.829 tok/s=74.1
  qa_factual         pred= 192 draft= 208 acc= 120 rate=0.577 tok/s=60.8
  translation        pred= 192 draft= 183 acc= 130 rate=0.710 tok/s=71.1
  creative_short     pred= 192 draft= 239 acc= 110 rate=0.460 tok/s=53.0
  stepwise_math      pred= 192 draft= 195 acc= 124 rate=0.636 tok/s=63.9
  long_code_review   pred= 192 draft= 203 acc= 122 rate=0.601 tok/s=60.6

Aggregate: {
  "n_requests": 9,
  "total_predicted": 1728,
  "total_draft": 1792,
  "total_draft_accepted": 1113,
  "aggregate_accept_rate": 0.6211,
  "wall_s_total": 29.63
}