--- license: apache-2.0 language: - ja - en base_model: - google/gemma-4-12B-it-qat-q4_0-unquantized-assistant tags: - llama.cpp - mtp --- Gemma4は小さいモデルに先行させる事で推論を高速化するMTP(Multi-Token Prediction)という仕組みがあります。 500MB程度のメモリ使用量が増えますが1.5超の速度向上が期待できます このモデルは[dahara1/gemma-4-12B-it-qat-UD-japanese-imatrix](https://huggingface.co/dahara1/gemma-4-12B-it-qat-UD-japanese-imatrix)をMTPで動かすための小モデルです。 現時点ではQ4_0またはQ8_0が良さそうに見えます Gemma4 has a mechanism called MTP (Multi-Token Prediction) that speeds up inference by running a smaller model before the main model. While memory usage will increase by approximately 500MB, a speed improvement of over 1.5x can be expected. This model is a small model for running [dahara1/gemma-4-12B-it-qat-UD-japanese-imatrix](https://huggingface.co/dahara1/gemma-4-12B-it-qat-UD-japanese-imatrix) with MTP. At this point, Q4_0 or Q8_0 looks promising.　 # Benchmark Reslut (RTX 4060ti(16GB)) 私が使用したベンチマークスクリプトは以下です The benchmark script I used is as follows: [am17an/mtp-bench.py](https://gist.github.com/am17an/228edfb84ed082aa88e3865d6fa27090) [mtp_bench_jp.py](https://huggingface.co/dahara1/gemma-4-12B-it-qat-assistant/raw/main/mtp_bench_jp.py) japanese translated version by dahara1. ## llama-server command example ``` ./llama-server \ -m ./gemma-4-12B-it-UD-japanese-imatrix/gemma-4-12B-it-qat-ja-UD-Q4_K_XL.gguf \ --host 0.0.0.0 \ --port 8080 \ --temp 1.0 \ --top-p 0.95 \ --top-k 64 \ --min-p 0.0 \ --ctx-size 32000 \ --flash-attn on \ --jinja \ --model-draft ./gemma-4-423M-12b-it-qat-unquantized-assistant-Q4_0.gguf \ --chat-template-file ./gemma4-12b/chat_template.jinja \ --spec-type draft-mtp \ --reasoning on ``` ## Q4_0 mtp_bench.py result. ``` code_python pred= 192 draft= 185 acc= 129 rate=0.697 tok/s=77.1 code_cpp pred= 192 draft= 189 acc= 127 rate=0.672 tok/s=76.4 explain_concept pred= 192 draft= 189 acc= 127 rate=0.672 tok/s=77.2 summarize pred= 192 draft= 169 acc= 134 rate=0.793 tok/s=85.3 qa_factual pred= 192 draft= 184 acc= 129 rate=0.701 tok/s=77.6 translation pred= 192 draft= 197 acc= 125 rate=0.634 tok/s=72.3 creative_short pred= 192 draft= 237 acc= 111 rate=0.468 tok/s=60.1 stepwise_math pred= 192 draft= 165 acc= 136 rate=0.824 tok/s=87.7 long_code_review pred= 192 draft= 197 acc= 125 rate=0.634 tok/s=70.6 Aggregate: { "n_requests": 9, "total_predicted": 1728, "total_draft": 1712, "total_draft_accepted": 1143, "aggregate_accept_rate": 0.6676, "wall_s_total": 24.8 } ``` ## Q4_0 mtp_bench_jp.py result. ``` code_python pred= 192 draft= 184 acc= 129 rate=0.701 tok/s=78.3 code_cpp pred= 192 draft= 195 acc= 125 rate=0.641 tok/s=74.8 explain_concept pred= 192 draft= 212 acc= 120 rate=0.566 tok/s=71.0 summarize pred= 192 draft= 164 acc= 136 rate=0.829 tok/s=88.3 qa_factual pred= 192 draft= 204 acc= 121 rate=0.593 tok/s=71.1 translation pred= 192 draft= 189 acc= 128 rate=0.677 tok/s=77.2 creative_short pred= 192 draft= 239 acc= 110 rate=0.460 tok/s=60.9 stepwise_math pred= 192 draft= 195 acc= 124 rate=0.636 tok/s=73.3 long_code_review pred= 192 draft= 200 acc= 123 rate=0.615 tok/s=70.0 Aggregate: { "n_requests": 9, "total_predicted": 1728, "total_draft": 1782, "total_draft_accepted": 1116, "aggregate_accept_rate": 0.6263, "wall_s_total": 25.83 } ``` ## Q8_0 mtp_bench.py result. ``` code_python pred= 192 draft= 182 acc= 130 rate=0.714 tok/s=75.5 code_cpp pred= 192 draft= 180 acc= 130 rate=0.722 tok/s=77.6 explain_concept pred= 192 draft= 189 acc= 127 rate=0.672 tok/s=71.0 summarize pred= 192 draft= 169 acc= 134 rate=0.793 tok/s=79.1 qa_factual pred= 192 draft= 181 acc= 130 rate=0.718 tok/s=74.9 translation pred= 192 draft= 197 acc= 125 rate=0.634 tok/s=70.6 creative_short pred= 192 draft= 240 acc= 110 rate=0.458 tok/s=58.2 stepwise_math pred= 192 draft= 168 acc= 135 rate=0.804 tok/s=81.5 long_code_review pred= 192 draft= 200 acc= 124 rate=0.620 tok/s=66.8 Aggregate: { "n_requests": 9, "total_predicted": 1728, "total_draft": 1706, "total_draft_accepted": 1145, "aggregate_accept_rate": 0.6712, "wall_s_total": 25.93 } ``` ## Q8_0 mtp_bench_jp.py result. ``` code_python pred= 192 draft= 190 acc= 127 rate=0.668 tok/s=72.8 code_cpp pred= 192 draft= 198 acc= 124 rate=0.626 tok/s=70.8 explain_concept pred= 192 draft= 212 acc= 120 rate=0.566 tok/s=66.1 summarize pred= 192 draft= 164 acc= 136 rate=0.829 tok/s=85.4 qa_factual pred= 192 draft= 207 acc= 120 rate=0.580 tok/s=67.3 translation pred= 192 draft= 183 acc= 130 rate=0.710 tok/s=78.3 creative_short pred= 192 draft= 239 acc= 110 rate=0.460 tok/s=59.5 stepwise_math pred= 192 draft= 191 acc= 126 rate=0.660 tok/s=72.5 long_code_review pred= 192 draft= 203 acc= 122 rate=0.601 tok/s=66.7 Aggregate: { "n_requests": 9, "total_predicted": 1728, "total_draft": 1787, "total_draft_accepted": 1115, "aggregate_accept_rate": 0.624, "wall_s_total": 26.81 } ``` ## F16 mtp_bench.py result. ``` code_python pred= 192 draft= 182 acc= 130 rate=0.714 tok/s=68.5 code_cpp pred= 192 draft= 183 acc= 129 rate=0.705 tok/s=68.6 explain_concept pred= 192 draft= 189 acc= 127 rate=0.672 tok/s=66.3 summarize pred= 192 draft= 169 acc= 134 rate=0.793 tok/s=73.5 qa_factual pred= 192 draft= 184 acc= 129 rate=0.701 tok/s=67.9 translation pred= 192 draft= 197 acc= 125 rate=0.634 tok/s=63.7 creative_short pred= 192 draft= 237 acc= 111 rate=0.468 tok/s=53.1 stepwise_math pred= 192 draft= 168 acc= 135 rate=0.804 tok/s=75.6 long_code_review pred= 192 draft= 200 acc= 124 rate=0.620 tok/s=61.7 Aggregate: { "n_requests": 9, "total_predicted": 1728, "total_draft": 1709, "total_draft_accepted": 1144, "aggregate_accept_rate": 0.6694, "wall_s_total": 28.04 } ``` ## F16 mtp_bench_jp.py result. ``` code_python pred= 192 draft= 190 acc= 127 rate=0.668 tok/s=68.2 code_cpp pred= 192 draft= 198 acc= 124 rate=0.626 tok/s=65.5 explain_concept pred= 192 draft= 212 acc= 120 rate=0.566 tok/s=60.3 summarize pred= 192 draft= 164 acc= 136 rate=0.829 tok/s=77.2 qa_factual pred= 192 draft= 208 acc= 120 rate=0.577 tok/s=60.6 translation pred= 192 draft= 183 acc= 130 rate=0.710 tok/s=70.7 creative_short pred= 192 draft= 239 acc= 110 rate=0.460 tok/s=54.5 stepwise_math pred= 192 draft= 195 acc= 124 rate=0.636 tok/s=65.3 long_code_review pred= 192 draft= 203 acc= 122 rate=0.601 tok/s=61.2 Aggregate: { "n_requests": 9, "total_predicted": 1728, "total_draft": 1792, "total_draft_accepted": 1113, "aggregate_accept_rate": 0.6211, "wall_s_total": 29.13 } ``` ## BF16 mtp_bench.py result. ``` code_python pred= 192 draft= 182 acc= 130 rate=0.714 tok/s=68.9 code_cpp pred= 192 draft= 183 acc= 129 rate=0.705 tok/s=68.8 explain_concept pred= 192 draft= 189 acc= 127 rate=0.672 tok/s=64.7 summarize pred= 192 draft= 169 acc= 134 rate=0.793 tok/s=72.2 qa_factual pred= 192 draft= 184 acc= 129 rate=0.701 tok/s=67.5 translation pred= 192 draft= 197 acc= 125 rate=0.634 tok/s=63.3 creative_short pred= 192 draft= 237 acc= 111 rate=0.468 tok/s=53.2 stepwise_math pred= 192 draft= 168 acc= 135 rate=0.804 tok/s=75.8 long_code_review pred= 192 draft= 200 acc= 124 rate=0.620 tok/s=60.8 Aggregate: { "n_requests": 9, "total_predicted": 1728, "total_draft": 1709, "total_draft_accepted": 1144, "aggregate_accept_rate": 0.6694, "wall_s_total": 28.22 } ``` ## BF16 mtp_bench_jp.py result. ``` code_python pred= 192 draft= 190 acc= 127 rate=0.668 tok/s=65.9 code_cpp pred= 192 draft= 198 acc= 124 rate=0.626 tok/s=63.5 explain_concept pred= 192 draft= 212 acc= 120 rate=0.566 tok/s=59.9 summarize pred= 192 draft= 164 acc= 136 rate=0.829 tok/s=74.1 qa_factual pred= 192 draft= 208 acc= 120 rate=0.577 tok/s=60.8 translation pred= 192 draft= 183 acc= 130 rate=0.710 tok/s=71.1 creative_short pred= 192 draft= 239 acc= 110 rate=0.460 tok/s=53.0 stepwise_math pred= 192 draft= 195 acc= 124 rate=0.636 tok/s=63.9 long_code_review pred= 192 draft= 203 acc= 122 rate=0.601 tok/s=60.6 Aggregate: { "n_requests": 9, "total_predicted": 1728, "total_draft": 1792, "total_draft_accepted": 1113, "aggregate_accept_rate": 0.6211, "wall_s_total": 29.63 } ```