perplexity-testing.md

# Perplexity Testing

Qwen3.6-35B-A3B-12GB-2.6763bpw was designed to maximize quality within a strict 12 GB VRAM budget rather than target a specific quantisation format. On WikiText testing, the quant increases perplexity from 6.74 to 7.42 (+0.68 PPL / +10.1%) compared to a Q4_K_XL reference while reducing evaluation time. The result is a highly practical deployment option for RTX 3060-class GPUs, retaining most of the capabilities of the original model while fitting comfortably within a 12 GB memory envelope.

Absolute increase: +0.6821 PPL
Relative increase: +10.1%

C:\Program Files\llama.cpp>llama-perplexity -m e:/models/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf -f wiki.test.raw
0.00.206.620 I common_init_result: fitting params to device memory ...
0.00.206.628 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.08.798.197 W llama_model_loader: tensor overrides to CPU are used with mmap enabled - consider using --no-mmap for better performance
0.21.379.885 W llama_context: n_ctx_seq (512) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
0.21.443.555 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
0.21.765.486 I
0.21.765.547 I system_info: n_threads = 6 (n_threads_batch = 6) / 12 | CUDA : ARCHS = 500,610,700,750,800,860,890,900 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
0.21.765.852 I perplexity: tokenizing the input ..
0.22.471.832 I perplexity: tokenization took 705.973 ms
0.22.473.109 I perplexity: calculating perplexity over 580 chunks, n_ctx=512, batch_size=2048, n_seq=4
0.39.651.964 I perplexity: 17.18 seconds per pass - ETA 41.50 minutes
22.33.363.469 I Final estimate: PPL = 6.7359 +/- 0.04376


C:\Program Files\llama.cpp>llama-perplexity -m e:/models/Qwen3.6-35B-A3B-12Gb-2.763bpw.gguf -f wiki.test.raw
0.00.166.211 I common_init_result: fitting params to device memory ...
0.00.166.217 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.00.166.282 E gguf_init_from_file: failed to open GGUF file 'e:/models/Qwen3.6-35B-A3B-12Gb-2.763bpw.gguf' (No such file or directory)
0.00.166.360
C:\Program Files\llama.cpp>llama-perplexity -m e:/models/Qwen3.6-35B-A3B-12Gb-2.6763bpw.gguf -f wiki.test.raw
0.00.162.132 I common_init_result: fitting params to device memory ...
0.00.162.138 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.07.816.085 W llama_model_loader: tensor overrides to CPU are used with mmap enabled - consider using --no-mmap for better performance
0.17.394.894 W llama_context: n_ctx_seq (512) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
0.17.441.598 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
0.17.626.631 I
0.17.626.698 I system_info: n_threads = 6 (n_threads_batch = 6) / 12 | CUDA : ARCHS = 500,610,700,750,800,860,890,900 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
0.17.626.714 I perplexity: tokenizing the input ..
0.18.313.303 I perplexity: tokenization took 686.585 ms
0.18.313.784 I perplexity: calculating perplexity over 580 chunks, n_ctx=512, batch_size=2048, n_seq=4
0.22.264.606 I perplexity: 3.95 seconds per pass - ETA 9.53 minutes
7.59.836.982 I Final estimate: PPL = 7.4180 +/- 0.04936