perplexity-testing.md

Perplexity Testing

Qwen3.6-35B-A3B-12GB-2.6763bpw was designed to maximize quality within a strict 12 GB VRAM budget rather than target a specific quantisation format. On WikiText testing, the quant increases perplexity from 6.74 to 7.42 (+0.68 PPL / +10.1%) compared to a Q4_K_XL reference while reducing evaluation time. The result is a highly practical deployment option for RTX 3060-class GPUs, retaining most of the capabilities of the original model while fitting comfortably within a 12 GB memory envelope.

Absolute increase: +0.6821 PPL Relative increase: +10.1%

C:\Program Files\llama.cpp>llama-perplexity -m e:/models/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf -f wiki.test.raw 0.00.206.620 I common_init_result: fitting params to device memory ... 0.00.206.628 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on) 0.08.798.197 W llama_model_loader: tensor overrides to CPU are used with mmap enabled - consider using --no-mmap for better performance 0.21.379.885 W llama_context: n_ctx_seq (512) < n_ctx_train (262144) -- the full capacity of the model will not be utilized 0.21.443.555 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable) 0.21.765.486 I 0.21.765.547 I system_info: n_threads = 6 (n_threads_batch = 6) / 12 | CUDA : ARCHS = 500,610,700,750,800,860,890,900 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 0.21.765.852 I perplexity: tokenizing the input .. 0.22.471.832 I perplexity: tokenization took 705.973 ms 0.22.473.109 I perplexity: calculating perplexity over 580 chunks, n_ctx=512, batch_size=2048, n_seq=4 0.39.651.964 I perplexity: 17.18 seconds per pass - ETA 41.50 minutes 22.33.363.469 I Final estimate: PPL = 6.7359 +/- 0.04376

C:\Program Files\llama.cpp>llama-perplexity -m e:/models/Qwen3.6-35B-A3B-12Gb-2.763bpw.gguf -f wiki.test.raw 0.00.166.211 I common_init_result: fitting params to device memory ... 0.00.166.217 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on) 0.00.166.282 E gguf_init_from_file: failed to open GGUF file 'e:/models/Qwen3.6-35B-A3B-12Gb-2.763bpw.gguf' (No such file or directory) 0.00.166.360 C:\Program Files\llama.cpp>llama-perplexity -m e:/models/Qwen3.6-35B-A3B-12Gb-2.6763bpw.gguf -f wiki.test.raw 0.00.162.132 I common_init_result: fitting params to device memory ... 0.00.162.138 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on) 0.07.816.085 W llama_model_loader: tensor overrides to CPU are used with mmap enabled - consider using --no-mmap for better performance 0.17.394.894 W llama_context: n_ctx_seq (512) < n_ctx_train (262144) -- the full capacity of the model will not be utilized 0.17.441.598 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable) 0.17.626.631 I 0.17.626.698 I system_info: n_threads = 6 (n_threads_batch = 6) / 12 | CUDA : ARCHS = 500,610,700,750,800,860,890,900 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 0.17.626.714 I perplexity: tokenizing the input .. 0.18.313.303 I perplexity: tokenization took 686.585 ms 0.18.313.784 I perplexity: calculating perplexity over 580 chunks, n_ctx=512, batch_size=2048, n_seq=4 0.22.264.606 I perplexity: 3.95 seconds per pass - ETA 9.53 minutes 7.59.836.982 I Final estimate: PPL = 7.4180 +/- 0.04936