ggml_cuda_init: found 2 CUDA devices (Total VRAM: 32538 MiB): Device 0: Tesla P100-PCIE-16GB, compute capability 6.0, VMM: yes, VRAM: 16269 MiB Device 1: Tesla P100-PCIE-16GB, compute capability 6.0, VMM: yes, VRAM: 16269 MiB load_backend: loaded CUDA backend from /mnt/nvme1/llm/llama-eaddario-cuda-real/libggml-cuda.so load_backend: loaded RPC backend from /mnt/nvme1/llm/llama-eaddario-cuda-real/libggml-rpc.so load_backend: loaded CPU backend from /mnt/nvme1/llm/llama-eaddario-cuda-real/libggml-cpu-haswell.so common_init_result: fitting params to device memory, for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on llama_params_fit_impl: projected memory use with initial parameters [MiB]: llama_params_fit_impl: - CUDA0 (Tesla P100-PCIE-16GB): 16269 total, 23539 used, -7562 free vs. target of 2048 llama_params_fit_impl: - CUDA1 (Tesla P100-PCIE-16GB): 16269 total, 24083 used, -8102 free vs. target of 2048 llama_params_fit_impl: projected to use 47622 MiB of device memory vs. 31958 MiB of free device memory llama_params_fit_impl: cannot meet free memory targets on all devices, need to use 19760 MiB less in total llama_params_fit_impl: context size set by user to 2048 -> no change llama_params_fit_impl: with only dense weights in device memory there is a total surplus of 22234 MiB llama_params_fit_impl: filling dense-only layers back-to-front: llama_params_fit_impl: - CUDA1 (Tesla P100-PCIE-16GB): 41 layers, 5865 MiB used, 10115 MiB free llama_params_fit_impl: - CUDA0 (Tesla P100-PCIE-16GB): 0 layers, 801 MiB used, 15175 MiB free llama_params_fit_impl: converting dense-only layers to full layers and filling them front-to-back with overflow to next device/system memory: llama_params_fit_impl: - CUDA0 (Tesla P100-PCIE-16GB): 12 layers ( 1 overflowing), 13526 MiB used, 2450 MiB free llama_params_fit_impl: - CUDA1 (Tesla P100-PCIE-16GB): 29 layers (21 overflowing), 13747 MiB used, 2233 MiB free llama_params_fit: successfully fit params to free device memory llama_params_fit: fitting params to free memory took 6.25 seconds llama_model_load_from_file_impl: using device CUDA0 (Tesla P100-PCIE-16GB) (0000:01:00.0) - 16011 MiB free llama_model_load_from_file_impl: using device CUDA1 (Tesla P100-PCIE-16GB) (0000:04:00.0) - 16011 MiB free llama_model_loader: loaded meta data with 52 key-value pairs and 733 tensors from /mnt/ssd1/models/tvall43/Qwen3.6-35B-A3B-heretic/750-v1/Qwen3.6-35B-A3B-heretic-BPW11.00.gguf (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = qwen35moe llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.sampling.top_k i32 = 20 llama_model_loader: - kv 3: general.sampling.top_p f32 = 0.950000 llama_model_loader: - kv 4: general.sampling.temp f32 = 1.000000 llama_model_loader: - kv 5: general.name str = Qwen3.6 35B A3B Heretic llama_model_loader: - kv 6: general.finetune str = heretic llama_model_loader: - kv 7: general.basename str = Qwen3.6 llama_model_loader: - kv 8: general.size_label str = 35B-A3B llama_model_loader: - kv 9: general.license str = apache-2.0 llama_model_loader: - kv 10: general.license.link str = https://huggingface.co/Qwen/Qwen3.6-3... llama_model_loader: - kv 11: general.base_model.count u32 = 1 llama_model_loader: - kv 12: general.base_model.0.name str = Qwen3.6 35B A3B llama_model_loader: - kv 13: general.base_model.0.organization str = Qwen llama_model_loader: - kv 14: general.base_model.0.repo_url str = https://huggingface.co/Qwen/Qwen3.6-3... llama_model_loader: - kv 15: general.tags arr[str,7] = ["qwen3_5_moe", "qwen", "heretic", "u... llama_model_loader: - kv 16: qwen35moe.block_count u32 = 40 llama_model_loader: - kv 17: qwen35moe.context_length u32 = 262144 llama_model_loader: - kv 18: qwen35moe.embedding_length u32 = 2048 llama_model_loader: - kv 19: qwen35moe.attention.head_count u32 = 16 llama_model_loader: - kv 20: qwen35moe.attention.head_count_kv u32 = 2 llama_model_loader: - kv 21: qwen35moe.rope.dimension_sections arr[i32,4] = [11, 11, 10, 0] llama_model_loader: - kv 22: qwen35moe.rope.freq_base f32 = 10000000.000000 llama_model_loader: - kv 23: qwen35moe.attention.layer_norm_rms_epsilon f32 = 0.000001 llama_model_loader: - kv 24: qwen35moe.expert_count u32 = 256 llama_model_loader: - kv 25: qwen35moe.expert_used_count u32 = 8 llama_model_loader: - kv 26: qwen35moe.attention.key_length u32 = 256 llama_model_loader: - kv 27: qwen35moe.attention.value_length u32 = 256 llama_model_loader: - kv 28: qwen35moe.expert_feed_forward_length u32 = 512 llama_model_loader: - kv 29: qwen35moe.expert_shared_feed_forward_length u32 = 512 llama_model_loader: - kv 30: qwen35moe.ssm.conv_kernel u32 = 4 llama_model_loader: - kv 31: qwen35moe.ssm.state_size u32 = 128 llama_model_loader: - kv 32: qwen35moe.ssm.group_count u32 = 16 llama_model_loader: - kv 33: qwen35moe.ssm.time_step_rank u32 = 32 llama_model_loader: - kv 34: qwen35moe.ssm.inner_size u32 = 4096 llama_model_loader: - kv 35: qwen35moe.full_attention_interval u32 = 4 llama_model_loader: - kv 36: qwen35moe.rope.dimension_count u32 = 64 llama_model_loader: - kv 37: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 38: tokenizer.ggml.pre str = qwen35 llama_model_loader: - kv 39: tokenizer.ggml.tokens arr[str,248320] = ["!", "\"", "#", "$", "%", "&", "'", ... llama_model_loader: - kv 40: tokenizer.ggml.token_type arr[i32,248320] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 41: tokenizer.ggml.merges arr[str,247587] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",... llama_model_loader: - kv 42: tokenizer.ggml.eos_token_id u32 = 248046 llama_model_loader: - kv 43: tokenizer.ggml.padding_token_id u32 = 248055 llama_model_loader: - kv 44: tokenizer.ggml.bos_token_id u32 = 248044 llama_model_loader: - kv 45: tokenizer.chat_template str = {%- set image_count = namespace(value... llama_model_loader: - kv 46: general.quantization_version u32 = 2 llama_model_loader: - kv 47: general.file_type u32 = 1 llama_model_loader: - kv 48: quantize.imatrix.file str = /mnt/ssd1/models/tvall43/Qwen3.6-35B-... llama_model_loader: - kv 49: quantize.imatrix.dataset str = /home/duka/llm/models/tvall43/Qwen3.6... llama_model_loader: - kv 50: quantize.imatrix.entries_count u32 = 510 llama_model_loader: - kv 51: quantize.imatrix.chunks_count u32 = 197 llama_model_loader: - type f32: 301 tensors llama_model_loader: - type f16: 126 tensors llama_model_loader: - type q8_0: 306 tensors print_info: file format = GGUF V3 (latest) print_info: file type = F16 print_info: file size = 44.39 GiB (11.00 BPW) load: 0 unused tokens load: printing all EOG tokens: load: - 248044 ('<|endoftext|>') load: - 248046 ('<|im_end|>') load: - 248063 ('<|fim_pad|>') load: - 248064 ('<|repo_name|>') load: - 248065 ('<|file_sep|>') load: special tokens cache size = 33 load: token to piece cache size = 1.7581 MB print_info: arch = qwen35moe print_info: vocab_only = 0 print_info: no_alloc = 0 print_info: n_ctx_train = 262144 print_info: n_embd = 2048 print_info: n_embd_inp = 2048 print_info: n_layer = 40 print_info: n_head = 16 print_info: n_head_kv = 2 print_info: n_rot = 64 print_info: n_swa = 0 print_info: is_swa_any = 0 print_info: n_embd_head_k = 256 print_info: n_embd_head_v = 256 print_info: n_gqa = 8 print_info: n_embd_k_gqa = 512 print_info: n_embd_v_gqa = 512 print_info: f_norm_eps = 0.0e+00 print_info: f_norm_rms_eps = 1.0e-06 print_info: f_clamp_kqv = 0.0e+00 print_info: f_max_alibi_bias = 0.0e+00 print_info: f_logit_scale = 0.0e+00 print_info: f_attn_scale = 0.0e+00 print_info: n_ff = 0 print_info: n_expert = 256 print_info: n_expert_used = 8 print_info: n_expert_groups = 0 print_info: n_group_used = 0 print_info: causal attn = 1 print_info: pooling type = -1 print_info: rope type = 40 print_info: rope scaling = linear print_info: freq_base_train = 10000000.0 print_info: freq_scale_train = 1 print_info: n_ctx_orig_yarn = 262144 print_info: rope_yarn_log_mul = 0.0000 print_info: rope_finetuned = unknown print_info: mrope sections = [11, 11, 10, 0] print_info: ssm_d_conv = 4 print_info: ssm_d_inner = 4096 print_info: ssm_d_state = 128 print_info: ssm_dt_rank = 32 print_info: ssm_n_group = 16 print_info: ssm_dt_b_c_rms = 0 print_info: model type = 35B.A3B print_info: model params = 34.66 B print_info: general.name = Qwen3.6 35B A3B Heretic print_info: vocab type = BPE print_info: n_vocab = 248320 print_info: n_merges = 247587 print_info: BOS token = 248044 '<|endoftext|>' print_info: EOS token = 248046 '<|im_end|>' print_info: EOT token = 248046 '<|im_end|>' print_info: PAD token = 248055 '<|vision_pad|>' print_info: LF token = 198 'Ċ' print_info: FIM PRE token = 248060 '<|fim_prefix|>' print_info: FIM SUF token = 248062 '<|fim_suffix|>' print_info: FIM MID token = 248061 '<|fim_middle|>' print_info: FIM PAD token = 248063 '<|fim_pad|>' print_info: FIM REP token = 248064 '<|repo_name|>' print_info: FIM SEP token = 248065 '<|file_sep|>' print_info: EOG token = 248044 '<|endoftext|>' print_info: EOG token = 248046 '<|im_end|>' print_info: EOG token = 248063 '<|fim_pad|>' print_info: EOG token = 248064 '<|repo_name|>' print_info: EOG token = 248065 '<|file_sep|>' print_info: max token length = 256 load_tensors: loading model tensors, this can take a while... (mmap = false, direct_io = true) load_tensors: offloading output layer to GPU load_tensors: offloading 39 repeating layers to GPU load_tensors: offloaded 41/41 layers to GPU load_tensors: CUDA0 model buffer size = 12681.35 MiB load_tensors: CUDA1 model buffer size = 11675.82 MiB load_tensors: CUDA_Host model buffer size = 21093.31 MiB ................................................................................................ common_init_result: added <|endoftext|> logit bias = -inf common_init_result: added <|im_end|> logit bias = -inf common_init_result: added <|fim_pad|> logit bias = -inf common_init_result: added <|repo_name|> logit bias = -inf common_init_result: added <|file_sep|> logit bias = -inf llama_context: constructing llama_context llama_context: n_seq_max = 1 llama_context: n_ctx = 2048 llama_context: n_ctx_seq = 2048 llama_context: n_batch = 2048 llama_context: n_ubatch = 2048 llama_context: causal_attn = 1 llama_context: flash_attn = enabled llama_context: kv_unified = false llama_context: freq_base = 10000000.0 llama_context: freq_scale = 1 llama_context: n_ctx_seq (2048) < n_ctx_train (262144) -- the full capacity of the model will not be utilized llama_context: CUDA_Host output buffer size = 0.95 MiB llama_kv_cache: CUDA0 KV buffer size = 24.00 MiB llama_kv_cache: CUDA1 KV buffer size = 56.00 MiB llama_kv_cache: size = 80.00 MiB ( 2048 cells, 10 layers, 1/1 seqs), K (f32): 40.00 MiB, V (f32): 40.00 MiB llama_kv_cache: attn_rot_k = 0, n_embd_head_k_all = 256 llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 256 llama_memory_recurrent: CUDA0 RS buffer size = 18.84 MiB llama_memory_recurrent: CUDA1 RS buffer size = 43.97 MiB llama_memory_recurrent: size = 62.81 MiB ( 1 cells, 40 layers, 1 seqs), R (f32): 2.81 MiB, S (f32): 60.00 MiB sched_reserve: reserving ... sched_reserve: resolving fused Gated Delta Net support: sched_reserve: fused Gated Delta Net (autoregressive) enabled sched_reserve: fused Gated Delta Net (chunked) enabled sched_reserve: CUDA0 compute buffer size = 802.00 MiB sched_reserve: CUDA1 compute buffer size = 1972.00 MiB sched_reserve: CUDA_Host compute buffer size = 48.08 MiB sched_reserve: graph nodes = 3749 sched_reserve: graph splits = 87 (with bs=2048), 49 (with bs=1) sched_reserve: reserve took 47.24 ms, sched copies = 1 common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable) system_info: n_threads = 16 (n_threads_batch = 16) / 16 | CUDA : ARCHS = 600,610 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | FA_ALL_QUANTS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | kl_divergence: computing over 197 chunks, n_ctx=2048, batch_size=2048, n_seq=1 kl_divergence: 6.83 seconds per pass - ETA 22.42 minutes chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p 1 7.8120 ± 0.6296 -0.00031 ± 0.00442 0.00673 ± 0.00062 2.598 ± 0.255 % 96.188 ± 0.599 % 2 6.3932 ± 0.3481 -0.00002 ± 0.00288 0.00577 ± 0.00033 2.420 ± 0.142 % 96.285 ± 0.418 % 3 7.4971 ± 0.3546 -0.00023 ± 0.00250 0.00712 ± 0.00041 2.818 ± 0.223 % 95.960 ± 0.355 % 4 8.0902 ± 0.3421 0.00082 ± 0.00250 0.00826 ± 0.00086 2.775 ± 0.190 % 95.870 ± 0.311 % 5 7.9142 ± 0.2962 0.00084 ± 0.00215 0.00784 ± 0.00069 2.726 ± 0.156 % 95.816 ± 0.280 % 6 7.4267 ± 0.2506 -0.00020 ± 0.00186 0.00727 ± 0.00058 2.622 ± 0.136 % 96.074 ± 0.248 % 7 6.6214 ± 0.2012 -0.00002 ± 0.00171 0.00740 ± 0.00051 2.713 ± 0.128 % 95.531 ± 0.244 % 8 5.7769 ± 0.1577 -0.00080 ± 0.00153 0.00700 ± 0.00045 2.634 ± 0.117 % 94.550 ± 0.251 % 9 6.0352 ± 0.1569 -0.00032 ± 0.00143 0.00695 ± 0.00040 2.579 ± 0.106 % 94.580 ± 0.236 % 10 6.3013 ± 0.1579 0.00013 ± 0.00134 0.00683 ± 0.00036 2.542 ± 0.098 % 94.663 ± 0.222 % 11 6.7778 ± 0.1654 -0.00007 ± 0.00127 0.00677 ± 0.00033 2.530 ± 0.090 % 94.721 ± 0.211 % 12 6.9666 ± 0.1636 0.00038 ± 0.00120 0.00666 ± 0.00030 2.507 ± 0.084 % 94.835 ± 0.200 % 13 6.9180 ± 0.1561 0.00074 ± 0.00115 0.00651 ± 0.00028 2.503 ± 0.080 % 94.985 ± 0.189 % 14 7.0982 ± 0.1559 0.00071 ± 0.00110 0.00663 ± 0.00029 2.571 ± 0.104 % 95.064 ± 0.181 % 15 7.0589 ± 0.1498 0.00079 ± 0.00106 0.00674 ± 0.00028 2.604 ± 0.098 % 95.093 ± 0.174 % 16 7.0772 ± 0.1452 0.00072 ± 0.00102 0.00662 ± 0.00026 2.574 ± 0.093 % 95.192 ± 0.167 % 17 7.2190 ± 0.1448 0.00099 ± 0.00098 0.00732 ± 0.00076 2.553 ± 0.089 % 95.204 ± 0.162 % 18 7.0179 ± 0.1359 0.00129 ± 0.00095 0.00721 ± 0.00072 2.546 ± 0.084 % 95.297 ± 0.156 % 19 6.9872 ± 0.1319 0.00177 ± 0.00098 0.00739 ± 0.00075 2.596 ± 0.107 % 95.411 ± 0.150 % 20 7.0244 ± 0.1293 0.00205 ± 0.00095 0.00730 ± 0.00071 2.573 ± 0.102 % 95.464 ± 0.145 % 21 7.2329 ± 0.1310 0.00253 ± 0.00093 0.00728 ± 0.00068 2.579 ± 0.098 % 95.457 ± 0.142 % 22 7.2498 ± 0.1283 0.00243 ± 0.00090 0.00718 ± 0.00065 2.564 ± 0.094 % 95.490 ± 0.138 % 23 7.1043 ± 0.1228 0.00215 ± 0.00088 0.00736 ± 0.00067 2.603 ± 0.098 % 95.533 ± 0.135 % 24 7.1694 ± 0.1214 0.00212 ± 0.00085 0.00724 ± 0.00064 2.580 ± 0.095 % 95.556 ± 0.132 % 25 7.2642 ± 0.1208 0.00215 ± 0.00083 0.00717 ± 0.00061 2.563 ± 0.092 % 95.605 ± 0.128 % 26 7.2190 ± 0.1173 0.00201 ± 0.00081 0.00715 ± 0.00059 2.566 ± 0.089 % 95.631 ± 0.125 % 27 7.2116 ± 0.1147 0.00200 ± 0.00079 0.00710 ± 0.00057 2.555 ± 0.086 % 95.641 ± 0.123 % 28 7.2104 ± 0.1126 0.00219 ± 0.00078 0.00705 ± 0.00055 2.558 ± 0.084 % 95.619 ± 0.121 % 29 7.2337 ± 0.1110 0.00209 ± 0.00076 0.00709 ± 0.00054 2.562 ± 0.082 % 95.632 ± 0.119 % 30 7.1716 ± 0.1081 0.00234 ± 0.00075 0.00703 ± 0.00052 2.555 ± 0.080 % 95.627 ± 0.117 % 31 7.2613 ± 0.1079 0.00238 ± 0.00073 0.00697 ± 0.00050 2.541 ± 0.078 % 95.639 ± 0.115 % 32 7.2192 ± 0.1055 0.00271 ± 0.00072 0.00714 ± 0.00049 2.555 ± 0.076 % 95.620 ± 0.113 % 33 7.1534 ± 0.1027 0.00264 ± 0.00071 0.00728 ± 0.00051 2.578 ± 0.081 % 95.657 ± 0.111 % 34 7.2380 ± 0.1025 0.00276 ± 0.00070 0.00723 ± 0.00050 2.567 ± 0.079 % 95.644 ± 0.109 % 35 7.0844 ± 0.0985 0.00254 ± 0.00069 0.00722 ± 0.00049 2.575 ± 0.078 % 95.674 ± 0.108 % 36 7.0296 ± 0.0962 0.00260 ± 0.00068 0.00719 ± 0.00047 2.573 ± 0.076 % 95.683 ± 0.106 % 37 7.0962 ± 0.0960 0.00253 ± 0.00066 0.00716 ± 0.00046 2.561 ± 0.074 % 95.704 ± 0.104 % 38 7.0924 ± 0.0947 0.00255 ± 0.00065 0.00712 ± 0.00045 2.575 ± 0.075 % 95.727 ± 0.103 % 39 7.1438 ± 0.0943 0.00263 ± 0.00065 0.00732 ± 0.00050 2.563 ± 0.074 % 95.744 ± 0.101 % 40 7.1709 ± 0.0935 0.00265 ± 0.00064 0.00729 ± 0.00049 2.555 ± 0.072 % 95.740 ± 0.100 % 41 6.9809 ± 0.0895 0.00267 ± 0.00063 0.00723 ± 0.00048 2.553 ± 0.071 % 95.759 ± 0.098 % 42 7.0283 ± 0.0891 0.00265 ± 0.00063 0.00721 ± 0.00047 2.559 ± 0.070 % 95.771 ± 0.097 % 43 7.0248 ± 0.0881 0.00261 ± 0.00066 0.00744 ± 0.00047 2.606 ± 0.071 % 95.788 ± 0.096 % 44 6.9865 ± 0.0864 0.00263 ± 0.00065 0.00736 ± 0.00046 2.591 ± 0.070 % 95.812 ± 0.094 % 45 7.0413 ± 0.0862 0.00256 ± 0.00064 0.00731 ± 0.00045 2.580 ± 0.069 % 95.831 ± 0.093 % 46 7.0496 ± 0.0854 0.00249 ± 0.00063 0.00726 ± 0.00044 2.569 ± 0.068 % 95.822 ± 0.092 % 47 6.9861 ± 0.0842 0.00208 ± 0.00090 0.00939 ± 0.00063 2.863 ± 0.098 % 95.763 ± 0.092 % 48 7.0500 ± 0.0843 0.00206 ± 0.00088 0.00936 ± 0.00061 2.849 ± 0.096 % 95.746 ± 0.091 % 49 6.9203 ± 0.0816 0.00190 ± 0.00088 0.00966 ± 0.00060 2.970 ± 0.094 % 95.725 ± 0.090 % 50 6.9334 ± 0.0808 0.00203 ± 0.00087 0.00961 ± 0.00059 2.957 ± 0.093 % 95.728 ± 0.089 % 51 6.8958 ± 0.0795 0.00253 ± 0.00089 0.00992 ± 0.00063 2.966 ± 0.091 % 95.749 ± 0.088 % 52 6.8800 ± 0.0785 0.00264 ± 0.00087 0.00982 ± 0.00061 2.953 ± 0.090 % 95.768 ± 0.087 % 53 6.9210 ± 0.0782 0.00254 ± 0.00086 0.00996 ± 0.00063 2.972 ± 0.091 % 95.747 ± 0.087 % 54 6.8973 ± 0.0772 0.00252 ± 0.00085 0.00990 ± 0.00062 2.969 ± 0.089 % 95.768 ± 0.086 % 55 6.9322 ± 0.0769 0.00240 ± 0.00084 0.00981 ± 0.00061 2.956 ± 0.088 % 95.759 ± 0.085 % 56 6.9492 ± 0.0765 0.00244 ± 0.00082 0.00974 ± 0.00059 2.946 ± 0.087 % 95.764 ± 0.084 % 57 6.9612 ± 0.0760 0.00270 ± 0.00081 0.00969 ± 0.00058 2.943 ± 0.085 % 95.780 ± 0.083 % 58 7.0038 ± 0.0759 0.00262 ± 0.00080 0.00965 ± 0.00057 2.932 ± 0.084 % 95.761 ± 0.083 % 59 7.0393 ± 0.0757 0.00260 ± 0.00079 0.00962 ± 0.00056 2.932 ± 0.083 % 95.745 ± 0.082 % 60 7.0851 ± 0.0757 0.00269 ± 0.00078 0.00957 ± 0.00056 2.926 ± 0.082 % 95.738 ± 0.082 % 61 7.0593 ± 0.0748 0.00276 ± 0.00077 0.00951 ± 0.00055 2.927 ± 0.081 % 95.752 ± 0.081 % 62 7.0416 ± 0.0740 0.00273 ± 0.00077 0.00997 ± 0.00058 2.988 ± 0.084 % 95.732 ± 0.080 % 63 7.1021 ± 0.0742 0.00264 ± 0.00076 0.00990 ± 0.00057 2.975 ± 0.083 % 95.728 ± 0.080 % 64 7.0934 ± 0.0735 0.00278 ± 0.00076 0.00998 ± 0.00058 2.995 ± 0.085 % 95.720 ± 0.079 % 65 7.1307 ± 0.0733 0.00264 ± 0.00075 0.00994 ± 0.00057 2.992 ± 0.084 % 95.715 ± 0.079 % 66 7.1006 ± 0.0724 0.00261 ± 0.00075 0.00987 ± 0.00056 2.984 ± 0.083 % 95.730 ± 0.078 % 67 7.0521 ± 0.0713 0.00245 ± 0.00074 0.01005 ± 0.00056 3.028 ± 0.084 % 95.719 ± 0.077 % 68 7.0942 ± 0.0713 0.00258 ± 0.00073 0.01001 ± 0.00056 3.024 ± 0.083 % 95.718 ± 0.077 % 69 7.0723 ± 0.0705 0.00252 ± 0.00072 0.00994 ± 0.00055 3.013 ± 0.082 % 95.733 ± 0.076 % 70 6.9716 ± 0.0688 0.00256 ± 0.00071 0.00985 ± 0.00054 2.998 ± 0.081 % 95.772 ± 0.075 % 71 6.9681 ± 0.0683 0.00258 ± 0.00070 0.00978 ± 0.00053 2.985 ± 0.080 % 95.783 ± 0.075 % 72 6.9997 ± 0.0682 0.00255 ± 0.00070 0.00972 ± 0.00052 2.975 ± 0.080 % 95.795 ± 0.074 % 73 7.0342 ± 0.0681 0.00258 ± 0.00069 0.00967 ± 0.00052 2.966 ± 0.079 % 95.806 ± 0.073 % 74 7.0562 ± 0.0678 0.00252 ± 0.00068 0.00961 ± 0.00051 2.957 ± 0.078 % 95.805 ± 0.073 % 75 7.0056 ± 0.0668 0.00248 ± 0.00067 0.00954 ± 0.00050 2.949 ± 0.077 % 95.819 ± 0.072 % 76 7.0136 ± 0.0664 0.00258 ± 0.00067 0.00948 ± 0.00050 2.944 ± 0.076 % 95.830 ± 0.072 % 77 6.9963 ± 0.0658 0.00260 ± 0.00066 0.00943 ± 0.00049 2.939 ± 0.075 % 95.839 ± 0.071 % 78 7.0280 ± 0.0657 0.00260 ± 0.00065 0.00937 ± 0.00048 2.928 ± 0.075 % 95.834 ± 0.071 % 79 7.0594 ± 0.0656 0.00260 ± 0.00065 0.00939 ± 0.00048 2.931 ± 0.075 % 95.828 ± 0.070 % 80 7.0709 ± 0.0653 0.00254 ± 0.00064 0.00937 ± 0.00048 2.929 ± 0.074 % 95.821 ± 0.070 % 81 7.1035 ± 0.0653 0.00251 ± 0.00063 0.00933 ± 0.00047 2.920 ± 0.073 % 95.817 ± 0.070 % 82 7.1070 ± 0.0650 0.00253 ± 0.00063 0.00926 ± 0.00047 2.913 ± 0.073 % 95.830 ± 0.069 % 83 7.0548 ± 0.0641 0.00258 ± 0.00063 0.00934 ± 0.00046 2.934 ± 0.073 % 95.847 ± 0.068 % 84 7.0938 ± 0.0641 0.00255 ± 0.00062 0.00930 ± 0.00046 2.925 ± 0.073 % 95.837 ± 0.068 % 85 7.1062 ± 0.0639 0.00254 ± 0.00062 0.00927 ± 0.00045 2.917 ± 0.072 % 95.836 ± 0.068 % 86 7.1410 ± 0.0638 0.00258 ± 0.00061 0.00924 ± 0.00045 2.911 ± 0.071 % 95.823 ± 0.067 % 87 7.1506 ± 0.0636 0.00254 ± 0.00061 0.00920 ± 0.00044 2.904 ± 0.071 % 95.817 ± 0.067 % 88 7.1716 ± 0.0635 0.00258 ± 0.00061 0.00922 ± 0.00044 2.909 ± 0.070 % 95.814 ± 0.067 % 89 7.1756 ± 0.0632 0.00270 ± 0.00061 0.00928 ± 0.00044 2.921 ± 0.071 % 95.812 ± 0.066 % 90 7.1842 ± 0.0629 0.00275 ± 0.00061 0.00925 ± 0.00044 2.916 ± 0.071 % 95.818 ± 0.066 % 91 7.1692 ± 0.0624 0.00273 ± 0.00060 0.00919 ± 0.00044 2.907 ± 0.070 % 95.829 ± 0.066 % 92 7.1864 ± 0.0622 0.00274 ± 0.00060 0.00916 ± 0.00043 2.903 ± 0.070 % 95.826 ± 0.065 % 93 7.1669 ± 0.0617 0.00263 ± 0.00059 0.00915 ± 0.00043 2.903 ± 0.069 % 95.839 ± 0.065 % 94 7.2022 ± 0.0618 0.00260 ± 0.00059 0.00912 ± 0.00042 2.897 ± 0.068 % 95.825 ± 0.065 % 95 7.2172 ± 0.0616 0.00254 ± 0.00058 0.00910 ± 0.00042 2.892 ± 0.068 % 95.832 ± 0.064 % 96 7.2275 ± 0.0614 0.00236 ± 0.00058 0.00912 ± 0.00041 2.894 ± 0.067 % 95.835 ± 0.064 % 97 7.2446 ± 0.0613 0.00223 ± 0.00059 0.00921 ± 0.00042 2.907 ± 0.068 % 95.838 ± 0.063 % 98 7.2734 ± 0.0612 0.00220 ± 0.00058 0.00918 ± 0.00042 2.900 ± 0.067 % 95.836 ± 0.063 % 99 7.3071 ± 0.0613 0.00214 ± 0.00058 0.00915 ± 0.00041 2.893 ± 0.067 % 95.817 ± 0.063 % 100 7.2929 ± 0.0608 0.00221 ± 0.00058 0.00918 ± 0.00041 2.909 ± 0.067 % 95.819 ± 0.063 % 101 7.3090 ± 0.0607 0.00220 ± 0.00057 0.00915 ± 0.00041 2.902 ± 0.067 % 95.808 ± 0.062 % 102 7.3154 ± 0.0604 0.00225 ± 0.00057 0.00913 ± 0.00040 2.901 ± 0.066 % 95.816 ± 0.062 % 103 7.2914 ± 0.0599 0.00221 ± 0.00057 0.00909 ± 0.00040 2.896 ± 0.066 % 95.813 ± 0.062 % 104 7.2663 ± 0.0594 0.00233 ± 0.00056 0.00907 ± 0.00040 2.894 ± 0.065 % 95.819 ± 0.061 % 105 7.2504 ± 0.0589 0.00234 ± 0.00056 0.00906 ± 0.00039 2.890 ± 0.065 % 95.818 ± 0.061 % 106 7.2218 ± 0.0584 0.00236 ± 0.00055 0.00901 ± 0.00039 2.884 ± 0.064 % 95.824 ± 0.061 % 107 7.1983 ± 0.0579 0.00232 ± 0.00055 0.00903 ± 0.00039 2.895 ± 0.064 % 95.829 ± 0.060 % 108 7.1706 ± 0.0573 0.00238 ± 0.00055 0.00900 ± 0.00038 2.893 ± 0.064 % 95.834 ± 0.060 % 109 7.1832 ± 0.0571 0.00241 ± 0.00054 0.00898 ± 0.00038 2.888 ± 0.063 % 95.842 ± 0.060 % 110 7.1239 ± 0.0563 0.00240 ± 0.00054 0.00893 ± 0.00038 2.882 ± 0.063 % 95.863 ± 0.059 % 111 7.1244 ± 0.0560 0.00237 ± 0.00054 0.00890 ± 0.00037 2.882 ± 0.062 % 95.861 ± 0.059 % 112 7.1671 ± 0.0562 0.00241 ± 0.00053 0.00888 ± 0.00037 2.876 ± 0.062 % 95.864 ± 0.059 % 113 7.1480 ± 0.0558 0.00250 ± 0.00053 0.00886 ± 0.00037 2.875 ± 0.062 % 95.867 ± 0.059 % 114 7.1758 ± 0.0558 0.00255 ± 0.00053 0.00883 ± 0.00036 2.868 ± 0.061 % 95.870 ± 0.058 % 115 7.1852 ± 0.0556 0.00253 ± 0.00052 0.00880 ± 0.00036 2.863 ± 0.061 % 95.864 ± 0.058 % 116 7.1903 ± 0.0554 0.00255 ± 0.00052 0.00878 ± 0.00036 2.860 ± 0.060 % 95.861 ± 0.058 % 117 7.2119 ± 0.0554 0.00253 ± 0.00052 0.00876 ± 0.00036 2.857 ± 0.060 % 95.862 ± 0.058 % 118 7.2182 ± 0.0552 0.00255 ± 0.00051 0.00874 ± 0.00035 2.854 ± 0.060 % 95.857 ± 0.057 % 119 7.2384 ± 0.0551 0.00254 ± 0.00051 0.00871 ± 0.00035 2.849 ± 0.059 % 95.866 ± 0.057 % 120 7.2425 ± 0.0549 0.00255 ± 0.00051 0.00868 ± 0.00035 2.844 ± 0.059 % 95.872 ± 0.057 % 121 7.2604 ± 0.0548 0.00250 ± 0.00050 0.00867 ± 0.00034 2.842 ± 0.058 % 95.874 ± 0.057 % 122 7.2384 ± 0.0544 0.00251 ± 0.00050 0.00864 ± 0.00034 2.838 ± 0.058 % 95.885 ± 0.056 % 123 7.2262 ± 0.0541 0.00254 ± 0.00050 0.00863 ± 0.00034 2.837 ± 0.058 % 95.890 ± 0.056 % 124 7.2217 ± 0.0538 0.00256 ± 0.00049 0.00864 ± 0.00034 2.843 ± 0.057 % 95.881 ± 0.056 % 125 7.1498 ± 0.0530 0.00279 ± 0.00050 0.00878 ± 0.00034 2.884 ± 0.057 % 95.883 ± 0.056 % 126 7.1895 ± 0.0531 0.00275 ± 0.00050 0.00876 ± 0.00033 2.883 ± 0.057 % 95.879 ± 0.055 % 127 7.2002 ± 0.0530 0.00278 ± 0.00049 0.00874 ± 0.00033 2.879 ± 0.057 % 95.881 ± 0.055 % 128 7.2330 ± 0.0531 0.00276 ± 0.00049 0.00872 ± 0.00033 2.876 ± 0.056 % 95.885 ± 0.055 % 129 7.2077 ± 0.0527 0.00277 ± 0.00049 0.00870 ± 0.00032 2.872 ± 0.056 % 95.892 ± 0.055 % 130 7.2281 ± 0.0527 0.00283 ± 0.00049 0.00868 ± 0.00032 2.871 ± 0.056 % 95.887 ± 0.054 % 131 7.2496 ± 0.0526 0.00277 ± 0.00048 0.00872 ± 0.00032 2.876 ± 0.056 % 95.885 ± 0.054 % 132 7.2527 ± 0.0524 0.00280 ± 0.00048 0.00873 ± 0.00032 2.879 ± 0.056 % 95.880 ± 0.054 % 133 7.2490 ± 0.0522 0.00278 ± 0.00048 0.00872 ± 0.00032 2.875 ± 0.055 % 95.878 ± 0.054 % 134 7.2542 ± 0.0520 0.00278 ± 0.00048 0.00869 ± 0.00032 2.871 ± 0.055 % 95.881 ± 0.054 % 135 7.2501 ± 0.0518 0.00284 ± 0.00047 0.00867 ± 0.00031 2.867 ± 0.055 % 95.881 ± 0.053 % 136 7.2685 ± 0.0517 0.00280 ± 0.00047 0.00865 ± 0.00031 2.861 ± 0.055 % 95.874 ± 0.053 % 137 7.2717 ± 0.0516 0.00275 ± 0.00047 0.00863 ± 0.00031 2.859 ± 0.054 % 95.869 ± 0.053 % 138 7.2727 ± 0.0514 0.00276 ± 0.00046 0.00862 ± 0.00031 2.856 ± 0.054 % 95.866 ± 0.053 % 139 7.2705 ± 0.0512 0.00276 ± 0.00046 0.00860 ± 0.00031 2.853 ± 0.054 % 95.871 ± 0.053 % 140 7.2666 ± 0.0510 0.00269 ± 0.00046 0.00861 ± 0.00030 2.860 ± 0.053 % 95.875 ± 0.053 % 141 7.2790 ± 0.0509 0.00272 ± 0.00046 0.00859 ± 0.00030 2.854 ± 0.053 % 95.864 ± 0.052 % 142 7.3042 ± 0.0509 0.00275 ± 0.00046 0.00857 ± 0.00030 2.850 ± 0.053 % 95.862 ± 0.052 % 143 7.2591 ± 0.0504 0.00276 ± 0.00045 0.00853 ± 0.00030 2.845 ± 0.053 % 95.875 ± 0.052 % 144 7.2679 ± 0.0503 0.00279 ± 0.00045 0.00851 ± 0.00030 2.840 ± 0.052 % 95.879 ± 0.052 % 145 7.2720 ± 0.0501 0.00277 ± 0.00045 0.00850 ± 0.00029 2.837 ± 0.052 % 95.870 ± 0.052 % 146 7.2824 ± 0.0500 0.00275 ± 0.00045 0.00848 ± 0.00029 2.834 ± 0.052 % 95.864 ± 0.052 % 147 7.2867 ± 0.0499 0.00275 ± 0.00044 0.00847 ± 0.00029 2.831 ± 0.051 % 95.872 ± 0.051 % 148 7.2998 ± 0.0498 0.00274 ± 0.00044 0.00845 ± 0.00029 2.827 ± 0.051 % 95.873 ± 0.051 % 149 7.3049 ± 0.0497 0.00274 ± 0.00044 0.00844 ± 0.00029 2.823 ± 0.051 % 95.873 ± 0.051 % 150 7.2813 ± 0.0493 0.00287 ± 0.00044 0.00865 ± 0.00029 2.879 ± 0.053 % 95.876 ± 0.051 % 151 7.2799 ± 0.0491 0.00290 ± 0.00044 0.00863 ± 0.00029 2.876 ± 0.053 % 95.887 ± 0.051 % 152 7.2774 ± 0.0489 0.00291 ± 0.00044 0.00862 ± 0.00029 2.877 ± 0.053 % 95.897 ± 0.050 % 153 7.2781 ± 0.0488 0.00291 ± 0.00044 0.00862 ± 0.00029 2.877 ± 0.052 % 95.897 ± 0.050 % 154 7.2882 ± 0.0487 0.00298 ± 0.00044 0.00860 ± 0.00029 2.873 ± 0.052 % 95.889 ± 0.050 % 155 7.3090 ± 0.0487 0.00296 ± 0.00043 0.00859 ± 0.00029 2.868 ± 0.052 % 95.883 ± 0.050 % 156 7.3034 ± 0.0485 0.00295 ± 0.00043 0.00864 ± 0.00029 2.872 ± 0.052 % 95.878 ± 0.050 % 157 7.2988 ± 0.0483 0.00289 ± 0.00043 0.00863 ± 0.00029 2.869 ± 0.052 % 95.876 ± 0.050 % 158 7.3070 ± 0.0482 0.00288 ± 0.00043 0.00861 ± 0.00029 2.865 ± 0.051 % 95.878 ± 0.049 % 159 7.3162 ± 0.0481 0.00291 ± 0.00043 0.00858 ± 0.00029 2.861 ± 0.051 % 95.874 ± 0.049 % 160 7.3509 ± 0.0482 0.00295 ± 0.00042 0.00858 ± 0.00028 2.859 ± 0.051 % 95.871 ± 0.049 % 161 7.3674 ± 0.0482 0.00295 ± 0.00042 0.00855 ± 0.00028 2.856 ± 0.051 % 95.873 ± 0.049 % 162 7.3603 ± 0.0479 0.00295 ± 0.00042 0.00854 ± 0.00028 2.856 ± 0.050 % 95.879 ± 0.049 % 163 7.3540 ± 0.0477 0.00287 ± 0.00042 0.00853 ± 0.00028 2.859 ± 0.050 % 95.878 ± 0.049 % 164 7.2745 ± 0.0470 0.00286 ± 0.00042 0.00849 ± 0.00028 2.853 ± 0.050 % 95.894 ± 0.048 % 165 7.2915 ± 0.0470 0.00288 ± 0.00041 0.00848 ± 0.00028 2.850 ± 0.050 % 95.893 ± 0.048 % 166 7.3003 ± 0.0469 0.00285 ± 0.00041 0.00856 ± 0.00028 2.864 ± 0.051 % 95.891 ± 0.048 % 167 7.3175 ± 0.0469 0.00277 ± 0.00041 0.00857 ± 0.00028 2.862 ± 0.050 % 95.886 ± 0.048 % 168 7.3271 ± 0.0468 0.00275 ± 0.00042 0.00864 ± 0.00029 2.868 ± 0.051 % 95.889 ± 0.048 % 169 7.3288 ± 0.0467 0.00278 ± 0.00041 0.00862 ± 0.00029 2.864 ± 0.051 % 95.890 ± 0.048 % 170 7.3188 ± 0.0465 0.00260 ± 0.00042 0.00867 ± 0.00029 2.873 ± 0.051 % 95.888 ± 0.048 % 171 7.3041 ± 0.0463 0.00257 ± 0.00042 0.00867 ± 0.00029 2.876 ± 0.050 % 95.891 ± 0.047 % 172 7.3301 ± 0.0463 0.00258 ± 0.00042 0.00867 ± 0.00029 2.873 ± 0.050 % 95.875 ± 0.047 % 173 7.3422 ± 0.0463 0.00268 ± 0.00043 0.00898 ± 0.00030 2.927 ± 0.052 % 95.870 ± 0.047 % 174 7.3336 ± 0.0461 0.00268 ± 0.00042 0.00896 ± 0.00029 2.922 ± 0.052 % 95.868 ± 0.047 % 175 7.3485 ± 0.0461 0.00266 ± 0.00042 0.00895 ± 0.00029 2.919 ± 0.052 % 95.857 ± 0.047 % 176 7.3726 ± 0.0461 0.00268 ± 0.00042 0.00892 ± 0.00029 2.914 ± 0.051 % 95.847 ± 0.047 % 177 7.3689 ± 0.0459 0.00270 ± 0.00042 0.00893 ± 0.00029 2.915 ± 0.051 % 95.840 ± 0.047 % 178 7.3766 ± 0.0459 0.00271 ± 0.00042 0.00891 ± 0.00029 2.912 ± 0.051 % 95.837 ± 0.047 % 179 7.3114 ± 0.0453 0.00271 ± 0.00042 0.00888 ± 0.00029 2.909 ± 0.051 % 95.840 ± 0.047 % 180 7.3145 ± 0.0451 0.00273 ± 0.00041 0.00886 ± 0.00028 2.905 ± 0.050 % 95.837 ± 0.047 % 181 7.2905 ± 0.0448 0.00269 ± 0.00041 0.00884 ± 0.00028 2.906 ± 0.050 % 95.842 ± 0.046 % 182 7.2973 ± 0.0448 0.00272 ± 0.00041 0.00884 ± 0.00028 2.903 ± 0.050 % 95.844 ± 0.046 % 183 7.2975 ± 0.0447 0.00269 ± 0.00041 0.00890 ± 0.00029 2.906 ± 0.050 % 95.847 ± 0.046 % 184 7.2871 ± 0.0445 0.00270 ± 0.00041 0.00892 ± 0.00029 2.904 ± 0.050 % 95.848 ± 0.046 % 185 7.2898 ± 0.0443 0.00272 ± 0.00041 0.00891 ± 0.00029 2.901 ± 0.049 % 95.850 ± 0.046 % 186 7.3053 ± 0.0443 0.00272 ± 0.00040 0.00889 ± 0.00028 2.897 ± 0.049 % 95.850 ± 0.046 % 187 7.2962 ± 0.0441 0.00269 ± 0.00040 0.00891 ± 0.00028 2.904 ± 0.049 % 95.848 ± 0.046 % 188 7.3032 ± 0.0441 0.00265 ± 0.00040 0.00891 ± 0.00028 2.903 ± 0.049 % 95.835 ± 0.046 % 189 7.3181 ± 0.0441 0.00261 ± 0.00040 0.00890 ± 0.00028 2.900 ± 0.049 % 95.831 ± 0.045 % 190 7.3164 ± 0.0439 0.00257 ± 0.00040 0.00887 ± 0.00028 2.897 ± 0.048 % 95.834 ± 0.045 % 191 7.3354 ± 0.0439 0.00259 ± 0.00040 0.00885 ± 0.00028 2.892 ± 0.048 % 95.837 ± 0.045 % 192 7.3443 ± 0.0439 0.00257 ± 0.00040 0.00884 ± 0.00028 2.891 ± 0.048 % 95.838 ± 0.045 % 193 7.3266 ± 0.0436 0.00261 ± 0.00039 0.00886 ± 0.00028 2.898 ± 0.048 % 95.840 ± 0.045 % 194 7.3096 ± 0.0434 0.00267 ± 0.00039 0.00889 ± 0.00028 2.912 ± 0.049 % 95.842 ± 0.045 % 195 7.3285 ± 0.0434 0.00272 ± 0.00039 0.00888 ± 0.00027 2.911 ± 0.049 % 95.842 ± 0.045 % 196 7.3425 ± 0.0434 0.00273 ± 0.00039 0.00887 ± 0.00027 2.907 ± 0.048 % 95.844 ± 0.045 % 197 7.3735 ± 0.0435 0.00277 ± 0.00039 0.00885 ± 0.00027 2.903 ± 0.048 % 95.837 ± 0.044 % ====== Perplexity statistics ====== Mean PPL(Q) : 7.373544 ± 0.043529 Mean PPL(base) : 7.353122 ± 0.043360 Cor(ln(PPL(Q)), ln(PPL(base))): 99.78% Mean ln(PPL(Q)/PPL(base)) : 0.002773 ± 0.000389 Mean PPL(Q)/PPL(base) : 1.002777 ± 0.000390 Mean PPL(Q)-PPL(base) : 0.020422 ± 0.002869 ====== KL divergence statistics ====== Mean KLD: 0.008851 ± 0.000271 Maximum KLD: 15.418838 99.9% KLD: 0.575241 99.0% KLD: 0.057284 95.0% KLD: 0.019346 90.0% KLD: 0.012622 Median KLD: 0.003232 10.0% KLD: 0.000131 5.0% KLD: 0.000045 1.0% KLD: 0.000006 0.1% KLD: -0.000001 Minimum KLD: -0.000156 ====== Token probability statistics ====== Mean Δp: -0.044 ± 0.006 % Maximum Δp: 99.570% 99.9% Δp: 19.264% 99.0% Δp: 7.075% 95.0% Δp: 3.389% 90.0% Δp: 1.925% 75.0% Δp: 0.355% Median Δp: -0.001% 25.0% Δp: -0.440% 10.0% Δp: -2.096% 5.0% Δp: -3.566% 1.0% Δp: -7.339% 0.1% Δp: -18.180% Minimum Δp: -99.859% RMS Δp : 2.903 ± 0.048 % Same top p: 95.837 ± 0.044 % llama_perf_context_print: load time = 130297.35 ms llama_perf_context_print: prompt eval time = 1285110.71 ms / 403456 tokens ( 3.19 ms per token, 313.95 tokens per second) llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second) llama_perf_context_print: total time = 1319520.70 ms / 403457 tokens llama_perf_context_print: graphs reused = 196 llama_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted | llama_memory_breakdown_print: | - CUDA0 (P100-PCIE-16GB) | 16269 = 2257 + (13526 = 12681 + 42 + 801) + 485 | llama_memory_breakdown_print: | - CUDA1 (P100-PCIE-16GB) | 16269 = 1719 + (13747 = 11675 + 99 + 1972) + 802 | llama_memory_breakdown_print: | - Host | 21141 = 21093 + 0 + 48 |