[budget] waiting for GPU lock... [budget] holding GPU lock === evicting prod for the window === GPU drain: max used 0 MiB before boot === BOOT candidate: /home/ian/models/GLM-5.3-Flash-NVFP4-FP8ATTN-max on local/vllm-glm53:r6-max maxlen=524288 KVB=3300000000 === === BOOT LEDGER === (APIServer pid=56) INFO 08-30 20:49:44 [speculative.py:1256] Overriding draft model max model len from 1048576 to 524288 (APIServer pid=56) INFO 08-30 20:49:44 [vllm.py:1319] Auto-enabling VLLM_USE_BREAKABLE_CUDAGRAPH=1. Set VLLM_USE_BREAKABLE_CUDAGRAPH=0 to opt out. (APIServer pid=56) WARNING 08-30 20:49:44 [vllm.py:1849] max_num_scheduled_tokens is set to 1024 based on the speculative decoding settings. This may lead to suboptimal performance. Consider increasing max_num_batched_tokens to accommodate the additional draft token slots, or decrease num_speculative_tokens. (EngineCore pid=532) INFO 08-30 20:49:52 [core.py:122] Initializing a V1 LLM engine (v0.1.dev20051+g487ecf187) with config: model='/model', speculative_config=SpeculativeConfig(method='mtp', model='/model', num_spec_tokens=1), tokenizer='/model', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=524288, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=modelopt_mixed, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=fp8, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='glm45', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=glm-max, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [1024], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 2, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=False, enable_cutedsl_warmup=False, enable_jit_warmup=False, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto') (Worker pid=666) WARNING 08-30 20:50:04 [symm_mem.py:67] SymmMemCommunicator: Device capability 12.0 not supported, communicator is not available. (Worker pid=817) WARNING 08-30 20:50:04 [symm_mem.py:67] SymmMemCommunicator: Device capability 12.0 not supported, communicator is not available. (Worker_TP0 pid=666) INFO 08-30 20:50:05 [mm_encoder_attention.py:375] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention. (Worker_TP0 pid=666) WARNING 08-30 20:50:05 [vllm.py:1849] max_num_scheduled_tokens is set to 1024 based on the speculative decoding settings. This may lead to suboptimal performance. Consider increasing max_num_batched_tokens to accommodate the additional draft token slots, or decrease num_speculative_tokens. (Worker_TP0 pid=666) INFO 08-30 20:50:05 [deep_gemm.py:191] deep_gemm not found in site-packages, trying vendored vllm.third_party.deep_gemm (Worker_TP0 pid=666) INFO 08-30 20:50:05 [deep_gemm.py:218] DeepGEMM PDL enabled on vllm.third_party.deep_gemm. (Worker_TP0 pid=666) INFO 08-30 20:50:05 [deep_gemm.py:136] DeepGEMM E8M0 enabled on current platform. (Worker_TP0 pid=666) INFO 08-30 20:50:05 [weight_utils.py:858] Filesystem type for checkpoints: EXT4. Checkpoint size: 173.87 GiB. Available RAM: 396.76 GiB. [rank1]:[W830 20:50:55.322712359 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 1 while trying to allocate 603979776 bytes (free: 565706752, total: 101975851008). [rank0]:[W830 20:50:55.421468898 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 603979776 bytes (free: 565706752, total: 101975851008). (Worker_TP0 pid=666) INFO 08-30 20:50:55 [weight_utils.py:858] Filesystem type for checkpoints: EXT4. Checkpoint size: 173.87 GiB. Available RAM: 396.66 GiB. (Worker_TP0 pid=666) WARNING 08-30 20:50:58 [speculator.py:191] Draft model Glm5NextMTP does not support external multimodal embeddings. Embeddings from the target model will not be passed to the drafter; using text-only draft inputs instead. (Worker_TP0 pid=666) INFO 08-30 20:51:00 [encoder_runner.py:119] Encoder cache will be initialized with a budget of 7921 tokens, and profiled with 1 image items of the maximum feature size. (Worker_TP0 pid=666) WARNING 08-30 20:51:14 [rocm.py:42] Failed to import from amdsmi with ModuleNotFoundError("No module named 'amdsmi'") (Worker_TP0 pid=666) WARNING 08-30 20:51:14 [rocm.py:47] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'") (Worker_TP0 pid=666) WARNING 08-30 20:51:14 [rocm.py:58] Failed to import from vllm._rocm_C with ModuleNotFoundError("No module named 'vllm._rocm_C'") (Worker_TP1 pid=817) WARNING 08-30 20:51:14 [rocm.py:42] Failed to import from amdsmi with ModuleNotFoundError("No module named 'amdsmi'") (Worker_TP1 pid=817) WARNING 08-30 20:51:14 [rocm.py:47] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'") (Worker_TP1 pid=817) WARNING 08-30 20:51:14 [rocm.py:58] Failed to import from vllm._rocm_C with ModuleNotFoundError("No module named 'vllm._rocm_C'") (Worker_TP0 pid=666) INFO 08-30 20:51:18 [gpu_worker.py:492] Initial free memory 93.85 GiB, reserved 3.07 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly. (Worker_TP1 pid=817) INFO 08-30 20:51:18 [gpu_worker.py:492] Initial free memory 93.85 GiB, reserved 3.07 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly. (EngineCore pid=532) INFO 08-30 20:51:18 [kv_cache_utils.py:2141] GPU KV cache size: 548,560 tokens, Maximum concurrency for 524,288 tokens per request: 1.05x (Worker_TP0 pid=666) INFO 08-30 20:51:18 [kernel_warmup.py:95] Warming up ll_bf16 router GEMM kernels for shapes: ((4096, 288),). (Worker_TP0 pid=666) DeepGEMM warmup: 0%| | 0/475 [00:00= 524288) === G2: 391 === 391 answer: 391 | Let me compute: 17 × 23 = 17 × 20 + 17 × 3 = 340 + 51 = 391 === G4: decode >= 87 tok/s (MTP on, short ctx) === 512 completion tokens in 3.54s = 144.5 tok/s (single stream, MTP on) 512 completion tokens in 3.54s = 144.7 tok/s (single stream, MTP on) 512 completion tokens in 3.55s = 144.4 tok/s (single stream, MTP on) decode median: 144.5 tok/s (gate 87) === G3: MTP metrics === vllm:spec_decode_num_drafts_total{engine="0",model_name="glm-max"} 855.0 vllm:spec_decode_num_drafts_created{engine="0",model_name="glm-max"} 1.788123144753288e+09 vllm:spec_decode_num_draft_tokens_total{engine="0",model_name="glm-max"} 855.0 vllm:spec_decode_num_draft_tokens_created{engine="0",model_name="glm-max"} 1.7881231447533174e+09 vllm:spec_decode_num_accepted_tokens_total{engine="0",model_name="glm-max"} 819.0 vllm:spec_decode_num_accepted_tokens_created{engine="0",model_name="glm-max"} 1.7881231447533336e+09 vllm:spec_decode_num_accepted_tokens_per_pos_total{engine="0",model_name="glm-max",position="0"} 819.0 vllm:spec_decode_num_accepted_tokens_per_pos_created{engine="0",model_name="glm-max",position="0"} 1.7881231447533538e+09 MTP CHECK: drafts=855 accepted=819 acceptance=0.958 (floor 0.5) MTP CHECK: PASS — MTP ACTIVE === G5: validate + compare vs BF16 base === { "port": 18998, "model": "glm-max", "arith_correct": true, "arith_raw": "391", "long_gen_tokens": 2400, "long_gen_degenerate": false, "long_gen_tail": "lf of the twentieth century.\n\nPara 7 (Conclusion, ~70 words):\nThe shipping container is a monument to the power of standardization... Marc Levinson called it \"the box that changed the world.\" McLean died in 2001, having seen his humble steel box knit the world's economies together.\n\nLet me now write" } greedy prompts: 10, scored texts: 5 -> /home/ian/h3-v2/glm-max/results-max-0830-2049.json baseline=glm-fp8attn-base converted=glm-max arith 17*23: base=True conv=True long gen: base 2400 tok degenerate=False | conv 2400 tok degenerate=False greedy: 1/10 prompts identical for all 50 tok; agreement until divergence 73/82 = 89.02% greedy |dlogprob| on agreeing tokens: mean 0.05861 max 0.64639 (n=73) teacher-forced: n=271 mean |dlogprob| 0.15974 max 4.05421; top-1-status agreement 94.10% TF vs BF16 base: 94.10% (gate >= 95.0) ### GATE FAIL: G5-tf-gate --- info: delta vs prod accepted dump (recal) --- baseline=glm-5.3-flash-recal converted=glm-max arith 17*23: base=True conv=True long gen: base 2400 tok degenerate=False | conv 2400 tok degenerate=False greedy: 0/10 prompts identical for all 50 tok; agreement until divergence 34/44 = 77.27% greedy |dlogprob| on agreeing tokens: mean 0.13029 max 1.06888 (n=34) teacher-forced: n=271 mean |dlogprob| 0.14136 max 3.36681; top-1-status agreement 96.31% === G6: vision spot-set (>= 8/10) === [img01.png] OK img_toks=258 ans='Red' [img02.png] OK img_toks=258 ans='SATURN' [img03.png] OK img_toks=258 ans='4' [img04.png] OK img_toks=258 ans='Triangle' [img05.png] OK img_toks=258 ans='7318' [img06.png] OK img_toks=258 ans='3' [img07.png] OK img_toks=258 ans='Circle' [img08.png] OK img_toks=258 ans='MANGO' [img09.png] OK img_toks=258 ans='5' [img10.png] OK img_toks=258 ans='The black square is on the **left** side of the image.' VISION VERDICT: PASS (10/10 correct, gate >=8, hard_fail=False) wrote /home/ian/h3-v2/glm-max/results-vision-0830-2049.json === G7: longrep 2 x >= 250k === calibration: 0.18182 tok/char [round 1] context: 250870 tokens [round 1] prompt=250967 gen=2912 wall=76s finish=length | 4-gram max 3 (degenerate>=12: False) 8-gram max 1 line-repeat 1 [round 1] top 4-gram: 'require the model to' [round 2] context: 250838 tokens [round 2] prompt=250935 gen=2912 wall=71s finish=length | 4-gram max 2 (degenerate>=12: False) 8-gram max 1 line-repeat 1 [round 2] top 4-gram: 'at least 2,000 words.' VERDICT: PASS (worst 4-gram 3, worst 8-gram 1) wrote /home/ian/h3-v2/glm-max/results-longrep-max-0830-2049.json longrep prompt_tokens: [250967, 250935] === G8: 3 x >= 500k fresh prefills === calibration: 0.18182 tok/char on 9471 chars [round 1] built prompt: 434531 tokens (2390199 chars) [round 1] extended to 500974 tokens [round 1] prompt_tokens=501006 completion=64 wall=96.1s finish=length ok=True [round 1] reply: 'ACKNOWLEDGED. The text appears to be randomly shuffled filler content with no coherent meaning' [round 1] decode at 501006 ctx: 35 tok in 2.4s (1-tok call 2.3s) -> ~430.0 tok/s [round 2] built prompt: 434533 tokens (2390199 chars) [round 2] extended to 501041 tokens [round 2] prompt_tokens=501073 completion=64 wall=90.2s finish=length ok=True [round 2] reply: 'ACKNOWLEDGED. The text appears' [round 3] built prompt: 434566 tokens (2390199 chars) [round 3] extended to 501061 tokens [round 3] prompt_tokens=501093 completion=64 wall=90.2s finish=stop ok=True [round 3] reply: 'ACKNOWLEDGED. The text appears to be randomly generated filler content with no coherent meaning.' PASS 3/3 === G9: post-500k vision re-probe === [img02.png] OK img_toks=258 ans='SATURN' VISION VERDICT: PASS (1/1 correct, gate >=1, hard_fail=False) hard-OOM lines in log: 0 (recoverable-retry-ish lines: 0) === min free watermark (MiB): 270 === 0, 95973 MiB 1, 95973 MiB ############ ACCEPT-MAX: FAIL — [G5-tf-gate] ############ [rewarm] posting completion to prod glm-5.3-flash via 8080... [rewarm] answer: 391 The user wants me to calculate 17 × 23 and reply with just the number. 17 × 23 = 17 × 20 + 17 × 3 = 340 + 51 = 391 Let me verify: 17 × 23 = 391. Yes, that's correct. (16 × 24 = 384, and 17 × 23 [rewarm] running: glm-5.3-flash=ready [rewarm] PROD WARM OK [./accept-max.sh] released