Help!!!CUDA driver error: device not ready
APIServer pid=1) INFO 07-10 03:16:54 [api_utils.py:339]
(APIServer pid=1) INFO 07-10 03:16:54 [api_utils.py:339] █ █ █▄ ▄█
(APIServer pid=1) INFO 07-10 03:16:54 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.24.0
(APIServer pid=1) INFO 07-10 03:16:54 [api_utils.py:339] █▄█▀ █ █ █ █ model /model
(APIServer pid=1) INFO 07-10 03:16:54 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=1) INFO 07-10 03:16:54 [api_utils.py:339]
(APIServer pid=1) INFO 07-10 03:16:54 [api_utils.py:273] non-default args: {'model_tag': '/model', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'model': '/model', 'trust_remote_code': True, 'max_model_len': 16384, 'served_model_name': ['Qwen3.6-27B'], 'attention_backend': 'flashinfer', 'reasoning_parser': 'qwen3', 'gpu_memory_utilization': 0.85, 'kv_cache_dtype': 'fp8_e4m3', 'enable_prefix_caching': True, 'max_num_batched_tokens': 4096, 'max_num_seqs': 4, 'enable_chunked_prefill': True}
(APIServer pid=1) WARNING 07-10 03:16:54 [envs.py:2004] Unknown vLLM environment variable detected: VLLM_NVFP4_GEMM_BACKEND
(APIServer pid=1) WARNING 07-10 03:16:54 [envs.py:2004] Unknown vLLM environment variable detected: VLLM_BUILD_COMMIT
(APIServer pid=1) WARNING 07-10 03:16:54 [envs.py:2004] Unknown vLLM environment variable detected: VLLM_BUILD_PIPELINE
(APIServer pid=1) WARNING 07-10 03:16:54 [envs.py:2004] Unknown vLLM environment variable detected: VLLM_BUILD_URL
(APIServer pid=1) WARNING 07-10 03:16:54 [envs.py:2004] Unknown vLLM environment variable detected: VLLM_IMAGE_TAG
(APIServer pid=1) INFO 07-10 03:17:00 [model.py:598] Resolved architecture: Qwen3_5ForConditionalGeneration
(APIServer pid=1) INFO 07-10 03:17:00 [model.py:1725] Using max model len 16384
(APIServer pid=1) INFO 07-10 03:17:00 [cache.py:279] Using fp8_e4m3 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor
(APIServer pid=1) INFO 07-10 03:17:00 [scheduler.py:252] Chunked prefill is enabled with max_num_batched_tokens=4096.
(APIServer pid=1) WARNING 07-10 03:17:00 [config.py:422] Mamba cache mode is set to 'align' for Qwen3_5ForConditionalGeneration by default when prefix caching is enabled
(APIServer pid=1) INFO 07-10 03:17:00 [config.py:442] Warning: Prefix caching in Mamba cache 'align' mode is currently enabled. Its support for Mamba layers is experimental. Please report any issues you may observe.
(APIServer pid=1) WARNING 07-10 03:17:00 [modelopt.py:384] Detected ModelOpt fp8 checkpoint (quant_algo=FP8). Please note that the format is experimental and could change.
(APIServer pid=1) WARNING 07-10 03:17:00 [modelopt.py:1028] Detected ModelOpt NVFP4 checkpoint (quant_algo=NVFP4). Please note that the format is experimental and could change in future.
(APIServer pid=1) WARNING 07-10 03:17:00 [modelopt.py:1028] Detected ModelOpt NVFP4 checkpoint (quant_algo=W4A16_NVFP4). Please note that the format is experimental and could change in future.
(APIServer pid=1) INFO 07-10 03:17:00 [vllm.py:1006] Asynchronous scheduling is enabled.
(APIServer pid=1) INFO 07-10 03:17:00 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=1) [transformers] The use_fast parameter is deprecated and will be removed in a future version. Use backend="torchvision" instead of use_fast=True, or backend="pil" instead of use_fast=False.
(EngineCore pid=148) INFO 07-10 03:17:10 [core.py:114] Initializing a V1 LLM engine (v0.24.0) with config: model='/model', speculative_config=None, tokenizer='/model', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=16384, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=modelopt_mixed, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=fp8_e4m3, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_verbose=False), seed=0, served_model_name=Qwen3.6-27B, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 8, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto')
(EngineCore pid=148) INFO 07-10 03:17:11 [parallel_state.py:1588] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.17.0.2:57473 backend=nccl
(EngineCore pid=148) INFO 07-10 03:17:11 [parallel_state.py:1923] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
(EngineCore pid=148) INFO 07-10 03:17:12 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=148) [transformers] The use_fast parameter is deprecated and will be removed in a future version. Use backend="torchvision" instead of use_fast=True, or backend="pil" instead of use_fast=False.
(EngineCore pid=148) INFO 07-10 03:17:15 [gpu_model_runner.py:5160] Starting to load model /model...
(EngineCore pid=148) INFO 07-10 03:17:15 [cuda.py:539] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(EngineCore pid=148) INFO 07-10 03:17:15 [mm_encoder_attention.py:373] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(EngineCore pid=148) INFO 07-10 03:17:16 [init.py:594] Selected FlashInferFP8ScaledMMLinearKernel for ModelOptFp8LinearMethod
(EngineCore pid=148) INFO 07-10 03:17:16 [deep_gemm.py:175] deep_gemm not found in site-packages, trying vendored vllm.third_party.deep_gemm
(EngineCore pid=148) INFO 07-10 03:17:16 [deep_gemm.py:202] DeepGEMM PDL enabled on vllm.third_party.deep_gemm.
(EngineCore pid=148) INFO 07-10 03:17:16 [deep_gemm.py:120] DeepGEMM E8M0 enabled on current platform.
(EngineCore pid=148) INFO 07-10 03:17:16 [qwen_gdn_linear_attn.py:228] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128).
(EngineCore pid=148) INFO 07-10 03:17:16 [cuda.py:420] Using AttentionBackendEnum.FLASHINFER backend.
(EngineCore pid=148) INFO 07-10 03:17:17 [weight_utils.py:849] Filesystem type for checkpoints: EXT4. Checkpoint size: 20.42 GiB. Available RAM: 21.05 GiB.
(EngineCore pid=148) INFO 07-10 03:17:17 [weight_utils.py:879] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre) and the checkpoint size (20.42 GiB) exceeds 90% of available RAM (21.05 GiB).
Loading safetensors checkpoint shards: 0% Completed | 0/3 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 33% Completed | 1/3 [00:03<00:07, 3.76s/it]
Loading safetensors checkpoint shards: 67% Completed | 2/3 [00:11<00:06, 6.04s/it]
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:11<00:00, 3.52s/it]
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:11<00:00, 3.97s/it]
(EngineCore pid=148)
(EngineCore pid=148) INFO 07-10 03:17:29 [default_loader.py:430] Loading weights took 11.91 seconds
(EngineCore pid=148) WARNING 07-10 03:17:29 [marlin.py:34] Your GPU does not have native support for FP4 computation but FP4 quantization is being used. Weight-only FP4 compression will be used leveraging the Marlin kernel. This may degrade performance for compute-heavy workloads.
(EngineCore pid=148) WARNING 07-10 03:17:30 [kv_cache.py:134] Checkpoint does not provide a q scaling factor. Setting it to k_scale. This only matters for FP8 Attention backends (flash-attn or flashinfer).
(EngineCore pid=148) WARNING 07-10 03:17:30 [kv_cache.py:148] Using KV cache scaling factor 1.0 for fp8_e4m3. If this is unintended, verify that k/v_scale scaling factors are properly set in the checkpoint.
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] EngineCore failed to start.
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] Traceback (most recent call last):
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1200, in run_engine_core
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] return func(*args, **kwargs)
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 966, in __init__
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] super().__init__(
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 123, in __init__
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] self.model_executor = executor_class(vllm_config)
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] return func(*args, **kwargs)
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 109, in __init__
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] self._init_executor()
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/uniproc_executor.py", line 68, in _init_executor
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] self.driver_worker.load_model()
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 384, in load_model
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] self.model_runner.load_model(load_dummy_weights=load_dummy_weights)
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] return func(*args, **kwargs)
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5176, in load_model
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] self.model = model_loader.load_model(
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] ^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] return func(*args, **kwargs)
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/base_loader.py", line 80, in load_model
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] process_weights_after_loading(model, model_config, target_device)
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/utils.py", line 112, in process_weights_after_loading
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] quant_method.process_weights_after_loading(module)
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/modelopt.py", line 1378, in process_weights_after_loading
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] self.kernel.process_weights_after_loading(layer)
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/kernels/linear/nvfp4/marlin.py", line 40, in process_weights_after_loading
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] prepare_fp4_layer_for_marlin(layer)
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/marlin_utils_fp4.py", line 282, in prepare_fp4_layer_for_marlin
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] weight_scale, scale_factor = nvfp4_marlin_process_scales(
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/marlin_utils_fp4.py", line 112, in nvfp4_marlin_process_scales
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] scale_factor = _nvfp4_compute_scale_factor(marlin_scales, a_dtype)
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/marlin_utils_fp4.py", line 54, in _nvfp4_compute_scale_factor
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] max_val = ws_float[nonzero_mask].max()
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] ~~~~~~~~^^^^^^^^^^^^^^
(EngineCore pid=148) ERROR 07-10 03:17:36 [core.py:1231] RuntimeError: CUDA driver error: device not ready
(EngineCore pid=148) Process EngineCore:
(EngineCore pid=148) Traceback (most recent call last):
(EngineCore pid=148) File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore pid=148) self.run()
(EngineCore pid=148) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore pid=148) self._target(*self._args, **self._kwargs)
(EngineCore pid=148) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1235, in run_engine_core
(EngineCore pid=148) raise e
(EngineCore pid=148) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1200, in run_engine_core
(EngineCore pid=148) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=148) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=148) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=148) return func(*args, **kwargs)
(EngineCore pid=148) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=148) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 966, in __init__
(EngineCore pid=148) super().__init__(
(EngineCore pid=148) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 123, in __init__
(EngineCore pid=148) self.model_executor = executor_class(vllm_config)
(EngineCore pid=148) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=148) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=148) return func(*args, **kwargs)
(EngineCore pid=148) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=148) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 109, in __init__
(EngineCore pid=148) self._init_executor()
(EngineCore pid=148) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/uniproc_executor.py", line 68, in _init_executor
(EngineCore pid=148) self.driver_worker.load_model()
(EngineCore pid=148) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 384, in load_model
(EngineCore pid=148) self.model_runner.load_model(load_dummy_weights=load_dummy_weights)
(EngineCore pid=148) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=148) return func(*args, **kwargs)
(EngineCore pid=148) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=148) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5176, in load_model
(EngineCore pid=148) self.model = model_loader.load_model(
(EngineCore pid=148) ^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=148) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=148) return func(*args, **kwargs)
(EngineCore pid=148) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=148) File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/base_loader.py", line 80, in load_model
(EngineCore pid=148) process_weights_after_loading(model, model_config, target_device)
(EngineCore pid=148) File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/utils.py", line 112, in process_weights_after_loading
(EngineCore pid=148) quant_method.process_weights_after_loading(module)
(EngineCore pid=148) File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/modelopt.py", line 1378, in process_weights_after_loading
(EngineCore pid=148) self.kernel.process_weights_after_loading(layer)
(EngineCore pid=148) File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/kernels/linear/nvfp4/marlin.py", line 40, in process_weights_after_loading
(EngineCore pid=148) prepare_fp4_layer_for_marlin(layer)
(EngineCore pid=148) File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/marlin_utils_fp4.py", line 282, in prepare_fp4_layer_for_marlin
(EngineCore pid=148) weight_scale, scale_factor = nvfp4_marlin_process_scales(
(EngineCore pid=148) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=148) File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/marlin_utils_fp4.py", line 112, in nvfp4_marlin_process_scales
(EngineCore pid=148) scale_factor = _nvfp4_compute_scale_factor(marlin_scales, a_dtype)
(EngineCore pid=148) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=148) File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/marlin_utils_fp4.py", line 54, in _nvfp4_compute_scale_factor
(EngineCore pid=148) max_val = ws_float[nonzero_mask].max()
(EngineCore pid=148) ~~~~~~~~^^^^^^^^^^^^^^
(EngineCore pid=148) RuntimeError: CUDA driver error: device not ready
[rank0]:[W710 03:17:37.876040998 ProcessGroupNCCL.cpp:1575] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
(APIServer pid=1) Traceback (most recent call last):
(APIServer pid=1) File "/usr/local/bin/vllm", line 10, in
(APIServer pid=1) sys.exit(main())
(APIServer pid=1) ^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 95, in main
(APIServer pid=1) args.dispatch_function(args)
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py", line 148, in cmd
(APIServer pid=1) uvloop.run(run_server(args))
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/uvloop/init.py", line 96, in run
(APIServer pid=1) return __asyncio.run(
(APIServer pid=1) ^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
(APIServer pid=1) return runner.run(main)
(APIServer pid=1) ^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=1) return self._loop.run_until_complete(task)
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/uvloop/init.py", line 48, in wrapper
(APIServer pid=1) return await main
(APIServer pid=1) ^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 663, in run_server
(APIServer pid=1) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 677, in run_server_worker
(APIServer pid=1) async with build_async_engine_client(
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
(APIServer pid=1) return await anext(self.gen)
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 99, in build_async_engine_client
(APIServer pid=1) async with build_async_engine_client_from_engine_args(
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
(APIServer pid=1) return await anext(self.gen)
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 135, in build_async_engine_client_from_engine_args
(APIServer pid=1) async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 217, in from_vllm_config
(APIServer pid=1) return cls(
(APIServer pid=1) ^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 146, in init
(APIServer pid=1) self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=1) return func(*args, **kwargs)
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 132, in make_async_mp_client
(APIServer pid=1) return AsyncMPClient(*client_args)
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=1) return func(*args, **kwargs)
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 963, in init
(APIServer pid=1) super().init(
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 573, in init
(APIServer pid=1) with launch_core_engines(
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/lib/python3.12/contextlib.py", line 144, in exit
(APIServer pid=1) next(self.gen)
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 1213, in launch_core_engines
(APIServer pid=1) wait_for_engine_startup(
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 1272, in wait_for_engine_startup
(APIServer pid=1) raise RuntimeError(
(APIServer pid=1) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
5090 running error