Failed to run on 8xB200 - AssertionError: Tried to load weights of size torch.Size([6144, 30720])to a parameter of size torch.Size([6144, 18432])
Hello,
Below is the issue encountered when running on 8×B200 GPUs.
cc @mgoin
Error logs:
Loading safetensors checkpoint shards: 89% Completed | 42/47 [01:15<00:07, 1.50s/it]
(Worker_TP2 pid=559535) INFO 07-25 10:10:23 [weight_utils.py:574] No model.safetensors.index.json found in remote.
(Worker_TP2 pid=559535) INFO 07-25 10:10:23 [weight_utils.py:811] Prefetching checkpoint files into page cache started (in background, num_threads=8, block_size=16777216 bytes)
(Worker_TP2 pid=559535) INFO 07-25 10:10:23 [weight_utils.py:806] Prefetching checkpoint files into page cache finished in 0.00s
(Worker_TP5 pid=559538) INFO 07-25 10:10:24 [weight_utils.py:574] No model.safetensors.index.json found in remote.
(Worker_TP5 pid=559538) INFO 07-25 10:10:24 [weight_utils.py:811] Prefetching checkpoint files into page cache started (in background, num_threads=8, block_size=16777216 bytes)
(Worker_TP5 pid=559538) INFO 07-25 10:10:24 [weight_utils.py:806] Prefetching checkpoint files into page cache finished in 0.00s
Loading safetensors checkpoint shards: 91% Completed | 43/47 [01:16<00:06, 1.51s/it]
(Worker_TP6 pid=559539) INFO 07-25 10:10:24 [weight_utils.py:574] No model.safetensors.index.json found in remote.
(Worker_TP6 pid=559539) INFO 07-25 10:10:24 [weight_utils.py:811] Prefetching checkpoint files into page cache started (in background, num_threads=8, block_size=16777216 bytes)
(Worker_TP6 pid=559539) INFO 07-25 10:10:24 [weight_utils.py:806] Prefetching checkpoint files into page cache finished in 0.00s
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] WorkerProc failed to start.
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] Traceback (most recent call last):
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 865, in worker_main
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] worker = WorkerProc(*args, **kwargs)
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] return func(*args, **kwargs)
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 634, in __init__
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] self.worker.load_model()
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 413, in load_model
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] self.model_runner.load_model(load_dummy_weights=load_dummy_weights)
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu/model_runner.py", line 295, in load_model
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] self.speculator.load_model(self.model)
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu/spec_decode/speculator.py", line 160, in load_model
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] self.model = self.load_draft_model(target_model, target_attn_layer_names)
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu/spec_decode/dspark/speculator.py", line 82, in load_draft_model
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] model = load_dspark_model(target_model, self.vllm_config)
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu/spec_decode/dspark/utils.py", line 31, in load_dspark_model
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] draft_model = get_model(
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] ^^^^^^^^^^
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/model_loader/__init__.py", line 140, in get_model
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] return loader.load_model(
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] return func(*args, **kwargs)
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/model_loader/base_loader.py", line 64, in load_model
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] self.load_weights(model, model_config)
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] return func(*args, **kwargs)
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/model_loader/default_loader.py", line 427, in load_weights
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] loaded_weights = model.load_weights(self.get_all_weights(model_config, model))
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/models/qwen3_dspark.py", line 184, in load_weights
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] loader.load_weights(model_weights.items())
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/model_loader/reload/torchao_decorator.py", line 50, in patched_model_load_weights
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] return original_load_weights(self, weights, *args, **kwargs)
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/models/utils.py", line 421, in load_weights
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] autoloaded_weights = set(self._load_module("", self.module, weights))
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/models/utils.py", line 355, in _load_module
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] yield from self._load_module(
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/models/utils.py", line 328, in _load_module
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] loaded_params = module_load_weights(weights)
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/models/qwen3_dflash.py", line 657, in load_weights
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] weight_loader(param, loaded_weight)
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/layers/linear.py", line 369, in weight_loader
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] assert param.size() == loaded_weight.size(), (
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP2 pid=559535) ERROR 07-25 10:10:24 [multiproc_executor.py:898] AssertionError: Tried to load weights of size torch.Size([6144, 30720])to a parameter of size torch.Size([6144, 18432])
(EngineCore pid=559457) INFO 07-25 10:10:24 [multiproc_executor.py:426] [shutdown] Executor: waiting for worker exit count=8
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] WorkerProc failed to start.
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] Traceback (most recent call last):
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 865, in worker_main
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] worker = WorkerProc(*args, **kwargs)
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] return func(*args, **kwargs)
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 634, in __init__
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] self.worker.load_model()
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 413, in load_model
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] self.model_runner.load_model(load_dummy_weights=load_dummy_weights)
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu/model_runner.py", line 295, in load_model
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] self.speculator.load_model(self.model)
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu/spec_decode/speculator.py", line 160, in load_model
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] self.model = self.load_draft_model(target_model, target_attn_layer_names)
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu/spec_decode/dspark/speculator.py", line 82, in load_draft_model
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] model = load_dspark_model(target_model, self.vllm_config)
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu/spec_decode/dspark/utils.py", line 31, in load_dspark_model
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] draft_model = get_model(
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] ^^^^^^^^^^
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/model_loader/__init__.py", line 140, in get_model
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] return loader.load_model(
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] return func(*args, **kwargs)
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/model_loader/base_loader.py", line 64, in load_model
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] self.load_weights(model, model_config)
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] return func(*args, **kwargs)
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/model_loader/default_loader.py", line 427, in load_weights
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] loaded_weights = model.load_weights(self.get_all_weights(model_config, model))
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/models/qwen3_dspark.py", line 184, in load_weights
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] loader.load_weights(model_weights.items())
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/model_loader/reload/torchao_decorator.py", line 50, in patched_model_load_weights
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] return original_load_weights(self, weights, *args, **kwargs)
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/models/utils.py", line 421, in load_weights
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] autoloaded_weights = set(self._load_module("", self.module, weights))
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/models/utils.py", line 355, in _load_module
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] yield from self._load_module(
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/models/utils.py", line 328, in _load_module
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] loaded_params = module_load_weights(weights)
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/models/qwen3_dflash.py", line 657, in load_weights
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] weight_loader(param, loaded_weight)
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/layers/linear.py", line 369, in weight_loader
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] assert param.size() == loaded_weight.size(), (
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP5 pid=559538) ERROR 07-25 10:10:25 [multiproc_executor.py:898] AssertionError: Tried to load weights of size torch.Size([6144, 30720])to a parameter of size torch.Size([6144, 18432])
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] WorkerProc failed to start.
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] Traceback (most recent call last):
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 865, in worker_main
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] worker = WorkerProc(*args, **kwargs)
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] return func(*args, **kwargs)
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 634, in __init__
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] self.worker.load_model()
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 413, in load_model
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] self.model_runner.load_model(load_dummy_weights=load_dummy_weights)
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu/model_runner.py", line 295, in load_model
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] self.speculator.load_model(self.model)
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu/spec_decode/speculator.py", line 160, in load_model
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] self.model = self.load_draft_model(target_model, target_attn_layer_names)
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu/spec_decode/dspark/speculator.py", line 82, in load_draft_model
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] model = load_dspark_model(target_model, self.vllm_config)
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu/spec_decode/dspark/utils.py", line 31, in load_dspark_model
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] draft_model = get_model(
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] ^^^^^^^^^^
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/model_loader/__init__.py", line 140, in get_model
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] return loader.load_model(
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] return func(*args, **kwargs)
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/model_loader/base_loader.py", line 64, in load_model
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] self.load_weights(model, model_config)
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] return func(*args, **kwargs)
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/model_loader/default_loader.py", line 427, in load_weights
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] loaded_weights = model.load_weights(self.get_all_weights(model_config, model))
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/models/qwen3_dspark.py", line 184, in load_weights
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] loader.load_weights(model_weights.items())
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/model_loader/reload/torchao_decorator.py", line 50, in patched_model_load_weights
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] return original_load_weights(self, weights, *args, **kwargs)
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/models/utils.py", line 421, in load_weights
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] autoloaded_weights = set(self._load_module("", self.module, weights))
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/models/utils.py", line 355, in _load_module
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] yield from self._load_module(
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/models/utils.py", line 328, in _load_module
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] loaded_params = module_load_weights(weights)
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/models/qwen3_dflash.py", line 657, in load_weights
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] weight_loader(param, loaded_weight)
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/layers/linear.py", line 369, in weight_loader
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] assert param.size() == loaded_weight.size(), (
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP6 pid=559539) ERROR 07-25 10:10:25 [multiproc_executor.py:898] AssertionError: Tried to load weights of size torch.Size([6144, 30720])to a parameter of size torch.Size([6144, 18432])
Loading safetensors checkpoint shards: 94% Completed | 44/47 [01:18<00:04, 1.53s/it]
Loading safetensors checkpoint shards: 96% Completed | 45/47 [01:18<00:02, 1.26s/it]
Loading safetensors checkpoint shards: 100% Completed | 47/47 [01:18<00:00, 1.68s/it]
(Worker_TP0 pid=559533)
(Worker_TP0 pid=559533) INFO 07-25 10:10:26 [default_loader.py:430] Loading weights took 78.91 seconds
(Worker_TP0 pid=559533) WARNING 07-25 10:10:26 [kv_cache.py:134] Checkpoint does not provide a q scaling factor. Setting it to k_scale. This only matters for FP8 Attention backends (flash-attn or flashinfer).
(Worker_TP0 pid=559533) WARNING 07-25 10:10:26 [kv_cache.py:148] Using KV cache scaling factor 1.0 for fp8_e4m3. If this is unintended, verify that k/v_scale scaling factors are properly set in the checkpoint.
(Worker_TP0 pid=559533) INFO 07-25 10:10:26 [nvfp4.py:543] Using MoEPrepareAndFinalizeNoDPEPMonolithic
(EngineCore pid=559457) WARNING 07-25 10:10:29 [multiproc_executor.py:438] [shutdown] Executor: workers still running after grace period; sending SIGTERM count=5
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] WorkerProc failed to start.
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] Traceback (most recent call last):
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 865, in worker_main
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] worker = WorkerProc(*args, **kwargs)
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] return func(*args, **kwargs)
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 634, in __init__
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] self.worker.load_model()
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 413, in load_model
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] self.model_runner.load_model(load_dummy_weights=load_dummy_weights)
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu/model_runner.py", line 283, in load_model
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] self.model = model_loader.load_model(
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] return func(*args, **kwargs)
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/model_loader/base_loader.py", line 64, in load_model
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] self.load_weights(model, model_config)
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] return func(*args, **kwargs)
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/model_loader/default_loader.py", line 427, in load_weights
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] loaded_weights = model.load_weights(self.get_all_weights(model_config, model))
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/models/deepseek_v2.py", line 1903, in load_weights
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] return loader.load_weights(weights)
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/model_loader/reload/torchao_decorator.py", line 50, in patched_model_load_weights
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] return original_load_weights(self, weights, *args, **kwargs)
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/models/utils.py", line 421, in load_weights
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] autoloaded_weights = set(self._load_module("", self.module, weights))
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/models/utils.py", line 355, in _load_module
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] yield from self._load_module(
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/models/utils.py", line 328, in _load_module
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] loaded_params = module_load_weights(weights)
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/models/deepseek_v2.py", line 1555, in load_weights
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] for name, loaded_weight in weights:
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] ^^^^^^^
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/models/utils.py", line 230, in
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] for parts, weights_data in group
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] ^^^^^
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/models/utils.py", line 220, in
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] for weight_name, weight_data in weights
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] ^^^^^^^
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/models/utils.py", line 418, in
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] (name, weight) for name, weight in weights if not self._can_skip(name)
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] ^^^^^^^
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/models/utils.py", line 137, in apply
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] for name, data in weights:
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] ^^^^^^^
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/model_loader/default_loader.py", line 333, in get_all_weights
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] yield from self._get_weights_iterator(primary_weights)
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/model_loader/default_loader.py", line 319, in
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] return ((source.prefix + name, tensor) for (name, tensor) in weights_iterator)
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/model_executor/model_loader/weight_utils.py", line 953, in safetensors_weights_iterator
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] param = f.get_tensor(name)
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] ^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=559536) ERROR 07-25 10:10:30 [multiproc_executor.py:898] ValueError: could not determine the shape of object type 'torch.storage.UntypedStorage'
[rank0]:[W725 10:10:31.309884130 ProcessGroupNCCL.cpp:1575] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
(EngineCore pid=559457) WARNING 07-25 10:10:33 [multiproc_executor.py:448] [shutdown] Executor: workers still running after SIGTERM; sending SIGKILL count=0
(EngineCore pid=559457) ERROR 07-25 10:10:33 [core.py:1231] EngineCore failed to start.
(EngineCore pid=559457) ERROR 07-25 10:10:33 [core.py:1231] Traceback (most recent call last):
(EngineCore pid=559457) ERROR 07-25 10:10:33 [core.py:1231] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/v1/engine/core.py", line 1200, in run_engine_core
(EngineCore pid=559457) ERROR 07-25 10:10:33 [core.py:1231] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=559457) ERROR 07-25 10:10:33 [core.py:1231] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=559457) ERROR 07-25 10:10:33 [core.py:1231] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=559457) ERROR 07-25 10:10:33 [core.py:1231] return func(*args, **kwargs)
(EngineCore pid=559457) ERROR 07-25 10:10:33 [core.py:1231] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=559457) ERROR 07-25 10:10:33 [core.py:1231] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/v1/engine/core.py", line 966, in init
(EngineCore pid=559457) ERROR 07-25 10:10:33 [core.py:1231] super().init(
(EngineCore pid=559457) ERROR 07-25 10:10:33 [core.py:1231] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/v1/engine/core.py", line 123, in init
(EngineCore pid=559457) ERROR 07-25 10:10:33 [core.py:1231] self.model_executor = executor_class(vllm_config)
(EngineCore pid=559457) ERROR 07-25 10:10:33 [core.py:1231] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=559457) ERROR 07-25 10:10:33 [core.py:1231] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 108, in init
(EngineCore pid=559457) ERROR 07-25 10:10:33 [core.py:1231] super().init(vllm_config)
(EngineCore pid=559457) ERROR 07-25 10:10:33 [core.py:1231] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=559457) ERROR 07-25 10:10:33 [core.py:1231] return func(*args, **kwargs)
(EngineCore pid=559457) ERROR 07-25 10:10:33 [core.py:1231] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=559457) ERROR 07-25 10:10:33 [core.py:1231] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/v1/executor/abstract.py", line 109, in init
(EngineCore pid=559457) ERROR 07-25 10:10:33 [core.py:1231] self._init_executor()
(EngineCore pid=559457) ERROR 07-25 10:10:33 [core.py:1231] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 201, in _init_executor
(EngineCore pid=559457) ERROR 07-25 10:10:33 [core.py:1231] self.workers = WorkerProc.wait_for_ready(unready_workers)
(EngineCore pid=559457) ERROR 07-25 10:10:33 [core.py:1231] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=559457) ERROR 07-25 10:10:33 [core.py:1231] File "/home/shakhizat/.vllm/lib64/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 762, in wait_for_ready
(EngineCore pid=559457) ERROR 07-25 10:10:33 [core.py:1231] raise e from None
(EngineCore pid=559457) ERROR 07-25 10:10:33 [core.py:1231] Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause.
(EngineCore pid=559457) Process EngineCore:
Hi @shakhizat - I traced this; it's a draft-model config translation bug in vLLM, not your hardware or the checkpoint.
Root cause: the speculator was trained with 5 auxiliary hidden-state layers (aux_hidden_state_layer_ids: [2, 20, 39, 58, 75] in config.json), so its fc weight is [6144, 30720] (5 x 6144). At load time vLLM fails to pick up those layer ids and falls back to sizing fc from the draft's num_hidden_layers: 3 - [6144, 18432]. Hence the exact assertion you hit.
Specifically, vLLM's speculators-format translator update_dspark() (vllm/transformers_utils/configs/speculators/algos.py) doesn't populate dflash_config.target_layer_ids, while update_dflash() does - which is why the RedHatAI .dflash checkpoints with the same 5-aux/3-draft-layer layout load fine and this .dspark one doesn't. This is the bug fixed in open PR vllm-project/vllm#48932 ("Fix speculators-format DSpark loading"), which reports the same weight-size assertion.
Workaround until the PR lands - add this to the draft model's config.json (local copy), keeping DSpark's i-1 layer-id semantics (vLLM adds +1 back internally):
"dflash_config": {
"mask_token_id": 154856,
"target_layer_ids": [1, 19, 38, 57, 74]
}
or patch your vLLM install with the algos.py hunk from the PR (a ~9-line change to update_dspark).
Two side notes on the log: the ValueError: could not determine the shape of object type 'torch.storage.UntypedStorage' on Worker_TP3 happens after the executor begins shutdown - it's teardown noise, not a second issue. And No model.safetensors.index.json found in remote is expected for this repo layout.
Would be great to get #48932 reviewed/merged - happy to help validate the fix on multi-GPU Blackwell once it's in a nightly. cc @mgoin
@shakhizat @rigoretto Could you pull the latest main from vllm? This issue should have been resolved. Thanks! Let me know if bug still exists.
hi @shanjiaz , failed to run using v0.26.0, and haven't managed to build from the source, will try it later on.
Yeah the new version did not include the fix https://github.com/vllm-project/vllm/pull/48524
Hi @shanjiaz ,
I cherry-picked PR #48524 into my build and confirmed the dspark weight-loading issue is fixed (thanks!). However, the engine still fails to start with the same setup:
NotImplementedError: Layer model.layers.0.self_attn.indexer.k_cache: page size is not divisible by the maximum page size and cannot be padded.
Start command:
docker run -d \
--runtime /usr/bin/nvidia-container-runtime \
--name glm-5.2 \
--gpus all \
--ipc=host \
-p 33000:33000 \
--ulimit memlock=-1 \
--ulimit stack=67108864 \
--restart always \
-v /models:/models \
-e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
vllm-openai:glm52-dspark-pr48524 \
--model /models/GLM-5.2-FP8 \
--host 0.0.0.0 \
--port 33000 \
--served-model-name glm-5.2 \
--trust-remote-code \
--enable-chunked-prefill \
--enable-prefix-caching \
--max-model-len auto \
--max-num-seqs 20 \
--max-num-batched-tokens 8192 \
--tensor-parallel-size 8 \
--tool-call-parser glm47 \
--enable-auto-tool-choice \
--reasoning-parser glm45 \
--safetensors-load-strategy=prefetch \
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
--speculative-config '{"model": "/models/GLM-5.2-speculator.dspark","num_speculative_tokens": 7,"method": "dspark"}'
What should I do next to get this running? Should I open a separate issue, or is there a config flag / patch I’m missing?
Ah thanks for trying! I think you might need the latest main or https://github.com/vllm-project/vllm/pull/49704.
Hello @shanjiaz , It now shows a CUDAOOM error. Shouldn't it fit on 8xB200 GPUs?
Here's the command to reproduce it.
vllm serve nvidia/GLM-5.2-NVFP4
--tensor-parallel-size 8
--enable-expert-parallel
--spec-model RedHatAI/GLM-5.2-speculator.dspark
--spec-method dspark
--spec-tokens 7
--trust-remote-code
--reasoning-parser glm45
--tool-call-parser glm47
--enable-auto-tool-choice
--kv-cache-dtype fp8_e4m3
--host 0.0.0.0 --port 8000
It should fit, taking a look!
@shanjiaz , I attached an error log here: https://github.com/vllm-project/vllm/issues/49475, a vanilla command without speculator works fine.