The config.json file is missing
#5
by klamas - opened
Vllm nightly requires a config.json file for this speculative model
I ran into issues as well.
Originally i was trying to use the cu129-nightly container which told me it did not support this speculative model.
However after pulling specifically:nightly it worked as is.
hey this error suggests that's it's trying to load hf format which is not the default for vLLM for a while now. This suggests that either you're not running a nightly version, or you're passing args to the serving command to change the loading format.
Yes, then I realized what the error was: I had copied the arguments from one of the quantized model cards, and there was an argument "--config-format hf" that I hadn't noticed. Without it, there wouldn't have been an error.
klamas changed discussion status to closed