How is the model supposed to be deployed with vLLM?
When I tried to deploy the model with VLLM, there is error AttributeError: Model DotsOCRForCausalLM does not support BitsAndBytes quantization yet. No 'packed_modules_mapping' found
Was faced with the same issue
Hello, sorry for late response.
Currently, vllm is not supporting dots.ocr ( DotsOCRForCausalLM ). If there is any update, I can update this model or release a similar model that supports vllm as well.
You might follow as:
https://huggingface.co/rednote-hilab/dots.ocr/discussions/20
Same issue. It seems the PR has been merged into vLLM 4 days ago.
The latest vllm nightly build wheel has support dots_ocr and can use bitsandbytes inflight.
using vllm 0.13.0 load this model, but have error like: self.data.shape == loaded_weight.shape
any idea how i implement via llama_cpp ?
seem there is no compatible chathandler...