Possible tokenizer bug and gibberish output.
#1
by mantafloppy - opened
Its my first time trying mlx-vlm, so i'm most likely the issue, but still.
Output is gibberish.
https://i.imgur.com/aVpffYC.png
Using Transformer 5 or a custom script bypass the error.
pip install transformers==5.0.0rc0
# Minimal fix for transformers tokenizer bug
import transformers.tokenization_utils_base as tub
original_method = tub.PreTrainedTokenizerBase._set_model_specific_special_tokens
def patched_method(self, special_tokens=None):
if isinstance(special_tokens, list):
special_tokens = {token: token for token in special_tokens}
return original_method(self, special_tokens)
tub.PreTrainedTokenizerBase._set_model_specific_special_tokens = patched_method
# Now run the chat UI
from mlx_vlm.chat_ui import main
main()
(mlx) /Volumes/SSD2/llm-model/mlx-community/Devstral-Small-2-24B-Instruct-2512-4bit % python3 -m mlx_vlm.chat_ui --model /Volumes/SSD2/llm-model/mlx-community/Devstral-Small-2-24B-Instruct-2512-4bit
Using `use_fast=True` but `torchvision` is not available. Falling back to the slow image processor.
You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama_fast.LlamaTokenizerFast'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 - if you loaded a llama tokenizer from a GGUF file you can ignore this message.
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/Volumes/SSD2/miniforge3/envs/mlx/lib/python3.12/site-packages/mlx_vlm/chat_ui.py", line 27, in <module>
model, processor = load(args.model, processor_kwargs={"trust_remote_code": True})
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Volumes/SSD2/miniforge3/envs/mlx/lib/python3.12/site-packages/mlx_vlm/utils.py", line 312, in load
processor = load_processor(model_path, True, eos_token_ids=eos_token_id, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Volumes/SSD2/miniforge3/envs/mlx/lib/python3.12/site-packages/mlx_vlm/utils.py", line 370, in load_processor
processor = AutoProcessor.from_pretrained(model_path, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Volumes/SSD2/miniforge3/envs/mlx/lib/python3.12/site-packages/transformers/models/auto/processing_auto.py", line 396, in from_pretrained
return processor_class.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Volumes/SSD2/miniforge3/envs/mlx/lib/python3.12/site-packages/transformers/processing_utils.py", line 1394, in from_pretrained
args = cls._get_arguments_from_pretrained(pretrained_model_name_or_path, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Volumes/SSD2/miniforge3/envs/mlx/lib/python3.12/site-packages/transformers/processing_utils.py", line 1453, in _get_arguments_from_pretrained
args.append(attribute_class.from_pretrained(pretrained_model_name_or_path, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Volumes/SSD2/miniforge3/envs/mlx/lib/python3.12/site-packages/transformers/models/auto/tokenization_auto.py", line 1156, in from_pretrained
return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Volumes/SSD2/miniforge3/envs/mlx/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 2113, in from_pretrained
return cls._from_pretrained(
^^^^^^^^^^^^^^^^^^^^^
File "/Volumes/SSD2/miniforge3/envs/mlx/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 2359, in _from_pretrained
tokenizer = cls(*init_inputs, **init_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Volumes/SSD2/miniforge3/envs/mlx/lib/python3.12/site-packages/transformers/models/llama/tokenization_llama_fast.py", line 154, in __init__
super().__init__(
File "/Volumes/SSD2/miniforge3/envs/mlx/lib/python3.12/site-packages/transformers/tokenization_utils_fast.py", line 178, in __init__
super().__init__(**kwargs)
File "/Volumes/SSD2/miniforge3/envs/mlx/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1472, in __init__
self._set_model_specific_special_tokens(special_tokens=self.extra_special_tokens)
File "/Volumes/SSD2/miniforge3/envs/mlx/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1210, in _set_model_specific_special_tokens
self.SPECIAL_TOKENS_ATTRIBUTES = self.SPECIAL_TOKENS_ATTRIBUTES + list(special_tokens.keys())
^^^^^^^^^^^^^^^^^^^
AttributeError: 'list' object has no attribute 'keys'
Same issue in LM Studio for me on different mlx quants.
Same issue in LM Studio for me on different mlx quants.
I normally use Lm Studio too, and it give :
🥲 Failed to load the model
Failed to load model
Error when loading model: AttributeError: 'list' object has no attribute 'keys'
I did tried the 8 bit also, same result. I posted here because the 4 bit was publish first.
But Lm Studio sometime is slow (slow as in not the same day :) ) in updating their Runtime, so i wanted to test the recommended method of those publishing this model MLX.
The GGUF of Mistral 3 don't work in Lm Studio either, so it could have been an un-updated runtime.
I got that same "AttributeError: 'list' object has no attribute 'keys'" error, using both LM Studio and mlx-vlm generate.
Yes there is a tokenizer issue!
This quant appears to be working now in LM Studio, but it's the only MLX quant of Devstral2 that works for some reason.