How to run with vLLM?
#13
by thies - opened
Hello, I tried to run this model with vLLM 0.15.1.
--runner pooling --dtype float16 --trust-remote-code --pooler-config '{"seq_pooling_type": "LAST", "normalize": true}'
It says:
(APIServer pid=1) INFO 02-25 06:06:15 [utils.py:261] non-default args: {'model_tag': '/model', 'api_server_count': 1, 'model': '/model', 'runner': 'pooling', 'trust_remote_code': True, 'dtype': 'float16', 'pooler_config': PoolerConfig(pooling_type=None, seq_pooling_type='LAST', tok_pooling_type=None, normalize=True, dimensions=None, enable_chunked_processing=None, max_embed_len=None, softmax=None, activation=None, use_activation=True, logit_bias=None, step_tag_id=None, returned_token_ids=None)}
(APIServer pid=1) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(APIServer pid=1) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(APIServer pid=1) INFO 02-25 06:06:21 [model.py:871] Resolved `--convert auto` to `--convert embed`. Pass the value explicitly to silence this message.
(APIServer pid=1) INFO 02-25 06:06:27 [model.py:541] Resolved architecture: Qwen3Model
(APIServer pid=1) INFO 02-25 06:06:27 [config.py:772] Found sentence-transformers modules configuration.
(APIServer pid=1) INFO 02-25 06:06:27 [config.py:799] Found pooling configuration.
(APIServer pid=1) WARNING 02-25 06:06:27 [model.py:1885] Casting torch.bfloat16 to torch.float16.
(APIServer pid=1) INFO 02-25 06:06:27 [model.py:1561] Using max model len 32768
(APIServer pid=1) INFO 02-25 06:06:27 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=2048.
(APIServer pid=1) INFO 02-25 06:06:27 [vllm.py:624] Asynchronous scheduling is enabled.
(APIServer pid=1) WARNING 02-25 06:06:27 [vllm.py:741] Pooling models do not support full cudagraphs. Overriding cudagraph_mode to PIECEWISE.
(APIServer pid=1) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(APIServer pid=1) Traceback (most recent call last):
(APIServer pid=1) File "/usr/local/bin/vllm", line 10, in <module>
(APIServer pid=1) sys.exit(main())
(APIServer pid=1) ^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 73, in main
(APIServer pid=1) args.dispatch_function(args)
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py", line 111, in cmd
(APIServer pid=1) uvloop.run(run_server(args))
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 96, in run
(APIServer pid=1) return __asyncio.run(
(APIServer pid=1) ^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
(APIServer pid=1) return runner.run(main)
(APIServer pid=1) ^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=1) return self._loop.run_until_complete(task)
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=1) return await main
(APIServer pid=1) ^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 919, in run_server
(APIServer pid=1) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 938, in run_server_worker
(APIServer pid=1) async with build_async_engine_client(
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=1) return await anext(self.gen)
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 147, in build_async_engine_client
(APIServer pid=1) async with build_async_engine_client_from_engine_args(
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=1) return await anext(self.gen)
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 188, in build_async_engine_client_from_engine_args
(APIServer pid=1) async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 228, in from_vllm_config
(APIServer pid=1) return cls(
(APIServer pid=1) ^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 137, in __init__
(APIServer pid=1) self.input_processor = InputProcessor(self.vllm_config)
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/input_processor.py", line 72, in __init__
(APIServer pid=1) self.input_preprocessor = InputPreprocessor(
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/inputs/preprocess.py", line 58, in __init__
(APIServer pid=1) self.renderer = renderer_from_config(model_config)
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/renderers/registry.py", line 84, in renderer_from_config
(APIServer pid=1) return RENDERER_REGISTRY.load_renderer(
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/renderers/registry.py", line 62, in load_renderer
(APIServer pid=1) return renderer_cls.from_config(config, tokenizer_kwargs)
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/renderers/hf.py", line 489, in from_config
(APIServer pid=1) return cls(config, tokenizer_kwargs)
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/renderers/hf.py", line 505, in __init__
(APIServer pid=1) cached_get_tokenizer(
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/tokenizers/registry.py", line 214, in get_tokenizer
(APIServer pid=1) tokenizer = tokenizer_cls_.from_pretrained(tokenizer_name, *args, **kwargs)
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/tokenizers/hf.py", line 79, in from_pretrained
(APIServer pid=1) tokenizer = AutoTokenizer.from_pretrained(
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/tokenization_auto.py", line 1156, in from_pretrained
(APIServer pid=1) return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/transformers/tokenization_utils_base.py", line 2113, in from_pretrained
(APIServer pid=1) return cls._from_pretrained(
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/transformers/tokenization_utils_base.py", line 2359, in _from_pretrained
(APIServer pid=1) tokenizer = cls(*init_inputs, **init_kwargs)
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/transformers/models/qwen2/tokenization_qwen2_fast.py", line 120, in __init__
(APIServer pid=1) super().__init__(
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/transformers/tokenization_utils_fast.py", line 178, in __init__
(APIServer pid=1) super().__init__(**kwargs)
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/transformers/tokenization_utils_base.py", line 1472, in __init__
(APIServer pid=1) self._set_model_specific_special_tokens(special_tokens=self.extra_special_tokens)
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/transformers/tokenization_utils_base.py", line 1210, in _set_model_specific_special_tokens
(APIServer pid=1) self.SPECIAL_TOKENS_ATTRIBUTES = self.SPECIAL_TOKENS_ATTRIBUTES + list(special_tokens.keys())
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) AttributeError: 'list' object has no attribute 'keys'
Hi,
which model did you try to run with vllm?
As explained in the README, the model version with the adapter switching logic does not work well together with vLLM. Therefore, we provided task-specific variants:
- jina-embeddings-v5-text-small-retrieval
- jina-embeddings-v5-text-small-text-matching
- jina-embeddings-v5-text-small-classification
- jina-embeddings-v5-text-small-clustering
If you follow the links you can find specific instructions on how to use them.
We tested the model with the following requirements:
transformers>=5.1.0torch>=2.8.0vllm>=0.15.1
If you need to get it run with different package versions, you can let me know them and I can look into it if I find the time.
Hello, apologies for not mentioning the model. I used "jina-embeddings-v5-text-small-retrieval".
My vllm docker image is v0.15.1, and its transformers version is 4.57.6.
The problem are the extra_special_tokens in the file tokenzier_config.json (apperently the model expects a dict). When deleting this list the model loads fine and also inference works, but I'm not sure if this has an influence on the quality.
Yes extra_special_tokens has changed in transformers version 5 so it was causing trouble with version 4. I think that problem should be fixed by now because I ran into similar problems when using the onnx models (I renamed it to additional_special_tokens) , but it should not have an influence if you remove it.
Thank you for this confirmation!
thies changed discussion status to closed