Instructions to use OpenVINO/starcoder2-7b-int4-ov with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use OpenVINO/starcoder2-7b-int4-ov with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="OpenVINO/starcoder2-7b-int4-ov")

# Load model directly
from transformers import AutoTokenizer, AutoModelForMultimodalLM

tokenizer = AutoTokenizer.from_pretrained("OpenVINO/starcoder2-7b-int4-ov")
model = AutoModelForMultimodalLM.from_pretrained("OpenVINO/starcoder2-7b-int4-ov")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use OpenVINO/starcoder2-7b-int4-ov with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "OpenVINO/starcoder2-7b-int4-ov"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OpenVINO/starcoder2-7b-int4-ov",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/OpenVINO/starcoder2-7b-int4-ov

SGLang

How to use OpenVINO/starcoder2-7b-int4-ov with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "OpenVINO/starcoder2-7b-int4-ov" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OpenVINO/starcoder2-7b-int4-ov",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "OpenVINO/starcoder2-7b-int4-ov" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OpenVINO/starcoder2-7b-int4-ov",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use OpenVINO/starcoder2-7b-int4-ov with Docker Model Runner:
```
docker model run hf.co/OpenVINO/starcoder2-7b-int4-ov
```

Padtoken

by ECaleb - opened Dec 18, 2024

Discussion

ECaleb

Dec 18, 2024

How i can set the pad_token_id to stop open generation?

Echo9Zulu

OpenVINO Toolkit org Dec 21, 2024

There are numerous ways to achieve this.

Sometimes one can be inferred from Autotokenizers, other times you will need to explicitly define the id by viewing the tokenizer files. Maybe there are other ways to do it, but the per-model-basis method leverages the structure of the OpenVINO IR for your model.

Review "tokenizer_config.json" and look for pad_token (or something similar); I opened up a converted Qwen2.5-32B -Coder which uses the Qwen2Tokenizer. Here we see

 "pad_token": "<|endoftext|>"

or something similar; then open "tokenizer.json" and look for the "<|endoftext|>" token object. In this example, we get the token ID 151643. You would set that value explicitly and for models which share tokenizers you wont have an issue with something like

if tokenizer.pad_token_id is None:
    tokenizer.pad_token = tokenizer.eos_token  # or any other appropriate token
pad_token_id = tokenizer.pad_token_id

Hope this helps.

Also, check out my repo for more converted models. The official Intel repos have lots of outdated/"vanilla" models. I will be hosting a space soon with a conversion tool that makes it much easier to build the Optimum CLI commands, which can be quite difficult to configure when trying to access more advanced quantization strategies or special cases.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment