Instructions to use migtissera/Tess-4-27B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use migtissera/Tess-4-27B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="migtissera/Tess-4-27B")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("migtissera/Tess-4-27B")
model = AutoModelForMultimodalLM.from_pretrained("migtissera/Tess-4-27B", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use migtissera/Tess-4-27B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "migtissera/Tess-4-27B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "migtissera/Tess-4-27B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/migtissera/Tess-4-27B

SGLang

How to use migtissera/Tess-4-27B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "migtissera/Tess-4-27B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "migtissera/Tess-4-27B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "migtissera/Tess-4-27B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "migtissera/Tess-4-27B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use migtissera/Tess-4-27B with Docker Model Runner:
```
docker model run hf.co/migtissera/Tess-4-27B
```

Thank you!

#14

by ercangorgulu - opened Jul 20

Discussion

ercangorgulu

Jul 20

I was excessively using this for a long time and just wanted to thank you for this great model.
I had a lot of issues with Qwen 27B and 35B A3B quantized models. I tried almost every version of them like Ornith, ThinkingCap, Qwopus etc. They were either not doing exactly what I want, getting stuck, or stopping in the middle of the run.
This model is way better, not just 1-2% better like it's seen on benchmarks. I never had a problem on stopping prematurely or doing something else that I didn't ask. It sometimes gets stuck and loops infinitely but compared to other counterparts, it is way less. That looping usually happens when I run ralph loop that works minimum 8 hours, so for that case I think it is normal especially since I am using the quantized version.
Thank you so much again, this model is so underrated.

migtissera

Owner Jul 20

You're welcome! I only saw the looping behaviour when I used SGLang for serving -- apparently the looping behaviour for the Qwen models is a thing, there's a couple of issues opened on SGLang for this as well.

I'm glad you're enjoying the model! I have the 35B-A3B set to public as well, I have not announced it yet. For the GGUF version, I will release it with a custom runtime (fork of llama.cpp with custom kernels) that can do 1600 tok/sec prefill and 112 tok/sec decode -- But that's only for Macs. What computer are you using?

migtissera

Owner Jul 21

Check this out for the looping behaviour: https://github.com/noonghunna/club-3090/discussions/662#discussioncomment-17707827

HuggingFace007Galeiq

Jul 21

Can't wait for the custom runtime with the 35B-A3B, the 1600 tok/sec prefill and 112 tok/sec decode - was this on an M1 Max 64GB by any chance? That's the machine I'm running. Haven't seen even MLX reach those numbers if it is the m1 max

ercangorgulu

Jul 21

I have RTX 4090, I overclocked the memory and GPU to make it a little faster. I already have 2500 tps prefill and 110 tps decode on 27B model with MTP. As I understood 35B A3B model will fly then 😀 Looping behavior only occurs when it tries to do something and gets error (Like web-fetch or grep). Most of the time it tries something else but sometimes it tries the same thing again and again. I guess repetition penalty would fix it, but didn't try it yet.

migtissera

Owner Jul 21

Can't wait for the custom runtime with the 35B-A3B, the 1600 tok/sec prefill and 112 tok/sec decode - was this on an M1 Max 64GB by any chance? That's the machine I'm running. Haven't seen even MLX reach those numbers if it is the m1 max

It's on my M4 Max 128GB, but your 64GB is plenty. The numbers above are for the 8-bit quant, with the MTP head spliced in the GGUF.

migtissera

Owner Jul 21

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment