Instructions to use meta-models/Muse-Glimmer-30B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use meta-models/Muse-Glimmer-30B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="meta-models/Muse-Glimmer-30B")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("meta-models/Muse-Glimmer-30B")
model = AutoModelForMultimodalLM.from_pretrained("meta-models/Muse-Glimmer-30B", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use meta-models/Muse-Glimmer-30B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "meta-models/Muse-Glimmer-30B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "meta-models/Muse-Glimmer-30B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/meta-models/Muse-Glimmer-30B

SGLang

How to use meta-models/Muse-Glimmer-30B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "meta-models/Muse-Glimmer-30B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "meta-models/Muse-Glimmer-30B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "meta-models/Muse-Glimmer-30B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "meta-models/Muse-Glimmer-30B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use meta-models/Muse-Glimmer-30B with Docker Model Runner:
```
docker model run hf.co/meta-models/Muse-Glimmer-30B
```

5090 dflash results - can you share details?

#21

by csabakecskemeti - opened 1 day ago

Discussion

csabakecskemeti

1 day ago

•

edited 1 day ago

First of all, thanks for shipping this model, and especially for releasing the DFlash drafter alongside it — it's great to see speculative decoding supported out of the box.
I'm trying to reproduce the RTX 5090 row from the model card:

GPU	Baseline (tok/s)	With DFlash (tok/s)	Speedup
Nvidia RTX 5090	74.9	233.4	3.1x

I'm peaking at 137 tok/s, using your own GGUFs for both the target and the drafter, so I don't think this is a conversion or quantization issue on my side.

Setup: RTX 5090, llama.cpp master d2f83055d, CUDA build with CMAKE_CUDA_ARCHITECTURES=120, Release. muse-glimmer-30B-kquant-17gb.gguf + dflash-kquant.gguf, both from this repo. 512 tokens generated, greedy, batch 1, 71-token prompt.

	claimed	mine
Baseline	74.9 tok/s	82.91 tok/s
With DFlash	233.4 tok/s	137.14 tok/s
Speedup	3.1x	1.65x

llama-server
-m muse-glimmer-30B-kquant-17gb.gguf
-md dflash-kquant.gguf
--spec-type draft-dflash --spec-draft-n-max 15
-ngl 99 -ngld 99 -fa on -c 8192 -np 1 --temp 0 --top-k 1 --jinja

ruanslv

Meta Inc. org 1 day ago

Could you try with https://github.com/ggml-org/llama.cpp/pull/26842? It's an important optimization, we haven't managed to merge it in time but it's work in progress (as you can see from the PR description)

csabakecskemeti

1 day ago

•

edited 1 day ago

Inluding the optimization improved my results but still far form the claimed level

Run	tok/s	Acceptance	Mean len	ms/pass
baseline (no draft)	82.52	—	—	12.12
master + DFlash	129.35	0.13673 (341/2494)	3.01	23.27
PR 26842 + DFlash	139.79	0.13673 (341/2494)	3.01	21.53

RTX 6000 PRO
151.39 tokens per second
llama-server -m muse-glimmer-30B-kquant-17gb.gguf -md dflash-kquant.gguf --spec-type draft-dflash --spec-draft-n-max 15 -ngl 99 -ngld 99 -fa on -c 8192 -np 1 --temp 0 --top-k 1 --jinja

ruanslv

Meta Inc. org 1 day ago

•

edited 1 day ago

Thanks for checking. So 233.4 tok/s is the mean, it's a wide interval as you can see from the image:

Your request is definitely on the lower end, given the acceptance you are reporting. In general the dflash head has been mostly optimized for agentic and coding flows, so that's where I'd expect the highest speed-ups. Open-ended or subjective prompts I'd expect to be among the worst. But still we do see variance even within a specific category. If you were to sweep across a wider range of prompts I'd expect you to reach numbers closer to the mean we reported

csabakecskemeti

1 day ago

Confirmed coding prompt performs really good with dflash
238.83t/s generation
draft acceptance = 0.28100 (784 accepted / 2790 generated), mean len = 5.22

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment