Instructions to use meta-models/Muse-Glimmer-30B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use meta-models/Muse-Glimmer-30B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="meta-models/Muse-Glimmer-30B")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("meta-models/Muse-Glimmer-30B")
model = AutoModelForMultimodalLM.from_pretrained("meta-models/Muse-Glimmer-30B", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use meta-models/Muse-Glimmer-30B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "meta-models/Muse-Glimmer-30B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "meta-models/Muse-Glimmer-30B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/meta-models/Muse-Glimmer-30B

SGLang

How to use meta-models/Muse-Glimmer-30B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "meta-models/Muse-Glimmer-30B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "meta-models/Muse-Glimmer-30B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "meta-models/Muse-Glimmer-30B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "meta-models/Muse-Glimmer-30B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use meta-models/Muse-Glimmer-30B with Docker Model Runner:
```
docker model run hf.co/meta-models/Muse-Glimmer-30B
```

How to get Muse-Glimmer working on a Mac / MacBook Pro in LM Studio and the M5 Metal Tensor issue

#42

by Portll-LLM - opened about 17 hours ago

Discussion

Portll-LLM

about 17 hours ago

•

edited about 16 hours ago

If LM Studio fails to load this GGUF with "Engine protocol runtime llama-server exited before becoming healthy. exitCode=1", it isn't your Mac, your RAM, or a bad download. The real error is only visible in the server logs (~/.lmstudio/server-logs/):

**"error loading model: unknown model architecture: 'muse-glimmer'". **

Muse Glimmer is a new architecture and llama.cpp only learned it recently. LM Studio's bundled engine 2.27.1 and older predate it and reject the file near-instantly (~0.2 seconds), before any memory is allocated - which is why changing context length or GPU offload settings does nothing.

The fix takes about a minute.

Open Settings, go to Runtimes, and update the llama.cpp runtime (or run "lms runtime update" in a terminal). You need llama.cpp-mac-arm64-apple-metal-advsimd version 2.28.2 or newer.
Check it shows as selected for GGUF ("lms runtime ls" should show a check mark next to 2.28.2). Then just load the model again - no app restart needed, because every model load spawns a fresh engine process from the currently selected runtime. Verified on a MacBook Pro M5 Pro (38 GB, macOS 26.4, LM Studio 0.4.20): on 2.27.1 the model is rejected, on 2.28.2 the same file loads in seconds and serves normally, including the mmproj multimodal projector.

Prompt processing on the M5 by default is currently slower than the hardware can achieve. Every load logs:

"ggml_metal_library_init_from_source: error compiling source"

followed by

"the tensor API is not supported in this environment - disabling".

The M5's GPU Neural Accelerators are driven through Apple's Metal 4 tensor API and llama.cpp supports it, but LM Studio's engine builds are compiled against the macOS 14 SDK, which makes the runtime shader compile default to Metal 3.2, where the tensor API does not exist.

The capability check fails and llama.cpp silently falls back to the classic kernels. What you lose is roughly 2-3x on prompt processing, i.e. time to first token on long prompts, RAG and agentic workloads. It doesn't affect generation speed - tokens per second are memory-bandwidth-bound and unchanged, so chat feels normal and outputs are identical. This is tracked with a confirmed root cause at github.com/lmstudio-ai/lmstudio-bug-tracker/issues/2040. There is nothing to tune on your side for GGUF; if an MLX conversion of this model exists, LM Studio's MLX "nax" runtime does use the M5 accelerators today.

The architecture fix applies to every Mac: any Apple-silicon machine needs runtime 2.28.2 or newer to load Muse Glimmer at all.

The tensor API slowdown is M5-only: the tensor API targets the GPU Neural Accelerators introduced with the M5 generation (A19 on iPhone), and llama.cpp explicitly disables it for pre-M5 devices because the hardware is not there.

Sizing note: this quant is about 17 GB of weights plus context, comfortable on 32 GB or more of unified memory. On 16 or 18 GB machines it will page or fail to fit regardless of everything above.

Portll-LLM changed discussion title from How to get Muse-Glimmer working on a Mac / MacBook Pro M5 chip in LM Studio, and the Metal Tensor issue to How to get Muse-Glimmer working on a Mac / MacBook Pro in LM Studio and the M5 Metal Tensor issue about 17 hours ago

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment