Instructions to use cerebras/GLM-4.6-REAP-218B-A32B-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use cerebras/GLM-4.6-REAP-218B-A32B-FP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="cerebras/GLM-4.6-REAP-218B-A32B-FP8")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("cerebras/GLM-4.6-REAP-218B-A32B-FP8")
model = AutoModelForCausalLM.from_pretrained("cerebras/GLM-4.6-REAP-218B-A32B-FP8", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use cerebras/GLM-4.6-REAP-218B-A32B-FP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "cerebras/GLM-4.6-REAP-218B-A32B-FP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "cerebras/GLM-4.6-REAP-218B-A32B-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/cerebras/GLM-4.6-REAP-218B-A32B-FP8

SGLang

How to use cerebras/GLM-4.6-REAP-218B-A32B-FP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "cerebras/GLM-4.6-REAP-218B-A32B-FP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "cerebras/GLM-4.6-REAP-218B-A32B-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "cerebras/GLM-4.6-REAP-218B-A32B-FP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "cerebras/GLM-4.6-REAP-218B-A32B-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use cerebras/GLM-4.6-REAP-218B-A32B-FP8 with Docker Model Runner:
```
docker model run hf.co/cerebras/GLM-4.6-REAP-218B-A32B-FP8
```

Definitely does NOT run on dual RTX PRO 6000 96GB

by aaron-newsome - opened Oct 25, 2025

Discussion

aaron-newsome

Oct 25, 2025

I can confirm the model won't load on a dual RTX PRO 60000 rig. Which is a real bummer. I'm still hoping for an actual 'Air' model that will fit on my setup. GLM 4.5 Air is a kickass model but 4.6 is a more competent coder. I'm able to load the GLM 4.6 Q2 quant but it's not great. Thanks for posting this one though. I was hopeful, but failed to run it.

Lockout

Oct 25, 2025

Huh? I can run this in 4.25bpw.. was like 128gb. Are you loading full context? I have half your vram. On your system should be able to do bnb-4bit from these weights themselves.

aaron-newsome

Oct 26, 2025

The context I tried was 128K which is really the bare minimum, as it fills up very quickly when coding, plus the "compact" takes far too long when it gets anywhere near ~70k. Also anything above ~70k context and things get extremely slow. It's always a struggle between losing context, running out of room, or running into a performance wall. I was able to run the cerebras 4.5, using bits and bytes. I posted about it here:

https://huggingface.co/cerebras/GLM-4.5-Air-REAP-82B-A12B/discussions/11

With full ctx, it barely fits.

Lockout

Oct 26, 2025

Yea.. I mean that's air. Different class of model. With this much context you're stuck with GGUF/EXL3/AWQ and harder quantizations. Bare minimum for me is 32k. Attention over 128k is poor anyway, imo. You're using VLM so you can halve the cache to FP8, could work for you.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment