--- license: mit base_model: - zerofata/GLM-4.5-Iceblink-v2-106B-A12B datasets: - zerofata/Instruct-Anime - zerofata/Roleplay-Anime-Characters - zerofata/Instruct-Anime-CreativeWriting - zerofata/Summaries-Anime-FandomPages pipeline_tag: text-generation tags: - text adventure - roleplay - rpg - creative writing - conversational - vllm --- # GLM-4.5-Iceblink-v2-106B-A12B (W8A8 FP8 with 2D-block quantization) This repo contains GLM-4.5-Iceblink-v2-106B-A12B quantized with mixed FP8/BF16 precision following state-of-the-art Mixture-Of-Expert quantization. - Original Model: - [zerofata/GLM-4.5-Iceblink-v2-106B-A12B](https://huggingface.co/zerofata/GLM-4.5-Iceblink-v2-106B-A12B) The model requires Ada (4000 series), Hopper (H100) or Blackwell (5000 series) GPUs for hardware FP8 support. ## 📥 Usage & Running Instructions The model was tested with vLLM + 2x RTX Pro 6000, here is a script suitable for such configuration with 131072 context length. ### Recommendations It is however recommended to use only 65K context to avoid significant degradation (https://fiction.live/stories/Fiction-liveBench-Sept-29-2025/oQdzQvKHw8JyXbN87) The recommended sampler is "min-p" sampling, this sampling is available through both the oldest Text completions API and the Chat completions API (and there is a new Response API), however most LLM frontends only support modifying min-p when using Text completions. You can however use `--override-generation-config "${SAMPLER_JSONCONFIG}"` to override the sampler (which is a merge of generation_config.json and vLLM defaults) ### Running script ```bash # Model configuration (Mandatory) MODEL="mratsim/GLM-4.5-Iceblink-v2-106B-A12B-FP8" MODELNAME="GLM-4.5-Iceblink-v2" GPU_UTIL=0.75 # Sampling configuration (Optional, if departing from `generation_config.json`) # Note that top_p=0.95 seems to lead to a serious paragraph repetition issue SAMPLER_OVERRIDE='{"temperature": 0.8, "min_p": 0.05, "top_p": 1}' # Prevent memory fragmentation export PYTORCH_ALLOC_CONF=expandable_segments:True,max_split_size_mb:512 # Prevent vLLM from using 100% CPU when idle (Very Recommended) export VLLM_SLEEP_WHEN_IDLE=1 # Use FlashInfer backend (fastest, recommended, "instant" context reprocessing) # export VLLM_ATTENTION_BACKEND=FLASHINFER vllm serve "${MODEL}" \ --tensor-parallel 2 \ --served-model-name "${MODELNAME}" \ --gpu-memory-utilization ${GPU_UTIL} \ --override-generation-config "${SAMPLER_OVERRIDE}" ``` > ℹ️ The FlashInfer backend may fail with an error similar to > `Failed to allocate memory for batch_prefill_tmp_v with size XYZ and alignment 16 in AlignedAllocator`. > > A workaround is running a sed replacement command within vllm install to increase buffer space > ```bash > sed -i 's/FLASHINFER_WORKSPACE_BUFFER_SIZE = 256 \* 1024 \* 1024/FLASHINFER_WORKSPACE_BUFFER_SIZE = 768 \* 1024 \* 1024/g' vllm/v1/attention/backends/flashinfer.py > ``` > This will be fixed by PR https://github.com/vllm-project/vllm/pull/25344 or https://github.com/vllm-project/vllm/pull/28269 ## 🔬 Quantization method The llmcompressor library was used with the following recipe: ``` scheme=QuantizationScheme( targets=["Linear"], weights=QuantizationArgs( num_bits=8, type=QuantizationType.FLOAT, dynamic=False, symmetric=True, strategy=QuantizationStrategy.BLOCK, block_structure=[32, 32], ), input_activations=QuantizationArgs( num_bits=8, type=QuantizationType.FLOAT, strategy=QuantizationStrategy.GROUP, symmetric=True, dynamic=True, observer=None, group_size=128, ), ), ignore=[ "lm_head", "model.embed_tokens", "model.norm", "re:.*input_layernorm$", "re:.*post_attention_layernorm$", "re:.*self_attn.*", "re:.*shared_experts.*", "re:.*mlp\\.gate$", # MoE router "re:model.layers.0.*", # Keep first block, (GLM-4.5-Air first_k_dense_replace = 1), also weird loading here https://github.com/vllm-project/vllm/blob/v0.11.0/vllm/model_executor/models/glm4_moe.py#L525-L547 "re:model.layers.46.*" # MTP layer (Multi-token prediction, cannot be loaded by huggingface/transformers) ], ``` FP8 quantization does not require calibration ### Deep-dive Quantization should be focused on Linear layer (also called Dense or Fully-Connected layers i.e. MatMu+Bias) In particular quantizing LayerNorm/RMSnorm layer is strongly discouraged, see [1] > LayerNorm in Quantization. Kovaleva et al. (2021); Wei et al. (2022) find that outliers in the > LayerNorm parameters of BERT (Devlin et al., 2019) cause difficulties in model compression. > Given the importance of LayerNorm, all the quantization methods we discuss above leave LayerNorm unquantized. _Note: Experts layers might not be stored as a `Linear` layer, meaning they might be skipped if using `llmcompressor` with a `Linear` target._ Some layers have a higher impact on LLM performance. According to [2], spending more bits in attention layers results in large gain compared to spending them in FFN layers. According to [3] on 2-bit quantization: - quantizing expert FFN layers do not seriously impact model quality - quantizing cross-attention has some impact - quantizing self-attention has a large impact - quantizing dense FFN has a very significant impact Hence to preserve model quality we choose not to quantize dense FFN layers (i.e. shared experts) and self-attention layers. We notice that: - official MXFP4 weights of gpt-oss-120b from OpenAI keep self-attention in BF16: - https://huggingface.co/openai/gpt-oss-120b/blob/main/model.safetensors.index.json - NVFP4 weights of DeepSeek-R1 quantized by Nvidia also keep self-attention in BF16: - https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4/blob/main/model.safetensors.index.json According to [2], giving more bits to the first `k` blocks have a significantly higher impact on model quality than for the same last `k` blocks. In this case, we keep the first layer unquantized as `"first_k_dense_replace": 1` in [config.json](config.json) ### References 1. Why Do Some Inputs Break Low-Bit LLM Quantization? (2025)\ Ting-Yun Chang, Muru Zhang, Jesse Thomason, Robin Jia\ https://arxiv.org/pdf/2506.12044 2. Examining Post-Training Quantization for Mixture-of-Experts: A Benchmark (2024)\ Pingzhi Li, Xiaolong Jin, Yu Cheng, Tianlong Chen\ https://arxiv.org/pdf/2406.08155v1 3. Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness (2023)\ Young Jin Kim, Raffy Fahim, Hany Hassan Awadalla\ https://arxiv.org/pdf/2310.02410 *** Iceblink

ICEBLINK

VERSION 2
![image](https://cdn-uploads.huggingface.co/production/uploads/65b19c6c638328850e12d38c/zA2QsjdJNOC-62-bZiOun.png)

Overview

Another re-attempt at GLM 4.5 Air. This time using a different training framework, some updated data and better hyperparameters.

This model is a creative writing and RP model. It's pretty verbose. The intent is to keep the behavior of the original model, but to improve writing, dialogue & creativity.

Compared to the original Iceblink, the effect on this one is more pronounced, with hopefully minimal impact on the intelligence.

SillyTavern Settings

Recommended Roleplay Format

> Actions: In plaintext
> Dialogue: "In quotes"
> Thoughts: *In asterisks*

Recommended Samplers

> Temp: 0.8 - 0.9
> MinP: 0.05
> TopP: 0.95 - 1.00

Instruct

GLM4.5 (no thinking): SillyTavern Preset

Quantizations

GGUF

> iMatrix (ddh0)

Creation Process

Creation Process: SFT

SFT on approx 13 million tokens, SFW / NSFW RP, stories, creative instruct & chat data. Some of the SFW datasets are public and can be found in the model datasets list.

I've switched over from Axolotl to MS-Swift w/ Megatron to train MoE models now. There's a roughly 5-10x speedup in training the models, thanks to escaping the naive MoE implementation in TRL. The training time for this run took only 40 minutes, excluding environment setup time.

A low LR for GLM Air appears to be king. Going any higher, I've found it extremely easy to begin overcooking the model.

> MS-Swift config

Not optimized for cost / performance efficiency, YMMV.

SFT (8*H200)
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \
NPROC_PER_NODE=8 \
WANDB_API_KEY=wandb_key \
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
megatron sft \
    --load '/workspace/glm-4.5-air-mcore' \
    --dataset '/workspace/joined_dataset_cleaned_modified.jsonl' \
    --load_from_cache_file true \
    --train_type lora \
    --lora_rank 256 \
    --lora_alpha 16 \
    --use-rslora true \
    --target_modules all-linear \
    --split_dataset_ratio 0.01 \
    --moe_permute_fusion true \
    --tensor_model_parallel_size 8 \
    --expert_tensor_parallel_size 1 \
    --expert_model_parallel_size 8 \
    --moe_grouped_gemm true \
    --moe_shared_expert_overlap true \
    --moe_aux_loss_coeff 6e-5 \
    --micro_batch_size 4 \
    --global_batch_size 32 \
    --recompute_granularity full \
    --recompute_method uniform \
    --recompute_num_layers 1 \
    --max_epochs 2 \
    --cross_entropy_loss_fusion true \
    --lr 6e-6 \
    --lr_warmup_fraction 0.05 \
    --min_lr 6e-7 \
    --save megatron_output/Iceblink-v3-SFT-3 \
    --eval_interval 20 \
    --save_interval 25 \
    --finetune true \
    --packing true \
    --max_length 10280 \
    --num_workers 8 \
    --dataset_num_proc 8 \
    --no_save_optim true \
    --no_save_rng true \
    --sequence_parallel true \
    --wandb_project Megatron-Air-SFT \
    --wandb_exp_name Iceblink-v3-SFT-3 \
    --attention_backend flash

Special Thanks

A shoutout to the people in BeaverAI discord that helped me test this model and my intermediate versions.

ddh0 (Madison), Ambius, Dysfunctional & my dude.