arXiv: REAM: Merging Improves Pruning of Experts in LLMs

Qwen3.5-122B-A10B-REAM

This model is a compressed version of Qwen/Qwen3.5-122B-A10B. It is obtained by reducing the number of experts in each MoE layer from 256 to 192. This reduction is achieved by the REAM method described in https://bknyaz.github.io/blog/2026/moe/.

Compared to other models obtained in this collection, more code data is used in the calibration data during pruning/merging to better preserve original's model coding abilities. Specifically, the ratio between c4, math and coding data (see https://bknyaz.github.io/blog/2026/moe/) is 0.0, 0.3, 0.7. The calibration data used here is the same as in Qwen3-Coder-Next-REAM. Compared to other REAM models, here we used C=32 (number of experts in groups) instead of C=16, which we found to work better.

The compressed model has 94B params (190GB) instead of 122B (250GB) of the original model, reducing storage and GPU memory requirements by roughly 25%. At the same time, the model retains >=98% of the original model's performance on the benchmarks (see Results section below). Additional efficiency optimization (e.g., quantization) can be added similarly to the original model.

See additional details at Qwen3-30B-A3B-Instruct-2507-REAM.

The MTP layer is also merged using our code. You can use it by setting --speculative-config '{"method": "mtp", "num_speculative_tokens": N}' when launching the API server.

Evaluation

Evaluation on IFeval/GSM8K of both the original and REAM model is done as:

python -m vllm.entrypoints.openai.api_server --model $model \
      --tensor-parallel-size 4 \
      --dtype auto \
      --gpu-memory-utilization 0.9 \
      --reasoning-parser qwen3 \
      --port $PORT \
      --max-model-len 65536

python -m lm_eval --model local-chat-completions \
      --model_args base_url=http://localhost:$PORT/v1/chat/completions,model=${model},tokenized_requests=False \
      --tasks ifeval \
      --batch_size 1 \
      --apply_chat_template \
      --gen_kwargs '{"chat_template_kwargs": {"enable_thinking": false}, "max_tokens": 8192}'

For HumanEval, it is evaluated as in Qwen3-30B-A3B-Instruct-2507-REAM.

Results

Model	IFeval	GSM8K	HumanEval	AVG
Qwen3.5-122B-A10B	94.7	97.0	95.1	95.6
Qwen3.5-122B-A10B-REAM	93.4	96.3	93.3	94.3

License

Please refer to the license of the original model Qwen/Qwen3.5-122B-A10B.