Qwen

/

Qwen3.8-27B

like 13.9k

Follow

Qwen 102k

Yea, need 35b Moe.

I don't want to raise false hopes, deny or confirm any information, but someone on Reddit found a hint about the 35b model
https://www.reddit.com/r/LocalLLaMA/comments/1voxppd/qwen_38_35ba3b_spotted/

I don't want to raise false hopes, deny or confirm any information, but someone on Reddit found a hint about the 35b model
https://www.reddit.com/r/LocalLLaMA/comments/1voxppd/qwen_38_35ba3b_spotted/

Unfortunately, this is just a typo (35B-A3B was written instead of 27B), which was fixed in the commit https://github.com/modelscope/ms-swift/commit/a45f1d4f73157ba59062a7fd1f55a40dae759156, and the repository, as well as the person who submitted this commit, are in no way associated with the Qwen company.

So it seems that the only possible solution good for single 12GB GPUs are q3_k_xl quant from unsloth + --no-kv-offload + --spec-draft-device cpu. The performance won't be great, however it is still possible to survive.

Qwen 3.8 35B-A3B/A5B (in my opinion, the A5B version is better) 🌟
+++

Seems like we won't get a lower MoE version =(

Seems like we won't get a lower MoE version =(

ngl will feel betrayed if they don't

I'm also really looking forward to the release of the Qwen3.8-35B-A3B model and I hope it will be. Of course, A5B proved to be much better and smarter. I have 5060 on 8gb and 32 ddr5, I launched qwen3.8-27B and achieved 6t/s in a small context. but when the context increases at least to 30,000, it becomes simply impossible, the speed drops to 3t/s and in this regard the MoE model from Qwen3.6-35B-A3B simply has no equal, it shows itself perfectly on weak wildeocards. So the release of Qwen3.8-35B-A3B is a huge support for such geek engineers like me.

Yeah! That will be really necessary! Please! <3

I'm also really looking forward to the release of the Qwen3.8-35B-A3B model and I hope it will be. Of course, A5B proved to be much better and smarter. I have 5060 on 8gb and 32 ddr5, I launched qwen3.8-27B and achieved 6t/s in a small context. but when the context increases at least to 30,000, it becomes simply impossible, the speed drops to 3t/s and in this regard the MoE model from Qwen3.6-35B-A3B simply has no equal, it shows itself perfectly on weak wildeocards. So the release of Qwen3.8-35B-A3B is a huge support for such geek engineers like me.

So it seems that the only possible solution good for single 12GB GPUs are q3_k_xl quant from unsloth + --no-kv-offload + --spec-draft-device cpu. The performance won't be great, however it is still possible to survive.

just search for a reap, probably will be one, if you want code, reap the whole vision tower from the model, test it, and see what you get.

I've launched unsloth/Qwen3.8-27B-GGUF:IQ4_NL on my single RX 7600 XT 16GB. The setup is the following (important options):

--no-mmproj
--no-warmup
--no-ui
--parallel "1"
--kv-unified
--batch-size "512"
--ubatch-size "256"
--no-kv-offload
--spec-type "draft-mtp"
--spec-draft-n-max "4"
--fit "off"
--n-gpu-layers "all"
--reasoning-effort "low"

Points:

1. --no-kv-offload I have no limitations about KV cache, it sits on CPU; I don't need to quantize it, I can launch 128k or 256k, doesn't matter.
2. no-mmproj means the visual encoder/decoder is disabled and won't be loaded.
3. spec-draft-n-max "4" gives me about 6-10 t/s output.
4. --reasoning-effort "low" means each task uses up to 4k tokens and up to 6-11 minutes, which is enough to survive somehow.

However, I have the second machine with dual 9060 XT 16GB, it runs unsloth/Qwen3.8-27B-GGUF:Q8_0 at about 35-45 t/s output. Dual 9060 XT 16GB is today seems to be the minimum for open weight LLMs.

If they can make 2B and 0.8B that actually work without having to dig for Base GGUFs and stuff like that, that's cool. Or they can go the Ling 3.0 Tiny route which was proven to be successful if the right lab did it. They can easily make a Qwen3.8-7B-A1.5B for example.

With 16Gb VRam, I hope 35B A5B MOE.

Qwen 3.6 35b a3b has only 1.1b active ffn. 0.7 transformer, 1b(huge) embeddings, 0.3 mtp etc. Alibaba did like this because if you have a api server, you cant spend high effort for all customers. İt is a trick. Attention layer is same for everyone, embeddings too. Hard part is ffn. Usually companies dont think us when they make a llm. They must think to cut their api prices first. Like deepseek. İt is bad but it is normal. But i want 256 top 16+1 anyway. Shitty scaling laws.

We need this model.

I've launched unsloth/Qwen3.8-27B-GGUF:IQ4_NL on my single RX 7600 XT 16GB. The setup is the following (important options):

--no-mmproj
--no-warmup
--no-ui
--parallel "1"
--kv-unified
--batch-size "512"
--ubatch-size "256"
--no-kv-offload
--spec-type "draft-mtp"
--spec-draft-n-max "4"
--fit "off"
--n-gpu-layers "all"
--reasoning-effort "low"

Points:

1. --no-kv-offload I have no limitations about KV cache, it sits on CPU; I don't need to quantize it, I can launch 128k or 256k, doesn't matter.
2. no-mmproj means the visual encoder/decoder is disabled and won't be loaded.
3. spec-draft-n-max "4" gives me about 6-10 t/s output.
4. --reasoning-effort "low" means each task uses up to 4k tokens and up to 6-11 minutes, which is enough to survive somehow.

However, I have the second machine with dual 9060 XT 16GB, it runs unsloth/Qwen3.8-27B-GGUF:Q8_0 at about 35-45 t/s output. Dual 9060 XT 16GB is today seems to be the minimum for open weight LLMs.

Hi, I'm running on 9060 XT as well. 35-45 t/s is nice. Do you think you can share an llama bench result or the token ingestion speed? My biggest problem is token ingestion speed because of Hermes prompt size. With Qwen 3.6 A3B I'm getting around 3500 which is really good. But with dense models I am getting very low.

Hi, I'm running on 9060 XT as well. 35-45 t/s is nice. Do you think you can share an llama bench result or the token ingestion speed? My biggest problem is token ingestion speed because of Hermes prompt size. With Qwen 3.6 A3B I'm getting around 3500 which is really good. But with dense models I am getting very low.

I am taking real log right now.

3.22.790.277 I slot print_timing: id  0 | task 668 | prompt processing, n_tokens =   2560, progress = 0.13, t =   3.23 s / 792.77 tokens per second
3.23.486.157 I slot print_timing: id  0 | task 668 | prompt processing, n_tokens =   3072, progress = 0.16, t =   3.92 s / 783.63 tokens per second
...
3.51.331.617 I slot print_timing: id  0 | task 668 | prompt processing, n_tokens =  19280, progress = 0.99, t =  31.76 s / 607.07 tokens per second
3.51.830.926 I slot print_timing: id  0 | task 668 | prompt processing, n_tokens =  19376, progress = 1.00, t =  32.39 s / 598.21 tokens per second
3.55.360.998 I slot print_timing: id  0 | task 668 | n_gen =    126, tg =  41.46 t/s, tg_3s =  41.79 t/s
3.58.410.794 I slot print_timing: id  0 | task 668 | n_gen =    261, tg =  42.87 t/s, tg_3s =  44.27 t/s
...
5.43.105.272 I slot print_timing: id  0 | task 668 | n_gen =   4231, tg =  38.19 t/s, tg_3s =  32.62 t/s
5.46.197.051 I slot print_timing: id  0 | task 668 | n_gen =   4329, tg =  38.01 t/s, tg_3s =  31.70 t/s

Input starts from almost 800 t/s and goes down to 600 t/s. Output starts from 42 t/s and goes up to 45 t/s and then goes down to 32 t/s. However, I am not using maximum cards power; my current systemd script looks as follows:

/etc/systemd/system/amd-gpu.service:

[Unit]
Description=AMD GPU service

[Service]
Type=simple

ExecStartPre=/bin/bash -c 'for i in {1..60}; do amd-smi &>/dev/null && exit 0 || sleep 1; done; exit 1'
ExecStart=/bin/bash -c 'set -e; for gpu in 0 1; do amd-smi set --gpu $gpu --perf-level MANUAL && amd-smi set --gpu $gpu --clk-level sclk 0 1 && amd-smi set --gpu $gpu --power-cap ppt0 127; done'

[Install]
WantedBy=multi-user.target

Each card consumes about 90 watts. So per month it gives 90 million output tokens, consuming about 55 kWh. It is fine. I am not using any industrial agents like hermes because they are consuming too much tokens (both input and output), I am using my own.

Adding a measurement rather than a +1.

Same box (RTX 4090, llama.cpp, Q4-class GGUF), and the two files are nearly the same size on disk — 16.5 GB for Qwen3.6-35B-A3B, 16.4 GB for Qwen3.8-27B:

The dense model is not misconfigured: reading 16.4 GB of weights per token, it runs at roughly 76% of what a 4090's memory bandwidth allows. That is the ceiling for dense on a single consumer card, and A3B sparsity is the only thing that lifts it — same disk footprint, ~3.6× the throughput.

This matters most for what these models are increasingly used for: long agentic tool loops, where the whole conversation is re-read every turn and a run is measured in hours rather than seconds.

And to be fair to 3.8: on our own process-quality measurements it is the better worker — it beat 3.6 on walk discipline by a margin above the run's own spread (our scenes, small n, so read it as a direction and not a leaderboard). That is exactly why the missing slot stings. The 27B is the model we want; A3B is the shape we can run while working. Qwen3.8-Flash-Next is MoE, but at 180B it does not reach a 24 GB card.

A 3.8-generation 35B-A3B (or A5B) would be the single release that changes what one workstation GPU can do.

@silvashenath — you asked for a llama-bench result and, more to the point, for prompt ingestion numbers on a dense model. I do not have one yet (the card is busy for the next few hours), but I do have something adjacent that may be more useful anyway: server-side timings from real requests, so the numbers include the chat template, the KV type and the actual prompt rather than a synthetic sweep. Labelling that plainly so nobody reads it as llama-bench.

RTX 4090, llama.cpp, -ngl 99 -ctk q8_0 -ctv q8_0 --jinja, cache_prompt: false.
Model: Qwen3.8-27B-UD-Q4_K_XL (16.4 GB).

On your actual problem — ingestion, not decode. You said A3B gives you ~3,500 t/s prefill and dense drops hard. That gap is the same phenomenon from the other side: on this box the dense 27B prefills at ~2,700 t/s against 7,632 t/s for Qwen3.6-35B-A3B on the identical flags. Dense costs you at both ends, and for a long system prompt the ingestion end is the one you feel first.

Two more things worth having:

MTP pays more on long prompts than short ones — 2.17× at 8.4K in, 1.74× at 246 in. If you benchmark it with a short prompt you will understate it for agent workloads, where the prompt is the whole conversation.

It was lossless in this test. Same greedy request with and without MTP produced byte-identical text (md5 1c7c2029, three runs each). That is a check worth repeating locally rather than assuming — it is a speculative path, and "same speed, different output" would be the thing to catch.

The cost is context: with MTP enabled I could not go past 65K on this card, against 131K with it off.

For scale, the same box on Qwen3.6-35B-A3B-UD-IQ4_XS (16.5 GB — nearly the same file size) does 168 t/s decode and 7,632 t/s prefill. Same bytes on disk, ~3.6× the throughput, and that is the whole argument for the missing A3B in this generation.

If a real llama-bench table is still useful once the GPU frees up, say so and I will run it for both models with the same flags.