loops
i'm trying 2.04 bpw with latest exllamav3, and (when it doesn't loop) quality seems comparable with
https://huggingface.co/anoane/DeepSeek-V4-Flash-0731-vllm-moet-sm120-cache
but on vllm-moet i get ZERO loops, did 18h benchmarks, zero loops, pi.dev usage, zero loops
while with exllamav3 and exl3 it's totally unusable, it loops half of the time, what could be the cause?
i tried min_p 0.05 and it helps a little but still unusable, any way or recommendation or setting to make it stable?
i recompiled everything from scratch for cuda 13.3.X
It often comes down to sampler settings, and what your frontend/harness is setting by default. It's really all incredibly messy and hard to figure out exactly what settings TabbyAPI should respond to, where they appear in the request, and so on. Can you say more specifically what you're doing, like what harness are you using, what settings, what are your settings in TabbyAPI (or are you using it outside of TabbyAPI?), etc.
It often comes down to sampler settings, and what your frontend/harness is setting by default. It's really all incredibly messy and hard to figure out exactly what settings TabbyAPI should respond to, where they appear in the request, and so on. Can you say more specifically what you're doing, like what harness are you using, what settings, what are your settings in TabbyAPI (or are you using it outside of TabbyAPI?), etc.
very likely it was due to low quantization, i switched to 2.52 bpw (it loaded fine on the 6000 PRO without dspark), and with exact same settings i got 0 loops so far, but i'm at first hour of benchmarks, i'll use it 1 full day and see
if it's stable i'll try some surgery, maybe switch some layers that don't affect perplexity that much to lower quantization on those and try to make space for dspark, but hey, without dspark i get ~86 tok/s with max reasoning which is already not bad at all, very usable
Do be mindful of sampler settings, but yes, 2 bpw is very aggressive quantization. If you're struggling to fit the DSpark model, in the quants I uploaded all the DSpark modules are quantized to 3 bpw. It may work just as well with the routed experts in 2 bpw (util/convert_mtp.py takes only a single bitrate, which applies to attn and shared experts as well, so while it can convert the MTP layers, you'll want to transplant only the routed experts from the output.)
i will definitely try, btw this is the small/quick shim i'm using for my (pre-existing) benchmarks
(sorry couldn't upload the zip here)
openAI shim and preliminary metrics (still didn't finish running, can't compare quality to vllm-moet yet)
https://limewire.com/d/iV5RL#mM03cjdUkn