Works with 8 RTX3090
Many thanks for your great work. Thanks to your model variant and great instruction in the model card, I managed to get the model running on my LLM rig with 8 RTX3090. I had to lower the context to 240k because I received an OOM message at startup, but now it works. You got a new follower. Looking forward on upcoming models and your support for the RTX3090 people :D
Appeared that 4bit kv-cache attention generation speed is almost twice slower on 3090 than fp8 - tried both turboquant_4bit_nc and int4_per_token_head. Taking a look into --enable-experts-parallel mode with group-size 128 for experts to get more space for fp8 KV cache.
update: i have tested with gs=128 for expert modules that work only with --enable-expert-paralleland the available context-size with fp8 kv cache with--gpu-memory-utilization 0.945 is:
- current checkpoint (tp over expert modules as well): 145K
- gs128 checkpoint (ep): 175K, but in my case token generation slower by 10%
also appeared that autoround RTN quantization does not require patched vllm for fp8 kv and takes around 10 minutes, does not require complex hardware, preserves MTP layer. receipe for gs 128 experts:
auto_round /home/ubuntu/models/Hy3-tencent \
--model_free \
--scheme W4A16 \
--group_size 128 \
--layer_config "{ '.*shared_mlp':{'group_size':64}, 'experts':{'group_size':128}, '.*eh_proj':{'bits':16,'data_type':'float'} }" \
--ignore_layers "model.layers.0" \
--output_dir /home/ubuntu/models/autoround/tencent-Hy3-RTN-EP
to preserve speculative decoder quality ignore layer 80 as well.
for me MTP works slower and uses more VRAM, so less space for kv cache.
Would you release this AutoRound version with gs=128? Using turboquant here degrades the speed too much (70% slower than FP8), so it would be wonderful to have this version with more space for FP8 kv cache
Thank you so much! I am going to test it right now
Tencent released their own GPTQ variant. What do you say about it? https://huggingface.co/AngelSlim/Hy3-GPTQ-Int4
Is it beneficial to use this one ?
In my hardware, with the Tencent model I would get 96k maximum of kv cache. With avtc's model I get 163k. So it's a considerable diference. I wouldnt be able to tell the difference in accuracy between this too models but I think it wouldn't be very different.
I just ran the AA_LCR benchmark with the avtc model and it got 74% which is on par with the BF16. I am trying to run other benchmarks to assess the inteligence but the AA_LCR score was very good. ** I need to make a correction. When I say avtc model I mean the AutoRound version with FP8 kvcache, not this GPTQ version. I am running the SWE-bench right now.
** I need to make a correction. When I say avtc model I mean the AutoRound version with FP8 kvcache, not this GPTQ version.
I have chery-picked commit to allow using KVarN near lossless 3bit cache, it seems it does not hurt token generation speed, but have not tested yet with large context length (on 10-30% context length I see parity with fp8 in speed).
the branch to build vllm from (readme updated): https://github.com/avtc/vllm/tree/fix/load-quantized-hy3-mimo-v2.5-kvarn-0.26
Excelent!! Do you have a command/script example to run this 3bit cache?
Yes, i have updated readme - so need to build from another branch and run with --kv-cache-dtype kvarn_k4v2_g128
Perfect! Thanks!!
It works like a charm. Does it work with the AutoRound model or the AngelSlim GPTQ model?