Improved quality by changing the chat_template.jinja

#1
by Neiko2002 - opened

I have changed the chat_template.jinja to the official one and the scores on the https://benchlocal.com/ bench packs went from an average of 60.4 points to 82 points (Qwen/Qwen3.6-27B-FP8 has 82.5). Afterwards it has the highest Hermesagent scores of all tested 27B models. Might be worth changing it here in the repo.

That’s surprising! Let me confirm: do you mean that we should use https://huggingface.co/Qwen/Qwen3.6-27B/blob/main/chat_template.jinja instead?

Yes. You can install benchlocal on your machine and run your model as it is, than replace the jinja file and try again. You will see a huge difference. The currenty version get 60.4 points across all benchmarks, which is less than Qwen 3.5 9B (76.9 points).

I sincerely appreciate your testing and feedback! I may not have enough time to reproduce the issue myself in the near future, so I directly updated the chat_template.jinja according to your suggestion and added a corresponding note to the Model Card.

I have to thank you for the quant. Its on par with the one from Lorbus, but thanks to the Qwopus finetune it uses less tokens and is therefore faster:
image

And here you can also see the high Hermes score, which is amazing.

Neiko2002 changed discussion status to closed

Sign up or log in to comment