Recommended system prompt
Hello. I have tried a few of the 27b finetunes and this one is quite good, but all of them have a problem with logical errors for me.
Things like someone is already in the room, but open the door to enter, body positioning, just doing things that dont make sense. Its hard to picture the story in your mind when things like this happen.
Lowering the temp helps a little, but it happens far too often still. Do you have a recommended system prompt to use with this that might help?
What quant are you running?
Q8_0_ATTN
I use kobold with sillytavern. I have this problem with other 27b finetunes also, not just this one. So i was thinking maybe my prompt is bad.
What's your prompt?
I use llamaception (the prompt) with chat completion with a custom narrator-style blurb.
Also if you're not using thinking you -really- should be. It changes the output dramatically.
Ive always used text completion and i just tried chat completion. Not sure if im doing it right. If i have request model reasoning checked, it leaves empty think tags at the top of every message. If i have it unchecked it does think, but it gives me refusals. I think i just dont know how it works. Do i write my system prompt into where it says main prompt or do i insert an additional promp above it?
I went back to text completion and tried with thinking on. i did get it to work, but the problem now is i have to have my response tokens somewhat low, at like 500, and keep hitting continue 5-6 times to get through the thousands of thinking tokens to get a response. If i just set the response tokens to 5000 it wont think and will just give me a bunch of repeating paragraphs. Not only that i wont be able to control how long the response is even if it does think properly.
Thank you for the help, but I have determined models that need thinking arent for me. Ill just go back to my other models and forget about qwen :(
You can control the reasoning tokens in llamacpp so thinking doesn't take up entire blobs of text ;)
--reasoning on
--reasoning-budget 1000
--reasoning-budget-message "...\nI think I've explored this enough, time to respond.\n"