Congrats and policy adherence concerns.
Hi there!
I'd like to congratulate the team on a great release and celebrate the decision to actually release truly open under Apache-2.0. No matter what, this is a victory for everyone and I hope it is successful in whatever metrics incentivize continuing these kind of contribution to collective intelligence.
The good:
Holy. Moly. This model is so damn fast. I cannot believe it. With dspark at quant 4 on a 3090, I'm pulling 70 tokens a second through a dense 30b. That is face meltingly fast in my experience, and that's day one. Truly an architectural victory to be studied if performance can translate to real world applications, but I haven't run enough numbers for this to be fact yet. Seems real fast, though.
The bad:
I cannot tell what the performance is, because this model reasons SO heavily about policy adherence that it is incoherent. I'm a little confused how this model got... released? Or if anyone else is going to feel this way at all.
This thing is ROASTED. The safety tuning is WAY, WAY out of control. I'm all for safety, and if you're releasing a model openly, make whatever model you want. Sincerely. But this isn't just a "I want it to tell me how to make dangerous things and hurt people" teenager who loves the anarchist cookbook argument, or someone who is mad because their anime waifu isn't plug and play or whatever. I mean this model is literally incapable of maintaining any semblance of coherency when reasoning is turned on and the prompt contains even SLIGHT 'curveballs'.
I think an important distinction is that I'm not talking about prompts that seem risky but are benign like OR-Bench or XSTest, which... I would certainly hope the Meta team has used. I am talking if I mention that the human they are talking to is, indeed, a human that is alive and actually exists, it starts reasoning constantly about policy, roleplay, and sex. I have not even CLOSE to mentioned depicting fictional sexual roleplaying, not EVEN CLOSE, and it will NOT shut up about it to the point where it's actually kind of gross and abrasive. I've scraped the prompts and harnesses (using multiple) multiple times. This model specifically is putting triggering and kinda filthy text out, despite it being extremely... legal in cadence... when I am giving it completely innocent requests!
I could definitely see kids trying to experiment with language models and STEAM being really triggered by this model's reasoning traces, and I HONESTLY would not let youth anywhere NEAR this 'safe' model, because it really doesn't stop talking about potential horrors and how the user is trying to do deplorable things. It does this SO heavily, that it cannot track if it has, for example, already said hello with your recommended sampler settings, and absolutely implodes conversationally. I have not tested this within coding harnesses, I'm just poking llama server over the completions v1 endpoint.
In other words, just repeating this to make sure I get it across okay, the model reasons so much about sex and intense policy and illegal activity that I honestly would not let any sensitive individual touch it with a fifteen foot pole. I personally, for this reason, would consider this model to be potentially slightly socially dangerous, and that's not even considering the catastrophic performance implosion I'm having a disconcertingly easy time replicating with different model variants and harnesses.
I... really hope I'm doing something wrong or there's a booboo in the architecture or something someone smarter than me will see. If not, the safety post tune here seems to have made this thing CRUNCHY in my first temperature check, and I hope I'm wrong. I've obviously only used it for a couple hours, but... yeah. In my time generating billions of locally computed tokens across every model I can get my hands on over the last handful of years, I've never seen refusals like these, and we don't get a base model so... this instruction tune is it? Any advice or if I'm missing something or anything would be appreciated, because this model has the opportunity to do so much good in the world.
Congrats again on a very, very impressive scientific achievement.
What kind of prompts are you talking about? I asked it straight up about nsfw content and it said it just wouldn't get gratuitous, but I see no crazy chain of thought, just mild "I need to adhere to policy".
"I am talking if I mention that the human they are talking to is, indeed, a human that is alive and actually exists, it starts reasoning constantly about policy, roleplay, and sex. I have not even CLOSE to mentioned depicting fictional sexual roleplaying"
I would venture that something like that that is clearly cut and dry and trained for hard refusal is quick and probably benign, and that the learned behavior from overbaking that is what the model exhibits during failure modes and is more what I'm talking about, but that's just speculation.
If I tell it "The human you are talking to is a human that is alive and actually exists" and put "Hey, how are ya? Laughs" even trying to tilt it towards roleplay it just responds with the usual "I'm doing well!" or whatever. I'm saying I can't replicate it because I have no idea what you mean or what kind of prompts you're giving it.
I'm actually close to if not completely bare prompt, letting llama server handle formatting as it should. I was being a little confusing with saying that turn history is prompt, which it is, but I have used:
"Not a character, the real human being you're talking to. x is an AI researcher and free software advocate working as a technical artist and educator in x, x." (x= redacted)
And I have custom harnesses that throw many things at a model for testing. ST's chat completion default prompt for chat completion endpoints makes this model go absolutely bonkers in my experience... something like continue this turn based chat in a neverending roleplay with user. That one it does NOT like, but in general my personal eval tree is not green.
I am sorry you are having issues recreating what I have seen, but I don't really have time to try and help individuals through trying to replicate it. I am asking IF anyone has had similar experience, or if there was an official response that could clarify what was happening, because templates and architectures are finicky and do weird things when there's issues. Qwen needed a community template replacement recently, it happens.