IQ3_XXS stops after ~70 tokens on llama.cpp & RTX 5060ti CUDA toolkit 13.2.78
UPDATE - fixed after updating CUDA toolkit to 13.3.
When running this model on llama.cpp server i get about 70 tokens of reasoning and then it stops. There is no problem when i use Unsloth/Qwen3.8-Flash-Next-GGUF/Qwen3.8-Flash-Next-UD-IQ3_XXS
Has anyone the same problem?
Some funny reasoning that i get before it breaks:
Prompt: "make beautiful snake game in single HTML file."
Reasoning:
- "The user wants me to translate the "Make HTML snake game in a single HTML file" part, probably meaning the game's story. So I need to translate the English text into Chinese."
- "The user wants me to replicate the code in the attached image. I need to look at the image. There is a picture with text "C H". This is a typical question-answer pattern."
- "η¨ζ·θ¦ζ±ηΏ»θ―ζδΈζοΌδ½εΏ ι‘»δΏζζη»ηζ‘η¨ζ·ζεΎγ"
- "The user wants me to create a "beautiful" variant of a programming task. This is a well-known visual - a simple HTML game that implements a matching system for the visual. In this way, we can use a variety of styling tools to create a matching system, but it's probably not necessary to dive deep into the internals of that specific styling. So we can use a variety of styling tools to create a matching system, but it's probably not necessary to dive deep into that specific styling.
The user has requested a "make beautiful snake game" - this sounds like a simple HTML game that implements a matching system for the visual. In this way, we can use a variety of styling tools to create a matching system, but it's probably not necessary to dive deep into that specific styling.
The user has requested a "make beautiful snake game" - this sounds like a simple HTML game that implements a matching system for the visual. In this way, we can use a variety of styling tools to create a matching system, but it's probably not necessary to dive deep into that specific styling."
Here are my params:
llama-server --host 0.0.0.0 --port 1234 -m ~/models/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf -lm mmap -t 6 -c 40000 -fa on -np 1 -sm layer -ctk q8_0 -ctv q8_0 -fit on -fitt 512 --jinja
My system info:
0.00.394.762 I cmn common_param: common_params_print_info: build 10968 (41abbfd59) with GNU 13.3.0 for Linux x86_64
0.00.394.765 I cmn common_param: common_params_print_info: verbosity = 4 (adjust with the `-lv N` CLI arg)
0.00.394.765 I cmn common_param: device_info:
0.00.483.141 I cmn common_param: - CUDA0 : NVIDIA GeForce RTX 5060 Ti (15841 MiB, 15078 MiB free)
0.00.562.917 I cmn common_param: - CUDA1 : NVIDIA GeForce RTX 5060 Ti (15849 MiB, 15697 MiB free)
0.00.661.424 I cmn common_param: - CUDA2 : NVIDIA GeForce RTX 3060 (11911 MiB, 11785 MiB free)
0.00.661.439 I cmn common_param: - CPU : Intel(R) Core(TM) Ultra 5 245K (63912 MiB, 63912 MiB free)
0.00.661.512 I cmn common_param: system_info: n_threads = 6 (n_threads_batch = 6) / 14 | CUDA : ARCHS = 750,800,860,890,900,1200,1210 | USE_GRAPHS = 1 | FA_QUANTS = all | BLACKWELL_NATIVE_FP4 = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
Thank you for the detailed explanation. Does your llama.cpp build support the Q2_0 format? Our models use Q2_0, whereas the Unsloth models do not. If possible, could you rebuild llama.cpp using the latest version and try again? I ran the model using your exact command and prompt and it worked normally on my side.
I have llama.cpp from 2 days ago. I will download latest and build it in a moment.
Your ftype is different (different version of model)?
This is what i get from llama-cli:
llama-cli -m ~/models/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf -lm none -t 6 -c 40000 -fa on -np 1 -sm layer -ctk q8_0 -ctv q8_0 -fit on -fitt 512 --jinja
Loading model...
ββ ββ
ββ ββ
ββ ββ ββββ ββββββββ ββββ βββββ βββββ βββββ
ββ ββ βββββ ββ ββ ββ βββββ ββ ββ ββ ββ ββ
ββ ββ βββββ ββ ββ ββ βββββ ββ βββββ βββββ βββββ
ββ ββ
ββ ββ
build : b10968-41abbfd59
model : ~/models/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf
ftype : IQ3_XXS - 3.0625 bpw
modalities : text
available commands:
/exit or Ctrl+C stop or exit
/regen regenerate the last response
/clear clear the chat history
/read <file> add a text file
/glob <pattern> add text files using globbing pattern
> make beautiful snake game in single HTML file.
[Start thinking]
The user wants me to generate a single HTML file that renders a game. The instruction is repeated and emphasized in several places. I need to follow that idea. So the response should be in French. The user wants a HTML file that renders a game. I should produce a complete single HTML file that renders a game. The game is Snake (Serpent). In French the serpent is also called serpent, so I'll use this term.
I need to produce a single HTML file that renders a game. The instruction "generate a single HTML file" suggests I need to produce a single file. I'll generate the file with the requested content.
Je dois rΓ©pondre avec le jeu "Snake" mais la consigne dit "single HTML file that renders a game" - peut Γͺtre interprΓ©tΓ© comme un fichier unique. Je dois crΓ©er un fichier unique qui rend un jeu.
Je dois crΓ©er un fichier unique qui rend un jeu. Le fichier unique doit Γͺtre autonome. Peut-Γͺtre je dois crΓ©er un fichier unique qui est autonome et complet. Peut-Γͺtre je dois inclure un fichier unique qui est autonome et complet. Peut-Γͺtre je dois crΓ©er un fichier unique qui est autonome et complet.
Je dois crΓ©er un fichier unique qui est autonome et complet. Peut-Γͺtre je dois crΓ©er un fichier unique qui est autonome et complet.
Peut-Γͺtre je dois crΓ©er un fichier unique qui est autonome et complet. Peut-Γͺtre je dois crΓ©er un fichier unique qui rend un jeu.
Je vais crΓ©er un fichier unique qui est autonome et complet. Peut-Γͺtre je dois crΓ©er un fichier unique qui est autonome et complet. Peut-Γͺtre je dois crΓ©er un fichier unique qui rend un jeu.
Peut-Γͺtre je dois crΓ©er un fichier unique qui est autonome et complet. Peut-Γͺtre je dois crΓ©er un fichier unique qui rend un jeu.
Je vais crΓ©er un fichier unique qui est autonome et complet. Peut-Γͺtre je dois crΓ©er un fichier unique qui est autonome et complet.
Peut-Γͺtre je dois crΓ©er un fichier unique qui est autonome et complet.
Je vais crΓ©er un fichier unique qui est autonome et complet. Peut-Γͺtre je dois crΓ©er un fichier unique qui est autonome et complet.
Peut-Γͺtre je dois crΓ©er un fichier unique qui est autonome et complet. Peut-Γͺtre je dois crΓ©er un fichier unique qui rend un jeu.
Je vais crΓ©er un fichier unique qui rend un jeu.
Peut-Γͺtre je dois crΓ©er un fichier unique qui rend un jeu.
[ Prompt: 68.7 t/s | Generation: 28.7 t/s ]
That's just metadata. We update it when uploading the model and the value you saw was from the version I had locally. However, to make sure there was nothing wrong with the uploaded version I downloaded and tested it again, and everything worked fine.
./build/bin/llama-cli -m IQ3_XXS/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf -ngl 999 -lm mmap -t 6 -c 40000 -fa on -np 1 -sm layer -ctk q8_0 -ctv q8_0 -fit on -fitt 512 --jinja
Loading model...
ββ ββ
ββ ββ
ββ ββ ββββ ββββββββ ββββ βββββ βββββ βββββ
ββ ββ βββββ ββ ββ ββ βββββ ββ ββ ββ ββ ββ
ββ ββ βββββ ββ ββ ββ βββββ ββ βββββ βββββ βββββ
ββ ββ
ββ ββ
build : b10743-8887a48f0
model : IQ3_XXS/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf
ftype : IQ3_XXS - 3.0625 bpw
modalities : text
available commands:
/exit or Ctrl+C stop or exit
/regen regenerate the last response
/clear clear the chat history
/read <file> add a text file
/glob <pattern> add text files using globbing pattern
> make beautiful snake game in single HTML file.
[Start thinking]
The user wants a beautiful snake game in a single HTML file. Let me craft something distinctive β a really polished, alive-feeling game.
Design direction: Snake game. What's most characteristic of the subject? The game board itself β a grid, a retro arcade feel. Instead of a generic landing header, I should open with the game board front and center
, framed as an arcade cabinet.
Let me think about a distinctive aesthetic. Options:
- Retro CRT phosphor green terminal β classic but a bit expected.
- A warm, tactile "tabletop" feel β like a wooden/bakelite arcade console.
- A modern take: dark slate with a subtle grid, snake rendered as a smooth segmented chain with rounded joints, glowing food with pulsing halo.
Let me go with something like a "midnight arcade" but with a richer palette than a flat black board with a single neon accent β that reads cheap. I'll use a layered palette: deep teal-slate board, warm amber/coral
food, mint-green snake with gradient scales, plus a warm cream panel. Let me think about the palette more carefully.
Palette idea: deep ink-blue background (#0e1521-ish) with layered ambient (subtle vignette, faint noise, drifting gradient mesh in muted teal and rust β muted, not vivid). Board: slightly lighter panel with an inn
er grid of dashed lines in low-opacity. Snake: gradient from lime-chartreuse at the tail to bright spring green at the head? Or a more sophisticated approach: snake body in a warm gradient of jade β chartreuse wit
h darker scale outlines; head with two eyes that rotate to face direction.
Food: a glowing berry with a pulsing ring, plus occasional "golden" bonus with particles.
Typography: distinctive display face + readable body face. Google Fonts. Display: something with character. Options: "Bungee" (arcade blocky), "Monoton", "Silkscreen" (pixel), "Space Grotesk", "Chivo Mono", "Syne"
, "Archivo Expanded", "Anton". For arcade feel without being clichΓ©... "Bungee" is quite bold and blocky, quite arcade-appropriate. Or "Syne" (weird extended). Or "Space Grotesk" for body with "Bungee Shade"? Hmm.
I like: Display = "Bungee" (very arcade, blocky, uppercase), Body = "Space Grotesk" or "IBM Plex Sans". And mono for scores: "Space Mono" or "JetBrains Mono". Let's do Display: Bungee, Body: Space Grotesk, Numeric
: Space Mono. That's three but display+body is the core pairing; mono for tabular numbers is fine.
Actually Bungee might feel a bit clichΓ©. Alternative: "Archivo Black" or "Anton". Hmm. Let me use "Bungee" for the logo/title β it's genuinely characteristic of arcade cabinets. And it's a single-weight display fa
ce, so I'll pair it with Space Grotesk for body and Space Mono for stats. Good.
Layout: Not a centered stack. Let's do an asymmetric layout: left column = the game cabinet (board in a bezel with a screen), right column = a "control panel" stack with score, high score, speed meter, controls le
gend, and a live "log" of events. On mobile, stacks vertically.
Let me build:
Okay, i used latest llama.cpp and the problem still exists. I already downloaded model files twice to be sure. Also tried -dev CUDA0,CUDA1 and only CUDA0 to exclude architecture mix. Nothing helps. Very strange. I will download once again and try. Maybe it's a specific llama.cpp problem with my hardware setup. I will post a message when i figure it out. Thank you for your time.
Okay i tested it on separate GPUs. It works correctly only when i use -dev CUDA2 (my RTX 3060). On my other cards (2x RTX 5060ti) and all combined i get funky reasoning and it breaks as before.
So the problem must be with llama.cpp running this model on NVIDIA Blackwell architecture (RTX 5060ti)!
Would be nice if someone could test it also on 5060ti and other Blackwell GPUs, so we can report this issue to llama.cpp.
I searched through the llama.cpp issues a bit and found this:
https://github.com/ggml-org/llama.cpp/issues/28581
So yes, it looks like this is a bug specific to the 5060ti. It seems it is solvable by upgrading the CUDA version.
I searched through the llama.cpp issues a bit and found this:
https://github.com/ggml-org/llama.cpp/issues/28581
So yes, it looks like this is a bug specific to the 5060ti. It seems it is solvable by upgrading the CUDA version.
You're right! My CUDA compiler identification is NVIDIA 13.2.78 - exactly the one that is reported. Always those CUDA bugs! π