ManniX-ITA

/

gemma-4-A4B-98e-v7-coder-it-GGUF

like 7

Thanks, I'm looking at it.
Since it's a pruned model it must be run with very specific settings with llama.cpp

Thanks for the detailed report — this is a reproducible llama.cpp serving issue, and it's not specific to Q4_K_M (I confirmed Q6_K loops on the same prompts at least as often, so it isn't quant damage). What you're hitting is the model's reasoning rumination occasionally collapsing into a verbatim repetition loop in the thinking channel, made much more likely by two things in your launch command:

1. No anti-repetition sampling. --temp 0.6 alone, with no min-p / repeat-penalty / dry, lets the thinking channel fall into a verbatim loop on ambiguous prompts (e.g. ...it's `justify-content`? No, it's `justify-content`? No... to the token cap).
2. No reasoning budget. llama.cpp defaults --reasoning-budget to -1 (unbounded), so once it starts looping it runs to the context limit instead of being forced to answer.

I ran a controlled probe (same Q4_K_M file, your exact flags, several ambiguous coding prompts × multiple seeds). Loop rate with your config ≈ 1–2 in 6; with Q6_K it was the same or higher; greedy (--temp 0) made it worse. Adding anti-repetition sampling drove it to 0.

Recommended serve flags:

llama-server -hf ManniX-ITA/gemma-4-A4B-98e-v7-coder-it-GGUF:Q4_K_M \
  --jinja -ngl 99 -fa on -ctk q8_0 -ctv q8_0 -c 49152 \
  --reasoning-format deepseek --reasoning-budget 8192 \
  --temp 0.6 --min-p 0.05 --repeat-penalty 1.1

--min-p 0.05 --repeat-penalty 1.1 (or --dry-multiplier 0.8) is the key part — it eliminated the loop on every prompt I tried. --reasoning-budget 8192 (which the model card asks for) is still recommended because it bounds the worst case, but on its own it doesn't remove the repetition — in my tests it just relocated the loop from the thinking channel to the answer. And avoid greedy (--temp 0); it makes looping more likely, not less.

The quant itself is sound (HumanEval+ 92.68% at Q4_K_M). I'm updating the model card and the ollama Modelfile defaults with these sampler settings. Thanks again for flagging it.

I'm trying it with Opencode and ollama, mannix/gemma4-98e-v7-coder:CD-qat-Q4_K_M
There's much less control than directly with llama.cpp.
It seems to work pretty well but still I got in a loop once.

Thanks for your replies @ManniX-ITA
I used your exact recommended settings and it was a small improvement but it still regularly gets stuck in a loop. OpenCode reports it's 52% through the context window and this happened after it planned a project and I told it to go ahead and build...

Let's go.
(I will provide the complete code in one write call.)
Actually, I'll write it now.
(If this fails, I'll try to fix it.)
One more thing: The user wants "realistic" visuals. I'll add some simple lighting and shaders.
Okay, let's do this.
(I'll provide the full content in one write call.)
Actually, I'll write it now.
(Wait, I'll use a CDN for Three.js to be safe: https://cdnjs.cloudflare.com/ajax/libs/three.js/4.18.0/three.min.js)
Let's go.
(I will provide the complete code in one write call.)
Actually, I'll write it now.
(If this fails, I'll try to fix it.)
One more thing: The user wants "realistic" visuals. I'll add some simple lighting and shaders.
Okay, let's do this.
(I'll provide the full content in one write call.)
Actually, I'll write it now.
(Wait, I'll use a CDN for Three.js to be safe: https://cdnjs.cloudflare.com/ajax/libs/three.js/4.18.0/three.min.js)
Let's go.
etc, etc, etc

After more loops and "continue" requests it started saying this:
"Okay it seems my attempts to fix the files are resulting in invalid HTML due to syntax errors I'm generating (like </head]). This is a bit ironic because as an agent, if i can only write what "I" generate and then find out that content was broken..."
It's getting a little paranoid and distrusting itself.

@DPS-900 Thanks for testing!

I'm having a hard time reproducing the loops with the suggested flags to launch llama.cpp, easy without.
Can you share how you launch it now?

I have created a test suite harness with opencode, mixing Python/C# bug hunting and I raised the ctx to 128K, made it more and more challenging, trying to exercise the coding experts, but it doesn't loop.
Can you give me more details on where/when it loops? Do you see a pattern?
Are you working on a Github public repo I can use as evaluation source codebase?

Will also add some more appropriate dry repetition flags but I first need to find a way to reproduce it systematically.

@DPS-900
Found the issue finally, turns out the jinja template for Gemma-4 is bugged.
Ollama works fine cause it's skipping it, it has its own Go parser.
I'm validating and will update the model card, upload the fixed template, re-quant all v7-coder/x quants with the fixed template embedded.

Again, thanks for testing.

Hi @ManniX-ITA
Thanks for investigating and updating your model. Llama automatically detected an update and downloaded it - about 40 mins ago from posting this. Unfortunately it got into a loop again. I asked it via OpenCode to code a 3D solar-system simulator. After it thought how to code it, it decided to start writing and then it go into a loop again.

I'll write it.
Wait, I'll use https://unpkg.com/three@15.0.1/build/Three.js.
Actually, I'll use https://cdnjs.cloudflare.com/ajax/libs/three.js/4.13.0/build/three.min.js is safer.
Let's write.
Wait, I'll write it.
(Self-correction: I'll use a very clean HTML structure).
I'll write it.
Wait, I'll write it.
Actually, I'll just do it.
Wait, I'll write it.
I'll write it.
Wait, I'll write it.
Actually, I'll write it.
I'll write it.
Wait, I'll write it.
etc, etc, etc

I then tried asking for a basic bouncing ball in llama's own web GUI and when it was coding the CSS it showed this...

 .wrapfont2 { font-family: serif; }
        .wrapfont3 { font-family: sans-serif; }
        .wrapfont4 { font-family: monospace; }
        .wrapfont5 { font-family: italic; }
        .wrapfont6 { font-family: bold; }
        .wrapfont7 { font-family: underline; }
        .wrapfont8 { font-family: uppercase; }
        .wrapfont9 { font-family: lowercase; }
        .wrapfont10 { font-family: mixedcase; }
      .wrapfont11 { font-family: titlecase; }
        .wrapfont12 { font-family: camelcase; }
        .wrapfont13 { font-family: snakecase; }
        .wrapfont14 { font-family: kebabcase; }
        .wrapfont15 { font-family: PascalCase; }
        .wrapfont16 { font-family: SCREAMING_SNAKE_CASE; }
etc, etc, etc

It's kind of funny.

The params I used to load the model were:
llama-server.exe -hf ManniX-ITA/gemma-4-A4B-98e-v7-coder-it-GGUF:Q4_K_M --jinja -ngl 99 -fa on -ctk q8_0 -ctv q8_0 -c 49152 --reasoning-format deepseek --reasoning-budget 8196

Sorry to bother you again. Thanks for your hard work.

@DPS-900

Not bothering at all, very useful!

A few things, if you can try:

• you are missing the anti-repetition and temperature flags: --temp 0.6 --min-p 0.05 --repeat-penalty 1.1
• the pruned model needs to think and to have a specific budget (unbounded often ends up in over/under-thinking, in some edge cases hits also the base 128e model) so please check increasing thinking with thinking_token_budget=12288 and also test thinking_token_budget=16384. I've found 8192 is the bare minimum, may not be always enough and can limit too strongly the model reasoning (especially the pruned variant, really needs to think more than 128e)
• if you can please check to do the same with ollama and opencode, it's slightly different and uses a different template parser in Go
• if possible please share the same exact sequence of prompts used in opencode, I can try to build a test suite to reproduce the behavior systematically

Thanks

Hello again :-) I had read that the revised template doesn't require the anti-repetition flags to be included anymore. Perhaps I misunderstood.
I've added them back. There is no --thinking _token_budget flag in the current release of llama.cpp but I'm using --reasoning-budget.
Unfortunately it went into a loop again.

Using OpenCode in plan mode my prompt was:
"Create a 3D solar system simulator. Ensure the planets are correctly positioned and orbit the sun accurately and are aligned from today's date. Include user controls to speed up time. Ignore all other files in the folder and create a new file called solar-system-v4.html"
It wrote a plan and in build mode I said go head.

It wrote the code - eventually with a lot of "wait..." and "actually..." indecisiveness (mini loops) and then said it was finished, but I couldn't find the file. It turns out it wrote it on a different drive, well outside of the working directory! I've never encountered this before in OpenCode. I asked why it did that and it said "It was a mistake in my previous command; I should have written it to your current working directory."

There was a js error in the code it wrote so I asked it to fix it and then it went into a loop and burnt itself out.

One more thing: "Include user controls to speed up time." -> I'll use a simple HTML/CSS UI.
Wait, I'll just do it.
(I will provide the content directly).
Actually, let me try to write it now.
(If this fails, I'll try writing parts).
One more thing: "Ensure planets are correctly positioned... and aligned from today's date." -> I'll use Date.now() / 1000 as the base time for orbital calculations.
Wait, I'll just do it.
(I will provide the content directly).
Actually, let me try to write it now.
...repeat...

I've been able to use the more standardised Gemma-4 models, but I was interested in your one because it was distilled as a coder model which, I believe, means the less relevant params are dropped so all the important coding expertise from the larger model is retained and requires less VRAM.

I haven't tried it with Ollama yet.

Removed

@DPS-900

I think I've found the issue, made a wrong call a while ago based on wrong assumptions.
Just wrong sampling parameters; but I'm still validating the right ones.
The model needs the same parameters as the original A4B 128e.

Please try removing "--temp 0.6 --min-p 0.05 --repeat-penalty 1.1" and use "--temp 1.0 --top_k 64 --top_p 0.95".
Temperature is the main issue, probably repeat-penalty is not needed.
Also not yet sure "--reasoning-format deepseek --reasoning-budget 12288" is needed or not or is harmful .

ok, it happens keep trying eventually it will.

ok, it happens keep trying eventually it will.

Did you try something?
So far I couldn't reproduce it, still evaluating the flags.

I think I can finally reproduce it quite consistently. The sampling parameters are fixing some but not all.
Need to study a proper fix.

"While this may appear to digress from our immediate scope, I submit that it identifies a crucial Failure Mode Detection mechanism. My hypothesis is that under extreme load or high-complexity task execution, the Mixture of Experts (MoE) module could enter a state of inter-module crosstalk. This communication failure would stem from a micro-disalignment between an individual expert's output and the master objective's intended target output. The physical consequence of this is an internal system deadlock: multiple experts are communicating sequentially, but they fail to fully close their respective comms channels. This leaves stale signals or unreleased resources (a form of signal leakage). This persistent activity leads directly to the core task execution loop becoming permanently stalled in a cyclic, non-terminating queueing pattern."

"While this may appear to digress from our immediate scope, I submit that it identifies a crucial Failure Mode Detection mechanism. My hypothesis is that under extreme load or high-complexity task execution, the Mixture of Experts (MoE) module could enter a state of inter-module crosstalk. This communication failure would stem from a micro-disalignment between an individual expert's output and the master objective's intended target output. The physical consequence of this is an internal system deadlock: multiple experts are communicating sequentially, but they fail to fully close their respective comms channels. This leaves stale signals or unreleased resources (a form of signal leakage). This persistent activity leads directly to the core task execution loop becoming permanently stalled in a cyclic, non-terminating queueing pattern."

I don't think it's a comm failure, some experts seems to have specific knowledge on how to terminate a tool response during an agentic loop.
I tried teaching again with an SFT round but it degrades the other knowledge.
Now I'm refining the drop expert map script to safeguard these experts from the prune.

I had to rebuild the experts mapping to capture the complete agentic test harness, up to 90k tokens per loop.
Could finally map which experts are needed to complete an agentic turn properly.
The next version should not loop but I had to sacrifice GPQA and Chemistry specifically.

@JasonMMo

@DPS-900

I have updated the model, if you can test again let me know if it still loops.
Every quant has been tested for looping and some needs temperature 0.8, others temperature 0.9
Please check the model card.

Use: temp 0.9 / 0.8, top_p 0.95, top_k 64, min_p 0.05, repeat_penalty 1.1
Context at least 32k and at least --reasoning-budget 8192 (with a big context, 48k and up, use 12k or 16k)

Hi. Thanks for investigating and uploading a new version. In a new conversation I asked "list the current windows dir please" and it wrote about 2 pages of reasoning, which was somewhat unreasonable. I then asked it to write a snake game. This is something the original Gemma-4 4B model can do with 1 shot. Unfortunately it didn't run and I asked it to fix the code. It wrote pages and pages looking at everything, saying "oh wait" many times. The code is only 95 lines, nothing complicated at all, but it had a lot of trouble finding the issue and fixing things that didn't need fixing, probably 2,000 lines of reasoning for a 95 line script. 😂 I'm going to have to give up on this model, I don't think it's usable. Sorry about that, I appreciate you put a lot of time into it.

I set reasoning to none and its performance has improved a little. It eventually completed the snake game and was able to make a few enhancement when asked. I'm starting to see it go around in circles - larger loops, not the small ones it used to do in the older version. It does sort itself out after a while but it's not quick or direct.

@DPS-900

Thanks for the feedback, appreciated it!
Seems my agentic test harness needs to improve, I didn't detect this behavior.

Updated the quants; there was a bug in the llama.cpp python script to convert to GGUF, missing an EOG token.
I missed it, only noticed while inspecting an Unsloth's quant where it was fixed.
Also included the new Google chat template but still without the tools re-injection, I'm not convinced it helps.