BoldingBuilds/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ShimQuant-GGUF

Nemotron-3.5-Lightning-30B-A3B, ShimQuant tuned, 11.77 GiB

This file will not load in stock llama.cpp, LM Studio, Ollama, or any other unpatched runtime. It needs the ShimQuant patch. The failure is immediate and explicit, not silent corruption:

check_tensor_dims: tensor 'blk.0.ssm_in.weight' has wrong shape;
                   expected  2688, 10304, got  2816, 10304
llama_model_load_from_file_impl: failed to load model

Read Requirements before downloading 11.77 GiB.

A 3.07 bits-per-weight build of Nemotron-3.5-Lightning that runs on a 16 GB consumer card at 262,144 tokens of context. The stock quantizer cannot produce a file this size for this model at any label.

11.77 GiB = 12.63 GB. Both units appear below because comparable published work uses decimal GB and mixing them makes files look smaller than they are.

About the name. The file says IQ3_XXS because IQ3_XXS is nominally 3.0625 bpw and this file measures 3.069, which is closer than any other standard label gets. The tensors are not all IQ3_XXS. The recipe is a Q6_K base with iq2_xxs on gate and up, iq2_s on down, and q8_0 pinned on blk.52, and GGUF has no field that can say that. general.file_type is set to IQ3_XXS to match the name. That declares the bit budget, not the tensor types, which is the closest honest thing the format allows. llama-quantize leaves it at the base type you passed, so it read Q6_K until I corrected it by hand.

Why this exists

Nemotron-3.5-Lightning has n_embd = 2688 and expert widths of 1856 and 3712. None divide by 256, so no k-quant or i-quant can legally apply to about 99% of its parameters. llama-quantize substitutes a 32-block type instead and keeps the requested filename, so every published low-bit quant of this model measures ~4.70 bpw regardless of its name. The four IQ2 rungs in the popular repos are the same file under four labels. Census across 25 repos and 443 quants: ggufaudit.

ShimQuant shims each affected tensor row out to the next multiple of 256, so the low-bit types actually apply. The shim is zero-padding written at quantize time and sliced back off the activations at inference. The expert banks here shim 1856 to 2048, which is 9.4% overhead. That small distance to the next multiple is why this model is a good candidate and others are not.

Prior report of the underlying mechanism on this model: worthant, 2026-08-12.

Divergence from the Q8_0 reference

Mean per-token KL divergence in nats against one shared answer key: Q8_0 logits dumped once from the stock build, teacher-forced over a fixed corpus, identical for every row.

build	measured bpw	size	KLD vs Q8_0
stock IQ3_XXS	4.70	18.01 GiB	0.0398
stock IQ2_M	4.70	18.00 GiB	0.2722
shimmed IQ3_XXS	3.58	13.71 GiB	0.2541
shimmed IQ2_M	3.13	12.01 GiB	0.5229
this file (shimmed, tuned)	3.07	11.77 GiB	0.1230

Note the two middle rows. Shimming on its own is worse than stock, and by a wide margin. It does not improve anything; all it does is make the low-bit types legal on these tensor widths. The tuned recipe is what turns that access into a usable file, and without it this technique looks like a bad trade.

Read that table honestly. Against stock IQ2_M this file is 6.2 GiB smaller and less divergent, which is a strict improvement. Against stock IQ3_XXS it is 6.2 GiB smaller and three times more divergent, which is a trade. This file does not dominate the published ladder and the claim here is narrower: below about 18 GiB the stock quantizer produces nothing at all for this model, and this is a usable file in that gap.

Capability

HumanEval, greedy decoding, executed tests, 6000-token cap, 164 problems.

	pass@1	hit token cap	accuracy when code was emitted
this file	91.5%	5.5%	97.4%

Against other published builds of this model

Same harness, same 164 problems, same greedy settings, same token budget, same machine. Sizes in decimal GB. Comparison files are from AtomicChat, whose published divergence figures are measured against a different corpus and are therefore not comparable to the table above. This is why they were re-scored here rather than quoted.

build	maker	size	pass@1	hit token cap
AD-Q2_0	AtomicChat	11.39 GB	0.0%	59.8%
this file	ShimQuant	12.63 GB	91.5%	5.5%
AD-Q2_0_L	AtomicChat	15.52 GB	37.8%	39.6%
AD-IQ4_NL	AtomicChat	19.65 GB	91.5%	4.9%

AD-Q2_0 produced no parseable function in 164 attempts, which matches its maker's own assessment of it as unusable. AD-IQ4_NL is that repo's recommended build and it ties this file exactly, at 7.02 GB larger. The AtomicChat repo states plainly that "there is currently no good 16 GB option for this model, from anyone." That was accurate when written.

Measured on a 16 GB card

RTX 5080, 16,303 MiB, full offload, patched build a41d55cb8.

	value
maximum context loaded	262,144
peak VRAM at that context	13,458 MiB of 16,303
peak VRAM at 65,536 context	13,171 MiB

Context is nearly free on this model because most of its layers are SSM rather than attention, so there is no KV cache growing with sequence length. Going from 65,536 to 262,144 tokens costs 287 MiB. 262,144 was the largest value tested, not a measured ceiling.

Throughput

A single generation number is not meaningful for this file, because the two things that move it most are prompt depth and whether MTP is on. All of the following is one card, one file, -c 16384, greedy, full offload.

prompt	workload	no MTP	with MTP	gain
906 tok	code	139.8 tok/s	148.1 tok/s	1.06x
12,346 tok	prose	73.6 tok/s	144.4 tok/s	1.96x
14,306 tok	code	64.9 tok/s	109.4 tok/s	1.69x

Prefill runs 397 to 745 tok/s without MTP and 582 to 667 with it, rising with prompt length in both cases.

The gain grows with prompt depth. MTP buys little on a short prompt and close to a doubling at 12,000 tokens and up, so an agent working in a real codebase gets far more from it than a chat turn does. A separate sweep at shorter prompts and a 120-token budget measured 1.2x fairly flatly from 231 to 906 tokens, so the shallow figure depends on how much you generate as well as on prompt length; treat 1.06x and 1.2x as the same weak result rather than a discrepancy.

Every number above is wall-clock verified. That matters more than it sounds: llama-server's reported timings.predicted_per_second does not account for draft-side work, and on a different model we measured it reporting 38.6 tok/s during a request that took 537 seconds. It agrees with wall clock for this file, but do not benchmark any speculative config from that field alone.

To turn MTP on, add --spec-type draft-mtp --spec-draft-n-max 3. No draft model argument is needed; the head is blk.52 inside this file, which is why it is pinned to q8_0 in the recipe below rather than pruned out.

On quality: speculative decoding is supposed to be output-preserving, since the draft only proposes and the target verifies. Scored on the full 164-problem harness with and without the flag, 9 of 164 outcomes changed, 6 down and 3 up. The measurement noise floor on this harness is exactly 0.0000 over six identical runs, so those 9 flips are real differences caused by MTP rather than variance. Exact McNemar on the paired outcomes gives p = 0.51, so the net direction is not established, and conditional accuracy is unchanged at 0.9735 against 0.9740. Four of the six losses are problems that hit the token cap only with MTP, so what shifts is which problems run long, not whether the model writes correct code.

That is a weaker claim than "free" and a weaker claim than "costly", and it is the one the evidence supports.

Tool calling

Agentic use needs more than pass@1. A coding worker reads files, runs commands and searches, which means emitting a well-formed tool call with the right name and the right arguments. That is a stricter target than a function body, because a wrong token inside a JSON argument is a hard failure with no partial credit.

170 cases: 160 across four tools (read_file, run_command, search_code, list_dir), plus 10 questions that need no tool at all.

emitted a well-formed tool call	160/160
chose the correct tool	159/160
correct tool and correct arguments	159/160, 0.994
did not call a tool when none was needed	10/10

The single failure routes "What does Makefile contain?" to list_dir instead of read_file.

The restraint row matters as much as the accuracy row. Over-calling is the usual failure mode of small quantized models, and a benchmark that only rewards calling would score a model that calls constantly as perfect. Ten cases is a small sample and this row should be read as indicative rather than precise.

For context, the same harness run across a full qwen3-30b-a3b ladder moves from 1.000 at Q8_0 to 0.981 at IQ2_M, with no rung significantly different from its own Q8_0 reference under exact McNemar. Tool calling appears to be considerably more robust to quantization than code generation is, which was the opposite of what we expected going in.

Requires --jinja. Without it llama-server ignores the tools array and replies in prose.

Requirements

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout e70802a01f03f0ed31a26338a5664796f3824371
git apply /path/to/shimquant.patch      # github.com/JoshBolding/shimquant
cmake -B build -DGGML_CUDA=ON && cmake --build build -j

The commit is pinned because every number here was measured on a build of it. As of 2026-08-28 the patch also applies cleanly to upstream master (50f068fff), verified with git apply --check, but that build is unmeasured.

./build/bin/llama-server -m Nemotron-3.5-Lightning-30B-A3B-ShimQuant-IQ3_XXS.gguf -ngl 99 -c 32768

Backends

backend	status
CUDA	measured on RTX 3090 and RTX 5080
Metal	builds and runs correctly on Apple M2 Pro; perplexity on a shimmed toy model matches CPU to within 0.002%
CPU	works
Vulkan, ROCm	untested, no hardware

Shimming does not change the math

The concern with a graph-level change is that it quietly alters results. It does not. The same model quantized to Q8_0 twice, once normally and once with shimming forced on for eight tensors including a 320 to 512 widening, gives identical perplexity:

Q8_0, no shim    PPL = 260.7309 +/- 0.54015
Q8_0, shimmed    PPL = 260.7309 +/- 0.54015

The recipe

Q6_K everywhere except the expert banks, which carry about 93% of the parameters and are the cheapest class to crush on this model, measured by single-class ablation:

llama-quantize --imatrix nemotron.imatrix \
  --tensor-type "blk.52.=q8_0" \
  --tensor-type "ffn_(gate|up)_exps=iq2_xxs" \
  --tensor-type "ffn_down_exps=iq2_s" \
  BF16.gguf output.gguf Q6_K 24

blk.52 is the MTP block. It is decode-only, gets no imatrix data, and llama-quantize refuses low-bit types on tensors without importance statistics, so it must be pinned.

Limitations

• Patched build only. Stated three times because it is the real cost of using this.
• Does not beat stock IQ3_XXS on divergence. See the table above. The claim is about a size range the stock quantizer cannot reach, not about winning everywhere.
• Reasoning is on by default and --reasoning-budget 0 does not disable it on this architecture. About 5% of HumanEval problems exhaust the token budget inside the thinking block without emitting an answer. Raise max_tokens if you see empty replies. Doubling the budget to 12,000 raises pass@1 to 93.9% and cuts the capped rate to 3.0%.
• Evaluated on code and general text. No multilingual, long-context, or agentic evaluation, despite the context figure above being a load test rather than a quality test.
• One model, one recipe. ShimQuant does not help everywhere. On Qwen3.8-Flash-Next the same technique produced a worse result than the existing published quants, because 29% of that model is a per-layer embedding table that shimming cannot touch. Documented in the patch repo.

Credits

• NVIDIA for the original model, released under OpenMDW-1.1, which governs this derivative.
• Unsloth for the BF16 GGUF conversion this was quantized from.
• worthant / AtomicChat for reporting the mechanism on this model first, and for publishing a measured ladder careful enough to compare against.
• The llama.cpp maintainers, and PR #3747, which introduced the dimension fallback this works around. Deliberate behaviour, not a bug.