dburner/Qwen3.5-4B-Q8_0-FlashNgram-MTP

Qwen3.5-4B Q8_0 FlashNgram MTP

This repository contains an experimental GGUF built from the Unsloth Qwen3.5-4B MTP Q8_0 model with a native Flash-Next n-gram table grafted into the residual stream.

Important compatibility notice

This file is not a drop-in model for stock llama.cpp, LM Studio, Ollama, or Transformers. The n-gram table is only used by the custom runtime branch:

https://github.com/dburner/llama.cpp/tree/pr/qwen35-flashngram-prompt-logprobs

An ordinary GGUF loader may recognize the base model tensors, but it will not execute the n-gram lookup and residual injection. Use the matching Vulkan llama-server or llama-cli binary built from that branch.

Vision and modality

The published artifact is text-only. The original official Qwen3.5 source checkpoint contained model.visual.* weights and a vision_config, but those vision weights were not included by the local text-model conversion used to create the Q8 base. The later FlashNgram merge only appended the n-gram table; it did not remove any vision tensors from that converted base. No image input, vision tower, or MMProj file is included or supported by this artifact. Do not use it for multimodal inference or claim compatibility with multimodal benchmarks.

Files

File	Description
Qwen3.5-4B-Q8_0-FlashNgram-MTP.gguf	Q8_0 Qwen3.5-4B MTP base plus the embedded n-gram table
Qwen3.5-4B-Q8_0-FlashNgram-MTP.gguf.report.json	Local merge and tensor validation report

Artifact details:

• Size: 33,410,719,936 bytes
• SHA-256: 988FA0C4AC30C510570FB0CEC94F2F32605E59B00617A3D024F2132B37D12D98
• 442 tensors total: 441 tensors from the Unsloth base and one added table
• Full 32-layer Qwen3.5 backbone retained
• MTP tensors retained under blk.32.nextn.*

How the graft works

The base is the Unsloth Qwen3.5-4B-MTP-GGUF Q8_0 artifact. The added table is the native per_layer_token_embd.weight table from the validated Flash-Next source split. Its contract is:

• Shape: [160, 320001536]
• GGML type: IQ4_NL (type 20)
• Payload: 28,800,138,240 bytes
• 3-gram history, 16 hash heads, 160 features per head, 2,560 features total

At runtime the patched Qwen3.5 implementation hashes the current three-token history, gathers rows from the embedded table, reshapes the 16 × 160 result to the 2,560-wide hidden stream, and applies:

residual entering zero-based block 2 = base residual + alpha * v_ngram

The default embedded gate is alpha=0. Pass a nonzero --ngram-gate to enable the experimental graft. Alpha 0 is the gate-zero control; it is mathematically equivalent to the Q8 base for this injection. No Qwen3.5 layers were removed, and no LoRA or additional training was used.

This is a direct hashed-table residual graft, not a complete reproduction of the original Flash-Next PLE module. The incompatible Flash-Next projection, normalization, value, and convolution tensors were intentionally not copied.

Running with Vulkan

Build the custom branch with Vulkan and the server enabled, then run from the directory containing the downloaded file:

llama-server.exe `
  -m .\Qwen3.5-4B-Q8_0-FlashNgram-MTP.gguf `
  --n-gpu-layers 99 `
  --ctx-size 38192 `
  --flash-attn on `
  --jinja `
  --reasoning on `
  --ngram-layer 2 `
  --ngram-gate 0.5 `
  --port 8080

The table is embedded. Do not pass --ngram-file; doing so attempts to load a duplicate per_layer_token_embd.weight tensor. The MTP tensors are preserved in the GGUF. Speculative MTP decoding is optional and was slower in the local test; omit --spec-type draft-mtp for the faster baseline:

--spec-type draft-mtp --spec-draft-n-max 1

Validation and local results

The artifact was created by streaming the Unsloth Q8 base and appending only the table tensor. Local validation confirmed the Qwen3.5 architecture, 2,560 hidden width, all backbone blocks, MTP tensors, matching tokenizer IDs, and the native IQ4_NL table contract. The merge report is included beside the GGUF.

On an RTX 5070 Ti 16 GB using Vulkan, the local paired benchmark measured approximately 120 tokens/s with MTP enabled.

The graft was also tested through the custom server using prompt log-probabilities: changing alpha from 0.0 to 0.99 changed the returned token probabilities on all five tested prompts. This demonstrates that the table path is active in the custom runtime; it does not demonstrate that a particular alpha improves model quality.

A five-example canonical GPQA-Diamond smoke test reported:

Alpha	Examples	Accuracy	Normalized accuracy
0.0	5	0.40 (2/5)	0.40 (2/5)
0.5	5	0.40 (2/5)	0.40 (2/5)

This is too small to support a quality claim. No preferred alpha has been established.

Known limitations

• The n-gram table is approximately 28.8 GB and is normally CPU-mmapped; ensure sufficient RAM and disk space.
• This is an experimental graft and has no claim of improved accuracy, perplexity, or reasoning quality.
• The published artifact is text-generation focused; multimodal/vision use has not been validated here.
• The custom runtime branch is required for n-gram behavior.
• alpha=0 remains the safest control. Nonzero alpha values are experimental.
• Full GPQA-Diamond evaluation, broader regression testing, and alpha tuning remain future work.

Reproducibility and attribution

The source base model is Unsloth Qwen3.5-4B-MTP-GGUF. The original model's authors, license, and usage restrictions remain applicable. The n-gram table was sourced from the locally validated Flash-Next Q3 split; its upstream authors and license terms must be reviewed and credited before redistribution. The publisher should fill in the exact license and attribution notices for both upstream artifacts before making this repository public.

The local project contains the merge validator, streaming composer, runtime handoff, and evaluation records. Recreate or verify the artifact with the included report and SHA-256:

Get-FileHash .\Qwen3.5-4B-Q8_0-FlashNgram-MTP.gguf -Algorithm SHA256

The custom runtime source and prompt-logprob support are available at:

https://github.com/dburner/llama.cpp/tree/pr/qwen35-flashngram-prompt-logprobs