--- tags: - gguf - qwen3.5 - qwen - ngram - mtp library_name: llama.cpp pipeline_tag: text-generation --- # Qwen3.5-4B Q8_0 FlashNgram MTP This repository contains an experimental GGUF built from the Unsloth Qwen3.5-4B MTP Q8_0 model with a native Flash-Next n-gram table grafted into the residual stream. ## Important compatibility notice This file is **not a drop-in model for stock llama.cpp, LM Studio, Ollama, or Transformers**. The n-gram table is only used by the custom runtime branch: An ordinary GGUF loader may recognize the base model tensors, but it will not execute the n-gram lookup and residual injection. Use the matching Vulkan `llama-server` or `llama-cli` binary built from that branch. ## Vision and modality The published artifact is **text-only**. The original official Qwen3.5 source checkpoint contained `model.visual.*` weights and a `vision_config`, but those vision weights were not included by the local text-model conversion used to create the Q8 base. The later FlashNgram merge only appended the n-gram table; it did not remove any vision tensors from that converted base. No image input, vision tower, or MMProj file is included or supported by this artifact. Do not use it for multimodal inference or claim compatibility with multimodal benchmarks. ## Files | File | Description | | --- | --- | | `Qwen3.5-4B-Q8_0-FlashNgram-MTP.gguf` | Q8_0 Qwen3.5-4B MTP base plus the embedded n-gram table | | `Qwen3.5-4B-Q8_0-FlashNgram-MTP.gguf.report.json` | Local merge and tensor validation report | Artifact details: - Size: `33,410,719,936` bytes - SHA-256: `988FA0C4AC30C510570FB0CEC94F2F32605E59B00617A3D024F2132B37D12D98` - 442 tensors total: 441 tensors from the Unsloth base and one added table - Full 32-layer Qwen3.5 backbone retained - MTP tensors retained under `blk.32.nextn.*` ## How the graft works The base is the Unsloth `Qwen3.5-4B-MTP-GGUF` Q8_0 artifact. The added table is the native `per_layer_token_embd.weight` table from the validated Flash-Next source split. Its contract is: - Shape: `[160, 320001536]` - GGML type: `IQ4_NL` (type 20) - Payload: `28,800,138,240` bytes - 3-gram history, 16 hash heads, 160 features per head, 2,560 features total At runtime the patched Qwen3.5 implementation hashes the current three-token history, gathers rows from the embedded table, reshapes the 16 × 160 result to the 2,560-wide hidden stream, and applies: ```text residual entering zero-based block 2 = base residual + alpha * v_ngram ``` The default embedded gate is `alpha=0`. Pass a nonzero `--ngram-gate` to enable the experimental graft. Alpha `0` is the gate-zero control; it is mathematically equivalent to the Q8 base for this injection. No Qwen3.5 layers were removed, and no LoRA or additional training was used. This is a direct hashed-table residual graft, not a complete reproduction of the original Flash-Next PLE module. The incompatible Flash-Next projection, normalization, value, and convolution tensors were intentionally not copied. ## Running with Vulkan Build the custom branch with Vulkan and the server enabled, then run from the directory containing the downloaded file: ```powershell llama-server.exe ` -m .\Qwen3.5-4B-Q8_0-FlashNgram-MTP.gguf ` --n-gpu-layers 99 ` --ctx-size 38192 ` --flash-attn on ` --jinja ` --reasoning on ` --ngram-layer 2 ` --ngram-gate 0.5 ` --port 8080 ``` The table is embedded. Do **not** pass `--ngram-file`; doing so attempts to load a duplicate `per_layer_token_embd.weight` tensor. The MTP tensors are preserved in the GGUF. Speculative MTP decoding is optional and was slower in the local test; omit `--spec-type draft-mtp` for the faster baseline: ```text --spec-type draft-mtp --spec-draft-n-max 1 ``` ## Validation and local results The artifact was created by streaming the Unsloth Q8 base and appending only the table tensor. Local validation confirmed the Qwen3.5 architecture, 2,560 hidden width, all backbone blocks, MTP tensors, matching tokenizer IDs, and the native IQ4_NL table contract. The merge report is included beside the GGUF. On an RTX 5070 Ti 16 GB using Vulkan, the local paired benchmark measured approximately 120 tokens/s with MTP enabled. The graft was also tested through the custom server using prompt log-probabilities: changing alpha from `0.0` to `0.99` changed the returned token probabilities on all five tested prompts. This demonstrates that the table path is active in the custom runtime; it does not demonstrate that a particular alpha improves model quality. A five-example canonical GPQA-Diamond smoke test reported: | Alpha | Examples | Accuracy | Normalized accuracy | | ---: | ---: | ---: | ---: | | 0.0 | 5 | 0.40 (2/5) | 0.40 (2/5) | | 0.5 | 5 | 0.40 (2/5) | 0.40 (2/5) | This is too small to support a quality claim. No preferred alpha has been established. ## Known limitations - The n-gram table is approximately 28.8 GB and is normally CPU-mmapped; ensure sufficient RAM and disk space. - This is an experimental graft and has no claim of improved accuracy, perplexity, or reasoning quality. - The published artifact is text-generation focused; multimodal/vision use has not been validated here. - The custom runtime branch is required for n-gram behavior. - `alpha=0` remains the safest control. Nonzero alpha values are experimental. - Full GPQA-Diamond evaluation, broader regression testing, and alpha tuning remain future work. ## Reproducibility and attribution The source base model is [Unsloth Qwen3.5-4B-MTP-GGUF](https://huggingface.co/unsloth/Qwen3.5-4B-MTP-GGUF). The original model's authors, license, and usage restrictions remain applicable. The n-gram table was sourced from the locally validated Flash-Next Q3 split; its upstream authors and license terms must be reviewed and credited before redistribution. The publisher should fill in the exact license and attribution notices for both upstream artifacts before making this repository public. The local project contains the merge validator, streaming composer, runtime handoff, and evaluation records. Recreate or verify the artifact with the included report and SHA-256: ```powershell Get-FileHash .\Qwen3.5-4B-Q8_0-FlashNgram-MTP.gguf -Algorithm SHA256 ``` The custom runtime source and prompt-logprob support are available at: