MIThridate/Qwen3.5-9B-24L-GGUF

Qwen3.5-9B Layer-Pruned GGUF Suite (24L & 28L) - Experimental

Experimental layer-pruned GGUF checkpoints derived from Qwen/Qwen3.5-9B. Evaluated for local inference on consumer hardware with 4GB–6GB VRAM (NVIDIA GTX 1060, RTX 3050 Laptop) using llama.cpp.

Repository Structure Note:
This repository contains an experimental pruning suite: the aggressive 24-layer reduction (optimized to fit completely inside ~4GB VRAM) alongside a conservative 28-layer reference cut (which preserves more semantic depth for 6GB GPUs).

---

Experimental Scope & Research Disclaimer

These builds are exploratory proof-of-concept (PoC) checkpoints developed to test the physical limits of hardware-constrained inference.

• Heuristic Layer Ablation: Layers were excised directly from the GGUF binary graph without subsequent fine-tuning or full retraining.
• Expected Trade-Offs: While basic syntactic structure and short-context generation remain stable, aggressive layer excision inherently degrades multi-step reasoning, long-context coherence, and subtle edge-case logic.
• Intended Use: Strictly for edge hardware benchmarking, VRAM ceiling stress-testing, and local experimentation. It is not an enterprise-certified or production-validated release.
• Community Validation: If you test these checkpoints and encounter repeating loops, syntax breakdown, or perplexity spikes, please report them in the Discussions tab.

---

Available Checkpoints

Filename	Layers	Physical Params	Quantization Profile	File Size	Active VRAM (2k ctx)	Generation Speed (GTX 1060)	Profile Notes
Qwen3.5-9B-28L-Q4_K_M.gguf	28 / 32	~8.3B	Q4_K_M	4.81 GB	~4.20 GB	~18.0 tok/s	Conservative Cut: 4 intermediate layers removed. Retains higher semantic capacity and attention structure.
Qwen3.5-9B-24L-SAMI-Q4_K_M.gguf	24 / 32	~7.1B	SAMI-Q4 (Mixed)	4.06 GB	~3.92 GB	~21.5 tok/s	Aggressive Edge: 8 layers removed with mixed quantization. Fits fully inside 4GB VRAM.
Qwen3.5-9B-24L-Q4_K_M.gguf	24 / 32	~7.1B	Q4_K_M	4.31 GB	~4.11 GB	~20.8 tok/s	Standard 24-layer baseline cut.
Qwen3.5-9B-24L-Q3_K_M.gguf	24 / 32	~7.1B	Q3_K_M	3.53 GB	~3.46 GB	~17.7 tok/s	High-compression cut for strictly constrained memory budgets.

Note on Model Size Badge: The global Hugging Face repository badge automatically displays 7B params because the primary 24-layer variants (representing 75% of repository builds) are pruned to ~7.1B parameters, whereas the 28-layer cut retains ~8.3B parameters.

---

Quick Start (llama.cpp)

Requires llama.cpp (b11300 or newer).

1. Launch llama-server (28-Layer Baseline)

./llama-server \
  -m Qwen3.5-9B-28L-Q4_K_M.gguf \
  -ngl 99 \
  -c 2048 \
  -t 4 \
  --port 8080

2. Interactive CLI Run (24-Layer Aggressive Cut)

./llama-cli \
  -m Qwen3.5-9B-24L-SAMI-Q4_K_M.gguf \
  -ngl 99 \
  -c 2048 \
  -t 4 \
  -p "Explain the concept of memory alignment in low-level systems programming."

---

Layer Pruning Mechanism

The base Qwen3.5-9B architecture employs an interleaved block structure alternating between State-Space Model (Mamba-2) blocks and Full Attention (GQA) blocks (a 3:1 pattern over a 4-layer cycle):

• 28-Layer Slicing (Conservative):
  • Excises exactly one full 4-layer period from the middle stages (Layers 16–19, L16–L19).
  • Preserves the global ratio between linear recurrence and dense self-attention while lowering VRAM requirements below 4.5 GB.
• 24-Layer Slicing (Aggressive):
  • Excises two full 4-layer periods (Layers 12–19, L12–L19), reducing total layer count from 32 to 24.
  • Lowers parameter count from 9.46B to ~7.09B (7.1B), enabling the entire model to run resident inside 4GB VRAM without CPU paging overhead.

---

Qualitative Smoke Tests (Sanity Checks)

During local validation on an Intel i5-8400 + NVIDIA GTX 1060 6GB:

• Basic factual querying, structured code extraction, and Romanian/English syntax were maintained without infinite output loops.
• Sustained inference throughput ranged between 18.0 and 21.5 tok/s with all layers resident in GPU memory.

---

Tooling & Reproducibility

The layer-slicing procedure for this suite was executed directly on the GGUF binary format using custom open-source tooling:

• Repository: MIThriade/sami-surgical-pruner
• Script: sami_gguf_layer_pruner.py
• Author: Valentin Spiridon (TSV Consulting SRL, Timișoara, Romania)

Feel free to inspect the slicing implementation, reproduce the 28L/24L cuts, or adapt the script to test layer pruning on other hybrid architectures.

---

License

This model release is distributed under the original Apache-2.0 license governing the base Qwen/Qwen3.5-9B weights.