File size: 4,644 Bytes
19747d3 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 | ---
license: apache-2.0
base_model: Jackrong/Qwopus3.6-35B-A3B-Coder
tags:
- nvfp4
- quantized
- vllm
- moe
- coding
- tool-use
- qwen3
language:
- en
---
# Qwopus3.6-35B-A3B-Coder — NVFP4
NVFP4 quantization of [Jackrong/Qwopus3.6-35B-A3B-Coder](https://huggingface.co/Jackrong/Qwopus3.6-35B-A3B-Coder), a practical coding-agent fine-tune of Qwen3.6-35B-A3B built for fast agent loops and tool-calling workflows.
Quantized with [llmcompressor](https://github.com/vllm-project/llm-compressor) using the NVFP4 scheme. Fits in ~22 GB unified memory vs ~70 GB for BF16, enabling it to run alongside an embedding model on a single 128 GB DGX Spark (GB10).
## Model Details
| Property | Value |
|---|---|
| Base fine-tune | [Jackrong/Qwopus3.6-35B-A3B-Coder](https://huggingface.co/Jackrong/Qwopus3.6-35B-A3B-Coder) |
| Base architecture | Qwen3.6-35B-A3B (GDN hybrid MoE) |
| Total parameters | 35B |
| Active parameters | ~3B per token |
| Quantization | NVFP4 (compressed-tensors) |
| Disk size | ~22 GB |
| Context | 262 144 tokens (tested) |
## Performance (from original model card)
- **SWE-bench**: 62.4% on 300-case submitted-patch evaluation (Q5_K_M, thinking off)
## vLLM Usage
Requires vLLM with Marlin NVFP4 backend. Tested on vLLM ≥ v0.22.1rc1 on NVIDIA GB10 (SM121, arm64).
```bash
vllm serve <repo-id> \
--served-model-name qwopus \
--trust-remote-code \
--moe-backend marlin \
--linear-backend marlin \
--attention-backend TRITON_ATTN \
--tool-call-parser qwen3_coder \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--load-format instanttensor \
--gpu-memory-utilization 0.30 \
--max-model-len 262144 \
--max-num-seqs 2
```
**Required environment variable:**
```bash
VLLM_NVFP4_GEMM_BACKEND=marlin # or use --linear-backend marlin (preferred in v0.22+)
```
**Notes:**
- `--attention-backend TRITON_ATTN` is required — this model uses GDN (GatedDeltaNet) hybrid linear-attention layers that are incompatible with FlashInfer
- `--load-format instanttensor` is strongly recommended for correct weight loading
- MTP speculative decoding works: `--speculative-config '{"method":"mtp","num_speculative_tokens":1,"moe_backend":"triton"}'` (note `moe_backend:triton` for the unquantized MTP draft head)
- Do NOT set `CUTE_DSL_ARCH=sm_121a` on GB10 — causes hangs
## Fixes Applied During Quantization
Three model-level fixes were required to make this checkpoint load correctly in vLLM. They are already applied in this repo — you do not need to do anything extra.
### 1. VL wrapper config format
The original Jackrong BF16 model uses `Qwen3_5MoeForCausalLM` architecture. vLLM's Marlin NVFP4 MoE path requires the VL wrapper (`Qwen3_5MoeForConditionalGeneration`) with nested `text_config`/`vision_config`. `config.json` and `processor_config.json` were updated accordingly.
### 2. GDN quantisation ignore list
This model's 30 GDN (GatedDeltaNet) linear-attention layers have split HF weight tensors (`in_proj_qkv`, `in_proj_z`, `in_proj_b`, `in_proj_a`) that vLLM fuses at load time into `in_proj_qkvz` and `in_proj_ba`. The llmcompressor recipe only ignored the split names; the fused names must also be in the `quantization_config.ignore` list or vLLM creates quantized modules for them and fails to load. Added to `config.json`:
```
"re:.*in_proj_qkvz.*"
"re:.*in_proj_ba.*"
"re:.*linear_attn.*out_proj.*"
```
### 3. BF16 shared_expert_gate injection
vLLM hardcodes `shared_expert_gate` as BF16 (`quant_config=None`) regardless of the checkpoint's quantization config. llmcompressor quantized it to NVFP4, so the checkpoint contained `weight_packed`/`weight_scale` tensors but no `weight` tensor — leaving the gate uninitialized (random). This produces NaN logits → `!!!!` output on every prompt.
Fix: BF16 gate weights were extracted from the original BF16 model and saved as `shared_expert_gates.safetensors` (41 tensors: 40 layers + 1 MTP). The NVFP4 artefacts (`weight_packed`, `weight_scale`, `input_global_scale`, `weight_global_scale`) for `shared_expert_gate` were stripped from `model.safetensors`. `model.safetensors.index.json` now points both shards so vLLM loads them together.
## Attribution
- Fine-tune by **Jackrong** ([Qwopus3.6-35B-A3B-Coder](https://huggingface.co/Jackrong/Qwopus3.6-35B-A3B-Coder)) — Apache 2.0
- Base architecture by **Alibaba Cloud Qwen Team** ([Qwen3](https://huggingface.co/Qwen)) — Apache 2.0
- Fine-tuning acceleration by **Unsloth**
- NVFP4 quantization using [llmcompressor](https://github.com/vllm-project/llm-compressor)
## License
Apache 2.0 — same as the base fine-tune and Qwen3 base model.
|