README.md

---
license: apache-2.0
base_model: Jackrong/Qwopus3.6-35B-A3B-Coder
tags:
  - nvfp4
  - quantized
  - vllm
  - moe
  - coding
  - tool-use
  - qwen3
language:
  - en
---

# Qwopus3.6-35B-A3B-Coder — NVFP4

NVFP4 quantization of [Jackrong/Qwopus3.6-35B-A3B-Coder](https://huggingface.co/Jackrong/Qwopus3.6-35B-A3B-Coder), a practical coding-agent fine-tune of Qwen3.6-35B-A3B built for fast agent loops and tool-calling workflows.

Quantized with [llmcompressor](https://github.com/vllm-project/llm-compressor) using the NVFP4 scheme. Fits in ~22 GB unified memory vs ~70 GB for BF16, enabling it to run alongside an embedding model on a single 128 GB DGX Spark (GB10).

## Model Details

| Property | Value |
|---|---|
| Base fine-tune | [Jackrong/Qwopus3.6-35B-A3B-Coder](https://huggingface.co/Jackrong/Qwopus3.6-35B-A3B-Coder) |
| Base architecture | Qwen3.6-35B-A3B (GDN hybrid MoE) |
| Total parameters | 35B |
| Active parameters | ~3B per token |
| Quantization | NVFP4 (compressed-tensors) |
| Disk size | ~22 GB |
| Context | 262 144 tokens (tested) |

## Performance (from original model card)

- **SWE-bench**: 62.4% on 300-case submitted-patch evaluation (Q5_K_M, thinking off)

## vLLM Usage

Requires vLLM with Marlin NVFP4 backend. Tested on vLLM ≥ v0.22.1rc1 on NVIDIA GB10 (SM121, arm64).

```bash
vllm serve <repo-id> \
  --served-model-name qwopus \
  --trust-remote-code \
  --moe-backend marlin \
  --linear-backend marlin \
  --attention-backend TRITON_ATTN \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --load-format instanttensor \
  --gpu-memory-utilization 0.30 \
  --max-model-len 262144 \
  --max-num-seqs 2
```

**Required environment variable:**
```bash
VLLM_NVFP4_GEMM_BACKEND=marlin   # or use --linear-backend marlin (preferred in v0.22+)
```

**Notes:**
- `--attention-backend TRITON_ATTN` is required — this model uses GDN (GatedDeltaNet) hybrid linear-attention layers that are incompatible with FlashInfer
- `--load-format instanttensor` is strongly recommended for correct weight loading
- MTP speculative decoding works: `--speculative-config '{"method":"mtp","num_speculative_tokens":1,"moe_backend":"triton"}'` (note `moe_backend:triton` for the unquantized MTP draft head)
- Do NOT set `CUTE_DSL_ARCH=sm_121a` on GB10 — causes hangs

## Fixes Applied During Quantization

Three model-level fixes were required to make this checkpoint load correctly in vLLM. They are already applied in this repo — you do not need to do anything extra.

### 1. VL wrapper config format

The original Jackrong BF16 model uses `Qwen3_5MoeForCausalLM` architecture. vLLM's Marlin NVFP4 MoE path requires the VL wrapper (`Qwen3_5MoeForConditionalGeneration`) with nested `text_config`/`vision_config`. `config.json` and `processor_config.json` were updated accordingly.

### 2. GDN quantisation ignore list

This model's 30 GDN (GatedDeltaNet) linear-attention layers have split HF weight tensors (`in_proj_qkv`, `in_proj_z`, `in_proj_b`, `in_proj_a`) that vLLM fuses at load time into `in_proj_qkvz` and `in_proj_ba`. The llmcompressor recipe only ignored the split names; the fused names must also be in the `quantization_config.ignore` list or vLLM creates quantized modules for them and fails to load. Added to `config.json`:

```
"re:.*in_proj_qkvz.*"
"re:.*in_proj_ba.*"
"re:.*linear_attn.*out_proj.*"
```

### 3. BF16 shared_expert_gate injection

vLLM hardcodes `shared_expert_gate` as BF16 (`quant_config=None`) regardless of the checkpoint's quantization config. llmcompressor quantized it to NVFP4, so the checkpoint contained `weight_packed`/`weight_scale` tensors but no `weight` tensor — leaving the gate uninitialized (random). This produces NaN logits → `!!!!` output on every prompt.

Fix: BF16 gate weights were extracted from the original BF16 model and saved as `shared_expert_gates.safetensors` (41 tensors: 40 layers + 1 MTP). The NVFP4 artefacts (`weight_packed`, `weight_scale`, `input_global_scale`, `weight_global_scale`) for `shared_expert_gate` were stripped from `model.safetensors`. `model.safetensors.index.json` now points both shards so vLLM loads them together.

## Attribution

- Fine-tune by **Jackrong** ([Qwopus3.6-35B-A3B-Coder](https://huggingface.co/Jackrong/Qwopus3.6-35B-A3B-Coder)) — Apache 2.0
- Base architecture by **Alibaba Cloud Qwen Team** ([Qwen3](https://huggingface.co/Qwen)) — Apache 2.0
- Fine-tuning acceleration by **Unsloth**
- NVFP4 quantization using [llmcompressor](https://github.com/vllm-project/llm-compressor)

## License

Apache 2.0 — same as the base fine-tune and Qwen3 base model.