Qwopus3.6-35B-A3B-Coder — NVFP4

NVFP4 quantization of Jackrong/Qwopus3.6-35B-A3B-Coder, a practical coding-agent fine-tune of Qwen3.6-35B-A3B built for fast agent loops and tool-calling workflows.

Quantized with llmcompressor using the NVFP4 scheme. Fits in ~22 GB unified memory vs ~70 GB for BF16, enabling it to run alongside an embedding model on a single 128 GB DGX Spark (GB10).

Model Details

Property	Value
Base fine-tune	Jackrong/Qwopus3.6-35B-A3B-Coder
Base architecture	Qwen3.6-35B-A3B (GDN hybrid MoE)
Total parameters	35B
Active parameters	~3B per token
Quantization	NVFP4 (compressed-tensors)
Disk size	~22 GB
Context	262 144 tokens (tested)

Performance (from original model card)

SWE-bench: 62.4% on 300-case submitted-patch evaluation (Q5_K_M, thinking off)

vLLM Usage

Requires vLLM with Marlin NVFP4 backend. Tested on vLLM ≥ v0.22.1rc1 on NVIDIA GB10 (SM121, arm64).

vllm serve <repo-id> \
  --served-model-name qwopus \
  --trust-remote-code \
  --moe-backend marlin \
  --linear-backend marlin \
  --attention-backend TRITON_ATTN \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --load-format instanttensor \
  --gpu-memory-utilization 0.30 \
  --max-model-len 262144 \
  --max-num-seqs 2

Required environment variable:

VLLM_NVFP4_GEMM_BACKEND=marlin   # or use --linear-backend marlin (preferred in v0.22+)

Notes:

--attention-backend TRITON_ATTN is required — this model uses GDN (GatedDeltaNet) hybrid linear-attention layers that are incompatible with FlashInfer
--load-format instanttensor is strongly recommended for correct weight loading
MTP speculative decoding works: --speculative-config '{"method":"mtp","num_speculative_tokens":1,"moe_backend":"triton"}' (note moe_backend:triton for the unquantized MTP draft head)
Do NOT set CUTE_DSL_ARCH=sm_121a on GB10 — causes hangs

Fixes Applied During Quantization

Three model-level fixes were required to make this checkpoint load correctly in vLLM. They are already applied in this repo — you do not need to do anything extra.

1. VL wrapper config format

The original Jackrong BF16 model uses Qwen3_5MoeForCausalLM architecture. vLLM's Marlin NVFP4 MoE path requires the VL wrapper (Qwen3_5MoeForConditionalGeneration) with nested text_config/vision_config. config.json and processor_config.json were updated accordingly.

2. GDN quantisation ignore list

This model's 30 GDN (GatedDeltaNet) linear-attention layers have split HF weight tensors (in_proj_qkv, in_proj_z, in_proj_b, in_proj_a) that vLLM fuses at load time into in_proj_qkvz and in_proj_ba. The llmcompressor recipe only ignored the split names; the fused names must also be in the quantization_config.ignore list or vLLM creates quantized modules for them and fails to load. Added to config.json:

"re:.*in_proj_qkvz.*"
"re:.*in_proj_ba.*"
"re:.*linear_attn.*out_proj.*"

3. BF16 shared_expert_gate injection

vLLM hardcodes shared_expert_gate as BF16 (quant_config=None) regardless of the checkpoint's quantization config. llmcompressor quantized it to NVFP4, so the checkpoint contained weight_packed/weight_scale tensors but no weight tensor — leaving the gate uninitialized (random). This produces NaN logits → !!!! output on every prompt.

Fix: BF16 gate weights were extracted from the original BF16 model and saved as shared_expert_gates.safetensors (41 tensors: 40 layers + 1 MTP). The NVFP4 artefacts (weight_packed, weight_scale, input_global_scale, weight_global_scale) for shared_expert_gate were stripped from model.safetensors. model.safetensors.index.json now points both shards so vLLM loads them together.