Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,102 @@
+---
+license: apache-2.0
+base_model: Jackrong/Qwopus3.6-35B-A3B-Coder
+tags:
+  - nvfp4
+  - quantized
+  - vllm
+  - moe
+  - coding
+  - tool-use
+  - qwen3
+language:
+  - en
+---
+
+# Qwopus3.6-35B-A3B-Coder — NVFP4
+
+NVFP4 quantization of [Jackrong/Qwopus3.6-35B-A3B-Coder](https://huggingface.co/Jackrong/Qwopus3.6-35B-A3B-Coder), a practical coding-agent fine-tune of Qwen3.6-35B-A3B built for fast agent loops and tool-calling workflows.
+
+Quantized with [llmcompressor](https://github.com/vllm-project/llm-compressor) using the NVFP4 scheme. Fits in ~22 GB unified memory vs ~70 GB for BF16, enabling it to run alongside an embedding model on a single 128 GB DGX Spark (GB10).
+
+## Model Details
+
+| Property | Value |
+|---|---|
+| Base fine-tune | [Jackrong/Qwopus3.6-35B-A3B-Coder](https://huggingface.co/Jackrong/Qwopus3.6-35B-A3B-Coder) |
+| Base architecture | Qwen3.6-35B-A3B (GDN hybrid MoE) |
+| Total parameters | 35B |
+| Active parameters | ~3B per token |
+| Quantization | NVFP4 (compressed-tensors) |
+| Disk size | ~22 GB |
+| Context | 262 144 tokens (tested) |
+
+## Performance (from original model card)
+
+- **SWE-bench**: 62.4% on 300-case submitted-patch evaluation (Q5_K_M, thinking off)
+
+## vLLM Usage
+
+Requires vLLM with Marlin NVFP4 backend. Tested on vLLM ≥ v0.22.1rc1 on NVIDIA GB10 (SM121, arm64).
+
+```bash
+vllm serve <repo-id> \
+  --served-model-name qwopus \
+  --trust-remote-code \
+  --moe-backend marlin \
+  --linear-backend marlin \
+  --attention-backend TRITON_ATTN \
+  --tool-call-parser qwen3_coder \
+  --reasoning-parser qwen3 \
+  --enable-auto-tool-choice \
+  --load-format instanttensor \
+  --gpu-memory-utilization 0.30 \
+  --max-model-len 262144 \
+  --max-num-seqs 2
+```
+
+**Required environment variable:**
+```bash
+VLLM_NVFP4_GEMM_BACKEND=marlin   # or use --linear-backend marlin (preferred in v0.22+)
+```
+
+**Notes:**
+- `--attention-backend TRITON_ATTN` is required — this model uses GDN (GatedDeltaNet) hybrid linear-attention layers that are incompatible with FlashInfer
+- `--load-format instanttensor` is strongly recommended for correct weight loading
+- MTP speculative decoding works: `--speculative-config '{"method":"mtp","num_speculative_tokens":1,"moe_backend":"triton"}'` (note `moe_backend:triton` for the unquantized MTP draft head)
+- Do NOT set `CUTE_DSL_ARCH=sm_121a` on GB10 — causes hangs
+
+## Fixes Applied During Quantization
+
+Three model-level fixes were required to make this checkpoint load correctly in vLLM. They are already applied in this repo — you do not need to do anything extra.
+
+### 1. VL wrapper config format
+
+The original Jackrong BF16 model uses `Qwen3_5MoeForCausalLM` architecture. vLLM's Marlin NVFP4 MoE path requires the VL wrapper (`Qwen3_5MoeForConditionalGeneration`) with nested `text_config`/`vision_config`. `config.json` and `processor_config.json` were updated accordingly.
+
+### 2. GDN quantisation ignore list
+
+This model's 30 GDN (GatedDeltaNet) linear-attention layers have split HF weight tensors (`in_proj_qkv`, `in_proj_z`, `in_proj_b`, `in_proj_a`) that vLLM fuses at load time into `in_proj_qkvz` and `in_proj_ba`. The llmcompressor recipe only ignored the split names; the fused names must also be in the `quantization_config.ignore` list or vLLM creates quantized modules for them and fails to load. Added to `config.json`:
+
+```
+"re:.*in_proj_qkvz.*"
+"re:.*in_proj_ba.*"
+"re:.*linear_attn.*out_proj.*"
+```
+
+### 3. BF16 shared_expert_gate injection
+
+vLLM hardcodes `shared_expert_gate` as BF16 (`quant_config=None`) regardless of the checkpoint's quantization config. llmcompressor quantized it to NVFP4, so the checkpoint contained `weight_packed`/`weight_scale` tensors but no `weight` tensor — leaving the gate uninitialized (random). This produces NaN logits → `!!!!` output on every prompt.
+
+Fix: BF16 gate weights were extracted from the original BF16 model and saved as `shared_expert_gates.safetensors` (41 tensors: 40 layers + 1 MTP). The NVFP4 artefacts (`weight_packed`, `weight_scale`, `input_global_scale`, `weight_global_scale`) for `shared_expert_gate` were stripped from `model.safetensors`. `model.safetensors.index.json` now points both shards so vLLM loads them together.
+
+## Attribution
+
+- Fine-tune by **Jackrong** ([Qwopus3.6-35B-A3B-Coder](https://huggingface.co/Jackrong/Qwopus3.6-35B-A3B-Coder)) — Apache 2.0
+- Base architecture by **Alibaba Cloud Qwen Team** ([Qwen3](https://huggingface.co/Qwen)) — Apache 2.0
+- Fine-tuning acceleration by **Unsloth**
+- NVFP4 quantization using [llmcompressor](https://github.com/vllm-project/llm-compressor)
+
+## License
+
+Apache 2.0 — same as the base fine-tune and Qwen3 base model.