--- license: apache-2.0 language: - en pipeline_tag: text-generation library_name: ninfer tags: - qwen3.6 - qwen3_5 - 27b - thinkingcap - token-efficient - efficient-thinking - conversational - image-text-to-text - multimodal - nvfp4 - w4a4 - groupwise-quantization - w4a16 - 4bit - llm - ninfer - single-file base_model: - Qwen/Qwen3.6-27B --- # ThinkingCap-Qwen3.6-27B Three single-file NInfer (`*.ninfer`) artifacts for **ThinkingCap-Qwen3.6-27B**, a token-efficient, thinking-oriented, multimodal (image+video→text) 27B finetune of [`Qwen/Qwen3.6-27B`](https://huggingface.co/Qwen/Qwen3.6-27B). Each artifact is a self-contained binary the NInfer engine serves directly. Artifacts 1 and 2 are built from the ThinkingCap-Qwen3.6-27B finetune family. Artifact 3 has the same weight layout but is produced from the Qwen3.8-27B source pair by a newer converter generation. None is a rename or re-label of another. ## Model architecture - `Qwen3_5ForConditionalGeneration` (`model_type: qwen3_5`), dense multimodal (image+video→text) finetune of [`Qwen/Qwen3.6-27B`](https://huggingface.co/Qwen/Qwen3.6-27B). Fully dense transformer: every decoder layer uses a dense SwiGLU MLP with no routing. - **Text:** 64 decoder layers; hybrid attention: 16 full-attention layers interleaved with 48 recurrent layers (short convolution, width 4), full-attention interval 4; hidden 5120, intermediate 17408, output matrix 248320 rows (tokenizer domain 248077); 24 query / 4 KV heads (head dim 256); 1 MTP draft layer; 262144 max position embeddings. - **Vision:** 27-layer ViT (hidden 1152, intermediate 4304, 16 heads), 16×16 spatial patch, 2-frame temporal patch, 2×2 spatial merge. - **Positional:** RoPE θ=10,000,000 with mRoPE section `[11, 11, 10]`. - Upstream base: [`bottlecapai/ThinkingCap-Qwen3.6-27B`](https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B), itself a finetune of [`Qwen/Qwen3.6-27B`](https://huggingface.co/Qwen/Qwen3.6-27B). ## Artifacts ### 1. Groupwise-int4 (W4A16) — `thinkingcap_qwen3_6_27b.ninfer` On-the-fly groupwise quantization from a single BF16 source; each linear layer is quantized at conversion time into a per-group 4/5/6-bit + W8 format selected by tensor size/role (`Q4G64_F16S`, `Q5G64_F16S`, `Q6G64_F16S`, `W8G32_F16S`). Weights are groupwise-quantized while activations stay 16-bit, making this a W4A16 build. Single-source provenance: base BF16 checkpoint `bottlecapai/ThinkingCap-Qwen3.6-27B` (rev `6a9e13bd`). Converter module `tools.convert.qwen3_6_27b.convert` (recipe `qwen3_6_27b-v2`, weights_id `groupwise-int`): ```bash python -m tools.convert.qwen3_6_27b.convert \ --model --out out/.ninfer --device cuda ``` Stats: objects 1124 (1118 tensors + 6 resources); tensor_bytes 17482342304; resource_bytes 12833441; alignment_bytes 14015; md5 `8c08f2f50ca93cab2e13508365d65147`. Histogram (sums to 1118): `BF16 582, Q5G64_F16S 246, Q4G64_F16S 183, FP32 96, W8G32_F16S 7, Q6G64_F16S 3, I32 1`. Layouts: `contiguous-le-v1 ×679, row-split-k128-v1 ×439`; 6 resources encoded `raw-bytes-v1`. ### 2. NVFP4 (W4A4) — `thinkingcap_qwen3_6_27b_nvfp4.ninfer` NVFP4 W4A4 mixed-precision: packed 4-bit FP4 (E2M1) weights with per-tensor group scales and `input_activations: tensor_group`, so activations are also 4-bit at inference. This is the true W4A4 path; the W4A16 build keeps 16-bit activations. Dual-source: base BF16 `bottlecapai/ThinkingCap-Qwen3.6-27B` (rev `6a9e13bd`) fused with vLLM-quantized NVFP4 partner [`sakamakismile/ThinkingCap-Qwen3.6-27B-NVFP4`](https://huggingface.co/sakamakismile/ThinkingCap-Qwen3.6-27B-NVFP4) (rev `9b5389d4`). The NVFP4-packed lines take 4-bit weight bytes + scales from the partner; the remaining 16-bit/resource tensors come from the base BF16 tree. Converter `tools.convert.qwen3_6_27b.convert_nvfp4` (recipe `qwen3_6_27b_nvfp4-v1`, weights_id `nvfp4`, model_id `qwen3.6-27b`): ```bash python3 -m tools.convert.qwen3_6_27b.convert_nvfp4 \ --model \ --nvfp4-model \ --out out/.ninfer --device cpu ``` Stats: objects 1307 (1301 tensors + 6 resources); tensor_bytes 18310883160; resource_bytes 12833441; alignment_bytes 138503; md5 `24a3d90b8aae9ea4f5296061439bd625`. Histogram (sums to 1301): `BF16 591, FP32 343, NVFP4 247, Q4G64_F16S 55, Q5G64_F16S 54, W8G32_F16S 9, Q6G64_F16S 1, I32 1`. Layouts: `blockscale-k16-m128x4-v1 ×247, contiguous-le-v1 ×935, row-split-k128-v1 ×119`; 6 resources `raw-bytes-v1`. ### 3. NVFP4 (W4A4) w8g32-generation — `thinkingcap_qwen3_6_27b_nvfp4_w8g32.ninfer` Built by the **W8G32-generation** converter `tools.convert.qwen3_8_27b.w8g32_nvfp4.convert_nvfp4` (recipe `qwen3_8_27b_nvfp4_w8g32-v1`, converter output basename `qwen3_8_27b_nvfp4_w8g32.ninfer`). This is a separate module from artifact 2's converter, not a re-quantization of it: it reuses the `qwen3_6_27b` recipe structure (additive NVFP4 MLP over 247 matrices, `blockscale-k16-m128x4-v1` packing, split GDN control, W8G32_F16S vocabulary endpoints) but pins the Qwen3.8 base model's frontend resources (its W8G32 resource profile, verified by SHA256 at conversion) instead of the Qwen3.6 resources artifact 2 embeds. Dual-source, with a different source pair than artifact 2, per the recorded conversion run: the base BF16 checkpoint [`Qwen/Qwen3.8-27B`](https://huggingface.co/Qwen/Qwen3.8-27B) (rev `1d4bf0f2`) supplies the direct tensors, split GDN control projections, and W8G32_F16S vocabulary endpoints; the NVFP4 partner [`sakamakismile/Qwen3.8-27B-MTP-NVFP4`](https://huggingface.co/sakamakismile/Qwen3.8-27B-MTP-NVFP4) supplies the packed 4-bit codes plus E4M3FN scales and divisors. The embedded identity (`model_id qwen3.8-27b`, `weights_id nvfp4_w8g32`) matches this Qwen3.8 source pair rather than the ThinkingCap-Qwen3.6-27B family of artifacts 1 and 2. With `input_activations: tensor_group` this is a true W4A4 build. It has the same object inventory as artifact 2 (1307 objects: 1301 tensors + 6 resources, the same format counts, the same layouts), but the files are not byte-identical: artifact 3 is 3840 bytes larger (18324067840 vs 18324064000) with a different SHA256, because its embedded frontend-resource bytes come from the Qwen3.8 base model. The histogram also includes one `Q6G64_F16S` object even though the filename omits `q6g64`; the missing token distinguishes this build from artifact 2 under the one-filename-per-(org, model, weight-types) convention. It does not mean the build lacks 6-bit weights. ```bash python -m tools.convert.qwen3_8_27b.w8g32_nvfp4.convert_nvfp4 \ --model \ --nvfp4-model \ --out out/.ninfer --device cuda ``` Stats: objects 1307 (1301 tensors + 6 resources); file bytes 18324067840; sha256 `00c3412b…cbd1`; layouts `blockscale-k16-m128x4-v1 ×247, contiguous-le-v1 ×935, row-split-k128-v1 ×119`. ## Differences between artifacts | | 1 — groupwise | 2 — NVFP4 | 3 — NVFP4 w8g32 | |---|---|---|---| | Quant class | W4A16 (groupwise-int weights, 16-bit activations) | W4A4 (NVFP4 4-bit weights, 4-bit activations) | W4A4 (NVFP4 4-bit weights, 4-bit activations) | | Sources | single (BF16 only) | two-source (BF16 + NVFP4 partner) | two-source (BF16 + NVFP4 partner) | | Source repos | `bottlecapai/ThinkingCap-Qwen3.6-27B` | `bottlecapai/ThinkingCap-Qwen3.6-27B` + `sakamakismile/ThinkingCap-Qwen3.6-27B-NVFP4` | `Qwen/Qwen3.8-27B` + `sakamakismile/Qwen3.8-27B-MTP-NVFP4` | | Converter | `qwen3_6_27b.convert` (groupwise recipe) | `qwen3_6_27b.convert_nvfp4` (`qwen3_6_27b_nvfp4-v1`) | `qwen3_8_27b.w8g32_nvfp4.convert_nvfp4` (`qwen3_8_27b_nvfp4_w8g32-v1`) | | Tensors | 1118 | 1301 | 1301 (same format counts as 2) | | Frontend resources | Qwen3.6 base | Qwen3.6 base | Qwen3.8 base (W8G32 profile) | Artifacts 2 and 3 share an identical 1301-tensor weight layout and the same weight types (both include one Q6G64), but are not byte-identical files: the w8g32 build is 3840 bytes larger, with a different SHA256, because its embedded source and frontend-resource bytes differ even though the format counts match. They are two distinct builds (different converter generation, different source pair, different payload bytes), which is why the naming convention (one filename per org + model + weight-types) gives them distinct names. Artifact 1 is the only non-NVFP4, W4A16 build. ## Files and checksums Byte sizes below are the authoritative LFS sizes from the Hugging Face repo tree. | File | Size (bytes) | SHA256 | Scheme | |---|---|---|---| | `thinkingcap_qwen3_6_27b.ninfer` | 17495365888 | `406d7ca3c8248ec304518563b9dbb00cf48d9cfc19d126e6df66d354e6d5b892` | Groupwise-int4 W4A16 | | `thinkingcap_qwen3_6_27b_nvfp4.ninfer` | 18324064000 | `ec149b15d29acc0bfdb6f5720e12a39f15746b688ac4e290a6ebf6661ac2c8e7` | NVFP4 W4A4 | | `thinkingcap_qwen3_6_27b_nvfp4_w8g32.ninfer` | 18324067840 | `00c3412b764c24bdf37058dea5209b677462fa68c517c8e21c89aec9f38dcbd1` | NVFP4 W4A4 (w8g32) | ## Serving All artifacts are served directly by the **NInfer** engine as self-contained single binaries, so no separate weights download is needed. Capability flags relevant to this model: multimodal image input (`--vision`), MTP decoding (1 MTP draft layer), and int8 KV cache. ## Known limitations No KL-divergence or perplexity benchmark has been run on any artifact, so quantization fidelity against the BF16 reference is unmeasured. Validation is structural only: format histogram, object/tensor/resource counts, and byte/alignment integrity from the converter's accounting. No downstream quality claims follow from it. ## References - Base checkpoint (artifacts 1–2): [`bottlecapai/ThinkingCap-Qwen3.6-27B`](https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B) (finetune of [`Qwen/Qwen3.6-27B`](https://huggingface.co/Qwen/Qwen3.6-27B)), rev `6a9e13bd` - NVFP4 partner (artifacts 1–2): [`sakamakismile/ThinkingCap-Qwen3.6-27B-NVFP4`](https://huggingface.co/sakamakismile/ThinkingCap-Qwen3.6-27B-NVFP4), rev `9b5389d4` - Base checkpoint (artifact 3): [`Qwen/Qwen3.8-27B`](https://huggingface.co/Qwen/Qwen3.8-27B), rev `1d4bf0f2` - NVFP4 partner (artifact 3): [`sakamakismile/Qwen3.8-27B-MTP-NVFP4`](https://huggingface.co/sakamakismile/Qwen3.8-27B-MTP-NVFP4) - NInfer engine: https://github.com/ninfer - Converter toolchain: `tools.convert.qwen3_6_27b.{convert,convert_nvfp4}` and `tools.convert.qwen3_8_27b.w8g32_nvfp4.convert_nvfp4`