Text Generation
NInfer
English
qwen3.6
qwen3_5
27b
thinkingcap
token-efficient
efficient-thinking
conversational
image-text-to-text
multimodal
nvfp4
w4a4
groupwise-quantization
w4a16
4bit
llm
single-file
Instructions to use knoopx/ThinkingCap-Qwen3.6-27B-NInfer with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- NInfer
How to use knoopx/ThinkingCap-Qwen3.6-27B-NInfer with NInfer:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -28,12 +28,12 @@ base_model:
|
|
| 28 |
|
| 29 |
# ThinkingCap-Qwen3.6-27B
|
| 30 |
|
| 31 |
-
Three single-file NInfer (`*.ninfer`) artifacts for **ThinkingCap-Qwen3.6-27B**, a token-efficient, thinking-oriented, multimodal (image+video→text) 27B finetune of [`Qwen/Qwen3.6-27B`](https://huggingface.co/Qwen/Qwen3.6-27B). Each artifact is a self-contained binary the NInfer engine serves directly.
|
| 32 |
|
| 33 |
## Model architecture
|
| 34 |
|
| 35 |
- `Qwen3_5ForConditionalGeneration` (`model_type: qwen3_5`), dense multimodal (image+video→text) finetune of [`Qwen/Qwen3.6-27B`](https://huggingface.co/Qwen/Qwen3.6-27B). Fully dense transformer: every decoder layer uses a dense SwiGLU MLP with no routing.
|
| 36 |
-
- **Text:** 64 decoder layers; hybrid attention
|
| 37 |
- **Vision:** 27-layer ViT (hidden 1152, intermediate 4304, 16 heads), 16×16 spatial patch, 2-frame temporal patch, 2×2 spatial merge.
|
| 38 |
- **Positional:** RoPE θ=10,000,000 with mRoPE section `[11, 11, 10]`.
|
| 39 |
- Upstream base: [`bottlecapai/ThinkingCap-Qwen3.6-27B`](https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B), itself a finetune of [`Qwen/Qwen3.6-27B`](https://huggingface.co/Qwen/Qwen3.6-27B).
|
|
@@ -42,7 +42,7 @@ Three single-file NInfer (`*.ninfer`) artifacts for **ThinkingCap-Qwen3.6-27B**,
|
|
| 42 |
|
| 43 |
### 1. Groupwise-int4 (W4A16) — `thinkingcap_qwen3_6_27b.ninfer`
|
| 44 |
|
| 45 |
-
On-the-fly groupwise quantization from a single BF16 source; each linear layer is quantized at conversion time into a per-group 4/5/6-bit + W8 format selected by tensor size/role (`Q4G64_F16S`, `Q5G64_F16S`, `Q6G64_F16S`, `W8G32_F16S`). Weights are groupwise-quantized while activations stay 16-bit
|
| 46 |
|
| 47 |
Single-source provenance: base BF16 checkpoint `bottlecapai/ThinkingCap-Qwen3.6-27B` (rev `6a9e13bd`). Converter module `tools.convert.qwen3_6_27b.convert` (recipe `qwen3_6_27b-v2`, weights_id `groupwise-int`):
|
| 48 |
|
|
@@ -57,9 +57,9 @@ Histogram (sums to 1118): `BF16 582, Q5G64_F16S 246, Q4G64_F16S 183, FP32 96, W8
|
|
| 57 |
|
| 58 |
### 2. NVFP4 (W4A4) — `thinkingcap_qwen3_6_27b_nvfp4.ninfer`
|
| 59 |
|
| 60 |
-
NVFP4 W4A4 mixed-precision: packed 4-bit FP4 (E2M1) weights with per-tensor group scales and `input_activations: tensor_group`, so activations are also 4-bit at inference.
|
| 61 |
|
| 62 |
-
Dual-source: base BF16 `bottlecapai/ThinkingCap-Qwen3.6-27B` (rev `6a9e13bd`) fused with vLLM-quantized NVFP4 partner [`sakamakismile/ThinkingCap-Qwen3.6-27B-NVFP4`](https://huggingface.co/sakamakismile/ThinkingCap-Qwen3.6-27B-NVFP4) (rev `9b5389d4`).
|
| 63 |
|
| 64 |
```bash
|
| 65 |
python3 -m tools.convert.qwen3_6_27b.convert_nvfp4 \
|
|
@@ -74,11 +74,11 @@ Histogram (sums to 1301): `BF16 591, FP32 343, NVFP4 247, Q4G64_F16S 55, Q5G64_F
|
|
| 74 |
|
| 75 |
### 3. NVFP4 (W4A4) w8g32-generation — `thinkingcap_qwen3_6_27b_nvfp4_w8g32.ninfer`
|
| 76 |
|
| 77 |
-
|
| 78 |
|
| 79 |
-
Dual-source, with a different source pair than
|
| 80 |
|
| 81 |
-
|
| 82 |
|
| 83 |
```bash
|
| 84 |
python -m tools.convert.qwen3_8_27b.w8g32_nvfp4.convert_nvfp4 \
|
|
@@ -87,7 +87,7 @@ python -m tools.convert.qwen3_8_27b.w8g32_nvfp4.convert_nvfp4 \
|
|
| 87 |
--out out/<name>.ninfer --device cuda
|
| 88 |
```
|
| 89 |
|
| 90 |
-
Stats: objects 1307 (1301 tensors + 6 resources);
|
| 91 |
|
| 92 |
## Differences between artifacts
|
| 93 |
|
|
@@ -97,14 +97,14 @@ Stats: objects 1307 (1301 tensors + 6 resources); byte total 18324067840; sha256
|
|
| 97 |
| Sources | single (BF16 only) | two-source (BF16 + NVFP4 partner) | two-source (BF16 + NVFP4 partner) |
|
| 98 |
| Source repos | `bottlecapai/ThinkingCap-Qwen3.6-27B` | `bottlecapai/ThinkingCap-Qwen3.6-27B` + `sakamakismile/ThinkingCap-Qwen3.6-27B-NVFP4` | `Qwen/Qwen3.8-27B` + `sakamakismile/Qwen3.8-27B-MTP-NVFP4` |
|
| 99 |
| Converter | `qwen3_6_27b.convert` (groupwise recipe) | `qwen3_6_27b.convert_nvfp4` (`qwen3_6_27b_nvfp4-v1`) | `qwen3_8_27b.w8g32_nvfp4.convert_nvfp4` (`qwen3_8_27b_nvfp4_w8g32-v1`) |
|
| 100 |
-
| Tensors | 1118 | 1301 | 1301 (same format
|
| 101 |
| Frontend resources | Qwen3.6 base | Qwen3.6 base | Qwen3.8 base (W8G32 profile) |
|
| 102 |
|
| 103 |
-
Artifacts 2 and 3 share an identical 1301-tensor weight layout and the same weight
|
| 104 |
|
| 105 |
## Files and checksums
|
| 106 |
|
| 107 |
-
Byte sizes below are authoritative LFS sizes.
|
| 108 |
|
| 109 |
| File | Size (bytes) | SHA256 | Scheme |
|
| 110 |
|---|---|---|---|
|
|
@@ -114,11 +114,11 @@ Byte sizes below are authoritative LFS sizes.
|
|
| 114 |
|
| 115 |
## Serving
|
| 116 |
|
| 117 |
-
All artifacts are served directly by the **NInfer** engine as self-contained single binaries
|
| 118 |
|
| 119 |
## Known limitations
|
| 120 |
|
| 121 |
-
No KL-divergence or perplexity benchmark has been run on any artifact
|
| 122 |
|
| 123 |
## References
|
| 124 |
|
|
|
|
| 28 |
|
| 29 |
# ThinkingCap-Qwen3.6-27B
|
| 30 |
|
| 31 |
+
Three single-file NInfer (`*.ninfer`) artifacts for **ThinkingCap-Qwen3.6-27B**, a token-efficient, thinking-oriented, multimodal (image+video→text) 27B finetune of [`Qwen/Qwen3.6-27B`](https://huggingface.co/Qwen/Qwen3.6-27B). Each artifact is a self-contained binary the NInfer engine serves directly. Artifacts 1 and 2 are built from the ThinkingCap-Qwen3.6-27B finetune family. Artifact 3 has the same weight layout but is produced from the Qwen3.8-27B source pair by a newer converter generation. None is a rename or re-label of another.
|
| 32 |
|
| 33 |
## Model architecture
|
| 34 |
|
| 35 |
- `Qwen3_5ForConditionalGeneration` (`model_type: qwen3_5`), dense multimodal (image+video→text) finetune of [`Qwen/Qwen3.6-27B`](https://huggingface.co/Qwen/Qwen3.6-27B). Fully dense transformer: every decoder layer uses a dense SwiGLU MLP with no routing.
|
| 36 |
+
- **Text:** 64 decoder layers; hybrid attention: 16 full-attention layers interleaved with 48 recurrent layers (short convolution, width 4), full-attention interval 4; hidden 5120, intermediate 17408, output matrix 248320 rows (tokenizer domain 248077); 24 query / 4 KV heads (head dim 256); 1 MTP draft layer; 262144 max position embeddings.
|
| 37 |
- **Vision:** 27-layer ViT (hidden 1152, intermediate 4304, 16 heads), 16×16 spatial patch, 2-frame temporal patch, 2×2 spatial merge.
|
| 38 |
- **Positional:** RoPE θ=10,000,000 with mRoPE section `[11, 11, 10]`.
|
| 39 |
- Upstream base: [`bottlecapai/ThinkingCap-Qwen3.6-27B`](https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B), itself a finetune of [`Qwen/Qwen3.6-27B`](https://huggingface.co/Qwen/Qwen3.6-27B).
|
|
|
|
| 42 |
|
| 43 |
### 1. Groupwise-int4 (W4A16) — `thinkingcap_qwen3_6_27b.ninfer`
|
| 44 |
|
| 45 |
+
On-the-fly groupwise quantization from a single BF16 source; each linear layer is quantized at conversion time into a per-group 4/5/6-bit + W8 format selected by tensor size/role (`Q4G64_F16S`, `Q5G64_F16S`, `Q6G64_F16S`, `W8G32_F16S`). Weights are groupwise-quantized while activations stay 16-bit, making this a W4A16 build.
|
| 46 |
|
| 47 |
Single-source provenance: base BF16 checkpoint `bottlecapai/ThinkingCap-Qwen3.6-27B` (rev `6a9e13bd`). Converter module `tools.convert.qwen3_6_27b.convert` (recipe `qwen3_6_27b-v2`, weights_id `groupwise-int`):
|
| 48 |
|
|
|
|
| 57 |
|
| 58 |
### 2. NVFP4 (W4A4) — `thinkingcap_qwen3_6_27b_nvfp4.ninfer`
|
| 59 |
|
| 60 |
+
NVFP4 W4A4 mixed-precision: packed 4-bit FP4 (E2M1) weights with per-tensor group scales and `input_activations: tensor_group`, so activations are also 4-bit at inference. This is the true W4A4 path; the W4A16 build keeps 16-bit activations.
|
| 61 |
|
| 62 |
+
Dual-source: base BF16 `bottlecapai/ThinkingCap-Qwen3.6-27B` (rev `6a9e13bd`) fused with vLLM-quantized NVFP4 partner [`sakamakismile/ThinkingCap-Qwen3.6-27B-NVFP4`](https://huggingface.co/sakamakismile/ThinkingCap-Qwen3.6-27B-NVFP4) (rev `9b5389d4`). The NVFP4-packed lines take 4-bit weight bytes + scales from the partner; the remaining 16-bit/resource tensors come from the base BF16 tree. Converter `tools.convert.qwen3_6_27b.convert_nvfp4` (recipe `qwen3_6_27b_nvfp4-v1`, weights_id `nvfp4`, model_id `qwen3.6-27b`):
|
| 63 |
|
| 64 |
```bash
|
| 65 |
python3 -m tools.convert.qwen3_6_27b.convert_nvfp4 \
|
|
|
|
| 74 |
|
| 75 |
### 3. NVFP4 (W4A4) w8g32-generation — `thinkingcap_qwen3_6_27b_nvfp4_w8g32.ninfer`
|
| 76 |
|
| 77 |
+
Built by the **W8G32-generation** converter `tools.convert.qwen3_8_27b.w8g32_nvfp4.convert_nvfp4` (recipe `qwen3_8_27b_nvfp4_w8g32-v1`, converter output basename `qwen3_8_27b_nvfp4_w8g32.ninfer`). This is a separate module from artifact 2's converter, not a re-quantization of it: it reuses the `qwen3_6_27b` recipe structure (additive NVFP4 MLP over 247 matrices, `blockscale-k16-m128x4-v1` packing, split GDN control, W8G32_F16S vocabulary endpoints) but pins the Qwen3.8 base model's frontend resources (its W8G32 resource profile, verified by SHA256 at conversion) instead of the Qwen3.6 resources artifact 2 embeds.
|
| 78 |
|
| 79 |
+
Dual-source, with a different source pair than artifact 2, per the recorded conversion run: the base BF16 checkpoint [`Qwen/Qwen3.8-27B`](https://huggingface.co/Qwen/Qwen3.8-27B) (rev `1d4bf0f2`) supplies the direct tensors, split GDN control projections, and W8G32_F16S vocabulary endpoints; the NVFP4 partner [`sakamakismile/Qwen3.8-27B-MTP-NVFP4`](https://huggingface.co/sakamakismile/Qwen3.8-27B-MTP-NVFP4) supplies the packed 4-bit codes plus E4M3FN scales and divisors. The embedded identity (`model_id qwen3.8-27b`, `weights_id nvfp4_w8g32`) matches this Qwen3.8 source pair rather than the ThinkingCap-Qwen3.6-27B family of artifacts 1 and 2. With `input_activations: tensor_group` this is a true W4A4 build.
|
| 80 |
|
| 81 |
+
It has the same object inventory as artifact 2 (1307 objects: 1301 tensors + 6 resources, the same format counts, the same layouts), but the files are not byte-identical: artifact 3 is 3840 bytes larger (18324067840 vs 18324064000) with a different SHA256, because its embedded frontend-resource bytes come from the Qwen3.8 base model. The histogram also includes one `Q6G64_F16S` object even though the filename omits `q6g64`; the missing token distinguishes this build from artifact 2 under the one-filename-per-(org, model, weight-types) convention. It does not mean the build lacks 6-bit weights.
|
| 82 |
|
| 83 |
```bash
|
| 84 |
python -m tools.convert.qwen3_8_27b.w8g32_nvfp4.convert_nvfp4 \
|
|
|
|
| 87 |
--out out/<name>.ninfer --device cuda
|
| 88 |
```
|
| 89 |
|
| 90 |
+
Stats: objects 1307 (1301 tensors + 6 resources); file bytes 18324067840; sha256 `00c3412b…cbd1`; layouts `blockscale-k16-m128x4-v1 ×247, contiguous-le-v1 ×935, row-split-k128-v1 ×119`.
|
| 91 |
|
| 92 |
## Differences between artifacts
|
| 93 |
|
|
|
|
| 97 |
| Sources | single (BF16 only) | two-source (BF16 + NVFP4 partner) | two-source (BF16 + NVFP4 partner) |
|
| 98 |
| Source repos | `bottlecapai/ThinkingCap-Qwen3.6-27B` | `bottlecapai/ThinkingCap-Qwen3.6-27B` + `sakamakismile/ThinkingCap-Qwen3.6-27B-NVFP4` | `Qwen/Qwen3.8-27B` + `sakamakismile/Qwen3.8-27B-MTP-NVFP4` |
|
| 99 |
| Converter | `qwen3_6_27b.convert` (groupwise recipe) | `qwen3_6_27b.convert_nvfp4` (`qwen3_6_27b_nvfp4-v1`) | `qwen3_8_27b.w8g32_nvfp4.convert_nvfp4` (`qwen3_8_27b_nvfp4_w8g32-v1`) |
|
| 100 |
+
| Tensors | 1118 | 1301 | 1301 (same format counts as 2) |
|
| 101 |
| Frontend resources | Qwen3.6 base | Qwen3.6 base | Qwen3.8 base (W8G32 profile) |
|
| 102 |
|
| 103 |
+
Artifacts 2 and 3 share an identical 1301-tensor weight layout and the same weight types (both include one Q6G64), but are not byte-identical files: the w8g32 build is 3840 bytes larger, with a different SHA256, because its embedded source and frontend-resource bytes differ even though the format counts match. They are two distinct builds (different converter generation, different source pair, different payload bytes), which is why the naming convention (one filename per org + model + weight-types) gives them distinct names. Artifact 1 is the only non-NVFP4, W4A16 build.
|
| 104 |
|
| 105 |
## Files and checksums
|
| 106 |
|
| 107 |
+
Byte sizes below are the authoritative LFS sizes from the Hugging Face repo tree.
|
| 108 |
|
| 109 |
| File | Size (bytes) | SHA256 | Scheme |
|
| 110 |
|---|---|---|---|
|
|
|
|
| 114 |
|
| 115 |
## Serving
|
| 116 |
|
| 117 |
+
All artifacts are served directly by the **NInfer** engine as self-contained single binaries, so no separate weights download is needed. Capability flags relevant to this model: multimodal image input (`--vision`), MTP decoding (1 MTP draft layer), and int8 KV cache.
|
| 118 |
|
| 119 |
## Known limitations
|
| 120 |
|
| 121 |
+
No KL-divergence or perplexity benchmark has been run on any artifact, so quantization fidelity against the BF16 reference is unmeasured. Validation is structural only: format histogram, object/tensor/resource counts, and byte/alignment integrity from the converter's accounting. No downstream quality claims follow from it.
|
| 122 |
|
| 123 |
## References
|
| 124 |
|