knoopx commited on
Commit
ab212ac
·
verified ·
1 Parent(s): e076255

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +14 -14
README.md CHANGED
@@ -28,12 +28,12 @@ base_model:
28
 
29
  # ThinkingCap-Qwen3.6-27B
30
 
31
- Three single-file NInfer (`*.ninfer`) artifacts for **ThinkingCap-Qwen3.6-27B**, a token-efficient, thinking-oriented, multimodal (image+video→text) 27B finetune of [`Qwen/Qwen3.6-27B`](https://huggingface.co/Qwen/Qwen3.6-27B). Each artifact is a self-contained binary the NInfer engine serves directly. All three trace back to the same finetune, but each is produced by a distinct converter module and carries a distinct recipe and weight profile so none is a re-label of another.
32
 
33
  ## Model architecture
34
 
35
  - `Qwen3_5ForConditionalGeneration` (`model_type: qwen3_5`), dense multimodal (image+video→text) finetune of [`Qwen/Qwen3.6-27B`](https://huggingface.co/Qwen/Qwen3.6-27B). Fully dense transformer: every decoder layer uses a dense SwiGLU MLP with no routing.
36
- - **Text:** 64 decoder layers; hybrid attention 16 full-attention layers interleaved with 48 recurrent layers (short convolution, width 4), full-attention interval 4; hidden 5120, intermediate 17408, output matrix 248320 rows (tokenizer domain 248077); 24 query / 4 KV heads (head dim 256); 1 MTP draft layer; 262144 max position embeddings.
37
  - **Vision:** 27-layer ViT (hidden 1152, intermediate 4304, 16 heads), 16×16 spatial patch, 2-frame temporal patch, 2×2 spatial merge.
38
  - **Positional:** RoPE θ=10,000,000 with mRoPE section `[11, 11, 10]`.
39
  - Upstream base: [`bottlecapai/ThinkingCap-Qwen3.6-27B`](https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B), itself a finetune of [`Qwen/Qwen3.6-27B`](https://huggingface.co/Qwen/Qwen3.6-27B).
@@ -42,7 +42,7 @@ Three single-file NInfer (`*.ninfer`) artifacts for **ThinkingCap-Qwen3.6-27B**,
42
 
43
  ### 1. Groupwise-int4 (W4A16) — `thinkingcap_qwen3_6_27b.ninfer`
44
 
45
- On-the-fly groupwise quantization from a single BF16 source; each linear layer is quantized at conversion time into a per-group 4/5/6-bit + W8 format selected by tensor size/role (`Q4G64_F16S`, `Q5G64_F16S`, `Q6G64_F16S`, `W8G32_F16S`). Weights are groupwise-quantized while activations stay 16-bit (W4A16-class build).
46
 
47
  Single-source provenance: base BF16 checkpoint `bottlecapai/ThinkingCap-Qwen3.6-27B` (rev `6a9e13bd`). Converter module `tools.convert.qwen3_6_27b.convert` (recipe `qwen3_6_27b-v2`, weights_id `groupwise-int`):
48
 
@@ -57,9 +57,9 @@ Histogram (sums to 1118): `BF16 582, Q5G64_F16S 246, Q4G64_F16S 183, FP32 96, W8
57
 
58
  ### 2. NVFP4 (W4A4) — `thinkingcap_qwen3_6_27b_nvfp4.ninfer`
59
 
60
- NVFP4 W4A4 mixed-precision: packed 4-bit FP4 (E2M1) weights with per-tensor group scales and `input_activations: tensor_group`, so activations are also 4-bit at inference. True W4A4 path (vs W4A16, where activations stay 16-bit).
61
 
62
- Dual-source: base BF16 `bottlecapai/ThinkingCap-Qwen3.6-27B` (rev `6a9e13bd`) fused with vLLM-quantized NVFP4 partner [`sakamakismile/ThinkingCap-Qwen3.6-27B-NVFP4`](https://huggingface.co/sakamakismile/ThinkingCap-Qwen3.6-27B-NVFP4) (rev `9b5389d4`). NFp4-packed lines take 4-bit weight bytes + scales from the partner; remaining 16-bit/resource tensors come from the base BF16 tree. Converter `tools.convert.qwen3_6_27b.convert_nvfp4` (recipe `qwen3_6_27b_nvfp4-v1`, weights_id `nvfp4`, model_id `qwen3.6-27b`):
63
 
64
  ```bash
65
  python3 -m tools.convert.qwen3_6_27b.convert_nvfp4 \
@@ -74,11 +74,11 @@ Histogram (sums to 1301): `BF16 591, FP32 343, NVFP4 247, Q4G64_F16S 55, Q5G64_F
74
 
75
  ### 3. NVFP4 (W4A4) w8g32-generation — `thinkingcap_qwen3_6_27b_nvfp4_w8g32.ninfer`
76
 
77
- How it was created: built by the **W8G32-generation** converter `tools.convert.qwen3_8_27b.w8g32_nvfp4.convert_nvfp4` (recipe `qwen3_8_27b_nvfp4_w8g32-v1`, converter output basename `qwen3_8_27b_nvfp4_w8g32.ninfer`). It is a separate converter module from Artifact 2's, not a re-quantisation of it: it reuses the `qwen3_6_27b` base recipe structure (additive NVFP4 MLP over 247 matrices, `blockscale-k16-m128x4-v1` packing, split GDN control, W8G32_F16S vocabulary endpoints) but pins the **Qwen3.8 base model's own frontend resources** (its W8G32 resource profile, checked by SHA256) rather than the Qwen3.6 resources Artifact 2 embeds.
78
 
79
- Dual-source, with a different source pair than Artifact 2 per the recorded conversion run: base BF16 checkpoint [`Qwen/Qwen3.8-27B`](https://huggingface.co/Qwen/Qwen3.8-27B) (rev `1d4bf0f2`) supplies the direct tensors, split GDN control projections and W8G32_F16S vocabulary endpoints; NVFP4 partner [`sakamakismile/Qwen3.8-27B-MTP-NVFP4`](https://huggingface.co/sakamakismile/Qwen3.8-27B-MTP-NVFP4) supplies the packed 4-bit codes + E4M3FN scales/divisors. The embedded identity (`model_id qwen3.8-27b`, `weights_id nvfp4_w8g32`) matches this Qwen3.8 source pair, not the ThinkingCap-Qwen3.6-27B family of Artifacts 12. With `input_activations: tensor_group` it is a true W4A4 build.
80
 
81
- Same object inventory as Artifact 2 1307 objects (1301 tensors + 6 resources) with the same format counts (BF16 591, FP32 343, Q4G64_F16S 55, Q5G64_F16S 54, Q6G64_F16S 1, W8G32_F16S 9, NVFP4 247, I32 1) and the same layouts but the files are different: distinct SHA256, +3840 bytes, and different embedded bytes, because the w8g32-generation pins the Qwen3.8 base model's W8G32 frontend resources (checked by SHA256 at conversion) instead of the Qwen3.6 resources Artifact 2 embeds. Its histogram also includes one `Q6G64_F16S` object even though its deploy name omits `q6g64`; the missing `q6g64` token is the naming-collision distinction from Artifact 2 (which derives the same weight-types), not an absence of a 6-bit weight.
82
 
83
  ```bash
84
  python -m tools.convert.qwen3_8_27b.w8g32_nvfp4.convert_nvfp4 \
@@ -87,7 +87,7 @@ python -m tools.convert.qwen3_8_27b.w8g32_nvfp4.convert_nvfp4 \
87
  --out out/<name>.ninfer --device cuda
88
  ```
89
 
90
- Stats: objects 1307 (1301 tensors + 6 resources); byte total 18324067840; sha256 `00c3412b…cbd1`; layouts `blockscale-k16-m128x4-v1 ×247, contiguous-le-v1 ×935, row-split-k128-v1 ×119`. Identity embedded in the artifact: model_id `qwen3.8-27b`, weights_id `nvfp4_w8g32`.
91
 
92
  ## Differences between artifacts
93
 
@@ -97,14 +97,14 @@ Stats: objects 1307 (1301 tensors + 6 resources); byte total 18324067840; sha256
97
  | Sources | single (BF16 only) | two-source (BF16 + NVFP4 partner) | two-source (BF16 + NVFP4 partner) |
98
  | Source repos | `bottlecapai/ThinkingCap-Qwen3.6-27B` | `bottlecapai/ThinkingCap-Qwen3.6-27B` + `sakamakismile/ThinkingCap-Qwen3.6-27B-NVFP4` | `Qwen/Qwen3.8-27B` + `sakamakismile/Qwen3.8-27B-MTP-NVFP4` |
99
  | Converter | `qwen3_6_27b.convert` (groupwise recipe) | `qwen3_6_27b.convert_nvfp4` (`qwen3_6_27b_nvfp4-v1`) | `qwen3_8_27b.w8g32_nvfp4.convert_nvfp4` (`qwen3_8_27b_nvfp4_w8g32-v1`) |
100
- | Tensors | 1118 | 1301 | 1301 (same format-counts as 2) |
101
  | Frontend resources | Qwen3.6 base | Qwen3.6 base | Qwen3.8 base (W8G32 profile) |
102
 
103
- Artifacts 2 and 3 share an identical 1301-tensor weight layout and the same weight-types (both include one Q6G64), and are not byte-identical files: the w8g32 build is +3840 bytes with a different SHA256 because its embedded source and frontend-resource bytes differ, even though the format counts match. They are therefore two distinct builds (different converter generation, different source pair, different payload bytes), which is why the naming convention (one filename per org+model+weight-types) requires them to carry distinct names. Artifact 1 is the only non-NVFP4, W4A16 build.
104
 
105
  ## Files and checksums
106
 
107
- Byte sizes below are authoritative LFS sizes.
108
 
109
  | File | Size (bytes) | SHA256 | Scheme |
110
  |---|---|---|---|
@@ -114,11 +114,11 @@ Byte sizes below are authoritative LFS sizes.
114
 
115
  ## Serving
116
 
117
- All artifacts are served directly by the **NInfer** engine as self-contained single binaries no separate weights download. Capability flags relevant to this model: multimodal image input (`--vision`), MTP decoding (1 MTP draft layer), and int8 KV cache.
118
 
119
  ## Known limitations
120
 
121
- No KL-divergence or perplexity benchmark has been run on any artifact; quantization fidelity is unmeasured against the BF16 reference. Validation performed is structural only format histogram, object/tensor/resource counts, and byte/alignment integrity from the converter's accounting. The artifacts are not numerically validated for downstream quality claims.
122
 
123
  ## References
124
 
 
28
 
29
  # ThinkingCap-Qwen3.6-27B
30
 
31
+ Three single-file NInfer (`*.ninfer`) artifacts for **ThinkingCap-Qwen3.6-27B**, a token-efficient, thinking-oriented, multimodal (image+video→text) 27B finetune of [`Qwen/Qwen3.6-27B`](https://huggingface.co/Qwen/Qwen3.6-27B). Each artifact is a self-contained binary the NInfer engine serves directly. Artifacts 1 and 2 are built from the ThinkingCap-Qwen3.6-27B finetune family. Artifact 3 has the same weight layout but is produced from the Qwen3.8-27B source pair by a newer converter generation. None is a rename or re-label of another.
32
 
33
  ## Model architecture
34
 
35
  - `Qwen3_5ForConditionalGeneration` (`model_type: qwen3_5`), dense multimodal (image+video→text) finetune of [`Qwen/Qwen3.6-27B`](https://huggingface.co/Qwen/Qwen3.6-27B). Fully dense transformer: every decoder layer uses a dense SwiGLU MLP with no routing.
36
+ - **Text:** 64 decoder layers; hybrid attention: 16 full-attention layers interleaved with 48 recurrent layers (short convolution, width 4), full-attention interval 4; hidden 5120, intermediate 17408, output matrix 248320 rows (tokenizer domain 248077); 24 query / 4 KV heads (head dim 256); 1 MTP draft layer; 262144 max position embeddings.
37
  - **Vision:** 27-layer ViT (hidden 1152, intermediate 4304, 16 heads), 16×16 spatial patch, 2-frame temporal patch, 2×2 spatial merge.
38
  - **Positional:** RoPE θ=10,000,000 with mRoPE section `[11, 11, 10]`.
39
  - Upstream base: [`bottlecapai/ThinkingCap-Qwen3.6-27B`](https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B), itself a finetune of [`Qwen/Qwen3.6-27B`](https://huggingface.co/Qwen/Qwen3.6-27B).
 
42
 
43
  ### 1. Groupwise-int4 (W4A16) — `thinkingcap_qwen3_6_27b.ninfer`
44
 
45
+ On-the-fly groupwise quantization from a single BF16 source; each linear layer is quantized at conversion time into a per-group 4/5/6-bit + W8 format selected by tensor size/role (`Q4G64_F16S`, `Q5G64_F16S`, `Q6G64_F16S`, `W8G32_F16S`). Weights are groupwise-quantized while activations stay 16-bit, making this a W4A16 build.
46
 
47
  Single-source provenance: base BF16 checkpoint `bottlecapai/ThinkingCap-Qwen3.6-27B` (rev `6a9e13bd`). Converter module `tools.convert.qwen3_6_27b.convert` (recipe `qwen3_6_27b-v2`, weights_id `groupwise-int`):
48
 
 
57
 
58
  ### 2. NVFP4 (W4A4) — `thinkingcap_qwen3_6_27b_nvfp4.ninfer`
59
 
60
+ NVFP4 W4A4 mixed-precision: packed 4-bit FP4 (E2M1) weights with per-tensor group scales and `input_activations: tensor_group`, so activations are also 4-bit at inference. This is the true W4A4 path; the W4A16 build keeps 16-bit activations.
61
 
62
+ Dual-source: base BF16 `bottlecapai/ThinkingCap-Qwen3.6-27B` (rev `6a9e13bd`) fused with vLLM-quantized NVFP4 partner [`sakamakismile/ThinkingCap-Qwen3.6-27B-NVFP4`](https://huggingface.co/sakamakismile/ThinkingCap-Qwen3.6-27B-NVFP4) (rev `9b5389d4`). The NVFP4-packed lines take 4-bit weight bytes + scales from the partner; the remaining 16-bit/resource tensors come from the base BF16 tree. Converter `tools.convert.qwen3_6_27b.convert_nvfp4` (recipe `qwen3_6_27b_nvfp4-v1`, weights_id `nvfp4`, model_id `qwen3.6-27b`):
63
 
64
  ```bash
65
  python3 -m tools.convert.qwen3_6_27b.convert_nvfp4 \
 
74
 
75
  ### 3. NVFP4 (W4A4) w8g32-generation — `thinkingcap_qwen3_6_27b_nvfp4_w8g32.ninfer`
76
 
77
+ Built by the **W8G32-generation** converter `tools.convert.qwen3_8_27b.w8g32_nvfp4.convert_nvfp4` (recipe `qwen3_8_27b_nvfp4_w8g32-v1`, converter output basename `qwen3_8_27b_nvfp4_w8g32.ninfer`). This is a separate module from artifact 2's converter, not a re-quantization of it: it reuses the `qwen3_6_27b` recipe structure (additive NVFP4 MLP over 247 matrices, `blockscale-k16-m128x4-v1` packing, split GDN control, W8G32_F16S vocabulary endpoints) but pins the Qwen3.8 base model's frontend resources (its W8G32 resource profile, verified by SHA256 at conversion) instead of the Qwen3.6 resources artifact 2 embeds.
78
 
79
+ Dual-source, with a different source pair than artifact 2, per the recorded conversion run: the base BF16 checkpoint [`Qwen/Qwen3.8-27B`](https://huggingface.co/Qwen/Qwen3.8-27B) (rev `1d4bf0f2`) supplies the direct tensors, split GDN control projections, and W8G32_F16S vocabulary endpoints; the NVFP4 partner [`sakamakismile/Qwen3.8-27B-MTP-NVFP4`](https://huggingface.co/sakamakismile/Qwen3.8-27B-MTP-NVFP4) supplies the packed 4-bit codes plus E4M3FN scales and divisors. The embedded identity (`model_id qwen3.8-27b`, `weights_id nvfp4_w8g32`) matches this Qwen3.8 source pair rather than the ThinkingCap-Qwen3.6-27B family of artifacts 1 and 2. With `input_activations: tensor_group` this is a true W4A4 build.
80
 
81
+ It has the same object inventory as artifact 2 (1307 objects: 1301 tensors + 6 resources, the same format counts, the same layouts), but the files are not byte-identical: artifact 3 is 3840 bytes larger (18324067840 vs 18324064000) with a different SHA256, because its embedded frontend-resource bytes come from the Qwen3.8 base model. The histogram also includes one `Q6G64_F16S` object even though the filename omits `q6g64`; the missing token distinguishes this build from artifact 2 under the one-filename-per-(org, model, weight-types) convention. It does not mean the build lacks 6-bit weights.
82
 
83
  ```bash
84
  python -m tools.convert.qwen3_8_27b.w8g32_nvfp4.convert_nvfp4 \
 
87
  --out out/<name>.ninfer --device cuda
88
  ```
89
 
90
+ Stats: objects 1307 (1301 tensors + 6 resources); file bytes 18324067840; sha256 `00c3412b…cbd1`; layouts `blockscale-k16-m128x4-v1 ×247, contiguous-le-v1 ×935, row-split-k128-v1 ×119`.
91
 
92
  ## Differences between artifacts
93
 
 
97
  | Sources | single (BF16 only) | two-source (BF16 + NVFP4 partner) | two-source (BF16 + NVFP4 partner) |
98
  | Source repos | `bottlecapai/ThinkingCap-Qwen3.6-27B` | `bottlecapai/ThinkingCap-Qwen3.6-27B` + `sakamakismile/ThinkingCap-Qwen3.6-27B-NVFP4` | `Qwen/Qwen3.8-27B` + `sakamakismile/Qwen3.8-27B-MTP-NVFP4` |
99
  | Converter | `qwen3_6_27b.convert` (groupwise recipe) | `qwen3_6_27b.convert_nvfp4` (`qwen3_6_27b_nvfp4-v1`) | `qwen3_8_27b.w8g32_nvfp4.convert_nvfp4` (`qwen3_8_27b_nvfp4_w8g32-v1`) |
100
+ | Tensors | 1118 | 1301 | 1301 (same format counts as 2) |
101
  | Frontend resources | Qwen3.6 base | Qwen3.6 base | Qwen3.8 base (W8G32 profile) |
102
 
103
+ Artifacts 2 and 3 share an identical 1301-tensor weight layout and the same weight types (both include one Q6G64), but are not byte-identical files: the w8g32 build is 3840 bytes larger, with a different SHA256, because its embedded source and frontend-resource bytes differ even though the format counts match. They are two distinct builds (different converter generation, different source pair, different payload bytes), which is why the naming convention (one filename per org + model + weight-types) gives them distinct names. Artifact 1 is the only non-NVFP4, W4A16 build.
104
 
105
  ## Files and checksums
106
 
107
+ Byte sizes below are the authoritative LFS sizes from the Hugging Face repo tree.
108
 
109
  | File | Size (bytes) | SHA256 | Scheme |
110
  |---|---|---|---|
 
114
 
115
  ## Serving
116
 
117
+ All artifacts are served directly by the **NInfer** engine as self-contained single binaries, so no separate weights download is needed. Capability flags relevant to this model: multimodal image input (`--vision`), MTP decoding (1 MTP draft layer), and int8 KV cache.
118
 
119
  ## Known limitations
120
 
121
+ No KL-divergence or perplexity benchmark has been run on any artifact, so quantization fidelity against the BF16 reference is unmeasured. Validation is structural only: format histogram, object/tensor/resource counts, and byte/alignment integrity from the converter's accounting. No downstream quality claims follow from it.
122
 
123
  ## References
124