README.md

---
license: apache-2.0
base_model:
- huihui-ai/Huihui-ThinkingCap-Qwen3.6-27B-abliterated
- bottlecapai/ThinkingCap-Qwen3.6-27B
- Qwen/Qwen3.6-27B
base_model_relation: quantized
pipeline_tag: image-text-to-text
library_name: transformers
tags:
- qwen3_5
- qwen3_6
- nvfp4
- compressed-tensors
- w4a4
- mtp
- speculative-decoding
- token-efficient
- efficient-thinking
- abliterated
- uncensored
- vllm
- quantization
language:
- en
- ja
- zh
---

# Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4

**NVFP4 (W4A4)** quantization of [`huihui-ai/Huihui-ThinkingCap-Qwen3.6-27B-abliterated`](https://huggingface.co/huihui-ai/Huihui-ThinkingCap-Qwen3.6-27B-abliterated) — Huihui's **abliterated (refusal-removed / uncensored)** finetune of [`bottlecapai/ThinkingCap-Qwen3.6-27B`](https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B), itself a **token-efficient** reasoning fine-tune of [Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B). Produced with [`llm-compressor`](https://github.com/vllm-project/llm-compressor) → `compressed-tensors`, with the native **MTP** speculative-decode head preserved (bf16) and the **Qwen3-VL vision tower** preserved (bf16).

**Why this pairing is nice.** You keep ThinkingCap's short-`<think>` token efficiency (the base cuts reasoning length by **~46 %** vs Qwen3.6-27B) *and* Huihui's abliteration (refusal directions removed), then NVFP4 + the MTP draft cut the cost of *every* token. Fewer thinking tokens × faster tokens × no refusal detours = a snappy, compliant local reasoner. Abliteration can shift behavior on some prompts — evaluate for your use case.

20.6 GB on disk (down from ~55.6 GB bf16). Serves on **stock vLLM 0.21+** — no `--quantization` flag needed (auto-detected).

## Architecture

`Qwen3_5ForConditionalGeneration` (model_type `qwen3_5`), **dense 27.4 B**:
- **Hybrid attention** — Gated-DeltaNet (linear) + full-attention layers, hidden 5120, **262 K** native context.
- **Vision** — Qwen3-VL ViT, kept **bf16**; serve text-only with `--limit-mm-per-prompt`.
- **Native MTP** (`mtp_num_hidden_layers=1`), kept **bf16** → drives vLLM speculative decoding.
- Thinking-by-default reasoning model (`<think>…</think>`, use `--reasoning-parser qwen3`) — but a *token-efficient*, *abliterated* one.

## Quantization recipe

```
QuantizationModifier(targets="Linear", scheme="NVFP4",  # W4A4, group_size 16
  ignore=["lm_head", "re:.*visual.*", "re:.*conv1d.*", "re:.*mtp.*"])
```
- Vision tower, DeltaNet causal `conv1d`, `lm_head`, and the entire MTP head stay **bf16**; everything else is NVFP4 W4A4. 32 calibration samples (`neuralmagic/calibration`), seq 8192, pure-CPU load (sequential-pipeline onload).
- This model ships the MTP head as a separate `model-base-aux.safetensors` (bf16 tensors). Those are **grafted** into the NVFP4 output (`model-mtp-bf16.safetensors`) and spliced into the safetensors index.
- **Note for re-bakers:** the grafted MTP modules **must** also be added to `quantization_config.ignore`, otherwise vLLM matches `mtp.*_proj` against `targets=["Linear"]`, expects NVFP4 scales that do not exist, and loads the `Qwen3_5MTP` draft as garbage → **0 % spec-decode acceptance**. This bake adds them automatically.

## Serving (vLLM ≥ 0.21)

```bash
vllm serve sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4 \
  --tensor-parallel-size 4 --max-model-len 131072 \
  --max-num-seqs 16 --gpu-memory-utilization 0.90 --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 --limit-mm-per-prompt '{"image":0,"video":0}' \
  --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'
```
On NVLink-less boxes add `NCCL_P2P_DISABLE=1` + `--disable-custom-all-reduce` (and `NCCL_CUMEM_ENABLE=0` if TP=8 CUDA-graph capture hangs). Drop `--speculative-config` for plain decode. The hybrid model's KV is light (only the full-attention layers cache), so **full 128 K context fits even at TP=2**.

- **Reasoning model → set `max_tokens` ≥ 4096 (prefer 8192+).** Even though ThinkingCap thinks less, at a tiny budget it can still spend it all inside `<think>` and return empty content.
- **Do not produce a W4A16 / NVFP4A16 variant** — it fails to serve on vLLM (`gptq_marlin_repack: size_n not divisible by tile_n_size=64`; the odd attention-head / DeltaNet dims violate Marlin's tile constraint). W4A4 avoids Marlin (NVFP4 cutlass/FlashInfer path).
- Sampling: the base recommends `temperature=1.0, top_p=0.95, top_k=20`.

## License & attribution

Apache-2.0, inherited from the base models. Abliteration by **huihui-ai**; token-efficiency fine-tune by **BottleCap AI**; base **Qwen3.6-27B** by the Qwen Team. NVFP4 quantization by **sakamakismile** (Lna-Lab), reusing the validated `qwen3_5` dense+MTP recipe shared with [`sakamakismile/ThinkingCap-Qwen3.6-27B-NVFP4`](https://huggingface.co/sakamakismile/ThinkingCap-Qwen3.6-27B-NVFP4) and [`sakamakismile/Qwen3.6-27B-MTP-pi-tune-NVFP4`](https://huggingface.co/sakamakismile/Qwen3.6-27B-MTP-pi-tune-NVFP4).

## Support the Base Model Author (huihui-ai)

If you find the abliterated base useful, please support huihui-ai:

- **Ko-fi:** [https://ko-fi.com/huihuiai](https://ko-fi.com/huihuiai)
- **Bitcoin:** `bc1qqnkhuchxw0zqjh2ku3lu4hq45hc6gy84uk70ge`