Instructions to use sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4")
model = AutoModelForMultimodalLM.from_pretrained("sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4

SGLang

How to use sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4 with Docker Model Runner:
```
docker model run hf.co/sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4
```

Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4 / README.md

sakamakismile

Huihui-ThinkingCap-Qwen3.6-27B-abliterated NVFP4 (W4A4) + MTP, from huihui-ai/Huihui-ThinkingCap-Qwen3.6-27B-abliterated

23e9a30 verified 4 days ago

preview code

Raw

History Blame Contribute Delete

5.24 kB

	---
	license: apache-2.0
	base_model:
	- huihui-ai/Huihui-ThinkingCap-Qwen3.6-27B-abliterated
	- bottlecapai/ThinkingCap-Qwen3.6-27B
	- Qwen/Qwen3.6-27B
	base_model_relation: quantized
	pipeline_tag: image-text-to-text
	library_name: transformers
	tags:
	- qwen3_5
	- qwen3_6
	- nvfp4
	- compressed-tensors
	- w4a4
	- mtp
	- speculative-decoding
	- token-efficient
	- efficient-thinking
	- abliterated
	- uncensored
	- vllm
	- quantization
	language:
	- en
	- ja
	- zh
	---

	# Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4

	NVFP4 (W4A4) quantization of [`huihui-ai/Huihui-ThinkingCap-Qwen3.6-27B-abliterated`](https://huggingface.co/huihui-ai/Huihui-ThinkingCap-Qwen3.6-27B-abliterated) — Huihui's abliterated (refusal-removed / uncensored) finetune of [`bottlecapai/ThinkingCap-Qwen3.6-27B`](https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B), itself a token-efficient reasoning fine-tune of [Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B). Produced with [`llm-compressor`](https://github.com/vllm-project/llm-compressor) → `compressed-tensors`, with the native MTP speculative-decode head preserved (bf16) and the Qwen3-VL vision tower preserved (bf16).

	Why this pairing is nice. You keep ThinkingCap's short-`<think>` token efficiency (the base cuts reasoning length by ~46 % vs Qwen3.6-27B) and Huihui's abliteration (refusal directions removed), then NVFP4 + the MTP draft cut the cost of every token. Fewer thinking tokens × faster tokens × no refusal detours = a snappy, compliant local reasoner. Abliteration can shift behavior on some prompts — evaluate for your use case.

	20.6 GB on disk (down from ~55.6 GB bf16). Serves on stock vLLM 0.21+ — no `--quantization` flag needed (auto-detected).

	## Architecture

	`Qwen3_5ForConditionalGeneration` (model_type `qwen3_5`), dense 27.4 B:
	- Hybrid attention — Gated-DeltaNet (linear) + full-attention layers, hidden 5120, 262 K native context.
	- Vision — Qwen3-VL ViT, kept bf16; serve text-only with `--limit-mm-per-prompt`.
	- Native MTP (`mtp_num_hidden_layers=1`), kept bf16 → drives vLLM speculative decoding.
	- Thinking-by-default reasoning model (`<think>…</think>`, use `--reasoning-parser qwen3`) — but a token-efficient, abliterated one.

	## Quantization recipe

	```
	QuantizationModifier(targets="Linear", scheme="NVFP4", # W4A4, group_size 16
	ignore=["lm_head", "re:.visual.", "re:.conv1d.", "re:.mtp."])
	```
	- Vision tower, DeltaNet causal `conv1d`, `lm_head`, and the entire MTP head stay bf16; everything else is NVFP4 W4A4. 32 calibration samples (`neuralmagic/calibration`), seq 8192, pure-CPU load (sequential-pipeline onload).
	- This model ships the MTP head as a separate `model-base-aux.safetensors` (bf16 tensors). Those are grafted into the NVFP4 output (`model-mtp-bf16.safetensors`) and spliced into the safetensors index.
	- Note for re-bakers: the grafted MTP modules must also be added to `quantization_config.ignore`, otherwise vLLM matches `mtp._proj` against `targets=["Linear"]`, expects NVFP4 scales that do not exist, and loads the `Qwen3_5MTP` draft as garbage → 0 % spec-decode acceptance*. This bake adds them automatically.

	## Serving (vLLM ≥ 0.21)

	```bash
	vllm serve sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4 \
	--tensor-parallel-size 4 --max-model-len 131072 \
	--max-num-seqs 16 --gpu-memory-utilization 0.90 --kv-cache-dtype fp8 \
	--reasoning-parser qwen3 --limit-mm-per-prompt '{"image":0,"video":0}' \
	--speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'
	```
	On NVLink-less boxes add `NCCL_P2P_DISABLE=1` + `--disable-custom-all-reduce` (and `NCCL_CUMEM_ENABLE=0` if TP=8 CUDA-graph capture hangs). Drop `--speculative-config` for plain decode. The hybrid model's KV is light (only the full-attention layers cache), so full 128 K context fits even at TP=2.

	- Reasoning model → set `max_tokens` ≥ 4096 (prefer 8192+). Even though ThinkingCap thinks less, at a tiny budget it can still spend it all inside `<think>` and return empty content.
	- Do not produce a W4A16 / NVFP4A16 variant — it fails to serve on vLLM (`gptq_marlin_repack: size_n not divisible by tile_n_size=64`; the odd attention-head / DeltaNet dims violate Marlin's tile constraint). W4A4 avoids Marlin (NVFP4 cutlass/FlashInfer path).
	- Sampling: the base recommends `temperature=1.0, top_p=0.95, top_k=20`.

	## License & attribution

	Apache-2.0, inherited from the base models. Abliteration by huihui-ai; token-efficiency fine-tune by BottleCap AI; base Qwen3.6-27B by the Qwen Team. NVFP4 quantization by sakamakismile (Lna-Lab), reusing the validated `qwen3_5` dense+MTP recipe shared with [`sakamakismile/ThinkingCap-Qwen3.6-27B-NVFP4`](https://huggingface.co/sakamakismile/ThinkingCap-Qwen3.6-27B-NVFP4) and [`sakamakismile/Qwen3.6-27B-MTP-pi-tune-NVFP4`](https://huggingface.co/sakamakismile/Qwen3.6-27B-MTP-pi-tune-NVFP4).

	## Support the Base Model Author (huihui-ai)

	If you find the abliterated base useful, please support huihui-ai:

	- Ko-fi: [https://ko-fi.com/huihuiai](https://ko-fi.com/huihuiai)
	- Bitcoin: `bc1qqnkhuchxw0zqjh2ku3lu4hq45hc6gy84uk70ge`