Instructions to use sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4")
model = AutoModelForMultimodalLM.from_pretrained("sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4

SGLang

How to use sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4 with Docker Model Runner:
```
docker model run hf.co/sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4
```

Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4 / README.md

sakamakismile

Huihui-ThinkingCap-Qwen3.6-27B-abliterated NVFP4 (W4A4) + MTP, from huihui-ai/Huihui-ThinkingCap-Qwen3.6-27B-abliterated

23e9a30 verified 4 days ago

preview code

Raw

History Blame Contribute Delete

5.24 kB

metadata

license: apache-2.0
base_model:
  - huihui-ai/Huihui-ThinkingCap-Qwen3.6-27B-abliterated
  - bottlecapai/ThinkingCap-Qwen3.6-27B
  - Qwen/Qwen3.6-27B
base_model_relation: quantized
pipeline_tag: image-text-to-text
library_name: transformers
tags:
  - qwen3_5
  - qwen3_6
  - nvfp4
  - compressed-tensors
  - w4a4
  - mtp
  - speculative-decoding
  - token-efficient
  - efficient-thinking
  - abliterated
  - uncensored
  - vllm
  - quantization
language:
  - en
  - ja
  - zh

Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4

NVFP4 (W4A4) quantization of huihui-ai/Huihui-ThinkingCap-Qwen3.6-27B-abliterated — Huihui's abliterated (refusal-removed / uncensored) finetune of bottlecapai/ThinkingCap-Qwen3.6-27B, itself a token-efficient reasoning fine-tune of Qwen3.6-27B. Produced with llm-compressor → compressed-tensors, with the native MTP speculative-decode head preserved (bf16) and the Qwen3-VL vision tower preserved (bf16).

Why this pairing is nice. You keep ThinkingCap's short-<think> token efficiency (the base cuts reasoning length by ~46 % vs Qwen3.6-27B) and Huihui's abliteration (refusal directions removed), then NVFP4 + the MTP draft cut the cost of every token. Fewer thinking tokens × faster tokens × no refusal detours = a snappy, compliant local reasoner. Abliteration can shift behavior on some prompts — evaluate for your use case.

20.6 GB on disk (down from ~55.6 GB bf16). Serves on stock vLLM 0.21+ — no --quantization flag needed (auto-detected).

Architecture

Qwen3_5ForConditionalGeneration (model_type qwen3_5), dense 27.4 B:

Hybrid attention — Gated-DeltaNet (linear) + full-attention layers, hidden 5120, 262 K native context.
Vision — Qwen3-VL ViT, kept bf16; serve text-only with --limit-mm-per-prompt.
Native MTP (mtp_num_hidden_layers=1), kept bf16 → drives vLLM speculative decoding.
Thinking-by-default reasoning model (<think>…</think>, use --reasoning-parser qwen3) — but a token-efficient, abliterated one.

Quantization recipe

QuantizationModifier(targets="Linear", scheme="NVFP4",  # W4A4, group_size 16
  ignore=["lm_head", "re:.*visual.*", "re:.*conv1d.*", "re:.*mtp.*"])

Vision tower, DeltaNet causal conv1d, lm_head, and the entire MTP head stay bf16; everything else is NVFP4 W4A4. 32 calibration samples (neuralmagic/calibration), seq 8192, pure-CPU load (sequential-pipeline onload).
This model ships the MTP head as a separate model-base-aux.safetensors (bf16 tensors). Those are grafted into the NVFP4 output (model-mtp-bf16.safetensors) and spliced into the safetensors index.
Note for re-bakers: the grafted MTP modules must also be added to quantization_config.ignore, otherwise vLLM matches mtp.*_proj against targets=["Linear"], expects NVFP4 scales that do not exist, and loads the Qwen3_5MTP draft as garbage → 0 % spec-decode acceptance. This bake adds them automatically.

Serving (vLLM ≥ 0.21)

vllm serve sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4 \
  --tensor-parallel-size 4 --max-model-len 131072 \
  --max-num-seqs 16 --gpu-memory-utilization 0.90 --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 --limit-mm-per-prompt '{"image":0,"video":0}' \
  --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'

On NVLink-less boxes add NCCL_P2P_DISABLE=1 + --disable-custom-all-reduce (and NCCL_CUMEM_ENABLE=0 if TP=8 CUDA-graph capture hangs). Drop --speculative-config for plain decode. The hybrid model's KV is light (only the full-attention layers cache), so full 128 K context fits even at TP=2.

Reasoning model → set max_tokens ≥ 4096 (prefer 8192+). Even though ThinkingCap thinks less, at a tiny budget it can still spend it all inside <think> and return empty content.
Do not produce a W4A16 / NVFP4A16 variant — it fails to serve on vLLM (gptq_marlin_repack: size_n not divisible by tile_n_size=64; the odd attention-head / DeltaNet dims violate Marlin's tile constraint). W4A4 avoids Marlin (NVFP4 cutlass/FlashInfer path).
Sampling: the base recommends temperature=1.0, top_p=0.95, top_k=20.

License & attribution

Apache-2.0, inherited from the base models. Abliteration by huihui-ai; token-efficiency fine-tune by BottleCap AI; base Qwen3.6-27B by the Qwen Team. NVFP4 quantization by sakamakismile (Lna-Lab), reusing the validated qwen3_5 dense+MTP recipe shared with sakamakismile/ThinkingCap-Qwen3.6-27B-NVFP4 and sakamakismile/Qwen3.6-27B-MTP-pi-tune-NVFP4.

Support the Base Model Author (huihui-ai)

If you find the abliterated base useful, please support huihui-ai:

Ko-fi: https://ko-fi.com/huihuiai
Bitcoin: bc1qqnkhuchxw0zqjh2ku3lu4hq45hc6gy84uk70ge