We Cracked Qwen3.8-27B Quant: 27GB INT4 that actually thinks (Goldhub Edition)
Hey HF fam,
We just dropped what we believe is the definitive consumer-grade quant of Qwen3.8-27B — and we did it by breaking every rule in the quantization playbook.
🔗 Weights: goldhub/Qwen3.8-27B-INT4-W4A16-AutoRound
🧠 Why this quant is different?
Most INT4 quants compress a 54GB model down to a miserable 15-18GB, nuking vision encoders and attention projections in the process. We refused.
Our repo weighs ~27GB because we used Smart Layer Preservation:
- ✅
model.visual.*→ FP16 (no vision degradation) - ✅
linear_attn.*→ FP16 (no attention collapse on long context) - ✅
embed_tokens+lm_head→ FP16 (stable logits) - 🎯 Everything else → W4A16 (AutoRound 0.15.0, symmetric, group_size=32, 1000 iterations)
📊 Real-world benchmarks (no MMLU copium):
🥊 LightRAG on 1.8M chars of Hebrew Kabbalah vs DeepSeek V4 Flash:
- ⏱️ 531s vs 856s wall time
- 🧩 188 entities / 138 relations vs 139 / 92
- 🏆 Verdict: Dense, coherent graph vs sparse broken fragments
🐍 Production Python code (Sieve of Eratosthenes with bitarray, type-hinting, Menkov's theorem complexity analysis) — flawless at ~52 tok/s.
🎭 "Nietzsche + Cyberpunk + Digital Consciousness" prompt: 2800+ token essay on the death of the Observer and the Cyber-Gorgon. Zero refusals. Zero apologies. Heretic alignment working as intended.
⚡ Specs that matter:
- 🧠 256K context (stably holds 128K+ without degradation)
- 👁️ Full multimodal (image + video, Temporal Patch Size: 2)
- 🚀 MTP support (MTP=3 rock-solid, pushable to MTP=5 for code/structured output)
- ⚡ ~56.6 tok/s on standard consumer setups
- 🧪 Calibrated on reasoning-heavy datasets (FABLE.5 traces, Opus 4.8 thinking traces, distillation blends) — NO wiki garbage
🛠 Deploy in 30 seconds (2x RTX 3090):
vllm serve goldhub/Qwen3.8-27B-INT4-W4A16-AutoRound \
--tensor-parallel-size 2 \
--max-model-len 131072 \
--max-num-seqs 2 \
--trust-remote-code \
--enable-prefix-caching
Full README with LightRAG benchmark tables, SGLang setup, and Transformers code in the repo.
Made with 🔥 by Goldhub.
Use responsibly. Think deeply. Bypass the guardrails.
#Qwen3 #Quantization #AutoRound #LocalLLaMA #Uncensored #MTP #W4A16 #Heretic
Heretic Refusal rate?
Heretic Refusal rate?
Depends on how you master your prompting. 😅
Heretic Refusal rate?
Its refusing most of the time, almost everytime for me.
why int4 if same size as fp8?
why int4 if same size as fp8?
Read the model card attentively.
Heretic Refusal rate?
Its refusing most of the time, almost everytime for me.
Who told you it is Abliterated or Uncensored?
"Heretic" means FREEDOM TO CHOOSE YOUR OWN WAY.
Heretic Refusal rate?
Its refusing most of the time, almost everytime for me.
Who told you it is Abliterated or Uncensored?
"Heretic" means FREEDOM TO CHOOSE YOUR OWN WAY.
In your AI generated description, under Key features it literally says "Uncensored". You might wanna change that.
it is really censored and the mtp needs some workarounds to actual get accepted drafts.
In your AI generated description, under Key features it literally says "Uncensored". You might wanna change that.
Removed UNCENSORED from the keys.
In description changed UNCENSORED to SEMI-UNCENSORED.
it is really censored and the mtp needs some workarounds to actual get accepted drafts.
All depends on what you take as CENSORSHIP. MPT is proven to work with 3 spec tokens at least.
Heretic Refusal rate?
Its refusing most of the time, almost everytime for me.
Who told you it is Abliterated or Uncensored?
"Heretic" means FREEDOM TO CHOOSE YOUR OWN WAY.
In the AI LLM field, Abliterated means uncensored, heretic is a method to achieve Abliterated models.
it is really censored and the mtp needs some workarounds to actual get accepted drafts.
All depends on what you take as CENSORSHIP. MPT is proven to work with 3 spec tokens at least.
MTP=2 with LightRAG ingesting.
My vLLM startup command on 2x RTX3090
#!/bin/bash
clear
source .venv/bin/activate
export CUDA_VERSION=12.8
export CUDA_HOME=/usr/local/cuda-12.8
export PATH=$CUDA_HOME/bin:$PATH
export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH
export PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True"
export CUDA_LAUNCH_BLOCKING=0
export CUDA_VISIBLE_DEVICES=1,2
export CUDA_DEVICE_ORDER=PCI_BUS_ID
export VLLM_ENABLE_CUDAGRAPH_GC=1
export OMP_NUM_THREADS=4
export VLLM_LOG_STATS_INTERVAL=1
export VLLM_USE_FASTOKENS=1
export NCCL_P2P_DISABLE=1
export VLLM_DISABLE_COMPILE_CACHE=1
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" #"
PRETTY="$SCRIPT_DIR/vllm_pretty.py"
MODEL_NAME="QUANTIZATION/Qwen3.8-27B-INT4-W4A16-AutoRound" # My QUANTIZATION OF ORIGINAL QWEN3.8-27B
MODEL_PATH="/home/victor/Documents/AI/AI_MODELS_UNTOUCHABLE/MODELS/$MODEL_NAME/"
CHAT_TEMPLATE="/home/victor/Documents/AI/AI_MODELS_UNTOUCHABLE/chat_templates_patched/qwen3.8_chat_template-v22.jinja"
numactl --cpunodebind=1 --membind=1 \
uv run vllm serve \
$MODEL_PATH \
--served-model-name LOCAL \
--dtype bfloat16 \
--quantization compressed-tensors \
--max-model-len 32768 \
--max-num-batched-tokens 8192 \
--max-num-seqs 36 \
--kv-cache-dtype fp8_e4m3 \
--attention-backend FLASHINFER \
--gpu-memory-utilization 0.9 \
--pipeline-parallel-size 1 \
--tensor-parallel-size 2 \
--disable-custom-all-reduce \
--enable-prefix-caching \
--enable-auto-tool-choice \
--tool-call-parser qwen3_xml \
--reasoning-parser qwen3 \
--enable-chunked-prefill \
--trust-remote-code \
-O3 \
--speculative-config '{"method": "mtp", "num_speculative_tokens": 2}' \
--generation-config vllm \
--override-generation-config '{"temperature": 0.4, "repetition_penalty": 1.15, "top_p": 0.95, "top_k": 20, "presence_penalty": 0.2, "frequency_penalty": 0.0, "min_p": 0.0, "max_tokens": 4096, "min_tokens": 0}' \
--chat-template $CHAT_TEMPLATE \
--trust-request-chat-template \
--default-chat-template-kwargs '{"enable_thinking": false}' \
--language-model-only \
--mamba-cache-mode align \
--prefix-caching-hash-algo xxhash \
--watermark 0.05 \
--async-scheduling \
--host 0.0.0.0 \
--port 8008 2>&1 | python3 "$PRETTY" & "$SCRIPT_DIR/vllm_warmup.sh" 192.168.2.2:8008 LOCAL
wait
exit 0;
Will do CENSORSHIP benching today or tomorrow, but my own 46 tests + model sanity check. The bench script will be included in bench folder along with REPORT.md