cjxzdzh/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP-GPTQ-INT4

Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP-GPTQ-INT4

GPTQ INT4 quantization of the original DavidAU/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP, optimized for dual-RTX-2080-Ti deployment.

Quantization Details

• Method: GPTQ (via modelcloud/gptqmodel v7.3.2)
• Base precision: INT4 (4-bit weights, group_size=128, symmetric, desc_act=true)
• Dynamic per-layer override: Layers 68-79 projection layers quantized at INT8 instead of INT4 to preserve reasoning quality in the upper transformer layers
• LM head: Not quantized (preserved at BF16)
• Total size: ~19 GB (vs ~56 GB BF16 original)

Deployment Configuration

This quantization was specifically optimized and tested on a dual-RTX-2080-Ti (22GB each) setup with NVLink interconnect, running a custom vLLM fork designed for consumer GPUs.

• Inference engine: weicj/vLLM-2080Ti-Definitive v0.1.14 (fork of vLLM with SM75-specific patches and INT4 GPTQ kernel optimizations)
• GPU config: 2× NVIDIA RTX 2080 Ti (22GB VRAM each, compute capability 7.5), connected via NVLink (2 links per GPU, 25.781 GB/s each)
• Tensor parallelism: TP=2
• Context length: 262K tokens
• MAX_NUM_SEQS: 2
• GPU memory utilization: 0.93
• MTP depth: 3

Benchmark Results

Measured on dual RTX 2080 Ti (NVLink) with the configuration above. All tests generate 1024 output tokens.

Prompt Length (tokens)	TTFT (ms)	ITL Avg (ms)	ITL Std (ms)	Prefill Time (ms)	Prefill Speed (tok/s)	Output Time (ms)	Decode Speed (tok/s)
4,096	3,265	39.99	0.50	3,260	1,256	15,358	66.68
8,192	6,730	40.62	0.79	6,725	1,218	11,903	86.03
16,384	14,034	41.46	0.64	14,029	1,168	17,537	58.39
32,768	29,943	42.84	1.10	29,937	1,095	16,194	63.23
65,536	66,835	45.29	1.25	66,829	981	16,531	61.94
131,072	160,941	50.18	2.40	160,936	814	19,620	52.19

• TTFT: Time To First Token
• ITL: Inter-Token Latency (average between consecutive output tokens)
• Prefill Speed: prompt tokens processed per second
• Decode Speed: output tokens generated per second

Usage

Load with vLLM:

from vllm import LLM

llm = LLM(
    model="cjxzdzh/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP-GPTQ-INT4",
    quantize="gptq",
    tensor_parallel_size=2,
    max_model_len=262144,
    gpu_memory_utilization=0.93,
    max_num_seqs=2,
)

Load with GPTQModel directly:

from gptqmodel import GPTQModel, QuantizeConfig

model = GPTQModel.from_quantized(
    "cjxzdzh/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP-GPTQ-INT4",
    device_map="auto",
)

Notes

• The dynamic INT8 override on layers 68-79 projection layers is intentional and required for maintaining reasoning quality on this model family. This is a per-layer selective quantization strategy, not a uniform INT8 quantization.
• The forked vLLM (weicj/vLLM-2080Ti-Definitive) includes critical patches for SM75 architecture and INT4 GPTQ kernel optimizations that are not present in upstream vLLM.
• For benchmark results on the base model, refer to the original repository.