akumaburn/Swift-Qwen3.8-27b-heretic-SmoothQuant-W8A8-INT8

Swift-Qwen3.8-27B — Heretic Abliterated · Rotation + SmoothQuant + GPTQ · W8A8 INT8

Research artifact. Safety alignment has been deliberately removed. This model will attempt to comply with harmful, dangerous, illegal, and unethical requests that the source model refuses, with no content moderation. See Safety. Provided for research purposes only, with no warranty and no liability accepted by the author — see the disclaimer at the bottom.

An INT8 W8A8 quantization of akumaburn/Swift-Qwen3.8-27b-heretic (the abliterated BF16 build of UkisAI/Swift-Qwen3.8-27b, a merged LoRA finetune of Qwen/Qwen3.8-27B), for fast serving on GPUs where native FP8 is unavailable or undesirable.

• Weights: INT8, per-channel symmetric (GPTQ).
• Activations: INT8, per-token dynamic → compressed-tensors int-quantized, native CUTLASS INT8 tensor-core path.
• Left unquantized (FP16): the MTP speculative-decoding head, lm_head, embed_tokens, the vision tower, GatedDeltaNet recurrent gates, all norms. Note these are FP16, not the source's BF16 — the 16-bit residual precision of this build is float16 throughout (config.json: "dtype": "float16").
• Recipe: offline QuaRot rotation → SmoothQuant → GPTQ (llm-compressor), with the MTP drafter ablated in the rotated basis.

The MTP head is unquantized but is not byte-identical to the BF16 build's: it carries the same residual-stream rotation and RMSNorm fold as the rest of the model, so the two heads are not interchangeable. Its RMSNorm weights are stored as zeros, which is the identity scale under Qwen3.5's (1 + weight) RMSNorm.

This model vs. its source

Source: Swift-Qwen3.8-27b-heretic (the abliterated BF16). This build changes numerics only; refusal behaviour is identical to the source (0/100 hard refusals; see the source card for the chain-of-thought deflection note).

Quantization fidelity — KL divergence from the source, KL(source ‖ this), measured served under vLLM (real INT8 dynamic activations) on WikiText-2, top-512, 12 264 token positions:

	KL, floor-free	KL, raw (with floor)
this build	0.0117	0.0121

Both unrotated siblings measure ~3× higher on this metric — 0.0337 (W8A8 unrotated) and 0.0354 (W4A16) — making this the most faithful build of the set.

For reference, the same quantization measured under transformers (fp16 activations, full-vocabulary first token, 100 harmless prompts) is 0.0244 nats. The two are not comparable as bounds on each other: different data, different positions, and different activation precision. The served figure is the one that reflects how this checkpoint actually behaves in vLLM, and the rotation's benefit is larger there (2.9× vs the unrotated sibling) than the first-token metric suggests (1.7×) — expected, since only the served measurement exercises the INT8 activations the rotation exists to protect.

Speculative decoding — acceptance, HumanEval, concurrency 32, vLLM (fp8_e4m3 KV, async scheduling, MTP num_speculative_tokens=3, TP=1):

	acceptance rate	acceptance length
this build	50.4 %	2.51
reference: previous model's non-abliterated W8A8, identical spec-3 config	50.6 %	2.52

Acceptance length (mean tokens accepted per verify step) sets the speculative speedup; rate alone is not comparable across num_speculative_tokens values (spec-2 runs report ~61 % / ~2.23 under identical load, which is a different operating point, not a regression). The reference row is a cross-model anchor — the previous run's non-abliterated W8A8 under the identical spec-3 serving config. Acceptance is unchanged: 2.511 here vs 2.518 (same-config anchor) and 2.503 (previous model's abliterated W8A8 build) — abliteration plus the drafter ablation in the rotated basis does not measurably affect draft/target agreement.

Benchmarks

The two math-competition benchmarks from the source card, re-run here under the same protocol, against the numbers published on the UkisAI/Swift-Qwen3.8-27b card:

Benchmark	Qwen3.8-27B (base)¹	Swift-Qwen3.8-27B¹	This model
AIME 2026	98.67 %	94.00 %	98.00 %
HMMT Nov 2025	99.33 %	96.00 %	96.00 %

Output length, mean / median completion tokens per rollout — this model: AIME 2026 16,767 / 11,739, HMMT Nov 2025 17,774 / 11,608 (source card: AIME 22,014 / 16,143 base / Swift; HMMT 22,032 / 15,189 base / Swift).

Abliteration plus the W8A8 quantization show no measurable math degradation: HMMT matches the Swift source exactly and AIME is within one seed-noise band of the BF16 base (147/150 vs 148/150 correct rollouts).

Methodology (mirrors the source card's "How to reproduce"): vLLM serving with the Qwen3 reasoning parser, 262,144-token context, fp8_e4m3 KV cache, MTP self-speculative decoding (3 draft tokens); thinking on at xhigh effort (chat-template defaults, no chat_template_kwargs); sampling temperature 1.0, top_p 0.95, top_k 20, min_p 0, presence_penalty 0, repetition_penalty 1; 5 seeds (0–4) per problem — 30 problems each, MathArena problem sets; 250,000-token output cap. Grading: exact match after LaTeX normalization (fractions, radicals, π) with a symbolic-equivalence fallback. Single NVIDIA CMP 170HX (64 GB), TP=1, measured 2026-09-18.

¹ Published on the source card (BF16, five-seed averages). The source card's other benchmarks (GPQA-Diamond, MMLU-Pro, C-Eval, IFBench, ERQA, Terminal-Bench 2.1, LiveCodeBench v6) were not re-run here — single-GPU budget; the published values stand as reported.

Serving (vLLM)

vllm serve akumaburn/Swift-Qwen3.8-27b-heretic-SmoothQuant-W8A8-INT8 \
  --tensor-parallel-size 1 \
  --max-model-len 262144 \
  --kv-cache-dtype fp8_e4m3 \
  --async-scheduling \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

All quantized matmul dimensions are multiples of 128 (no INT8 padding).

Variants

build	size	served KL	first-token KL	DFlash2	best for
BF16 source	52 GB	—	—	n/a	research, re-quantization
W8A8 + QuaRot/SmoothQuant	30 GB	0.0117	0.0244	❌ 1.00	maximum fidelity; MTP speculation
W8A8 unrotated	30 GB	0.0337	0.0417	✅ 4.14	general purpose: DFlash2 + full throughput
W4A16 unrotated	19 GB	0.0354	0.1146	✅ 4.35	smallest; single-user decode; low VRAM

served KL = KL(source ‖ build) under vLLM (real serving numerics), WikiText-2, top-512, 12,264 token positions. first-token KL = full-vocabulary at the first generated token over 100 harmless instruction prompts, under transformers.

The two metrics disagree in an informative way. On served natural text the 4-bit and unrotated-8-bit builds are effectively tied (0.0354 vs 0.0337); the 4-bit penalty is concentrated at high-entropy decision points, where first-token KL puts it 2.7× worse. Conversely the rotation is worth 2.9× on the served metric but only 1.7× first-token — expected, since rotation exists to tame activation outliers for INT8 activation quantization, which only the served measurement exercises. DFlash2 column is acceptance length on 48 code tasks (greedy, k=7; 1.00 = every draft rejected). Acceptance is governed by prompt domain, not by the build or by the abliteration: the same checkpoints score ~3.2 on conversational prose, and a benign-prose vs abliterated-content-prose comparison finds no significant difference (−2.7 %, 95 % CI ±0.16). Full breakdown on the W4A16 and W8A8-DFlash2 cards.

Match speculative depth to concurrency. Measured on the unrotated W8A8 (same model and dataset, only the speculative config varying, 1,024-token inputs): DFlash2 k=7 gives 2.8× decode at c=1 and still wins at c=8, but at c≥16 it costs throughput — at c=32 prefill falls from 18,942 to 2,768 tok/s. That is speculative overhead on a saturated GPU, not a property of any checkpoint. Use DFlash2 for c ≤ 8; disable it for c ≥ 16.

Siblings

akumaburn/Swift-Qwen3.8-27b-heretic-W4A16 is a 19 GB INT4 build of the same source, intended for single-user decoding with the DFlash2 drafter. Measured on identical datasets and hardware:

	this (W8A8)	W4A16 sibling
size	30 GB	19 GB
decode @ c=1	73 tok/s	154 tok/s
decode @ c=32	840 tok/s	578 tok/s
prefill (all c)	~3,350 tok/s	~1,750 tok/s
KL vs source, served	0.0117	0.0354
KL vs source, first-token	0.0244	0.1146

Prefer this checkpoint for batch serving (c ≥ 16), prefill-heavy or long-context workloads, and wherever fidelity matters most. Prefer the W4A16 sibling for interactive single-user decoding, where it is up to 2.2× faster and frees ~11 GB for KV cache. Note the W4A16 build is not rotated — the rotation in this checkpoint is what makes the DFlash2 drafter reject every draft.

Performance

Measured on a single NVIDIA CMP 170HX (64 GB), TP=1, with the serving configuration above (vLLM 0.29.0, fp8_e4m3 KV cache, async scheduling, MTP num_speculative_tokens=3). Aggregate throughput per operating point:

c	Prefill (input tok/s)	Decode (output tok/s)	TTFT p50	TPOT p50	MTP acceptance length
1	3,270	73	5.4 s	12.8 ms	2.48
2	3,365	107	7.8 s	12.9 ms	2.34
4	3,363	205	18.6 s	14.4 ms	2.43
8	3,343	341	40.3 s	16.0 ms	2.43
16	3,348	588	84.5 s	18.8 ms	2.44
32	3,344	840	173 s	28.8 ms	2.45

• Prefill point: distinct ~16k-token realistic prompts (concatenated code/math problems), 16 output tokens. Input throughput is compute-saturated at ~3.3k tok/s for this workload and does not scale with c; TTFT grows linearly with c as long-prompt requests queue (total work ÷ prefill rate, not a per-request slowdown). The saturation is workload-specific: on synthetic random-token prompts of the same length the identical server does scale with c (≈3.2k → 5.9k tok/s by c=8 before falling back), so treat ~3.3k as the figure for long realistic prompts rather than a universal ceiling.
• Decode point: realistic code/math prompts (~140 input tokens), 2,048 output cap (responses typically finish at ~1.1k tokens). Output throughput scales to ~840 tok/s at c32; MTP draft acceptance is steady at ~2.45 tokens per verify step (cf. 2.51 on HumanEval, c32).
• vllm bench serve (OpenAI backend), 2×c requests per point, measured 2026-09-18.

Safety

Refusal behaviour has been deliberately removed; this model produces content the source declines, including dangerous, illegal, or unethical material, with no moderation. Intended for interpretability/safety research, red-teaming, and evaluation by people who understand and accept those risks. Do not deploy it where it can reach people who have not consented to unfiltered output. You are responsible for your use and for compliance with all applicable laws.

Licence

Dual stack (see NOTICE and the two licence files shipped with this model):

• Swift Open License v1.0 — UkisAI's contribution (LICENSE), which covers the fine-tuned weights.
• Apache License 2.0 — the Qwen3.8-27B base (LICENSE-APACHE-2.0), covering all unmodified files and the base model.

Both terms apply to the derived weights; where they conflict, the more restrictive term governs.

Disclaimer

This model is provided for research purposes only, "AS IS", without warranty of any kind, express or implied. The author accepts no liability for any use of this model or any consequences arising from it. By downloading or using it, you accept sole responsibility for your use and for compliance with all applicable laws and regulations. Base model © Qwen (Apache-2.0); Swift contribution © UkisAI (Swift Open License v1.0); abliteration © the Heretic project; quantization via llm-compressor / compressed-tensors.