Qwythos-9B-Claude-Mythos-5-1M-AWQ (W4A16)

AWQ 4-bit (W4A16) quantization of empero-ai/Qwythos-9B-Claude-Mythos-5-1M — a Claude-Mythos/Fable-trace reasoning fine-tune of Qwen3.5-9B (qwen3_5: a dense hybrid GatedDeltaNet (linear-attention) + full-attention model, 3:1, with a vision tower and a 1M-token YaRN context).

Variant: AWQ W4A16 — 4-bit symmetric integer weights, group size 128, with activation-aware scaling. Activations stay BF16. Disk size: ~11.0 GB (vs ~18.8 GB BF16). The reduction is modest because the language model's GatedDeltaNet linear_attn layers (24 of 32), the large 248K-vocab embeddings + lm_head, and the vision tower are all kept in BF16; only the attention and MLP linears are int4. Quantized by: sahilchachra Tooling: llm-compressor 0.12 (AWQModifier + QuantizationModifier) → compressed-tensors pack-quantized

What is quantized

Quantized to int4 (128 modules):

self_attn.{q,k,v,o}_proj on the 8 full-attention layers — 32
mlp.{gate,up,down}_proj on all 32 layers — 96

Kept in BF16: the GatedDeltaNet linear_attn (mamba) layers, the vision tower, the MTP head, token embeddings, lm_head, and all norms.

Calibration

Calibrated on 64 sequences × 512 tokens of general instruction-following chat (HuggingFaceH4/ultrachat_200k) rendered through the model's chat template — broad activation coverage representative of the model's general inference distribution.

Serving (vLLM)

from vllm import LLM, SamplingParams
llm = LLM(model="sahilchachra/Qwythos-9B-Claude-Mythos-5-1M-AWQ",
          trust_remote_code=True, dtype="bfloat16")
print(llm.generate(["Explain a TCP SYN flood, briefly."],
                   SamplingParams(max_tokens=256))[0].outputs[0].text)

The GatedDeltaNet kernels (causal-conv1d / flash-linear-attention) fall back to a pure-PyTorch path where unavailable (e.g. ARM/Jetson) — correct, just slower. compressed-tensors == 0.17.1's eager decompress_model mis-reads group_size for pack-quantized checkpoints; load via vLLM (its own loader) or a fixed compressed-tensors. The weights/scales on disk are standard and correct.

Notes

Inherits the base model's uncensored behavior and 1M-token (YaRN) context.
The source repo omitted image/video *_processor_config.json; the standard Qwen3.5 (VL) processor configs are included here so the model loads in vLLM out of the box.
Format compressed-tensors pack-quantized; symmetric int4 for broad loadability.
Smoke-tested (loads in vLLM + coherent generation); not a full quality benchmark.