---
base_model:
- migtissera/Tess-4-27B
base_model_relation: quantized
license: apache-2.0
pipeline_tag: image-text-to-text
tags:
- qwen3_5
- fp8
- compressed-tensors
- mtp
- speculative-decoding
- multimodal
---

# Tess-4-27B-FP8

FP8 (`compressed-tensors`, `FP8_DYNAMIC` W8A8) quantisation of [`migtissera/Tess-4-27B`](https://huggingface.co/migtissera/Tess-4-27B),
**attnbf16 variant**: the entire self-attention path is kept in bf16 and only the MLPs are FP8. Ships a
**working MTP self-speculative-decoding head**.

## What's kept in bf16

`lm_head`, the **MTP head**, the **vision tower**, the whole `linear_attn` (Gated-DeltaNet / SSM) block,
and the **entire `self_attn` path** (incl. the attention output gate fused into `q_proj` on Qwen3.5/3.6).
Only the ~17 B MLP params are FP8. This keeps quantisation off the multiplicative attention gate and the
16 long-range full-attention layers, at a cost of ~+1.5 GiB (~4.6 %) vs a fully-quantised FP8 build. See
`recipe.yaml`.

Accuracy vs the bf16 parent: **KLD ≈ 0.0078 nats** (`kld_tess_attnbf16.json`), measured per-token on
`neuralmagic/calibration` (8 samples, seq 1024). This is markedly lower than a plain-attention FP8 build
of the same base.

## Speculative decoding (MTP)

The bf16 MTP head is declared in `quantization_config.ignore` so vLLM loads it correctly. (A bf16 MTP head
regrafted into a compressed-tensors quant is otherwise mis-loaded and yields 0 % draft acceptance — this
build fixes that.)

**Measured MTP acceptance:** **75.2 %** (vLLM 0.26.0, Blackwell, greedy,
`--speculative-config '{"method":"mtp","num_speculative_tokens":3}'`).

```bash
vllm serve huginnfork/Tess-4-27B-FP8 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --max-num-seqs 32
```

`--max-num-seqs 32` (or lower) is required — Qwen3.6 is a hybrid linear-attention model whose Mamba cache
otherwise runs out of blocks at the default `max_num_seqs`.