KebalBaguette/Mistral-7B-Instruct-v0.3-abliterated-AWQ

Mistral 7B Instruct v0.3 Abliterated — AWQ (W4A16)

AWQ 4-bit quantization of evolveon/Mistral-7B-Instruct-v0.3-abliterated, itself an abliteration of Mistral 7B Instruct v0.3.

Why this exists

The base abliterated model was only published in BF16 and GGUF. This AWQ quant serves natively in vLLM with the Marlin kernel for fast inference.

Quantization details

• Tool: llm-compressor (vLLM project)
• Format: compressed-tensors (auto-detected by vLLM)
• Scheme: W4A16_ASYM (4-bit asymmetric weights, group_size 128, 16-bit activations)
• Algorithm: AWQ (Activation-aware Weight Quantization)
• Calibration: 128 French samples × 192 tokens from wikimedia/wikipedia (20231101.fr)
• lm_head left in fp16 (preserves output quality)
• Note: v_proj → o_proj smoothing is skipped on all 32 layers due to GQA shape mismatch (32 query heads / 8 KV heads). Standard AWQ behaviour for GQA models — quality impact is marginal since the input_layernorm → q/k/v smoothing remains active.

Usage with vLLM

vllm serve KebalBaguette/Mistral-7B-Instruct-v0.3-abliterated-AWQ \
  --quantization compressed-tensors \
  --tool-call-parser mistral \
  --enable-auto-tool-choice \
  --max-model-len 8192

License

Apache 2.0, inherited from the upstream model and the abliterated base.