---
license: cc-by-nc-4.0
base_model: lenML/aya-expanse-8b-abliterated
tags:
  - cohere
  - aya
  - abliterated
  - uncensored
  - awq
  - compressed-tensors
  - quantized
language:
  - fr
  - en
---

# Aya Expanse 8B Abliterated — AWQ (W4A16)

AWQ 4-bit quantization of [`lenML/aya-expanse-8b-abliterated`](https://huggingface.co/lenML/aya-expanse-8b-abliterated),
itself an abliteration of [Aya Expanse 8B](https://huggingface.co/CohereForAI/aya-expanse-8b).

## Why this exists

The base abliterated model was only published in BF16 and GGUF. This AWQ quant
serves natively in vLLM with the Marlin kernel for fast inference.

## Quantization details

- Tool: [llm-compressor](https://github.com/vllm-project/llm-compressor) (vLLM project)
- Format: compressed-tensors (auto-detected by vLLM)
- Scheme: W4A16_ASYM (4-bit asymmetric weights, group_size 128, 16-bit activations)
- Algorithm: AWQ (Activation-aware Weight Quantization)
- Calibration: 128 French samples × 192 tokens from `wikimedia/wikipedia` (20231101.fr)
- `lm_head` left in fp16 (preserves output quality)
- Note: `v_proj → o_proj` smoothing is skipped on all 32 layers due to GQA shape mismatch (32 query heads / 8 KV heads). Standard AWQ behaviour for GQA models — quality impact is marginal since the `input_layernorm → q/k/v` smoothing remains active.

## Usage with vLLM

```bash
vllm serve KebalBaguette/aya-expanse-8b-abliterated-AWQ \
  --quantization compressed-tensors \
  --tool-call-parser hermes \
  --enable-auto-tool-choice \
  --max-model-len 8192
```

## License

**CC-BY-NC-4.0 — non-commercial use only.** Inherited from [Cohere Aya Expanse 8B](https://huggingface.co/CohereForAI/aya-expanse-8b) and [lenML's abliterated derivative](https://huggingface.co/lenML/aya-expanse-8b-abliterated). This abliterated derivative is for research, personal, and non-commercial evaluation use only. For commercial deployment, contact Cohere for licensing, or use an Apache-licensed alternative (e.g. Mistral 7B / Nemo).