---
license: other
license_name: meralion-public-license
license_link: https://huggingface.co/MERaLiON/MERaLiON-3-10B/blob/main/LICENSE
base_model: aisingapore/MERaLiON-AudioLLM-Whisper-SEA-LION-V3-3B
tags:
  - rotorquant
  - kv-cache-quantization
  - meralion
  - whisper
  - audio
  - speech-recognition
  - sea-lion
  - quantized
library_name: transformers
pipeline_tag: automatic-speech-recognition
---

> [!NOTE]
> **Status (2026-07-07): no weights published yet.** This repository currently contains only the model card — it marks a planned variant that has not been released. Follow the repo to be notified when files land.
<!-- status-note -->

> [!TIP]
> **KV-cache quantization without any fork (recommended, 2026):** upstream
> llama.cpp/Ollama now cover this natively — use `-ctk q8_0 -ctv q8_0`
> (~half KV memory, negligible quality loss: perplexity +0.002–0.05) or
> `-ctk q4_0 -ctv q4_0` (~quarter memory, ≈7.6% perplexity increase). In
> Ollama: `OLLAMA_KV_CACHE_TYPE=q8_0` with `OLLAMA_FLASH_ATTENTION=1`. Keep
> K and V types symmetric to stay on the fast fused Flash-Attention path.
> Since April 2026, mainline llama.cpp also applies Hadamard rotation to
> KV activations ([PR #21038](https://github.com/ggml-org/llama.cpp/pull/21038)),
> which greatly improves low-bit KV quality (opt-out:
> `LLAMA_ATTN_ROT_DISABLE=1`).
>
> The RotorQuant/TurboQuant fork flow below is **experimental/legacy**: the
> TurboQuant llama.cpp PR was closed without merging (June 2026) and the fork
> is unmaintained relative to mainline. It is NOT required to use this model.
<!-- kv-upstream-note -->

# MERaLiON-2-3B-RotorQuant

**RotorQuant KV cache compression** for [aisingapore/MERaLiON-AudioLLM-Whisper-SEA-LION-V3-3B](https://huggingface.co/aisingapore/MERaLiON-AudioLLM-Whisper-SEA-LION-V3-3B).

This is a **documentation repository** that explains how to combine MERaLiON-2-3B's weights with RotorQuant inference-time KV cache compression. No weights are stored here — use the base model directly and apply RotorQuant via the Python package or llama.cpp fork.

## Hardware compatibility

| Device | VRAM / RAM | Recommendation |
| --- | --- | --- |
| Any host that runs the base model | baseline + runtime savings | RotorQuant/TurboQuant is a KV-cache runtime modifier; pair with any weight variant |

## What is this?

KV cache compression reduces the memory used by the attention cache during inference. Unlike weight quantization (which is baked into the GGUF/MLX file), KV cache compression is applied at runtime — so the same base weights can be used with or without compression.

| Technique | Where it's applied | Savings |
|-----------|-------------------|---------|
| **RotorQuant KV cache** | At inference time | Reduces attention memory (critical for long context) |

Both can be combined for maximum efficiency.

## Quickstart

### Option A — Python / transformers

Install the `rotorquant` package:

```bash
pip install rotorquant
```

Then use it with the base model:

```python
import torch
from transformers import AutoModelForSpeechSeq2Seq, AutoTokenizer
from rotorquant import IsoQuantCache

tokenizer = AutoTokenizer.from_pretrained("aisingapore/MERaLiON-AudioLLM-Whisper-SEA-LION-V3-3B", trust_remote_code=True)
model = AutoModelForSpeechSeq2Seq.from_pretrained(
    "aisingapore/MERaLiON-AudioLLM-Whisper-SEA-LION-V3-3B",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

# Apply RotorQuant to the KV cache
cache = IsoQuantCache(bits=4)  # or bits=2 for more aggressive compression

inputs = tokenizer("Hello, how are you?", return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=128,
    past_key_values=cache,
    use_cache=True,
)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
```


## Model Specifications

| Property | Value |
|----------|-------|
| Base Model | [aisingapore/MERaLiON-AudioLLM-Whisper-SEA-LION-V3-3B](https://huggingface.co/aisingapore/MERaLiON-AudioLLM-Whisper-SEA-LION-V3-3B) |
| Architecture | Whisper-large-v3 + Gemma-2-2B-IT decoder |
| Parameters | ~3B (audio encoder + text decoder) |
| Context Length | 8K |
| BF16 Size | ~6 GB |
| Modalities | Audio + Text |
| License | other |

## What is RotorQuant?

[RotorQuant](https://github.com/scrya-com/rotorquant) is a KV cache compression method based on Clifford algebra (Cl(3,0)) rotors — a faster, more parameter-efficient alternative to Google's TurboQuant. Uses lightweight block-diagonal rotations (independent 2D/4D rotations per pair/quartet) achieving O(d) complexity instead of O(d log d), fully parallelisable with no inter-element dependencies.

**Benchmarks** (from the RotorQuant repository, Llama 3.1 8B on RTX 5090 — results vary by model and hardware):

- Prefill: 3,822 tok/s (vs TurboQuant 722 tok/s)
- Decode: 119 tok/s (vs TurboQuant 93 tok/s)
- Perplexity: 6.91 (vs TurboQuant 7.07)
- Parameters: 4 per rotor (vs TurboQuant 16,384)

> Benchmarks are from the RotorQuant repository using Llama 3.1 8B. Performance on MERaLiON-2-3B will differ. Please open a discussion if you have independent results.

## Current Ecosystem Support

| Runtime | RotorQuant Support | Notes |
|---------|----------------------|-------|
| Python transformers + `rotorquant` | ✅ Full | Drop-in cache class |
| llama.cpp upstream | ❌ Not merged | Use fork below |
| llama-cpp-turboquant fork | ✅ `planar3`, `iso3` | [GitHub](https://github.com/johndpope/llama-cpp-turboquant/tree/feature/planarquant-kv-cache) |
| LM Studio | ❌ [Requested](https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/1719) | Use `q8_0` as alternative |
| Ollama | ❌ Not supported | Use `OLLAMA_KV_CACHE_TYPE=q8_0` |
| vLLM | ❌ Not supported | — |
| koboldcpp | ❌ Not supported | — |

## Pre-quantized weight variants

If you want combined weight + KV cache compression, majentik hosts pre-quantized versions:

- [MLX (Apple Silicon)](https://huggingface.co/majentik?search=MERaLiON-2-3B+MLX)
- [GGUF (llama.cpp / Ollama / LM Studio)](https://huggingface.co/majentik?search=MERaLiON-2-3B+GGUF)

## See Also

- [RotorQuant GitHub](https://github.com/scrya-com/rotorquant)
- [TurboQuant paper (arXiv 2504.19874)](https://arxiv.org/abs/2504.19874)
- [llama-cpp-turboquant fork](https://github.com/johndpope/llama-cpp-turboquant/tree/feature/planarquant-kv-cache)
- [Base model: aisingapore/MERaLiON-AudioLLM-Whisper-SEA-LION-V3-3B](https://huggingface.co/aisingapore/MERaLiON-AudioLLM-Whisper-SEA-LION-V3-3B)

## Variants in this family

(Showing 8 sibling variants under `majentik/meralion2-3b-*`. The current variant — `RotorQuant` — is **bolded**.)

| Variant | Runtime | Approx size | Use case |
|---|---|---|---|
| **RotorQuant** | runtime modifier | n/a | KV-cache root (weight-agnostic) |
| [RotorQuant-MLX-2bit](https://huggingface.co/majentik/meralion2-3b-rotorquant-mlx-2bit) | mlx-lm | ~983 MB | Apple Silicon, smallest |
| [RotorQuant-MLX-4bit](https://huggingface.co/majentik/meralion2-3b-rotorquant-mlx-4bit) | mlx-lm | ~1.9 GB | Apple Silicon balanced |
| [RotorQuant-MLX-8bit](https://huggingface.co/majentik/meralion2-3b-rotorquant-mlx-8bit) | mlx-lm | ~3.5 GB | Apple Silicon reference |
| [TurboQuant](https://huggingface.co/majentik/meralion2-3b-turboquant) | runtime modifier | n/a | KV-cache root (weight-agnostic) |
| [TurboQuant-MLX-2bit](https://huggingface.co/majentik/meralion2-3b-turboquant-mlx-2bit) | mlx-lm | ~983 MB | Apple Silicon, smallest |
| [TurboQuant-MLX-4bit](https://huggingface.co/majentik/meralion2-3b-turboquant-mlx-4bit) | mlx-lm | ~1.9 GB | Apple Silicon balanced |
| [TurboQuant-MLX-8bit](https://huggingface.co/majentik/meralion2-3b-turboquant-mlx-8bit) | mlx-lm | ~3.5 GB | Apple Silicon reference |