README.md

---
license: apache-2.0
base_model: Zyphra/Zamba2-7B-instruct
tags:
  - zamba2
  - mamba
  - hybrid
  - compressed
  - hxq
  - helix-substrate
  - vector-quantization
  - 2d-vq
  - 12bit-packing
library_name: transformers
pipeline_tag: text-generation
model-index:
  - name: zamba2-7b-instruct-hxq
    results:
      - task:
          type: text-generation
        dataset:
          name: WikiText-2
          type: wikitext
        metrics:
          - name: Perplexity
            type: perplexity
            value: 3.8454
      - task:
          type: text-generation
        dataset:
          name: HellaSwag
          type: hellaswag
        metrics:
          - name: acc_norm
            type: acc_norm
            value: 0.8106
      - task:
          type: text-generation
        dataset:
          name: ARC-Challenge
          type: ai2_arc
        metrics:
          - name: acc_norm
            type: acc_norm
            value: 0.5811
      - task:
          type: text-generation
        dataset:
          name: ARC-Easy
          type: ai2_arc
        metrics:
          - name: acc_norm
            type: acc_norm
            value: 0.8190
---

# Zamba2-7B-Instruct-HXQ

> **2D Vector Quantization (k=4096) with 12-bit packed indices. 81-layer hybrid Mamba2+Transformer. Faster than dense at 60% less VRAM.**

Zamba2-7B-Instruct compressed with HXQ 2D VQ and 12-bit index packing (6 bits/weight). Beats bnb 4-bit NF4 on quality, speed, and requires no calibration data.

## Benchmark: Native HelixLinear Inference on RTX 3090

All numbers from a single session, same GPU, same WikiText-2 test set (50 chunks x 512 tokens).

| Method | PPL | Throughput | VRAM (load) | VRAM (peak) | Bits/weight |
|--------|-----|------------|-------------|-------------|-------------|
| Dense BF16 | 4.82 | 1,446 tok/s | 14,032 MB | 14,686 MB | 16 |
| bnb 8-bit | 4.85 | 515 tok/s | 7,831 MB | 8,635 MB | 8 |
| bnb 4-bit NF4 | 5.07 | 1,579 tok/s | 5,129 MB | 5,904 MB | 4 |
| **HXQ 12-bit packed** | **5.02** | **1,764 tok/s** | **5,657 MB** | **6,511 MB** | **6** |

### Why HXQ wins

- **Faster than dense** (1,764 vs 1,446 tok/s) -- fused Triton gather kernel eliminates memory bottleneck
- **60% less VRAM** than dense (5.7 GB vs 14.0 GB)
- **Better quality than bnb 4-bit** (5.02 vs 5.07 PPL) at comparable VRAM
- **No calibration data required** -- unlike GPTQ, AWQ, or bnb, HXQ compresses from weights alone

## Downstream Task Evaluation (lm-eval-harness v0.4.11)

Same GPU (RTX 3090), same harness, same settings. All metrics are `acc_norm`.

| Task | Dense BF16 | HXQ 2D VQ | Delta |
|------|-----------|-----------|-------|
| **HellaSwag** | 80.79% | 81.06% | +0.27% |
| **ARC-Challenge** | 59.39% | 58.11% | -1.28% |
| **ARC-Easy** | 83.21% | 81.90% | -1.31% |

Compression preserves task performance within noise. HellaSwag (commonsense reasoning) is slightly *better* under compression. ARC drops are within 1.3%.

## Install and Run

```bash
pip install "helix-substrate>=0.3.3"
```

```python
import helix_substrate  # registers the HXQ quantizer
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "EchoLabs33/zamba2-7b-instruct-hxq",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
    "EchoLabs33/zamba2-7b-instruct-hxq",
    trust_remote_code=True,
)

inputs = tokenizer("The capital of France is", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=32)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

**Note:** Tested with `transformers>=4.49`. `mamba-ssm` and `causal-conv1d` recommended for fast Mamba2 inference.

## Codec Details

| Property | Value |
|----------|-------|
| **Quantization** | 2D Vector Quantization |
| **Codebook size (k)** | 4,096 |
| **Vector dimension** | 2 (pairs of adjacent weights) |
| **Bits per weight** | 6 effective (12-bit packed index / 2 weights) |
| **Index packing** | 12-bit (3 bytes per 2 indices, lossless) |
| **Compressed modules** | 213 HelixLinear layers |
| **Exact tensors** | 573 (norms, embeddings, conv1d, A_log, D, dt_bias) |
| **Sidecar corrections** | Yes (sparse outlier compensation) |
| **Calibration data** | None required |
| **HXQ storage** | 5.7 GB (12-bit packed) |
| **Dense BF16** | 14.0 GB |

### Why 2D VQ

Standard scalar VQ assigns one codebook entry per weight. 2D VQ groups pairs of adjacent weights and clusters in R^2 -- the codebook captures the joint distribution. With k=4096 and 12-bit index packing, each pair of weights costs 12 bits = 6 bits per weight. This beats scalar k=256 (8 bits/weight) on both quality and compression.

On Zamba2-7B, 2D VQ k=4096 achieves PPL 3.8454 vs the scalar baseline's higher PPL on this architecture. SSM and hybrid architectures benefit from 2D VQ more than pure Transformers.

## Architecture

Zamba2-7B-Instruct is a hybrid architecture:
- **81 total layers** (Mamba2 + shared Transformer)
- **hidden_size=3584**, **attention_hidden_size=7168**, **32 attention heads**
- **mamba_d_state=64**, **mamba_d_conv=4**
- **vocab_size=32000**

213 linear layers compressed (Mamba projections, attention/MLP, LoRA adapters). Normalization layers, embeddings, conv1d, and Mamba-specific parameters stored at full precision.

## Verification

- **Benchmark receipt:** [`zamba2_7b_comparison.json`](https://huggingface.co/EchoLabs33/zamba2-7b-instruct-hxq/blob/main/zamba2_7b_comparison.json) -- RTX 3090, 2026-04-02
- **Conversion receipt:** [`conversion_receipt.json`](https://huggingface.co/EchoLabs33/zamba2-7b-instruct-hxq/blob/main/conversion_receipt.json) -- Gate 1 PASS
- **helix-substrate version:** 0.3.3 (12-bit packing, fused Triton gather)

## Companion Models

| Model | Architecture | PPL Delta |
|-------|-------------|-----------|
| [qwen2.5-14b-instruct-helix](https://huggingface.co/EchoLabs33/qwen2.5-14b-instruct-helix) | Transformer | pending |
| [qwen2.5-7b-instruct-helix](https://huggingface.co/EchoLabs33/qwen2.5-7b-instruct-helix) | Transformer | +6.34% |
| [qwen2.5-3b-instruct-helix](https://huggingface.co/EchoLabs33/qwen2.5-3b-instruct-helix) | Transformer | +0.69% |
| [zamba2-2.7b-instruct-helix](https://huggingface.co/EchoLabs33/zamba2-2.7b-instruct-helix) | Hybrid | +6.59% |
| [zamba2-1.2b-helix](https://huggingface.co/EchoLabs33/zamba2-1.2b-helix) | Hybrid | +2.90% |
| [mamba2-1.3b-helix](https://huggingface.co/EchoLabs33/mamba2-1.3b-helix) | Pure SSM | +8.0% |
| [tinyllama-1.1b-helix](https://huggingface.co/EchoLabs33/tinyllama-1.1b-helix) | Transformer | +0.78% |

## Citation

```bibtex
@software{helix_substrate_2026,
  title={Helix Substrate: Universal Weight Compression via HXQ},
  author={EchoLabs},
  year={2026},
  url={https://pypi.org/project/helix-substrate/}
}
```

## License

Apache 2.0 (inherited from [Zyphra/Zamba2-7B-instruct](https://huggingface.co/Zyphra/Zamba2-7B-instruct)).