Instructions to use EchoLabs33/zamba2-7b-instruct-hxq with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use EchoLabs33/zamba2-7b-instruct-hxq with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="EchoLabs33/zamba2-7b-instruct-hxq")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("EchoLabs33/zamba2-7b-instruct-hxq")
model = AutoModelForCausalLM.from_pretrained("EchoLabs33/zamba2-7b-instruct-hxq", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use EchoLabs33/zamba2-7b-instruct-hxq with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "EchoLabs33/zamba2-7b-instruct-hxq"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "EchoLabs33/zamba2-7b-instruct-hxq",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/EchoLabs33/zamba2-7b-instruct-hxq

SGLang

How to use EchoLabs33/zamba2-7b-instruct-hxq with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "EchoLabs33/zamba2-7b-instruct-hxq" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "EchoLabs33/zamba2-7b-instruct-hxq",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "EchoLabs33/zamba2-7b-instruct-hxq" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "EchoLabs33/zamba2-7b-instruct-hxq",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use EchoLabs33/zamba2-7b-instruct-hxq with Docker Model Runner:
```
docker model run hf.co/EchoLabs33/zamba2-7b-instruct-hxq
```

voidstream commited on Apr 2

Commit

f510725

verified ·

1 Parent(s): a6390f5

Upload README.md with huggingface_hub

Browse files

Files changed (1) hide show

README.md +24 -23

README.md CHANGED Viewed

@@ -10,6 +10,7 @@ tags:
   - helix-substrate
   - vector-quantization
   - 2d-vq
 library_name: transformers
 pipeline_tag: text-generation
 model-index:
@@ -28,33 +29,32 @@ model-index:
 # Zamba2-7B-Instruct-HXQ
-> **2D Vector Quantization (k=4096). 81-layer hybrid Mamba2+Transformer. Benchmarked against bitsandbytes on RTX 4090.**
-Zamba2-7B-Instruct compressed with HXQ 2D VQ: each pair of adjacent weights shares one codebook index. Beats bnb 4-bit NF4 on quality (+3.33% vs +4.57% PPL delta) at full inference speed.
-## Benchmark: Zamba2-7B-Instruct on RTX 4090
-All numbers from a single session. WikiText-2 test set, MAX_LENGTH=2048, STRIDE=512.
-| Method | PPL | PPL Delta | VRAM (load) | VRAM (peak) | Throughput |
-|--------|-----|-----------|-------------|-------------|------------|
-| Dense BF16 | 3.7216 | -- | 14.0 GB | 16.2 GB | 4,385 tok/s |
-| bitsandbytes 8-bit | 3.7335 | +0.32% | 7.8 GB | 10.0 GB | 2,165 tok/s |
-| bitsandbytes 4-bit NF4 | 3.8916 | +4.57% | 5.1 GB | 7.2 GB | 3,468 tok/s |
-| **HXQ 2D VQ k=4096** | **3.8454** | **+3.33%** | **14.0 GB** | **16.2 GB** | **4,385 tok/s** |
-**VRAM column shows materialized form** -- weights are expanded to BF16 before GPU inference. Native compressed inference via HelixLinear (fused Triton kernel, 7.4 GB VRAM) ships in `helix-substrate >= 0.3.1` but is not reflected in this table.
-### Where each method wins
-- **bnb 8-bit**: Best quality (+0.32%), lowest speed (2x slower), moderate VRAM
-- **bnb 4-bit NF4**: Smallest VRAM (5.1 GB), worst quality (+4.57%)
-- **HXQ 2D VQ**: Better quality than bnb-4bit, zero speed penalty, 4x disk savings
 ## Install and Run
 ```bash
-pip install "helix-substrate>=0.3.1"
 ```
 ```python
@@ -75,7 +75,7 @@ outputs = model.generate(**inputs, max_new_tokens=32)
 print(tokenizer.decode(outputs[0], skip_special_tokens=True))
 ```
-**Note:** Requires `transformers==4.49.0` for Zamba2 support. Later versions may have compatibility issues. `mamba-ssm` and `causal-conv1d` recommended for fast Mamba2 inference.
 ## Codec Details
@@ -84,17 +84,18 @@ print(tokenizer.decode(outputs[0], skip_special_tokens=True))
 | **Quantization** | 2D Vector Quantization |
 | **Codebook size (k)** | 4,096 |
 | **Vector dimension** | 2 (pairs of adjacent weights) |
-| **Bits per weight** | 8 effective (uint16 index / 2 weights) |
 | **Compressed modules** | 213 HelixLinear layers |
 | **Exact tensors** | 573 (norms, embeddings, conv1d, A_log, D, dt_bias) |
 | **Sidecar corrections** | Yes (sparse outlier compensation) |
 | **Calibration data** | None required |
-| **HXQ storage** | 7.0 GB (native format) |
 | **Dense BF16** | 14.0 GB |
 ### Why 2D VQ
-Standard scalar VQ assigns one codebook entry per weight. 2D VQ groups pairs of adjacent weights and clusters in R^2 -- the codebook captures the joint distribution. With k=4096, this matches or beats scalar k=256 on quality while using the same bits per weight.
 On Zamba2-7B, 2D VQ k=4096 achieves PPL 3.8454 vs the scalar baseline's higher PPL on this architecture. SSM and hybrid architectures benefit from 2D VQ more than pure Transformers.
@@ -110,9 +111,9 @@ Zamba2-7B-Instruct is a hybrid architecture:
 ## Verification
-- **Benchmark receipt:** [`zamba2_7b_comparison.json`](https://huggingface.co/EchoLabs33/zamba2-7b-instruct-hxq/blob/main/zamba2_7b_comparison.json) -- RTX 4090, 2026-04-01
 - **Conversion receipt:** [`conversion_receipt.json`](https://huggingface.co/EchoLabs33/zamba2-7b-instruct-hxq/blob/main/conversion_receipt.json) -- Gate 1 PASS
-- **helix-substrate version:** 0.3.1 (2D VQ support)
 ## Companion Models

   - helix-substrate
   - vector-quantization
   - 2d-vq
+  - 12bit-packing
 library_name: transformers
 pipeline_tag: text-generation
 model-index:
 # Zamba2-7B-Instruct-HXQ
+> **2D Vector Quantization (k=4096) with 12-bit packed indices. 81-layer hybrid Mamba2+Transformer. Faster than dense at 60% less VRAM.**
+Zamba2-7B-Instruct compressed with HXQ 2D VQ and 12-bit index packing (6 bits/weight). Beats bnb 4-bit NF4 on quality, speed, and requires no calibration data.
+## Benchmark: Native HelixLinear Inference on RTX 3090
+All numbers from a single session, same GPU, same WikiText-2 test set (50 chunks x 512 tokens).
+| Method | PPL | Throughput | VRAM (load) | VRAM (peak) | Bits/weight |
+|--------|-----|------------|-------------|-------------|-------------|
+| Dense BF16 | 4.82 | 1,446 tok/s | 14,032 MB | 14,686 MB | 16 |
+| bnb 8-bit | 4.85 | 515 tok/s | 7,831 MB | 8,635 MB | 8 |
+| bnb 4-bit NF4 | 5.07 | 1,579 tok/s | 5,129 MB | 5,904 MB | 4 |
+| **HXQ 12-bit packed** | **5.02** | **1,764 tok/s** | **5,657 MB** | **6,511 MB** | **6** |
+### Why HXQ wins
+- **Faster than dense** (1,764 vs 1,446 tok/s) -- fused Triton gather kernel eliminates memory bottleneck
+- **60% less VRAM** than dense (5.7 GB vs 14.0 GB)
+- **Better quality than bnb 4-bit** (5.02 vs 5.07 PPL) at comparable VRAM
+- **No calibration data required** -- unlike GPTQ, AWQ, or bnb, HXQ compresses from weights alone
 ## Install and Run
 ```bash
+pip install "helix-substrate>=0.3.3"
 ```
 ```python
 print(tokenizer.decode(outputs[0], skip_special_tokens=True))
 ```
+**Note:** Tested with `transformers>=4.49`. `mamba-ssm` and `causal-conv1d` recommended for fast Mamba2 inference.
 ## Codec Details
 | **Quantization** | 2D Vector Quantization |
 | **Codebook size (k)** | 4,096 |
 | **Vector dimension** | 2 (pairs of adjacent weights) |
+| **Bits per weight** | 6 effective (12-bit packed index / 2 weights) |
+| **Index packing** | 12-bit (3 bytes per 2 indices, lossless) |
 | **Compressed modules** | 213 HelixLinear layers |
 | **Exact tensors** | 573 (norms, embeddings, conv1d, A_log, D, dt_bias) |
 | **Sidecar corrections** | Yes (sparse outlier compensation) |
 | **Calibration data** | None required |
+| **HXQ storage** | 5.7 GB (12-bit packed) |
 | **Dense BF16** | 14.0 GB |
 ### Why 2D VQ
+Standard scalar VQ assigns one codebook entry per weight. 2D VQ groups pairs of adjacent weights and clusters in R^2 -- the codebook captures the joint distribution. With k=4096 and 12-bit index packing, each pair of weights costs 12 bits = 6 bits per weight. This beats scalar k=256 (8 bits/weight) on both quality and compression.
 On Zamba2-7B, 2D VQ k=4096 achieves PPL 3.8454 vs the scalar baseline's higher PPL on this architecture. SSM and hybrid architectures benefit from 2D VQ more than pure Transformers.
 ## Verification
+- **Benchmark receipt:** [`zamba2_7b_comparison.json`](https://huggingface.co/EchoLabs33/zamba2-7b-instruct-hxq/blob/main/zamba2_7b_comparison.json) -- RTX 3090, 2026-04-02
 - **Conversion receipt:** [`conversion_receipt.json`](https://huggingface.co/EchoLabs33/zamba2-7b-instruct-hxq/blob/main/conversion_receipt.json) -- Gate 1 PASS
+- **helix-substrate version:** 0.3.3 (12-bit packing, fused Triton gather)
 ## Companion Models