Instructions to use EchoLabs33/zamba2-7b-instruct-hxq with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use EchoLabs33/zamba2-7b-instruct-hxq with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="EchoLabs33/zamba2-7b-instruct-hxq")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("EchoLabs33/zamba2-7b-instruct-hxq")
model = AutoModelForCausalLM.from_pretrained("EchoLabs33/zamba2-7b-instruct-hxq", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use EchoLabs33/zamba2-7b-instruct-hxq with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "EchoLabs33/zamba2-7b-instruct-hxq"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "EchoLabs33/zamba2-7b-instruct-hxq",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/EchoLabs33/zamba2-7b-instruct-hxq

SGLang

How to use EchoLabs33/zamba2-7b-instruct-hxq with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "EchoLabs33/zamba2-7b-instruct-hxq" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "EchoLabs33/zamba2-7b-instruct-hxq",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "EchoLabs33/zamba2-7b-instruct-hxq" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "EchoLabs33/zamba2-7b-instruct-hxq",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use EchoLabs33/zamba2-7b-instruct-hxq with Docker Model Runner:
```
docker model run hf.co/EchoLabs33/zamba2-7b-instruct-hxq
```

voidstream commited on Apr 14

Commit

dd754d3

verified ·

1 Parent(s): 08e5d0c

fix: model card metadata (language, results, branding, ratios)

Browse files

Files changed (1) hide show

README.md +75 -141

README.md CHANGED Viewed

@@ -1,4 +1,5 @@
 ---
 license: apache-2.0
 base_model: Zyphra/Zamba2-7B-instruct
 tags:
@@ -9,192 +10,125 @@ tags:
   - hxq
   - helix-substrate
   - vector-quantization
-  - 2d-vq
-  - 12bit-packing
 library_name: transformers
 pipeline_tag: text-generation
 model-index:
   - name: zamba2-7b-instruct-hxq
-    results:
-      - task:
-          type: text-generation
-        dataset:
-          name: WikiText-2
-          type: wikitext
-        metrics:
-          - name: Perplexity
-            type: perplexity
-            value: 3.8454
-      - task:
-          type: text-generation
-        dataset:
-          name: HellaSwag
-          type: hellaswag
-        metrics:
-          - name: acc_norm
-            type: acc_norm
-            value: 0.8106
-      - task:
-          type: text-generation
-        dataset:
-          name: ARC-Challenge
-          type: ai2_arc
-        metrics:
-          - name: acc_norm
-            type: acc_norm
-            value: 0.5811
-      - task:
-          type: text-generation
-        dataset:
-          name: ARC-Easy
-          type: ai2_arc
-        metrics:
-          - name: acc_norm
-            type: acc_norm
-            value: 0.8190
 ---
 # Zamba2-7B-Instruct-HXQ
-> **2D Vector Quantization (k=4096) with 12-bit packed indices. 81-layer hybrid Mamba2+Transformer. Faster than dense at 60% less VRAM.**
-Zamba2-7B-Instruct compressed with HXQ 2D VQ and 12-bit index packing (6 bits/weight). Beats bnb 4-bit NF4 on quality, speed, and requires no calibration data.
-## Benchmark: Native HelixLinear Inference on RTX 3090
-All numbers from a single session, same GPU, same WikiText-2 test set (50 chunks x 512 tokens).
-| Method | PPL | Throughput | VRAM (load) | VRAM (peak) | Bits/weight |
-|--------|-----|------------|-------------|-------------|-------------|
-| Dense BF16 | 4.82 | 1,446 tok/s | 14,032 MB | 14,686 MB | 16 |
-| bnb 8-bit | 4.85 | 515 tok/s | 7,831 MB | 8,635 MB | 8 |
-| bnb 4-bit NF4 | 5.07 | 1,579 tok/s | 5,129 MB | 5,904 MB | 4 |
-| **HXQ 12-bit packed** | **5.02** | **1,764 tok/s** | **5,657 MB** | **6,511 MB** | **6** |
-### Cross-GPU Confirmation (RTX 4090)
-| Method | Throughput | VRAM (load) | VRAM (peak) | GPU |
-|--------|------------|-------------|-------------|-----|
-| **HXQ 12-bit packed** | **1,827 tok/s** | **5,692 MB** | **6,886 MB** | RTX 4090 |
-Independently reproduced on 2026-04-03. Receipt: [`triton_gather_speed_4090_fast.json`](https://huggingface.co/EchoLabs33/zamba2-7b-instruct-hxq/blob/main/triton_gather_speed_4090_fast.json).
-**Profiled breakdown (512-token prefill):**
-- HelixLinear (gather + matmul): 43% of forward time
-- Mamba SSM + attention + norms: 57% of forward time
-- Per-layer overhead: 0.66ms avg (gather 0.21ms + cuBLAS 0.32ms)
-### Why HXQ wins
-- **Faster than dense** (1,764 vs 1,446 tok/s) -- fused Triton gather kernel eliminates memory bottleneck
-- **60% less VRAM** than dense (5.7 GB vs 14.0 GB)
-- **Better quality than bnb 4-bit** (5.02 vs 5.07 PPL) at comparable VRAM
-- **No calibration data required** -- unlike GPTQ, AWQ, or bnb, HXQ compresses from weights alone
-## Downstream Task Evaluation (lm-eval-harness v0.4.11)
-Same GPU (RTX 3090), same harness, same settings. All metrics are `acc_norm`.
-| Task | Dense BF16 | HXQ 2D VQ | Delta |
-|------|-----------|-----------|-------|
-| **HellaSwag** | 80.79% | 81.06% | +0.27% |
-| **ARC-Challenge** | 59.39% | 58.11% | -1.28% |
-| **ARC-Easy** | 83.21% | 81.90% | -1.31% |
-Compression preserves task performance within noise. HellaSwag (commonsense reasoning) is slightly *better* under compression. ARC drops are within 1.3%.
 ## Install and Run
 ```bash
-pip install "helix-substrate>=0.3.3"
 ```
 ```python
-import helix_substrate  # registers the HXQ quantizer
 from transformers import AutoModelForCausalLM, AutoTokenizer
-model = AutoModelForCausalLM.from_pretrained(
-    "EchoLabs33/zamba2-7b-instruct-hxq",
-    trust_remote_code=True,
-)
-tokenizer = AutoTokenizer.from_pretrained(
-    "EchoLabs33/zamba2-7b-instruct-hxq",
-    trust_remote_code=True,
-)
-inputs = tokenizer("The capital of France is", return_tensors="pt").to(model.device)
-outputs = model.generate(**inputs, max_new_tokens=32)
 print(tokenizer.decode(outputs[0], skip_special_tokens=True))
 ```
-**Important:** Requires `mamba-ssm` and `causal-conv1d` for full speed. Without these, Mamba2 layers fall back to a naive implementation (~9x slower). Install from source:
-```bash
-pip install causal-conv1d
-pip install "mamba-ssm @ git+https://github.com/state-spaces/mamba.git@v2.2.2" --no-build-isolation
-```
-Tested with `transformers>=4.46,<5.0` and `helix-substrate>=0.3.3`.
-## Codec Details
-| Property | Value |
-|----------|-------|
-| **Quantization** | 2D Vector Quantization |
-| **Codebook size (k)** | 4,096 |
-| **Vector dimension** | 2 (pairs of adjacent weights) |
-| **Bits per weight** | 6 effective (12-bit packed index / 2 weights) |
-| **Index packing** | 12-bit (3 bytes per 2 indices, lossless) |
-| **Compressed modules** | 213 HelixLinear layers |
-| **Exact tensors** | 573 (norms, embeddings, conv1d, A_log, D, dt_bias) |
-| **Sidecar corrections** | Yes (sparse outlier compensation) |
-| **Calibration data** | None required |
-| **HXQ storage** | 5.7 GB (12-bit packed) |
-| **Dense BF16** | 14.0 GB |
-### Why 2D VQ
-Standard scalar VQ assigns one codebook entry per weight. 2D VQ groups pairs of adjacent weights and clusters in R^2 -- the codebook captures the joint distribution. With k=4096 and 12-bit index packing, each pair of weights costs 12 bits = 6 bits per weight. This beats scalar k=256 (8 bits/weight) on both quality and compression.
-On Zamba2-7B, 2D VQ k=4096 achieves PPL 3.8454 vs the scalar baseline's higher PPL on this architecture. SSM and hybrid architectures benefit from 2D VQ more than pure Transformers.
-## Architecture
-Zamba2-7B-Instruct is a hybrid architecture:
-- **81 total layers** (Mamba2 + shared Transformer)
 - **hidden_size=3584**, **attention_hidden_size=7168**, **32 attention heads**
 - **mamba_d_state=64**, **mamba_d_conv=4**
 - **vocab_size=32000**
-213 linear layers compressed (Mamba projections, attention/MLP, LoRA adapters). Normalization layers, embeddings, conv1d, and Mamba-specific parameters stored at full precision.
-## Verification
-- **RTX 3090 benchmark:** [`zamba2_7b_comparison.json`](https://huggingface.co/EchoLabs33/zamba2-7b-instruct-hxq/blob/main/zamba2_7b_comparison.json) -- 1,764 tok/s, 2026-04-02
-- **RTX 4090 confirmation:** [`triton_gather_speed_4090_fast.json`](https://huggingface.co/EchoLabs33/zamba2-7b-instruct-hxq/blob/main/triton_gather_speed_4090_fast.json) -- 1,827 tok/s, 2026-04-03
-- **Conversion receipt:** [`conversion_receipt.json`](https://huggingface.co/EchoLabs33/zamba2-7b-instruct-hxq/blob/main/conversion_receipt.json) -- Gate 1 PASS
-- **helix-substrate version:** 0.3.3 (12-bit packing, fused Triton gather)
 ## Companion Models
-| Model | Architecture | PPL Delta |
-|-------|-------------|-----------|
-| [qwen2.5-14b-instruct-hxq](https://huggingface.co/EchoLabs33/qwen2.5-14b-instruct-hxq) | Transformer | pending |
-| [qwen2.5-7b-instruct-hxq](https://huggingface.co/EchoLabs33/qwen2.5-7b-instruct-hxq) | Transformer | +6.34% |
-| [qwen2.5-3b-instruct-hxq](https://huggingface.co/EchoLabs33/qwen2.5-3b-instruct-hxq) | Transformer | +0.69% |
-| [zamba2-2.7b-instruct-hxq](https://huggingface.co/EchoLabs33/zamba2-2.7b-instruct-hxq) | Hybrid | +6.59% |
-| [zamba2-1.2b-hxq](https://huggingface.co/EchoLabs33/zamba2-1.2b-hxq) | Hybrid | +2.90% |
-| [mamba2-1.3b-hxq](https://huggingface.co/EchoLabs33/mamba2-1.3b-hxq) | Pure SSM | +8.0% |
-| [tinyllama-1.1b-hxq](https://huggingface.co/EchoLabs33/tinyllama-1.1b-hxq) | Transformer | +0.78% |
 ## Citation
 ```bibtex
 @software{helix_substrate_2026,
-  title={Helix Substrate: Universal Weight Compression via HXQ},
   author={EchoLabs},
   year={2026},
-  url={https://pypi.org/project/helix-substrate/}
 }
 ```

 ---
+language: en
 license: apache-2.0
 base_model: Zyphra/Zamba2-7B-instruct
 tags:
   - hxq
   - helix-substrate
   - vector-quantization
+  - helixcode
 library_name: transformers
 pipeline_tag: text-generation
 model-index:
   - name: zamba2-7b-instruct-hxq
+    results: []
 ---
 # Zamba2-7B-Instruct-HXQ
+> **2.0x smaller from BF16. 81-layer hybrid Mamba2+Transformer. Largest HXQ hybrid model.**
+>
+> Zamba2-7B-Instruct compressed from 14.7 GB (BF16) to 7.5 GB. 213 linear layers compressed, 573 exact tensors preserved. No calibration data. Just `pip install` and `from_pretrained()`.
 ## Install and Run
 ```bash
+pip install "helix-substrate[hf]"
 ```
 ```python
+import helix_substrate  # registers the HXQ quantizer with HuggingFace
 from transformers import AutoModelForCausalLM, AutoTokenizer
+model = AutoModelForCausalLM.from_pretrained("EchoLabs33/zamba2-7b-instruct-hxq")
+tokenizer = AutoTokenizer.from_pretrained("EchoLabs33/zamba2-7b-instruct-hxq")
+inputs = tokenizer("Explain the theory of relativity in simple terms:", return_tensors="pt")
+outputs = model.generate(**inputs, max_new_tokens=128)
 print(tokenizer.decode(outputs[0], skip_special_tokens=True))
 ```
+That's it. `import helix_substrate` registers the quantizer. `from_pretrained()` handles the rest automatically.
+## Benchmark
+| | Dense (BF16) | HXQ |
+|---|---|---|
+| **Size** | 14.7 GB | **7.5 GB** |
+| **Perplexity** (WikiText-2) | pending | pending |
+| **Compression ratio** | — | **2.0x** |
+| **Compressed modules** | — | 213 HelixLinear layers |
+| **Architecture** | Zamba2 (81 layers, Mamba2 + shared Transformer) | unchanged |
+## Good to Know
+- **GPU recommended** — 7.5 GB requires 10+ GB VRAM. Use `device_map="auto"` for multi-GPU.
+- **Not fine-tunable** — compressed weights are read-only (`is_trainable = False`).
+- **Requires `helix-substrate`** — the quantizer is not built into transformers. You need `pip install "helix-substrate[hf]"`.
+- **Requires `transformers >= 4.45`** — for Zamba2 architecture support.
+- **`mamba-ssm` recommended** — without it, falls back to a slower sequential code path.
+- **PPL pending** — requires cloud GPU eval (model doesn't fit on 4 GB T2000).
+## What is HelixCode?
+HelixCode is a universal weight compression codec based on vector quantization:
+- Each weight matrix is replaced by a **256-entry codebook** (float32) + **uint8 index matrix** + optional **sidecar corrections** for outlier values
+- The compressed form *is* the executable — `HelixLinear` performs `codebook[indices] @ x` directly, no decompression step
+- Works on any `nn.Linear` regardless of architecture (Transformer, Mamba, MLP, CNN)
+- **No calibration data required** — unlike GPTQ/AWQ, codebooks are fit from the weights alone
+## How It Works
+1. `import helix_substrate` registers the `hxq` quantizer with HuggingFace
+2. `from_pretrained()` reads `quantization_config.quant_method = "hxq"` from `config.json`
+3. The quantizer replaces 213 `nn.Linear` modules with `HelixLinear` shells before weight loading
+4. Safetensors populates the codebook, indices, and sidecar buffers directly
+5. The model runs in compressed form — no decompression needed
+## Architecture Details
+Zamba2-7B-Instruct is a hybrid architecture with:
+- **81 total layers** (Mamba2 + shared Transformer hybrid)
 - **hidden_size=3584**, **attention_hidden_size=7168**, **32 attention heads**
 - **mamba_d_state=64**, **mamba_d_conv=4**
 - **vocab_size=32000**
+213 linear layers compressed (162 Mamba projections, 38 attention/MLP, 26 LoRA adapters). Normalization layers, embeddings, conv1d, and Mamba-specific parameters (A_log, D, dt_bias) are stored at full precision.
+## Compression Receipt
+```
+Compressed modules:  213
+Exact tensors:       573  (norms, embeddings, conv1d, A_log, D, dt_bias, LoRA)
+Skip tensors:        243  (from original model)
+Total keys:          1425
+Dense size:          14.7 GB (BF16)
+Compressed size:     7.5 GB
+Compression ratio:   2.0x
+PPL delta:           pending (cloud GPU eval)
+Gate 1:              PASS (structural validation + SHA256)
+```
 ## Companion Models
+Same codec, same `pip install`, multiple architectures:
+| Model | Architecture | Ratio | PPL Delta |
+|-------|-------------|-------|-----------|
+| [qwen2.5-14b-instruct-helix](https://huggingface.co/EchoLabs33/qwen2.5-14b-instruct-helix) | Transformer | 3.4x | pending |
+| [qwen2.5-7b-instruct-helix](https://huggingface.co/EchoLabs33/qwen2.5-7b-instruct-helix) | Transformer | 2.2x | +6.34% |
+| [qwen2.5-3b-instruct-helix](https://huggingface.co/EchoLabs33/qwen2.5-3b-instruct-helix) | Transformer | 1.6x | +0.69% |
+| [qwen2.5-coder-3b-helix](https://huggingface.co/EchoLabs33/qwen2.5-coder-3b-helix) | Transformer (code) | 1.6x | +1.92% |
+| [qwen2.5-coder-1.5b-instruct-helix](https://huggingface.co/EchoLabs33/qwen2.5-coder-1.5b-instruct-helix) | Transformer (code) | 2.4x | +1.63% |
+| [tinyllama-1.1b-helix](https://huggingface.co/EchoLabs33/tinyllama-1.1b-helix) | Transformer | 4.0x | +0.78% |
+| [zamba2-2.7b-instruct-helix](https://huggingface.co/EchoLabs33/zamba2-2.7b-instruct-helix) | Hybrid (Mamba2+Transformer) | 1.8x | +6.59% |
+| [zamba2-1.2b-helix](https://huggingface.co/EchoLabs33/zamba2-1.2b-helix) | Hybrid (Mamba2+Transformer) | 1.7x | +2.90% |
+| [mamba2-1.3b-helix](https://huggingface.co/EchoLabs33/mamba2-1.3b-helix) | Pure SSM (Mamba2) | 2.1x | +8.0% |
+| [mamba-130m-helix](https://huggingface.co/EchoLabs33/mamba-130m-helix) | Pure SSM | 3.8x | +18.4% |
 ## Citation
 ```bibtex
 @software{helix_substrate_2026,
+  title={Helix Substrate: Universal Weight Compression via HelixCode},
   author={EchoLabs},
   year={2026},
+  url={https://github.com/echo313unfolding/helix-substrate}
 }
 ```