README.md

---
license: apache-2.0
base_model: Qwen/Qwen3.6-35B-A3B
base_model_relation: quantized
language:
  - en
pipeline_tag: text-generation
library_name: spiral
tags:
  - spiral
  - quantization
  - calibration-free
  - gguf
  - llama-cpp
  - qwen
  - qwen3.6
  - moe
  - apple-silicon
  - cuda
  - metal
---

# Qwen3.6-35B-A3B Spiral

**Release v0.3.0 — Spiral_4_5 scheme**

**Calibration-free 5 bpw compression of Qwen3.6-35B-A3B.** Competitive with unsloth Q4_K_M on HumanEval (91.5% vs 93.3% base pass@1) at similar artifact size — **without calibration data, without fine-tuning, without representative samples**. Production-stable on Apple Silicon (Metal) and NVIDIA H100 (CUDA). 91 tok/s decode on H100.

This is the v0.3.0 release of Spiral compression applied to [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) — the hybrid MoE + DeltaNet architecture from Qwen.

**v0.3.0 supersedes v0.2.0** (the prior INT3 release). The new release is a different point in the quality/size Pareto: cleaner inference, parity with Q4-class quants on coding benchmarks, and cross-platform shipping.

![Qwen3.6-35B-A3B Spiral v0.3.0 release results](spiral_release_slide.svg)

---

## What's new vs v0.2.0

| Aspect | v0.2.0 | v0.3.0 (this release) |
|---|---|---|
| Average compression | 3.5 bpw | ~5.0 bpw |
| NLL vs bf16 | +0.113 nats | ~0 nats |
| HumanEval pass@1 | not measured | 0.915 base / 0.866 plus |
| Multi-turn on Mac | broken | works |
| CUDA support | not shipped | shipped |
| Calibration data | not required | not required |
| Production status | research preview | shippable |

---

## Quality

### HumanEval pass@1

Measured on H100 with llama-server + evalplus, greedy decoding (`temp=0`), thinking mode disabled:

| Method | bpw | Calibration data | HumanEval base | HumanEval+ |
|---|---|---|---|---|
| **Spiral_4_5 (this model)** | **~5.0** | **None** | **0.915** | **0.866** |
| unsloth Q4_K_M (reference) | 4.6 | Yes (imatrix) | 0.933* | 0.902* |

\* Q4_K_M reference numbers from [unsloth's community leaderboard submission](https://github.com/evalplus/evalplus/issues/299).

Spiral_4_5 is **1.8 percentage points behind Q4_K_M on base HumanEval, 3.6 points behind on HumanEval+** — and is calibration-free. K-quants require representative data to construct importance matrices for outlier handling; Spiral derives its compression analytically. The model's coding capability is preserved within striking distance of one of the strongest community quantization schemes, without the calibration overhead.

### Perplexity (wikitext-2-raw-v1)

| Method | bpw | NLL (nats/token) | Gap vs bf16 |
|---|---|---|---|
| bf16 (reference) | 16.0 | 1.8968 | — |
| unsloth Q8_0 | 8.5 | 1.8962 | −0.0006 |
| unsloth Q4_K_XL | 5.2 | 1.8976 | +0.0008 |
| **Spiral_4_5** | **~5.0** | **≈1.8929** | **≈−0.004** |

Spiral_4_5 is competitive with Q4_K_XL on perplexity.

---

## Cross-platform validation

The same artifact runs on both Metal (Apple Silicon) and CUDA (NVIDIA H100). Output equivalence has been validated by side-by-side comparison on identical greedy prompts:

- Short prompts: byte-identical outputs across both platforms
- Long generations: semantically equivalent reasoning, with minor token-level drift from fp16 ordering differences in the deep stack

HumanEval is measured on H100 CUDA. Mac inference produces equivalent quality (validated by cross-platform diff) but has not been benchmarked separately on HumanEval; expect within 1-2 percentage points of the H100 number.

---

## Performance

### Apple M2 Max

| Configuration | Decode | Prefill |
|---|---|---|
| f16 KV + flash attention | ~30 tok/s | ~80 tok/s |

### NVIDIA H100 80GB

| Configuration | Decode | Prefill |
|---|---|---|
| f16 KV + flash attention + CUDA graphs | ~91 tok/s | 100-220 tok/s |

---

## Memory footprint

Qwen3.6-35B-A3B weights + 8K context KV cache:

| Method | Total |
|---|---|
| **Spiral_4_5** | **~20.2 GB** |
| Q4_K_XL | ~21.2 GB |
| Q8_0 | ~37.2 GB |
| bf16 | ~70.2 GB |

Fits comfortably in 24 GB of VRAM / unified memory for the 8K-context shipping config.

---

## Files

| File | Size |
|---|---|
| `Qwen3.6-35B-A3B-Spiral_4_5.gguf` | ~20 GB |
| `Qwen3.6-35B-A3B-Spiral_4_5.spiralcb` | 28 MB |

Both files are required. The `.spiralcb` sidecar is loaded by the Spiral runtime alongside the GGUF.

---

## Quick start

Spiral ships with three wrappers — `spiral-download`, `spiral-chat`, `spiral-serve` — that handle the model fetch and inference flags.

### Install

```bash
brew install reinforceai/spiral/spiral
```

The brew formula installs the Spiral fork of `llama.cpp` plus the wrappers. Standard upstream `llama.cpp` will not load Spiral-compressed models.

### Interactive chat

```bash
spiral-chat --model qwen-36-35b-spiral
```

First run auto-downloads the GGUF and codebook to `~/.spiral/models/qwen-36-35b-spiral/`. Subsequent runs use the local cache.

### Single prompt

```bash
spiral-chat --model qwen-36-35b-spiral \
    --prompt "Write a Python function to compute Fibonacci numbers iteratively" \
    --greedy
```

### OpenAI-compatible API server

```bash
spiral-serve --model qwen-36-35b-spiral --port 8080
```

```bash
curl http://localhost:8080/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -d '{
      "messages": [{"role": "user", "content": "Hello"}],
      "chat_template_kwargs": {"enable_thinking": false}
    }'
```

`enable_thinking: false` disables Qwen's thinking-block emission — matches how the HumanEval scores in this README were measured. Omit it for full thinking traces.

### Build from source

For CUDA, or to develop against the framework:

```bash
git clone https://github.com/ReinforceAI/spiral
cd spiral
cmake -B build -DGGML_METAL=ON   # or -DGGML_CUDA=ON
cmake --build build -j
```

Then `spiral-download` and `spiral-chat` from the build directory, or call `llama-cli` directly with `SPIRAL_CODEBOOK_PATH` set to the codebook file.

---

## How it works

Spiral applies a dense rotation to weight matrices before scalar quantization. The rotation produces near-Gaussian weight distributions with equalized variance, eliminating outlier channels — the primary source of quantization error in coder models. Quantization in the rotated space is then much more efficient than in the original basis.

Spiral_4_5 uses mixed bit depth across MoE expert tensors, with higher precision allocated to a subset of layers identified through per-layer sensitivity analysis. Attention and embedding weights remain in higher precision.

**No calibration data is required.** The compression is derived analytically from the model's own weights — no representative samples, no importance matrix construction, no fine-tuning.

Full method description is in development. Source: [github.com/ReinforceAI/spiral](https://github.com/ReinforceAI/spiral)

---

## Limitations

- **KV cache compression not in this release.** Long-context use cases that would benefit from KV compression are not addressed by v0.3.0; future releases will revisit.
- **HumanEval measured on H100 only.** Mac HumanEval is expected within 1-2pp based on cross-platform diff validation.
- **Custom llama.cpp build required.** Stock upstream `llama.cpp` does not load Spiral models.
- **Coding benchmarks only.** HumanEval is the primary evaluation. MBPP, LiveCodeBench, SWE-bench have not been measured in this release.
- **Research-grade release.** APIs may change before 1.0.

---

## Methodology

### HumanEval
- evalplus 0.3.1, llama-server on H100 80GB, CUDA backend
- Greedy decoding (`temperature=0`)
- Thinking mode disabled (`enable_thinking: false`)
- 164 problems, single sample per problem

### Perplexity
- wikitext-2-raw-v1, HF AutoTokenizer
- 64 sequences × 2048 tokens = 131,008 tokens
- Cross-method protocol: same tokenizer source across all comparisons, no GGUF tokenizer invocation for K-quants (avoids tokenizer mismatch as a confound)

### Cross-platform validation
- Side-by-side `llama-cli` with identical greedy prompts on Mac (M2 Max, Metal) and H100 (CUDA)
- Same `.gguf` + `.spiralcb` on both platforms

Reproduction scripts: [github.com/ReinforceAI/spiral](https://github.com/ReinforceAI/spiral)

---

## Acknowledgments

- **[llama.cpp](https://github.com/ggerganov/llama.cpp)** — inference engine, GGUF format, Metal and CUDA backends.
- **[unsloth](https://huggingface.co/unsloth)** — high-quality GGUF quantizations of Qwen3.6-35B-A3B and many other models, used as the reference comparison point in this release (Q4_K_M, Q4_K_XL, Q8_0).
- **Qwen Team** — Qwen3.6-35B-A3B under Apache 2.0.
- The broader open-source ML community — quantization theory, rotation methods, and product quantization research laid the groundwork.

---

## Citation

```bibtex
@misc{spiral2026,
  title={Spiral: Geometric Compression of Rotated Transformers},
  author={Deshwal, Viraj},
  year={2026},
  publisher={ReinforceAI},
  url={https://github.com/ReinforceAI/spiral}
}
```

## License

- Inference engine: Based on llama.cpp (MIT)
- Spiral compression framework: ReinforceAI
- Model weights: Apache 2.0 (inherited from Qwen3.6-35B-A3B)