Upload README.md with huggingface_hub

README.md

@@ -10,22 +10,70 @@ model_type: qwen2
 quantized_by: aaardpark
 ---
 
-# Qwen2.5-72B-Instruct — 3-bit GGUF (aaardpark)
+# Qwen2.5-72B-Instruct — GGUF (aaardpark)
 
-Qwen2.5-72B-Instruct quantized to 3-bit using a new importance-weighted quantization method. Produces significantly better quality at 3-bit than standard RTN or naive quantization approaches.
+**35 GB Q3_K_M GGUF. 88% GSM8K, where standard 3-bit quantization (RTN) gets 16% at the same size.**
 
-## Key Results (Base Model Benchmarks)
+> Looking for a smaller version? See [aaardpark/Qwen2.5-32B-Instruct-GGUF](https://huggingface.co/aaardpark/Qwen2.5-32B-Instruct-GGUF) — 15 GB, fits on a 24 GB machine.
+
+## Quick stats
+
+| File | Size | BPW | Min RAM | Speed (M5 Max, Metal) |
+|---|---|---|---|---|
+| `Qwen2.5-72B-Instruct-aaardpark-Q3_K_M.gguf` | 35 GB | 3.9 | 48 GB | ~5 tok/s |
+
+## How to use
+
+### Download
+
+```bash
+huggingface-cli download aaardpark/Qwen2.5-72B-Instruct-GGUF \
+  Qwen2.5-72B-Instruct-aaardpark-Q3_K_M.gguf --local-dir .
+```
+
+### Run
+
+**llama.cpp:**
+```bash
+llama-cli -m Qwen2.5-72B-Instruct-aaardpark-Q3_K_M.gguf -ngl 99 -p "Hello!"
+```
+
+**LM Studio:** Search for `aaardpark/Qwen2.5-72B-Instruct-GGUF` in the model browser.
+
+## Prompt format
+
+This model uses the ChatML template:
+
+```
+<|im_start|>system
+You are a helpful assistant.<|im_end|>
+<|im_start|>user
+{prompt}<|im_end|>
+<|im_start|>assistant
+```
+
+## Benchmarks
+
+### Base model evaluation (lm-evaluation-harness)
 
 | Metric | FP16 | This Quant (3-bit) | RTN 3-bit |
 |--------|------|---------------------|-----------|
-| **Perplexity** | 2.670 | **3.163** | 3.750 |
+| **Perplexity** (wikitext-2) | 2.670 | **3.163** | 3.750 |
 | **GSM8K** (5-shot) | 90% | **88%** | 16% |
 | **MMLU avg** (5-shot) | 77.6% | **76.8%** | 73.0% |
 | TruthfulQA | 58.5% | 56.9% | 56.3% |
 
-Benchmarks measured on Qwen2.5-72B (base) with lm-evaluation-harness. The quantization method is identical for both base and Instruct variants.
+Measured on Qwen2.5-72B (base) with lm-evaluation-harness. The quantization method is identical for base and Instruct variants.
 
-### vs Other Quantization Methods
+### GGUF perplexity (wikitext-2, llama.cpp)
+
+| Variant | PPL |
+|---------|-----|
+| Base Q8_0 (exact weights) | 3.028 |
+| Base Q3_K_M (this format) | 2.904 |
+| Instruct Q3_K_M | 3.962 |
+
+### vs other quantization methods
 
 | Method | Bits | PPL (72B) | GSM8K | Notes |
 |--------|------|-----------|-------|-------|
@@ -40,51 +88,31 @@ Benchmarks measured on Qwen2.5-72B (base) with lm-evaluation-harness. The quanti
 
 On smaller models (7B): GPTQ 3-bit PPL = 12.576, our 3-bit PPL = 6.148. GPTQ is unusable at 3-bit; ours is not.
 
-### GGUF Perplexity (wikitext-2, llama.cpp)
-
-| Variant | PPL |
-|---------|-----|
-| Base Q8_0 (exact weights) | 3.028 |
-| Base Q3_K_M (this format) | 2.904 |
-| Instruct Q3_K_M | 3.962 |
-
-## Why This Quant is Different
+## Why this quant is different
 
 Standard 3-bit quantization (RTN) rounds each weight to the nearest grid point uniformly. This destroys the precise weight values that control multi-step reasoning — GSM8K drops from 90% to 16%.
 
-Our method uses calibration data to identify which weights are critical for model quality, then allocates quantization precision accordingly. The result: 88% GSM8K at 3-bit, nearly matching FP16.
+Our method uses calibration data to identify which weights are critical for model quality, then allocates quantization precision accordingly. Same bit budget, dramatically different quality.
 
-## Details
-- **Method**: Importance-weighted per-group optimization
-- **Group size**: 128
-- **Quantization time**: ~20 minutes on a single GPU
-- **GGUF format**: Q3_K_M (converted via llama.cpp)
-- **File size**: 35 GB
-- **Context**: 128K tokens
+## Which file should I choose?
 
-## How to Use
+This file is 35 GB. Realistic RAM requirements:
 
-Works with llama.cpp, Ollama, LM Studio, or any GGUF-compatible runtime.
+- **≥64 GB RAM**: comfortable, full 128K context window
+- **48 GB RAM**: works with 16K-32K context
+- **32 GB RAM**: tight, short context only — consider the [32B variant](https://huggingface.co/aaardpark/Qwen2.5-32B-Instruct-GGUF) instead
+- **<32 GB RAM**: use the 32B variant (15 GB)
 
-```bash
-# llama.cpp
-llama-cli -m Qwen2.5-72B-Instruct-aaardpark-Q3_K_M.gguf -ngl 99 -p "Hello!"
+On Apple Silicon with Metal offload (`-ngl 99`), expect ~5 tok/s on M5 Max. NVIDIA GPUs need ~40 GB VRAM for full offload.
 
-# Ollama
-ollama run aaardpark/qwen2.5-72b-instruct
-```
+## Method
 
-## Chat Template
+Importance-weighted per-group optimization. Calibration data identifies which weights are critical for model quality, then quantization precision is allocated accordingly. ~20 minutes per quant on a single GPU. Output is standard Q3_K_M GGUF format — no custom kernels required.
 
-This model uses the ChatML template:
-```
-<|im_start|>system
-You are a helpful assistant.<|im_end|>
-<|im_start|>user
-{prompt}<|im_end|>
-<|im_start|>assistant
-```
+- **Group size**: 128
+- **GGUF format**: Q3_K_M (via llama.cpp)
+- **Context**: 128K tokens
 
 ## Acknowledgments
 
-Built on [Qwen2.5-72B-Instruct](https://huggingface.co/Qwen/Qwen2.5-72B-Instruct) by Alibaba Cloud.
+Built on [Qwen/Qwen2.5-72B-Instruct](https://huggingface.co/Qwen/Qwen2.5-72B-Instruct) by Alibaba Cloud.