---
language:
  - en
license: apache-2.0
library_name: llama.cpp
tags:
  - gguf
  - quantized
  - self-rag
  - on-device
  - edge-ai
  - llama-cpp
base_model: nuroai/Avalon-2B
pipeline_tag: text-generation
---

<div align="center">

# AVALON-2B GGUF

### Quantized for On-Device Deployment

[![Size](https://img.shields.io/badge/Size-1.5GB-blue)](https://huggingface.co/nuroai/Avalon-2B-GGUF)
[![Quantization](https://img.shields.io/badge/Quant-Q4__K__M-green)](https://huggingface.co/nuroai/Avalon-2B-GGUF)
[![Speed](https://img.shields.io/badge/Speed-40%2B%20tok%2Fs-orange)](https://huggingface.co/nuroai/Avalon-2B-GGUF)

[Full Model](https://huggingface.co/nuroai/Avalon-2B) | [Paper](https://github.com/Nuro-Labs/avalon-2b) | [GitHub](https://github.com/Nuro-Labs/avalon-2b)

</div>

## Overview

This is the **GGUF quantized version** of [AVALON-2B](https://huggingface.co/nuroai/Avalon-2B), the first sub-3B self-reflective language model. Optimized for on-device inference with llama.cpp, Ollama, LM Studio, and other GGUF-compatible runtimes.

## Available Files

| Filename | Quantization | Size | Quality | Use Case |
|----------|-------------|------|---------|----------|
| `avalon-2b-q4km.gguf` | Q4_K_M | 1.5 GB | Best balance | **Recommended** |

## Performance Benchmarks

Tested on Apple Silicon with llama.cpp:

| Device | Chip | RAM | Speed (tok/s) | Memory |
|--------|------|-----|---------------|--------|
| MacBook Air | Apple M3 | 16 GB | **40.2** | 2.1 GB |
| MacBook Pro | Apple M3 Pro | 18 GB | **52.4** | 2.1 GB |
| Mac Studio | Apple M2 Ultra | 64 GB | **78.6** | 2.0 GB |
| iPhone 15 Pro | Apple A17 Pro | 8 GB | **12.4** | 1.8 GB |

## Quality Retention

| Metric | Full Model | Q4_K_M | Retention |
|--------|-----------|--------|-----------|
| MMLU | 62.04% | 61.42% | 99.0% |
| Self-RAG Accuracy | 82.5% | 80.5% | 97.6% |
| Perplexity | Baseline | +1.6% | 98.4% |

## Usage

### Ollama (Easiest)

```bash
# Create Modelfile
cat > Modelfile << 'EOF'
FROM ./avalon-2b-q4km.gguf

TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant
{{ .Response }}<|im_end|>
"""

PARAMETER stop "<|im_end|>"
PARAMETER temperature 0.7

SYSTEM """You are AVALON, a self-reflective AI assistant. Generate [Retrieval] for time-sensitive questions and [No Retrieval] for factual knowledge. End responses with [Utility:X]."""
EOF

# Create and run
ollama create avalon-2b -f Modelfile
ollama run avalon-2b "What is quantum computing?"
```

### llama.cpp

```bash
# Download
wget https://huggingface.co/nuroai/Avalon-2B-GGUF/resolve/main/avalon-2b-q4km.gguf

# Run inference
./llama-cli -m avalon-2b-q4km.gguf \
    -p "<|im_start|>user\nWhat is the capital of France?<|im_end|>\n<|im_start|>assistant\n" \
    -n 128 --temp 0.7
```

### LM Studio

1. Download `avalon-2b-q4km.gguf`
2. Open LM Studio → Local Models → Import
3. Select the GGUF file
4. Configure chat template (Qwen/ChatML format)
5. Start chatting!

### Python (llama-cpp-python)

```python
from llama_cpp import Llama

llm = Llama(
    model_path="avalon-2b-q4km.gguf",
    n_ctx=4096,
    n_threads=8,
)

output = llm(
    "<|im_start|>user\nWhat is AI?<|im_end|>\n<|im_start|>assistant\n",
    max_tokens=256,
    temperature=0.7,
    stop=["<|im_end|>"]
)
print(output["choices"][0]["text"])
```

## Self-RAG Tokens

AVALON generates special reflection tokens:

| Token | Meaning | When Used |
|-------|---------|-----------|
| `[Retrieval]` | Needs external info | News, current events, prices |
| `[No Retrieval]` | Can answer from knowledge | Facts, math, definitions |
| `[Utility:1-5]` | Response quality rating | End of every response |

**Example Output:**
```
User: What is the capital of Japan?
Assistant: [No Retrieval]Tokyo is the capital of Japan.[Utility:5]

User: What's the current Bitcoin price?
Assistant: [Retrieval]I need current market data to answer this...[Utility:4]
```

## Hardware Requirements

| Platform | Minimum | Recommended |
|----------|---------|-------------|
| RAM | 4 GB | 8+ GB |
| Storage | 2 GB | 2 GB |
| GPU | Not required | Metal/CUDA for speed |

## Authors

- **Akhil Ponnada** - akhil@nuroailabs.com
- **Naga Sri Arvapalli** - nagasri3007@gmail.com

## License

Apache 2.0 - Same as the [full model](https://huggingface.co/nuroai/Avalon-2B).

## Citation

```bibtex
@article{ponnada2026avalon,
  title={AVALON-2B: The First Sub-3B Self-Reflective Language Model},
  author={Ponnada, Akhil and Arvapalli, Naga Sri},
  journal={arXiv preprint},
  year={2026}
}
```

## Links

- [Full Model (nuroai/Avalon-2B)](https://huggingface.co/nuroai/Avalon-2B)
- [GitHub Repository](https://github.com/Nuro-Labs/avalon-2b)
- [Nuro AI Labs](https://nuro.one)