README.md

---
language:
- en
license: apache-2.0
tags:
- gguf
- llama.cpp
- instella-moe
- mixture-of-experts
- amd
- quantized
- reasoning
- think
- text-generation
pipeline_tag: text-generation
base_model: amd/Instella-MoE-16B-A3B-Think
library_name: llama.cpp
---

# Instella-MoE-16B-A3B-Think GGUF

> GGUF quantizations of **AMD's Instella-MoE-16B-A3B-Think**, converted and optimized for local inference with llama.cpp-compatible runtimes that support the Instella-MoE architecture.

## Overview

This repository provides a full collection of GGUF quantizations for **[amd/Instella-MoE-16B-A3B-Think](la-MoE-16B-A3B-Think**.

Instella-MoE-16B-A3B-Think is a Mixture-of-Experts reasoning model featuring approximately **16B total parameters with ~3B active parameters per token**, designed for efficient high-quality inference while maintaining strong reasoning, coding, and instruction-following capabilities.

These GGUF files were generated to enable:

- Local inference
- CPU deployment
- GPU-accelerated llama.cpp inference
- Edge and workstation deployments
- Quantized execution with reduced memory requirements
- Reasoning-focused workloads

---

# ⚠️ Important Compatibility Notice

Instella-MoE is **not currently supported by upstream llama.cpp** at the time these GGUFs were produced.

The model introduces architecture components beyond standard DeepSeek-V3 implementations, including:

- Gated Attention
- FarSkip dual-residual connections

These quantizations were generated using the community fork:

**https://github.com/csabakecskemeti/llama.cpp**

Branch:

```text
instella-moe
```

This fork implements:

- `InstellaMoEForCausalLM`
- Gated attention runtime support
- FarSkip support
- GGUF export support for Instella-MoE

As a result, these files currently require:

```text
llama.cpp (instella-moe branch)
```

or any future upstream release that merges full Instella-MoE support.

---

# Model Details

| Property | Value |
|-----------|----------|
| Model | Instella-MoE-16B-A3B-Think |
| Organization | AMD |
| Architecture | Instella-MoE |
| Family | DeepSeek-V3 Derived |
| Total Parameters | ~16B |
| Active Parameters | ~3B |
| Format | GGUF |
| Purpose | Reasoning, Coding, General Assistant Tasks |
| Quantization | Multiple GGUF Variants |
| Base Model | amd/Instella-MoE-16B-A3B-Think |

---

# Available Quantizations

## Standard Quants

### Smallest

- Q2_K

### Q3 Family

- Q3_K_S
- Q3_K_M
- Q3_K_L

### Q4 Family

- Q4_0
- Q4_1
- Q4_K_S
- Q4_K_M

### Q5 Family

- Q5_K_S
- Q5_K_M

### High Quality

- Q6_K
- Q8_0

---

## IQ Quants

Importance Matrix (Imatrix) optimized quantizations:

- IQ2_M
- IQ3_XXS
- IQ3_XS
- IQ3_M
- IQ4_XS
- IQ4_NL

These quantizations generally achieve superior quality-to-size ratios compared to traditional quant methods.

---

# Recommended Quant

### For Low RAM Systems

```text
Q2_K
IQ2_M
```

### Best Balance

```text
Q4_K_M
IQ4_XS
```

### High Quality

```text
Q5_K_M
Q6_K
IQ4_NL
```

### Maximum Quality

```text
Q8_0
BF16
```

---

# Example Usage

## llama.cpp

```bash
./llama-cli \
  -m Instella-MoE-16B-A3B-Think-Q4_K_M.gguf \
  -p "Explain mixture-of-experts architectures."
```

---

## Server Mode

```bash
./llama-server \
  -m Instella-MoE-16B-A3B-Think-Q4_K_M.gguf \
  -c 32768
```

---

# Quantization Methodology

The conversion pipeline follows:

```text
Hugging Face Model
        ↓
Convert to BF16 GGUF
        ↓
Generate Imatrix
        ↓
Create Standard Quants
        ↓
Create IQ Quants
        ↓
Upload to Hugging Face
```

### BF16 Conversion

The original model weights were converted directly into GGUF BF16 format using the Instella-MoE-enabled llama.cpp conversion tools.

### Importance Matrix Generation

Importance matrix calibration was generated using:

```text
Salesforce/wikitext
wikitext-2-raw-v1
```

A lightweight calibration dataset was used to optimize IQ quantization quality while remaining practical on constrained hardware.

### IQ Quantization

IQ quant variants were produced using llama.cpp's importance-matrix-aware quantization pipeline.

---

# Build Environment

These GGUFs were generated on a resource-constrained environment designed to maximize reproducibility.

### System Constraints

- ~15 GB RAM
- No swap
- ~109 GB temporary storage
- 4 CPU cores

Because the BF16 GGUF is approximately:

```text
~32 GB
```

the importance matrix was computed from a smaller intermediate quantization to avoid memory exhaustion while still producing high-quality IQ variants.

---

# Repository Notes

Generation workflow includes:

- Automatic resume support
- Upload tracking
- Incremental quant generation
- Disk-space-aware cleanup
- Fault-tolerant upload recovery

Each quant is generated, uploaded, and safely removed locally before proceeding to the next file.

---

# Prompt Format

Instella-MoE-16B-A3B-Think is an instruction-tuned reasoning model.

Typical usage:

```text
User: Explain the difference between MoE and dense transformers.

Assistant:
```

For best results:

- Use clear instructions
- Allow sufficient context length
- Enable model reasoning when your frontend supports it
- Use lower temperatures for factual tasks
- Use higher temperatures for creative tasks

---

# Performance Expectations

General guidance:

| Quant | Quality | Memory Usage |
|---------|----------|----------|
| Q2_K | Lowest | Minimal |
| Q3_K_M | Good | Low |
| Q4_K_M | Very Good | Moderate |
| IQ4_XS | Excellent | Moderate |
| Q5_K_M | Excellent | Higher |
| Q6_K | Near BF16 | High |
| Q8_0 | Maximum | Very High |
| BF16 | Reference | Highest |

Actual results depend on:

- Prompt complexity
- Context length
- Hardware
- Backend implementation
- Future Instella-MoE runtime optimizations

---

# Acknowledgements

Thanks to:

- AMD for releasing Instella-MoE-16B-A3B-Think
- The llama.cpp community
- @csabakecskemeti for the Instella-MoE llama.cpp implementation
- The GGUF ecosystem and local AI community

---

# Disclaimer

This repository only provides GGUF conversions and quantizations.

Model behavior, weights, training methodology, benchmark performance, and intended use remain the responsibility of the original model authors.

Please refer to the upstream model card for official documentation:

👉 https://huggingface.co/amd/Instella-MoE-16B-A3B-Think

---

# Download Stats Welcome ⭐

If these quantizations help your projects, research, benchmarking, or local AI deployments, consider liking the repository and sharing feedback.

Happy inferencing 🚀