NANI-Nithin/Instella-MoE-16B-A3B-Think-GGUF

Instella-MoE-16B-A3B-Think GGUF

GGUF quantizations of AMD's Instella-MoE-16B-A3B-Think, converted and optimized for local inference with llama.cpp-compatible runtimes that support the Instella-MoE architecture.

Overview

This repository provides a full collection of GGUF quantizations for [amd/Instella-MoE-16B-A3B-Think](la-MoE-16B-A3B-Think.

Instella-MoE-16B-A3B-Think is a Mixture-of-Experts reasoning model featuring approximately 16B total parameters with ~3B active parameters per token, designed for efficient high-quality inference while maintaining strong reasoning, coding, and instruction-following capabilities.

These GGUF files were generated to enable:

• Local inference
• CPU deployment
• GPU-accelerated llama.cpp inference
• Edge and workstation deployments
• Quantized execution with reduced memory requirements
• Reasoning-focused workloads

---

⚠️ Important Compatibility Notice

Instella-MoE is not currently supported by upstream llama.cpp at the time these GGUFs were produced.

The model introduces architecture components beyond standard DeepSeek-V3 implementations, including:

• Gated Attention
• FarSkip dual-residual connections

These quantizations were generated using the community fork:

https://github.com/csabakecskemeti/llama.cpp

Branch:

instella-moe

This fork implements:

• InstellaMoEForCausalLM
• Gated attention runtime support
• FarSkip support
• GGUF export support for Instella-MoE

As a result, these files currently require:

llama.cpp (instella-moe branch)

or any future upstream release that merges full Instella-MoE support.

---

Model Details

Property	Value
Model	Instella-MoE-16B-A3B-Think
Organization	AMD
Architecture	Instella-MoE
Family	DeepSeek-V3 Derived
Total Parameters	~16B
Active Parameters	~3B
Format	GGUF
Purpose	Reasoning, Coding, General Assistant Tasks
Quantization	Multiple GGUF Variants
Base Model	amd/Instella-MoE-16B-A3B-Think

---

Available Quantizations

Standard Quants

Smallest

• Q2_K

Q3 Family

• Q3_K_S
• Q3_K_M
• Q3_K_L

Q4 Family

• Q4_0
• Q4_1
• Q4_K_S
• Q4_K_M

Q5 Family

• Q5_K_S
• Q5_K_M

High Quality

• Q6_K
• Q8_0

---

IQ Quants

Importance Matrix (Imatrix) optimized quantizations:

• IQ2_M
• IQ3_XXS
• IQ3_XS
• IQ3_M
• IQ4_XS
• IQ4_NL

These quantizations generally achieve superior quality-to-size ratios compared to traditional quant methods.

---

Recommended Quant

For Low RAM Systems

Q2_K
IQ2_M

Best Balance

Q4_K_M
IQ4_XS

High Quality

Q5_K_M
Q6_K
IQ4_NL

Maximum Quality

Q8_0
BF16

---

Example Usage

llama.cpp

./llama-cli \
  -m Instella-MoE-16B-A3B-Think-Q4_K_M.gguf \
  -p "Explain mixture-of-experts architectures."

---

Server Mode

./llama-server \
  -m Instella-MoE-16B-A3B-Think-Q4_K_M.gguf \
  -c 32768

---

Quantization Methodology

The conversion pipeline follows:

Hugging Face Model
        ↓
Convert to BF16 GGUF
        ↓
Generate Imatrix
        ↓
Create Standard Quants
        ↓
Create IQ Quants
        ↓
Upload to Hugging Face

BF16 Conversion

The original model weights were converted directly into GGUF BF16 format using the Instella-MoE-enabled llama.cpp conversion tools.

Importance Matrix Generation

Importance matrix calibration was generated using:

Salesforce/wikitext
wikitext-2-raw-v1

A lightweight calibration dataset was used to optimize IQ quantization quality while remaining practical on constrained hardware.

IQ Quantization

IQ quant variants were produced using llama.cpp's importance-matrix-aware quantization pipeline.

---

Build Environment

These GGUFs were generated on a resource-constrained environment designed to maximize reproducibility.

System Constraints

• ~15 GB RAM
• No swap
• ~109 GB temporary storage
• 4 CPU cores

Because the BF16 GGUF is approximately:

~32 GB

the importance matrix was computed from a smaller intermediate quantization to avoid memory exhaustion while still producing high-quality IQ variants.

---

Repository Notes

Generation workflow includes:

• Automatic resume support
• Upload tracking
• Incremental quant generation
• Disk-space-aware cleanup
• Fault-tolerant upload recovery

Each quant is generated, uploaded, and safely removed locally before proceeding to the next file.

---

Prompt Format

Instella-MoE-16B-A3B-Think is an instruction-tuned reasoning model.

Typical usage:

User: Explain the difference between MoE and dense transformers.

Assistant:

For best results:

• Use clear instructions
• Allow sufficient context length
• Enable model reasoning when your frontend supports it
• Use lower temperatures for factual tasks
• Use higher temperatures for creative tasks

---

Performance Expectations

General guidance:

Quant	Quality	Memory Usage
Q2_K	Lowest	Minimal
Q3_K_M	Good	Low
Q4_K_M	Very Good	Moderate
IQ4_XS	Excellent	Moderate
Q5_K_M	Excellent	Higher
Q6_K	Near BF16	High
Q8_0	Maximum	Very High
BF16	Reference	Highest

Actual results depend on:

• Prompt complexity
• Context length
• Hardware
• Backend implementation
• Future Instella-MoE runtime optimizations

---

Acknowledgements

Thanks to:

• AMD for releasing Instella-MoE-16B-A3B-Think
• The llama.cpp community
• @csabakecskemeti for the Instella-MoE llama.cpp implementation
• The GGUF ecosystem and local AI community

---

Disclaimer

This repository only provides GGUF conversions and quantizations.

Model behavior, weights, training methodology, benchmark performance, and intended use remain the responsibility of the original model authors.

Please refer to the upstream model card for official documentation:

👉 https://huggingface.co/amd/Instella-MoE-16B-A3B-Think

---

Download Stats Welcome ⭐

If these quantizations help your projects, research, benchmarking, or local AI deployments, consider liking the repository and sharing feedback.

Happy inferencing 🚀