Update README.md

README.md

@@ -1,303 +1,345 @@
 ---
-license: apache-2.0
 language:
-  - en
+- en
+license: apache-2.0
 tags:
-  - gguf
-  - llama-cpp
-  - text-generation
-  - reasoning
-  - mixture-of-experts
-  - moe
-  - instella
+- gguf
+- llama.cpp
+- instella-moe
+- mixture-of-experts
+- amd
+- quantized
+- reasoning
+- think
+- text-generation
 pipeline_tag: text-generation
-library_name: gguf
-base_model: NANI-Nithin/Instella-MoE-16B-A3B-Think
+base_model: amd/Instella-MoE-16B-A3B-Think
+library_name: llama.cpp
 ---
 
-# Instella-MoE-16B-A3B-Think-GGUF
+# Instella-MoE-16B-A3B-Think GGUF
+
+> GGUF quantizations of **AMD's Instella-MoE-16B-A3B-Think**, converted and optimized for local inference with llama.cpp-compatible runtimes that support the Instella-MoE architecture.
+
+## Overview
+
+This repository provides a full collection of GGUF quantizations for **[amd/Instella-MoE-16B-A3B-Think](la-MoE-16B-A3B-Think**.
 
-<p align="center">
-  <img src="https://huggingface.co/front/assets/huggingface_logo
-<h1 align="center">🚀 Instella-MoE-16B-A3B-Think-GGUF</h1>
+Instella-MoE-16B-A3B-Think is a Mixture-of-Experts reasoning model featuring approximately **16B total parameters with ~3B active parameters per token**, designed for efficient high-quality inference while maintaining strong reasoning, coding, and instruction-following capabilities.
 
-<p align="center">
-High-performance GGUF quantizations of <b>Instella-MoE-16B-A3B-Think</b>, optimized for efficient local inference across CPUs, consumer GPUs, edge devices, and production deployments.
-</p>
+These GGUF files were generated to enable:
 
-<p align="center">
-🤖 Mixture-of-Experts • 🧠 Reasoning Focused • ⚡ Fast Inference • 💾 Multiple Quantizations
-</p>
+- Local inference
+- CPU deployment
+- GPU-accelerated llama.cpp inference
+- Edge and workstation deployments
+- Quantized execution with reduced memory requirements
+- Reasoning-focused workloads
 
 ---
 
-## Overview
+# ⚠️ Important Compatibility Notice
 
-This repository provides **GGUF conversions and quantizations** of **Instella-MoE-16B-A3B-Think**, enabling efficient deployment with:
+Instella-MoE is **not currently supported by upstream llama.cpp** at the time these GGUFs were produced.
 
-- **llama.cpp**
-- **LM Studio**
-- **Ollama (GGUF import)**
-- **Jan**
-- **KoboldCpp**
-- **Text Generation WebUI**
-- **Open WebUI**
-- Any GGUF-compatible inference engine
+The model introduces architecture components beyond standard DeepSeek-V3 implementations, including:
 
-The collection includes a comprehensive range of **K-Quants** and **Importance-Aware Quantizations (IQ)**, allowing users to choose the optimal balance between:
+- Gated Attention
+- FarSkip dual-residual connections
 
-✅ Memory usage  
-✅ Inference speed  
-✅ Reasoning quality  
-✅ Hardware compatibility
+These quantizations were generated using the community fork:
 
----
+**https://github.com/csabakecskemeti/llama.cpp**
 
-# Model Highlights
+Branch:
 
-### 🧠 Strong Reasoning Capabilities
-Instella-MoE-16B-A3B-Think is designed for advanced reasoning workloads, including:
+```text
+instella-moe
+```
 
-- Mathematical problem solving
-- Coding assistance
-- Multi-step logical reasoning
-- Technical analysis
-- Knowledge-intensive tasks
-- Instruction following
+This fork implements:
 
-### ⚡ Efficient MoE Architecture
+- `InstellaMoEForCausalLM`
+- Gated attention runtime support
+- FarSkip support
+- GGUF export support for Instella-MoE
 
-The Mixture-of-Experts architecture activates only a subset of experts during inference, offering:
+As a result, these files currently require:
 
-- Better compute efficiency
-- Lower inference cost
-- Improved scaling characteristics
-- Strong performance-to-parameter ratio
+```text
+llama.cpp (instella-moe branch)
+```
 
-### 💻 Local AI Ready
+or any future upstream release that merges full Instella-MoE support.
 
-The available GGUF variants support a wide range of hardware:
+---
 
-- Modern laptops
-- Desktop CPUs
-- Consumer GPUs
-- Workstations
-- Home AI servers
+# Model Details
+
+| Property | Value |
+|-----------|----------|
+| Model | Instella-MoE-16B-A3B-Think |
+| Organization | AMD |
+| Architecture | Instella-MoE |
+| Family | DeepSeek-V3 Derived |
+| Total Parameters | ~16B |
+| Active Parameters | ~3B |
+| Format | GGUF |
+| Purpose | Reasoning, Coding, General Assistant Tasks |
+| Quantization | Multiple GGUF Variants |
+| Base Model | amd/Instella-MoE-16B-A3B-Think |
 
 ---
 
 # Available Quantizations
 
-## Standard K-Quants
-
-These quantizations are recommended for most users.
-
-| Quant | Quality | Speed | RAM Usage | Recommended For |
-|---------|----------|--------|------------|----------------|
-| Q2_K | Lowest | Fastest | Lowest | Maximum memory savings |
-| Q3_K_S | Low | Very Fast | Very Low | Entry-level systems |
-| Q3_K_M | Good | Fast | Low | Balanced performance |
-| Q3_K_L | Better | Fast | Low | Improved quality |
-| Q4_0 | Good | Fast | Moderate | Legacy favorite |
-| Q4_1 | Better | Fast | Moderate | General usage |
-| Q4_K_S | Very Good | Fast | Moderate | Daily inference |
-| Q4_K_M | Excellent | Fast | Moderate | Recommended baseline |
-| Q5_K_S | Excellent | Moderate | Higher | Quality-focused users |
-| Q5_K_M | Very High | Moderate | Higher | Strong reasoning tasks |
-| Q6_K | Near-lossless | Moderate | High | High-quality inference |
-| Q8_0 | Maximum | Slower | Highest | Closest to original model |
+## Standard Quants
 
----
+### Smallest
 
-## Importance-Aware (IQ) Quants
+- Q2_K
 
-IQ quants use advanced quantization techniques that often preserve quality better at similar file sizes.
+### Q3 Family
 
-| Quant | Quality Retention | Memory Efficiency |
-|---------|------------------|------------------|
-| IQ2_M | Excellent for size | Extremely High |
-| IQ3_XXS | Very High | Very High |
-| IQ3_XS | Very High | High |
-| IQ3_M | Excellent | High |
-| IQ4_XS | Near Q4 quality with lower memory |
-| IQ4_NL | Premium IQ quant with strong quality preservation |
+- Q3_K_S
+- Q3_K_M
+- Q3_K_L
 
----
+### Q4 Family
+
+- Q4_0
+- Q4_1
+- Q4_K_S
+- Q4_K_M
+
+### Q5 Family
 
-# Choosing the Right Quant
+- Q5_K_S
+- Q5_K_M
+
+### High Quality
+
+- Q6_K
+- Q8_0
+
+---
 
-### Low Memory Systems
+## IQ Quants
 
-Recommended:
+Importance Matrix (Imatrix) optimized quantizations:
 
 - IQ2_M
-- Q2_K
-- Q3_K_S
+- IQ3_XXS
+- IQ3_XS
+- IQ3_M
+- IQ4_XS
+- IQ4_NL
 
-### Balanced Setup
+These quantizations generally achieve superior quality-to-size ratios compared to traditional quant methods.
 
-Recommended:
+---
 
-- Q4_K_M ⭐
-- IQ4_XS ⭐
-- Q5_K_S
+# Recommended Quant
 
-### Best Quality
+### For Low RAM Systems
 
-Recommended:
+```text
+Q2_K
+IQ2_M
+```
 
-- Q5_K_M
-- Q6_K
-- Q8_0
-- IQ4_NL ⭐
+### Best Balance
+
+```text
+Q4_K_M
+IQ4_XS
+```
+
+### High Quality
+
+```text
+Q5_K_M
+Q6_K
+IQ4_NL
+```
+
+### Maximum Quality
+
+```text
+Q8_0
+BF16
+```
 
 ---
 
-# Usage with llama.cpp
+# Example Usage
 
-### Download
+## llama.cpp
 
 ```bash
-huggingface-cli download NANI-Nithin/Instella-MoE-16B-A3B-Think-GGUF \
-  --include "Instella-MoE-16B-A3B-Think-Q4_K_M.gguf" \
-  --local-dir .
+./llama-cli \
+  -m Instella-MoE-16B-A3B-Think-Q4_K_M.gguf \
+  -p "Explain mixture-of-experts architectures."
 ```
 
-### Run
+---
+
+## Server Mode
 
 ```bash
-./llama-cli \
+./llama-server \
   -m Instella-MoE-16B-A3B-Think-Q4_K_M.gguf \
-  -p "Explain the theory of relativity in simple terms."
+  -c 32768
 ```
 
 ---
 
-# Usage with LM Studio
+# Quantization Methodology
 
-1. Open **LM Studio**
-2. Navigate to **Models**
-3. Search for:
+The conversion pipeline follows:
 
 ```text
-NANI-Nithin/Instella-MoE-16B-A3B-Think-GGUF
+Hugging Face Model
+        ↓
+Convert to BF16 GGUF
+        ↓
+Generate Imatrix
+        ↓
+Create Standard Quants
+        ↓
+Create IQ Quants
+        ↓
+Upload to Hugging Face
 ```
 
-4. Download your preferred quantization.
-5. Load and chat.
+### BF16 Conversion
 
----
+The original model weights were converted directly into GGUF BF16 format using the Instella-MoE-enabled llama.cpp conversion tools.
 
-# Recommended Prompt Format
+### Importance Matrix Generation
 
-For best results, use clear and structured instructions.
-
-### Example
+Importance matrix calibration was generated using:
 
 ```text
-You are a helpful AI assistant.
+Salesforce/wikitext
+wikitext-2-raw-v1
+```
 
-Question:
-Explain how Mixture-of-Experts models work.
+A lightweight calibration dataset was used to optimize IQ quantization quality while remaining practical on constrained hardware.
 
-Provide:
-1. Simple explanation
-2. Technical explanation
-3. Practical advantages
-```
+### IQ Quantization
+
+IQ quant variants were produced using llama.cpp's importance-matrix-aware quantization pipeline.
 
 ---
 
-# Hardware Recommendations
+# Build Environment
 
-| Quant Family | Recommended RAM |
-|-------------|----------------|
-| Q2 / IQ2 | 8GB+ |
-| Q3 / IQ3 | 12GB+ |
-| Q4 / IQ4 | 16GB+ |
-| Q5 | 24GB+ |
-| Q6 | 32GB+ |
-| Q8 | 48GB+ |
+These GGUFs were generated on a resource-constrained environment designed to maximize reproducibility.
 
-Actual memory requirements may vary depending on context length, runtime, and offloading configuration.
+### System Constraints
 
----
+- ~15 GB RAM
+- No swap
+- ~109 GB temporary storage
+- 4 CPU cores
 
-# Benchmark Expectations
+Because the BF16 GGUF is approximately:
 
-Performance depends on:
+```text
+~32 GB
+```
 
-- CPU architecture
-- GPU acceleration
-- Context length
-- Inference backend
-- Quantization selection
+the importance matrix was computed from a smaller intermediate quantization to avoid memory exhaustion while still producing high-quality IQ variants.
+
+---
+
+# Repository Notes
+
+Generation workflow includes:
 
-Generally:
+- Automatic resume support
+- Upload tracking
+- Incremental quant generation
+- Disk-space-aware cleanup
+- Fault-tolerant upload recovery
 
-- Smaller quants → faster generation and lower memory usage
-- Larger quants → better reasoning fidelity and response quality
+Each quant is generated, uploaded, and safely removed locally before proceeding to the next file.
 
 ---
 
-# Compatibility
+# Prompt Format
 
-✅ llama.cpp
+Instella-MoE-16B-A3B-Think is an instruction-tuned reasoning model.
 
-✅ LM Studio
+Typical usage:
 
-✅ Ollama
+```text
+User: Explain the difference between MoE and dense transformers.
 
-✅ Jan
+Assistant:
+```
 
-✅ KoboldCpp
+For best results:
 
-✅ Open WebUI
+- Use clear instructions
+- Allow sufficient context length
+- Enable model reasoning when your frontend supports it
+- Use lower temperatures for factual tasks
+- Use higher temperatures for creative tasks
 
-✅ Text Generation WebUI
+---
+
+# Performance Expectations
+
+General guidance:
 
-✅ GGUF-compatible runtimes
+| Quant | Quality | Memory Usage |
+|---------|----------|----------|
+| Q2_K | Lowest | Minimal |
+| Q3_K_M | Good | Low |
+| Q4_K_M | Very Good | Moderate |
+| IQ4_XS | Excellent | Moderate |
+| Q5_K_M | Excellent | Higher |
+| Q6_K | Near BF16 | High |
+| Q8_0 | Maximum | Very High |
+| BF16 | Reference | Highest |
+
+Actual results depend on:
+
+- Prompt complexity
+- Context length
+- Hardware
+- Backend implementation
+- Future Instella-MoE runtime optimizations
 
 ---
 
 # Acknowledgements
 
-Special thanks to:
+Thanks to:
 
-- The Instella model creators
-- GGUF and llama.cpp contributors
-- The open-source AI community
-- Hugging Face for model hosting
+- AMD for releasing Instella-MoE-16B-A3B-Think
+- The llama.cpp community
+- @csabakecskemeti for the Instella-MoE llama.cpp implementation
+- The GGUF ecosystem and local AI community
 
 ---
 
 # Disclaimer
 
-This repository contains quantized GGUF versions of the original model.
+This repository only provides GGUF conversions and quantizations.
 
-Quantization can introduce minor differences in output quality compared to the original checkpoint. Users should select the quantization level that best matches their hardware constraints and quality requirements.
+Model behavior, weights, training methodology, benchmark performance, and intended use remain the responsibility of the original model authors.
 
----
+Please refer to the upstream model card for official documentation:
 
-# Support
+👉 https://huggingface.co/amd/Instella-MoE-16B-A3B-Think
 
-If you find this repository helpful:
+---
 
-⭐ Like the model  
-🤗 Follow **NANI-Nithin** on Hugging Face  
-🔄 Share feedback and benchmarks  
-🚀 Contribute to the open-source AI ecosystem
+# Download Stats Welcome ⭐
 
----
+If these quantizations help your projects, research, benchmarking, or local AI deployments, consider liking the repository and sharing feedback.
 
-## Citation
-
-```bibtex
-@misc{instella_moe_16b_a3b_think_gguf,
-  title={Instella-MoE-16B-A3B-Think-GGUF},
-  author={NANI-Nithin},
-  year={2026},
-  publisher={Hugging Face},
-  url={https://huggingface.co/NANI-Nithin/Instella-MoE-16B-A3B-Think-GGUF}
-}
-```
+Happy inferencing 🚀