---
license: apache-2.0
base_model: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
language: en
tags:
  - gguf
  - quantized
  - llama.cpp
  - nemotron
  - moe
pipeline_tag: text-generation
---

# NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF

GGUF quantizations of [nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16), converted with [llama.cpp](https://github.com/ggml-org/llama.cpp) for fast local inference on CPU/GPU.

## Files

| Quant | Use case |
|---|---|
| F16 | Full precision, reference quality |
| Q8_0 | Near-lossless, largest quant size |
| Q6_K | Very high quality, minimal loss |
| Q5_K_M / Q5_K_S | High quality, good balance |
| Q4_K_M / Q4_K_S | **Recommended default** — best speed/quality tradeoff |
| Q4_0 | Legacy 4-bit, faster on some hardware |
| Q3_K_L / Q3_K_M / Q3_K_S | Lower RAM, noticeable quality drop |
| Q2_K | Smallest, most compressed, quality degrades |

## Usage

Run with `llama.cpp`, [Ollama](https://ollama.com), [LM Studio](https://lmstudio.ai), or any GGUF-compatible runtime:

```bash
./llama-cli -m NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Q4_K_M.gguf -p "Your prompt here"
```

## Notes

- This is a Mixture-of-Experts (A3B) architecture — check RAM/VRAM requirements before choosing a quant.
- For most users, **Q4_K_M** offers the best balance of speed, size, and output quality.
- Quantized using automated pipeline on [Modal](https://modal.com) with `llama.cpp`'s conversion and quantization tools.

## Credits

- Base model by [NVIDIA](https://huggingface.co/nvidia)
- Quantization by [NANI-Nithin](https://huggingface.co/NANI-Nithin)