---
language:
- en
- multilingual
license: other
license_name: nvidia-license
license_link: https://huggingface.co/nvidia/llama-embed-nemotron-8b/blob/main/LICENSE
base_model: nvidia/llama-embed-nemotron-8b
tags:
- mlx
- embedding
- retrieval
- sentence-transformers
- feature-extraction
pipeline_tag: feature-extraction
library_name: mlx
---

# ncorder/llama-embed-nemotron-8b-mlx-2bit

MLX conversion of [`nvidia/llama-embed-nemotron-8b`](https://huggingface.co/nvidia/llama-embed-nemotron-8b) — the #1 embedding model under 20B parameters on the [MTEB leaderboard](https://huggingface.co/spaces/mteb/leaderboard), outperforming models 3x its size.

- **Parameters:** 7.5B
- **Quantization:** 2-bit affine (group_size=64)
- **Model size:** 2.4 GB
- **Architecture:** Llama-3.1-8B with bidirectional attention
- **Embedding dimension:** 4096
- **Max sequence length:** 32,768 tokens
- **Converted with:** [mlx-embeddings](https://github.com/Blaizzy/mlx-embeddings)

> **Note:** 2-bit quantization shows measurable quality degradation. The irrelevant document score inflates significantly compared to the reference. Suitable for coarse ranking or memory-constrained environments, but 4-bit is recommended for most use cases.

## All variants

| Variant | Size | Relevant ↑ | Irrelevant ↓ | Margin ↑ |
|---|---|---|---|---|
| [fp16](https://huggingface.co/ncorder/llama-embed-nemotron-8b-mlx-fp16) | 15 GB | 0.3763 | 0.0579 | 0.3184 |
| [8-bit](https://huggingface.co/ncorder/llama-embed-nemotron-8b-mlx-8bit) | 7.5 GB | 0.3780 | 0.0583 | 0.3197 |
| [**4-bit**](https://huggingface.co/ncorder/llama-embed-nemotron-8b-mlx-4bit) | 4.0 GB | 0.3826 | 0.0783 | 0.3043 |
| [2-bit](https://huggingface.co/ncorder/llama-embed-nemotron-8b-mlx-2bit) | 2.4 GB | 0.4799 | 0.2873 | 0.1926 |

Reference (original bf16 PyTorch): relevant=0.3771, irrelevant=0.0581, margin=0.3190

## Usage

```bash
pip install mlx-embeddings
```

```python
from mlx_embeddings.utils import load
import mlx.core as mx

model, tokenizer = load("ncorder/llama-embed-nemotron-8b-mlx-2bit")

query = "Instruct: Given a question, retrieve passages that answer the question\nQuery: How do neural networks learn patterns from examples?"
document = "Deep learning models adjust their weights through backpropagation."

def embed(text):
    inputs = tokenizer(text, return_tensors="np", padding=True)
    out = model(
        mx.array(inputs["input_ids"]),
        mx.array(inputs["attention_mask"])
    )
    return out.text_embeds

q_emb = embed(query)
d_emb = embed(document)
score = (q_emb @ d_emb.T).item()
print(f"Similarity: {score:.4f}")
```

### Query formatting

This model is instruction-aware. For retrieval, prefix queries with:

```
Instruct: {task_instruction}
Query: {your_query}
```

Documents are embedded without any prefix.

## License

This model inherits the [NVIDIA license](https://huggingface.co/nvidia/llama-embed-nemotron-8b/blob/main/LICENSE) from the original — **research/non-commercial use only**. Also subject to the [Llama 3.1 Community License](https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct/blob/main/LICENSE).

## Credits

- Original model by NVIDIA: [`nvidia/llama-embed-nemotron-8b`](https://huggingface.co/nvidia/llama-embed-nemotron-8b)
- Conversion via [`mlx-embeddings`](https://github.com/Blaizzy/mlx-embeddings) by Prince Canuma
- [Technical report](https://arxiv.org/abs/2511.07025)