groxaxo/Nemotron-3-Embed-8B-AWQ-W4A16-G32-ASYM

Nemotron 3 Embed 8B — 4-bit AWQ (W4A16, G32, Asymmetric)

A quality-first, deployment-ready 4-bit quantization of NVIDIA's multilingual 8B embedding model.

This checkpoint turns NVIDIA Nemotron-3-Embed-8B-BF16 into a single 5.36 GB safetensors artifact using asymmetric AWQ with a fine-grained group size of 32. It retains 4096-dimensional embeddings, loads natively through vLLM's compressed-tensors path, and was validated with real embedding requests on an RTX 3090.

This is an independently produced community quantization. It is not an official NVIDIA release.

Why this build

• 4-bit weights, BF16 activations — materially lower storage and weight memory than the BF16 source.
• Quality-first G32 quantization — smaller groups preserve more local weight detail than coarser AWQ configurations.
• Native vLLM serving — embedded compressed-tensors metadata enables Marlin-compatible execution without a manual quantization flag.
• Reproducible provenance — the exact source revision, calibration recipe, and quantizer settings ship with the model.
• Measured, not merely converted — validated for finite 4096D output, BF16-to-AWQ cosine fidelity, and code-retrieval quality.

Quick start with vLLM

Validated with vllm==0.25.0:

pip install "vllm==0.25.0" openai

vllm serve groxaxo/Nemotron-3-Embed-8B-AWQ-W4A16-G32-ASYM \
  --host 0.0.0.0 \
  --port 8000 \
  --served-model-name nemotron-embed-8b-awq \
  --dtype bfloat16 \
  --max-model-len 4096 \
  --max-num-batched-tokens 4096

Do not add --quantization awq. vLLM detects the checkpoint's compressed-tensors metadata automatically.

Embed queries and documents with the model's required role prefixes:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="local")

response = client.embeddings.create(
    model="nemotron-embed-8b-awq",
    input=[
        "query: How do I rotate an API key?",
        "passage: API keys can be rotated from the security settings page.",
    ],
)

query_vector, passage_vector = [item.embedding for item in response.data]
print(len(query_vector))  # 4096

For retrieval, use query: for queries and passage: for candidate documents. Compare L2-normalized vectors with cosine similarity or dot product.

Validation results

Quantization fidelity

The checkpoint was loaded with vLLM 0.25.0 on an RTX 3090 and served a real embeddings request. A four-vector BF16-versus-AWQ smoke comparison produced:

Check	Result
Output	Finite 4096-dimensional vectors
Mean BF16 ↔ AWQ cosine similarity	0.9925
Minimum BF16 ↔ AWQ cosine similarity	0.9910

These cosine figures measure output fidelity to the BF16 checkpoint; they are not a substitute for a full retrieval benchmark.

Code-retrieval evaluation

A controlled evaluation used the same five-repository corpus for both models: 10,403 function/method chunks and 200 labeled queries.

Model	Dimensions	R@1	R@5	R@10	MRR@10	Embeddings/s
Nemotron 3 Embed 8B AWQ	4096	68.5%	82.5%	84.5%	0.7482	15.7
Nemotron 3 Embed 1B AWQ comparison	2048	69.0%	82.0%	84.5%	0.7485	44.5

On this workload, retrieval quality was effectively tied. The 8B model should therefore be chosen for workloads where its larger capacity demonstrates an advantage on representative data—not simply because it has more parameters. Results are specific to this custom code corpus and are not RTEB scores.

Dimensionality and index compatibility

The native output is 4096 dimensions. The upstream model supports client-side prefix slicing (for example, keeping the first 2048 dimensions) followed by L2 normalization. However, in the validated vLLM 0.25.0 configuration, requesting output_dimension=2048 was rejected; client-side slicing after inference worked.

Do not mix 4096D and 2048D vectors in one index. Re-embed the corpus, slice queries and documents identically, normalize after slicing, and measure retrieval quality before migrating a production index.

Quantization details

Property	Value
Base model	nvidia/Nemotron-3-Embed-8B-BF16
Pinned source revision	8ca3ff382cf1de715e05acac8b553e0a084680d0
Algorithm	AWQ
Weight / activation precision	W4A16
Weight symmetry	Asymmetric
Group size	32
Observer	MSE
Quantized targets	Linear weights
Kept in BF16	Embedding table and normalization layers
Calibration data	1,024 balanced query/document samples from sentence-transformers/paq
Calibration sequence length	4096 tokens
Calibration seed	42
Serialization	compressed-tensors, pack-quantized
Toolchain	llmcompressor==0.12.0, compressed-tensors==0.17.1

See awq_build_settings.json and recipe.yaml for the machine-readable build configuration.

Intended use

This model is intended for multilingual retrieval, semantic search, dense indexing, clustering, and RAG systems that can serve a compressed 8B encoder and store 4096D vectors. Evaluate it on your own corpus before production use, especially for domain-specific, safety-critical, or latency-sensitive applications.

License and attribution

The upstream model and its configuration are licensed under the OpenMDW License Agreement 1.1. Review that license and the original NVIDIA model card before use. This repository contains a separately produced quantization of the pinned upstream checkpoint.