leonsarmiento/Nemotron-3.5-Lightning-30B-A3B-6bit-XL-mlx

Nemotron-3.5-Lightning-30B-A3B — 6-bit XL (MLX)

MLX 6/8-bit BaseQuant_XL quantization of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 for Apple Silicon inference.

About XL Quantization

BaseQuant_XL is a data-agnostic static mixed-precision recipe — no calibration data, no iMatrix, no activation-weighted sampling. It allocates precision by layer importance using a simple heuristic: routing-critical layers stay at full precision, every-token layers get 8-bit (near-lossless), and sparse MoE experts get 6-bit (natural redundancy from 128 experts, only 6 active per token). This contrasts with data-dependent methods (iMatrix, AWQ, GPTQ, oQ, oQ4e) that can skew representation toward well-represented domains.

Quantization Recipe (BaseQuant_XL 6/8)

Precision	Layers	Rationale
bf16	lm_head, shared expert	Routing/output-critical — errors here are qualitatively different
8-bit	embeddings, attention (q/k/v/o_proj), Mamba2 (in_proj, out_proj, conv1d)	Every-token layers — 8-bit is near-lossless
6-bit	routed experts (128 experts × 23 MoE layers + MTP MoE)	Bulk parameters — natural redundancy tolerates lower precision

The MoE router gate (gate.weight, e_score_correction_bias) is kept in full precision as bare arrays (not quantizable layers).

• 6.834 bits per weight, ~25 GB (6 shards)
• Group size: 64

Model Architecture

Nemotron-H — hybrid Mamba2 + Transformer + MoE architecture:

• 52 layers: 23 Mamba2 (SSM), 6 Attention, 23 MoE
• 1 NextN (MTP) prediction layer: [attention, moe] — Multi-Token Prediction for speculative decoding
• 30B total parameters, ~3B active per token (6 of 128 experts active)
• 1 shared expert with shared_expert_overlap: true
• mlp_hidden_act: relu2, routed_scaling_factor: 2.5
• tie_word_embeddings: false
• Context length: 262,144 tokens
• ChatML-style template with <think> reasoning and tool-call support

What Makes Lightning Different

• Mamba-2 hybrid architecture — interleaves SSM layers with MoE/Attention for higher throughput and lower memory than pure Transformer
• Multi-Token Prediction (MTP) — trained with MTP layers that predict multiple future tokens, enabling self-speculative decoding
• 3B active / 30B total — efficient for single-device deployment

Inference Parameters

temperature: 1.0
top_p: 0.95
top_k: 40
min_p: 0.01
repeat_penalty: 1.05
reasoning_parser: nemotron_v3
tool_call_parser: qwen3_coder

Thinking mode is controlled via the chat template kwarg enable_thinking (default: true).

Usage

from mlx_lm import load, generate

model, tokenizer = load("leonsarmiento/Nemotron-3.5-Lightning-30B-A3B-6bit-XL-mlx")

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Explain quantum entanglement."}],
    add_generation_prompt=True,
)

response = generate(model, tokenizer, prompt=prompt, max_tokens=512)
print(response)

Also compatible with LM Studio and oMLX — point it at the model directory.