majentik commited on
Commit
995dd33
·
verified ·
1 Parent(s): 6845655

docs: Tier 1 polish — frontmatter + quickstart + KV-root rewrite

Browse files
Files changed (1) hide show
  1. README.md +35 -1
README.md CHANGED
@@ -3,7 +3,13 @@ license: other
3
  license_name: nvidia-open-model-license
4
  license_link: https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf
5
  base_model: nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16
6
- tags: [nemotron, multimodal, mamba2, moe, quantized, turboquant, mlx]
 
 
 
 
 
 
7
  ---
8
 
9
  # Nemotron-3-Nano-Omni-30B-A3B-Reasoning - TurboQuant MLX 3-bit
@@ -21,6 +27,34 @@ see [`majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-TurboQuant-MLX-3bit-TQ-KV`
21
  For the runtime KV-cache modifier itself, see
22
  [`majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-TurboQuant`](https://huggingface.co/majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-TurboQuant).
23
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
24
  ## Modality matrix
25
 
26
  | Modality | Encoder | Quantization in this variant |
 
3
  license_name: nvidia-open-model-license
4
  license_link: https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf
5
  base_model: nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16
6
+ tags: [nemotron, multimodal, mamba2, moe, quantized, turboquant, mlx, apple-silicon,
7
+ mlx-lm, text-tower-only]
8
+ library_name: mlx
9
+ pipeline_tag: text-generation
10
+ language: [en]
11
+ datasets: [nvidia/Nemotron-Image-Training-v3]
12
+ inference: false
13
  ---
14
 
15
  # Nemotron-3-Nano-Omni-30B-A3B-Reasoning - TurboQuant MLX 3-bit
 
27
  For the runtime KV-cache modifier itself, see
28
  [`majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-TurboQuant`](https://huggingface.co/majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-TurboQuant).
29
 
30
+ ## Quickstart
31
+
32
+ ```python
33
+ # Today (mlx-lm 0.31.x): the NemotronH_Nano_Omni_Reasoning_V3 model class
34
+ # is not yet registered in mlx-lm. The cell below is the API shape that WILL
35
+ # work once upstream lands the class (track ml-explore/mlx-lm#386).
36
+
37
+ from mlx_lm import load, generate
38
+
39
+ model, tokenizer = load("majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-TurboQuant-MLX-3bit")
40
+
41
+ prompt = tokenizer.apply_chat_template(
42
+ [{"role": "user", "content": "Solve: 17 * 23"}],
43
+ add_generation_prompt=True,
44
+ enable_thinking=False, # set True to enable extended reasoning (default)
45
+ )
46
+
47
+ response = generate(
48
+ model, tokenizer,
49
+ prompt=prompt,
50
+ max_tokens=512,
51
+ sampler=lambda x: x.argmax(axis=-1), # or use mlx_lm.sample_utils.make_sampler(temp=0.6, top_p=0.95)
52
+ )
53
+ print(response)
54
+ ```
55
+
56
+ > ⚠️ This variant covers the **text tower only**. For multimodal inference (vision + audio + video), use the GGUF variants with `llama-mtmd-cli` — see the GGUF cards in this family.
57
+
58
  ## Modality matrix
59
 
60
  | Modality | Encoder | Quantization in this variant |