Text Generation
MLX
Safetensors
English
NemotronH_Nano_Omni_Reasoning_V3
nemotron
multimodal
mamba2
Mixture of Experts
quantized
turboquant
apple-silicon
mlx-lm
text-tower-only
conversational
custom_code
8-bit precision
Instructions to use majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-TurboQuant-MLX-8bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-TurboQuant-MLX-8bit with MLX:
# Make sure mlx-lm is installed # pip install --upgrade mlx-lm # Generate text with mlx-lm from mlx_lm import load, generate model, tokenizer = load("majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-TurboQuant-MLX-8bit") prompt = "Write a story about Einstein" messages = [{"role": "user", "content": prompt}] prompt = tokenizer.apply_chat_template( messages, add_generation_prompt=True ) text = generate(model, tokenizer, prompt=prompt, verbose=True) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Pi
How to use majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-TurboQuant-MLX-8bit with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-TurboQuant-MLX-8bit"
Configure the model in Pi
# Install Pi: npm install -g @mariozechner/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-TurboQuant-MLX-8bit" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Hermes Agent new
How to use majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-TurboQuant-MLX-8bit with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-TurboQuant-MLX-8bit"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-TurboQuant-MLX-8bit
Run Hermes
hermes
- OpenClaw new
How to use majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-TurboQuant-MLX-8bit with OpenClaw:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-TurboQuant-MLX-8bit"
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-TurboQuant-MLX-8bit" \ --custom-provider-id mlx-lm \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
- MLX LM
How to use majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-TurboQuant-MLX-8bit with MLX LM:
Generate or start a chat session
# Install MLX LM uv tool install mlx-lm # Interactive chat REPL mlx_lm.chat --model "majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-TurboQuant-MLX-8bit"
Run an OpenAI-compatible server
# Install MLX LM uv tool install mlx-lm # Start the server mlx_lm.server --model "majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-TurboQuant-MLX-8bit" # Calling the OpenAI-compatible server with curl curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-TurboQuant-MLX-8bit", "messages": [ {"role": "user", "content": "Hello"} ] }'
| import torch | |
| from typing import Tuple | |
| class EfficientVideoSampling: | |
| def compute_retention_mask( | |
| *, | |
| video_embeds: torch.FloatTensor, | |
| thw: torch.LongTensor, | |
| spatial_merge_size: int, | |
| q: float, | |
| ): | |
| """ | |
| Computes the retention mask for video embeddings based on the grid dimensions. | |
| Args: | |
| video_embeds (`torch.FloatTensor` of shape `(T * H * W, hidden_size)`): | |
| The video embeddings to compute the retention mask for. | |
| thw (`torch.LongTensor` of shape `(3)`): | |
| The temporal, height and width of feature shape of each video in LLM. | |
| spatial_merge_size (`int`): The spatial merge size of the video embeddings. | |
| If embeddings will be downsampled *later*, this should be the downsampling factor. | |
| q: (`float`): Pruning rate factor, indicating number of tokens to prune (remove) | |
| Returns: | |
| `torch.Tensor`: The retention mask for the video embeddings (T * H * W). | |
| 1 for tokens to keep, 0 for tokens to prune. | |
| """ | |
| T, H, W = thw | |
| # video_embeds = einops.rearrange( | |
| # video_embeds, | |
| # "(T H W) C -> T H W C", | |
| # T=T, | |
| # H=H // spatial_merge_size, | |
| # W=W // spatial_merge_size, | |
| # ) | |
| # Use reshape instead of einops to avoid graph breaks | |
| video_embeds = video_embeds.reshape( | |
| T, H // spatial_merge_size, W // spatial_merge_size, video_embeds.size(-1) | |
| ) | |
| # Core EVS | |
| similarity = torch.nn.functional.cosine_similarity( | |
| video_embeds[1:, ...], video_embeds[:-1, ...], dim=-1 | |
| ) | |
| dissimilarity = 1 - similarity | |
| # Always ensure we include all tokens from the first frame | |
| dissimilarity = torch.cat( | |
| [255 * torch.ones_like(video_embeds[:1, :, :, 0]), dissimilarity], dim=0 | |
| ) | |
| dissimilarity_flat = dissimilarity.view(-1) | |
| min_num_tokens = (H // spatial_merge_size) * (W // spatial_merge_size) # a single frame | |
| evs_num_tokens = int(T * min_num_tokens * (1 - q)) | |
| num_tokens_to_keep = max(min_num_tokens, evs_num_tokens) | |
| order = torch.argsort(dissimilarity_flat, | |
| dim=-1, | |
| descending=True, | |
| stable=True) | |
| topk_indices = order[:num_tokens_to_keep] | |
| retention_mask = torch.zeros_like(dissimilarity_flat, dtype=torch.bool) | |
| retention_mask[topk_indices] = True | |
| retention_mask = retention_mask.reshape(dissimilarity.size()) | |
| # print( | |
| # f"Computed retention mask of shape {retention_mask.shape=} with sparsity {retention_mask.float().mean().item():.4f} for {q=}", | |
| # ) | |
| mask = retention_mask.view(-1) # "T H W -> (T H W)" | |
| return mask | |