Instructions to use nvidia/Llama-3_3-Nemotron-Super-49B-GenRM-Multilingual with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use nvidia/Llama-3_3-Nemotron-Super-49B-GenRM-Multilingual with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="nvidia/Llama-3_3-Nemotron-Super-49B-GenRM-Multilingual", trust_remote_code=True) messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("nvidia/Llama-3_3-Nemotron-Super-49B-GenRM-Multilingual", trust_remote_code=True, dtype="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use nvidia/Llama-3_3-Nemotron-Super-49B-GenRM-Multilingual with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "nvidia/Llama-3_3-Nemotron-Super-49B-GenRM-Multilingual" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "nvidia/Llama-3_3-Nemotron-Super-49B-GenRM-Multilingual", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/nvidia/Llama-3_3-Nemotron-Super-49B-GenRM-Multilingual
- SGLang
How to use nvidia/Llama-3_3-Nemotron-Super-49B-GenRM-Multilingual with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "nvidia/Llama-3_3-Nemotron-Super-49B-GenRM-Multilingual" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "nvidia/Llama-3_3-Nemotron-Super-49B-GenRM-Multilingual", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "nvidia/Llama-3_3-Nemotron-Super-49B-GenRM-Multilingual" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "nvidia/Llama-3_3-Nemotron-Super-49B-GenRM-Multilingual", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use nvidia/Llama-3_3-Nemotron-Super-49B-GenRM-Multilingual with Docker Model Runner:
docker model run hf.co/nvidia/Llama-3_3-Nemotron-Super-49B-GenRM-Multilingual
| Field | Response |
|---|---|
| Intended Application & Domain: | Response Customization in Large Language Model Development |
| Model Type: | Text-to-Text Transformer |
| Intended User: | Developers customizing model response across different applications and domains. |
| Output: | Text |
| Describe how the model works: | Generates score based on the quality of the last assistant response in conversation |
| Name the adversely impacted groups this has been tested to deliver comparable outcomes regardless of: | Not Applicable |
| Verified to have met prescribed quality standards: | Yes |
| Technical Limitations: | This model may not work well beyond input sequence length of 8, 192 tokens. |
| Performance Metrics: | Throughput and Latency |
| Potential Known Risks: | The model was trained on data that contains toxic language, unsafe content, and societal biases originally crawled from the internet. Therefore, the model may amplify those biases and return toxic responses especially when prompted with toxic prompts. The model may generate answers that may be inaccurate, omit key information, or include irrelevant or redundant text producing socially unacceptable or undesirable text, even if the prompt itself does not include anything explicitly offensive. |
| Licensing: | GOVERNING TERMS: Use of this model is governed by the NVIDIA Open Model License . Additional Information: Llama 3.3 Community License Agreement. Built with Llama. |