Automatic Speech Recognition
NeMo
Safetensors
Transformers
PyTorch
nemotron3_5_asr
feature-extraction
speech-recognition
cache-aware ASR
streaming-asr
multilingual
speech
audio
FastConformer
RNNT
Parakeet
ASR
NeMo
Eval Results (legacy)
Eval Results
Instructions to use nvidia/nemotron-3.5-asr-streaming-0.6b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- NeMo
How to use nvidia/nemotron-3.5-asr-streaming-0.6b with NeMo:
import nemo.collections.asr as nemo_asr asr_model = nemo_asr.models.ASRModel.from_pretrained("nvidia/nemotron-3.5-asr-streaming-0.6b") transcriptions = asr_model.transcribe(["file.wav"]) - Transformers
How to use nvidia/nemotron-3.5-asr-streaming-0.6b with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("automatic-speech-recognition", model="nvidia/nemotron-3.5-asr-streaming-0.6b")# Load model directly from transformers import AutoProcessor, AutoModel processor = AutoProcessor.from_pretrained("nvidia/nemotron-3.5-asr-streaming-0.6b") model = AutoModel.from_pretrained("nvidia/nemotron-3.5-asr-streaming-0.6b", device_map="auto") - Inference
- Notebooks
- Google Colab
- Kaggle
Update model card
Browse filesSigned-off-by: Jinhan <jinhanw@nvidia.com>
- README.md +46 -30
- model_overview.png +2 -2
README.md
CHANGED
|
@@ -199,17 +199,34 @@ img {
|
|
| 199 |
</style>
|
| 200 |
|
| 201 |
<p align="center">
|
| 202 |
-
<a href="#model-architecture">
|
|
|
|
|
|
|
| 203 |
|
| 204 |
-
<a href="#model-architecture">
|
|
|
|
|
|
|
| 205 |
|
| 206 |
-
<a href="#supported-languages">
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 207 |
</p>
|
| 208 |
|
| 209 |
<p align="center">
|
| 210 |
<img src="model_overview.png" alt="Nemotron 3.5 ASR overview: multilingual audio across 40 language-locales is transcribed by a cache-aware FastConformer-RNNT model with language-ID prompting into punctuated text with an automatic language tag" width="900"/>
|
| 211 |
</p>
|
| 212 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 213 |
> [!Note]
|
| 214 |
> This model is the multilingual extension of [nvidia/nemotron-speech-streaming-en-0.6b](https://huggingface.co/nvidia/nemotron-speech-streaming-en-0.6b), adding language-ID prompt conditioning to support transcription across **40 language-locales** from a single model.
|
| 215 |
|
|
@@ -221,34 +238,10 @@ It was trained on a massive ASR dataset and is engineered to perform across dive
|
|
| 221 |
|
| 222 |
This model is ready for commercial use.
|
| 223 |
|
| 224 |
-
---
|
| 225 |
-
|
| 226 |
-
## License/Terms of Use
|
| 227 |
-
|
| 228 |
-
Governing Terms: Use of the model is governed by the [OpenMDW-1.1](https://openmdw.ai/license/1-1/) license.
|
| 229 |
-
|
| 230 |
-
## Deployment Geography
|
| 231 |
-
|
| 232 |
-
Global
|
| 233 |
-
|
| 234 |
-
## Use Case
|
| 235 |
-
|
| 236 |
-
This model is for transcription of multilingual audio.
|
| 237 |
-
|
| 238 |
## Release Date
|
| 239 |
|
| 240 |
- Hugging Face [06/04/2026] via https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b
|
| 241 |
|
| 242 |
-
## References
|
| 243 |
-
|
| 244 |
-
<a id="ref-1"></a>[1] [Stateful Conformer with Cache-based Inference for Streaming Automatic Speech Recognition](https://arxiv.org/abs/2312.17279)
|
| 245 |
-
|
| 246 |
-
<a id="ref-2"></a>[2] [Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition](https://arxiv.org/abs/2305.05084)
|
| 247 |
-
|
| 248 |
-
<a id="ref-3"></a>[3] [NVIDIA Granary](https://huggingface.co/datasets/nvidia/Granary)
|
| 249 |
-
|
| 250 |
-
<a id="ref-4"></a>[4] [NVIDIA NeMo Framework](https://github.com/NVIDIA/NeMo)
|
| 251 |
-
|
| 252 |
## Why Choose Nemotron 3.5 ASR?
|
| 253 |
|
| 254 |
- 🌍 **Single Multilingual Model:** Transcribes 40 language-locales from one model through language-ID prompt conditioning, with optional automatic language detection.
|
|
@@ -257,6 +250,8 @@ This model is for transcription of multilingual audio.
|
|
| 257 |
- 🎛️ **Dynamic Runtime Flexibility:** Choose the optimal operating point on the latency-accuracy Pareto curve at inference time. No re-training is required to adjust for different use-case requirements.
|
| 258 |
- 📝 **Punctuation & Capitalization:** Built-in support for punctuation and capitalization in output text.
|
| 259 |
|
|
|
|
|
|
|
| 260 |
---
|
| 261 |
|
| 262 |
## Supported Languages
|
|
@@ -294,7 +289,7 @@ This model consists of a cache-aware streaming Parakeet (FastConformer) encoder
|
|
| 294 |
|
| 295 |
The language-ID prompt is fused with the acoustic representation as follows:
|
| 296 |
|
| 297 |
-
- **FastConformer encoder** processes audio into an acoustic embedding of shape (D=
|
| 298 |
- **Language Encoding** expands a 128-dim one-hot language vector across the time axis → (K=128, T), broadcasting the language identity to every frame.
|
| 299 |
- **Concatenation** along the feature axis → fused tensor (D + K, T).
|
| 300 |
- **Projection layer** maps the fused features to the RNNT decoder.
|
|
@@ -359,7 +354,7 @@ Also, check out the following NVIDIA speech models:
|
|
| 359 |
|
| 360 |
## NVIDIA NeMo
|
| 361 |
|
| 362 |
-
To train, fine-tune or perform inference with this model, you will need to install [NVIDIA NeMo](https://github.com/NVIDIA/NeMo) [\[4\]](#ref-4). We recommend you install it after you've installed Cython and latest PyTorch version.
|
| 363 |
|
| 364 |
```bash
|
| 365 |
apt-get update && apt-get install -y libsndfile1 ffmpeg
|
|
@@ -579,8 +574,29 @@ _Languages are ordered by accuracy (lowest WER first)._
|
|
| 579 |
|
| 580 |
These **8 language-locales** are recognized by the tokenizer but are not tuned for production transcription out of the box: **Greek (el-GR), Hebrew (he-IL), Lithuanian (lt-LT), Slovenian (sl-SI), Latvian (lv-LV), Maltese (mt-MT), Thai (th-TH), and Norwegian Nynorsk (nn-NO)**. Fine-tuning on in-domain data is recommended to bring them to production quality.
|
| 581 |
|
| 582 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 583 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 584 |
|
| 585 |
---
|
| 586 |
|
|
|
|
| 199 |
</style>
|
| 200 |
|
| 201 |
<p align="center">
|
| 202 |
+
<a href="#model-architecture">
|
| 203 |
+
<img src="https://img.shields.io/badge/Model_Arch-FastConformer--CacheAware--RNNT-76b900?style=flat#model-badge" alt="Model architecture"/>
|
| 204 |
+
</a>
|
| 205 |
|
| 206 |
+
<a href="#model-architecture">
|
| 207 |
+
<img src="https://img.shields.io/badge/Params-600M-76b900?style=flat#model-badge" alt="Model size"/>
|
| 208 |
+
</a>
|
| 209 |
|
| 210 |
+
<a href="#supported-languages">
|
| 211 |
+
<img src="https://img.shields.io/badge/Language-Multilingual-76b900?style=flat#model-badge" alt="Language"/>
|
| 212 |
+
</a>
|
| 213 |
+
<a href="https://developer.nvidia.com/nemotron" target="_blank" style="margin: 2px;">
|
| 214 |
+
<img alt="Homepage" src="https://img.shields.io/badge/🏠Nemotron Developer Page-Learn More Here!-536af5?color=76B900&logoColor=white" style="display: inline-block; vertical-align: middle;"/>
|
| 215 |
+
</a>
|
| 216 |
+
<a href="https://discord.gg/9xpKQtVvrk" target="_blank" style="margin: 2px;">
|
| 217 |
+
<img alt="Discord" src="https://img.shields.io/badge/Discord-NVIDIA%20AI%20Developer-7289da?logo=discord&logoColor=white&color=7289da" style="display: inline-block; vertical-align: middle;"/>
|
| 218 |
+
</a>
|
| 219 |
</p>
|
| 220 |
|
| 221 |
<p align="center">
|
| 222 |
<img src="model_overview.png" alt="Nemotron 3.5 ASR overview: multilingual audio across 40 language-locales is transcribed by a cache-aware FastConformer-RNNT model with language-ID prompting into punctuated text with an automatic language tag" width="900"/>
|
| 223 |
</p>
|
| 224 |
|
| 225 |
+
<p align="center">
|
| 226 |
+
<img src="latency_vs_parallel.png" alt="Median final-token latency vs number of parallel requests on a single H100, Nemotron vs Parakeet RNNT across chunk sizes" width="900"/>
|
| 227 |
+
</p>
|
| 228 |
+
|
| 229 |
+
|
| 230 |
> [!Note]
|
| 231 |
> This model is the multilingual extension of [nvidia/nemotron-speech-streaming-en-0.6b](https://huggingface.co/nvidia/nemotron-speech-streaming-en-0.6b), adding language-ID prompt conditioning to support transcription across **40 language-locales** from a single model.
|
| 232 |
|
|
|
|
| 238 |
|
| 239 |
This model is ready for commercial use.
|
| 240 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 241 |
## Release Date
|
| 242 |
|
| 243 |
- Hugging Face [06/04/2026] via https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b
|
| 244 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 245 |
## Why Choose Nemotron 3.5 ASR?
|
| 246 |
|
| 247 |
- 🌍 **Single Multilingual Model:** Transcribes 40 language-locales from one model through language-ID prompt conditioning, with optional automatic language detection.
|
|
|
|
| 250 |
- 🎛️ **Dynamic Runtime Flexibility:** Choose the optimal operating point on the latency-accuracy Pareto curve at inference time. No re-training is required to adjust for different use-case requirements.
|
| 251 |
- 📝 **Punctuation & Capitalization:** Built-in support for punctuation and capitalization in output text.
|
| 252 |
|
| 253 |
+
- 🔧 **Fine-tuning** Check our [blog post](https://huggingface.co/blog/nvidia/fine-tuning-nemotron-35-asr) of **how to fine-tune Nemotron 3.5 ASR to improve these languages**, including before/after results.
|
| 254 |
+
|
| 255 |
---
|
| 256 |
|
| 257 |
## Supported Languages
|
|
|
|
| 289 |
|
| 290 |
The language-ID prompt is fused with the acoustic representation as follows:
|
| 291 |
|
| 292 |
+
- **FastConformer encoder** processes audio into an acoustic embedding of shape (D=1024, T).
|
| 293 |
- **Language Encoding** expands a 128-dim one-hot language vector across the time axis → (K=128, T), broadcasting the language identity to every frame.
|
| 294 |
- **Concatenation** along the feature axis → fused tensor (D + K, T).
|
| 295 |
- **Projection layer** maps the fused features to the RNNT decoder.
|
|
|
|
| 354 |
|
| 355 |
## NVIDIA NeMo
|
| 356 |
|
| 357 |
+
To train, fine-tune or perform inference with this model, you will need to install [NVIDIA NeMo](https://github.com/NVIDIA/NeMo) [\[4\]](#ref-4). We recommend you install it after you've installed python >3.11, Cython and latest PyTorch version.
|
| 358 |
|
| 359 |
```bash
|
| 360 |
apt-get update && apt-get install -y libsndfile1 ffmpeg
|
|
|
|
| 574 |
|
| 575 |
These **8 language-locales** are recognized by the tokenizer but are not tuned for production transcription out of the box: **Greek (el-GR), Hebrew (he-IL), Lithuanian (lt-LT), Slovenian (sl-SI), Latvian (lv-LV), Maltese (mt-MT), Thai (th-TH), and Norwegian Nynorsk (nn-NO)**. Fine-tuning on in-domain data is recommended to bring them to production quality.
|
| 576 |
|
| 577 |
+
---
|
| 578 |
+
|
| 579 |
+
## License/Terms of Use
|
| 580 |
+
|
| 581 |
+
Governing Terms: Use of the model is governed by the [OpenMDW-1.1](https://openmdw.ai/license/1-1/) license.
|
| 582 |
+
|
| 583 |
+
## Deployment Geography
|
| 584 |
+
|
| 585 |
+
Global
|
| 586 |
+
|
| 587 |
+
## Use Case
|
| 588 |
+
|
| 589 |
+
This model is for transcription of multilingual audio.
|
| 590 |
|
| 591 |
+
## References
|
| 592 |
+
|
| 593 |
+
<a id="ref-1"></a>[1] [Stateful Conformer with Cache-based Inference for Streaming Automatic Speech Recognition](https://arxiv.org/abs/2312.17279)
|
| 594 |
+
|
| 595 |
+
<a id="ref-2"></a>[2] [Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition](https://arxiv.org/abs/2305.05084)
|
| 596 |
+
|
| 597 |
+
<a id="ref-3"></a>[3] [NVIDIA Granary](https://huggingface.co/datasets/nvidia/Granary)
|
| 598 |
+
|
| 599 |
+
<a id="ref-4"></a>[4] [NVIDIA NeMo Framework](https://github.com/NVIDIA/NeMo)
|
| 600 |
|
| 601 |
---
|
| 602 |
|
model_overview.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|