Jinhanw commited on
Commit
9e6abce
·
verified ·
1 Parent(s): 24b151a

Update model card

Browse files

Signed-off-by: Jinhan <jinhanw@nvidia.com>

Files changed (2) hide show
  1. README.md +46 -30
  2. model_overview.png +2 -2
README.md CHANGED
@@ -199,17 +199,34 @@ img {
199
  </style>
200
 
201
  <p align="center">
202
- <a href="#model-architecture"><img src="https://img.shields.io/badge/Model_Arch-FastConformer--CacheAware--RNNT-lightgrey#model-badge" alt="Model architecture"/></a>
 
 
203
  &nbsp;
204
- <a href="#model-architecture"><img src="https://img.shields.io/badge/Params-600M-lightgrey#model-badge" alt="Model size"/></a>
 
 
205
  &nbsp;
206
- <a href="#supported-languages"><img src="https://img.shields.io/badge/Language-Multilingual-lightgrey#model-badge" alt="Language"/></a>
 
 
 
 
 
 
 
 
207
  </p>
208
 
209
  <p align="center">
210
  <img src="model_overview.png" alt="Nemotron 3.5 ASR overview: multilingual audio across 40 language-locales is transcribed by a cache-aware FastConformer-RNNT model with language-ID prompting into punctuated text with an automatic language tag" width="900"/>
211
  </p>
212
 
 
 
 
 
 
213
  > [!Note]
214
  > This model is the multilingual extension of [nvidia/nemotron-speech-streaming-en-0.6b](https://huggingface.co/nvidia/nemotron-speech-streaming-en-0.6b), adding language-ID prompt conditioning to support transcription across **40 language-locales** from a single model.
215
 
@@ -221,34 +238,10 @@ It was trained on a massive ASR dataset and is engineered to perform across dive
221
 
222
  This model is ready for commercial use.
223
 
224
- ---
225
-
226
- ## License/Terms of Use
227
-
228
- Governing Terms: Use of the model is governed by the [OpenMDW-1.1](https://openmdw.ai/license/1-1/) license.
229
-
230
- ## Deployment Geography
231
-
232
- Global
233
-
234
- ## Use Case
235
-
236
- This model is for transcription of multilingual audio.
237
-
238
  ## Release Date
239
 
240
  - Hugging Face [06/04/2026] via https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b
241
 
242
- ## References
243
-
244
- <a id="ref-1"></a>[1] [Stateful Conformer with Cache-based Inference for Streaming Automatic Speech Recognition](https://arxiv.org/abs/2312.17279)
245
-
246
- <a id="ref-2"></a>[2] [Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition](https://arxiv.org/abs/2305.05084)
247
-
248
- <a id="ref-3"></a>[3] [NVIDIA Granary](https://huggingface.co/datasets/nvidia/Granary)
249
-
250
- <a id="ref-4"></a>[4] [NVIDIA NeMo Framework](https://github.com/NVIDIA/NeMo)
251
-
252
  ## Why Choose Nemotron 3.5 ASR?
253
 
254
  - 🌍 **Single Multilingual Model:** Transcribes 40 language-locales from one model through language-ID prompt conditioning, with optional automatic language detection.
@@ -257,6 +250,8 @@ This model is for transcription of multilingual audio.
257
  - 🎛️ **Dynamic Runtime Flexibility:** Choose the optimal operating point on the latency-accuracy Pareto curve at inference time. No re-training is required to adjust for different use-case requirements.
258
  - 📝 **Punctuation & Capitalization:** Built-in support for punctuation and capitalization in output text.
259
 
 
 
260
  ---
261
 
262
  ## Supported Languages
@@ -294,7 +289,7 @@ This model consists of a cache-aware streaming Parakeet (FastConformer) encoder
294
 
295
  The language-ID prompt is fused with the acoustic representation as follows:
296
 
297
- - **FastConformer encoder** processes audio into an acoustic embedding of shape (D=512, T).
298
  - **Language Encoding** expands a 128-dim one-hot language vector across the time axis → (K=128, T), broadcasting the language identity to every frame.
299
  - **Concatenation** along the feature axis → fused tensor (D + K, T).
300
  - **Projection layer** maps the fused features to the RNNT decoder.
@@ -359,7 +354,7 @@ Also, check out the following NVIDIA speech models:
359
 
360
  ## NVIDIA NeMo
361
 
362
- To train, fine-tune or perform inference with this model, you will need to install [NVIDIA NeMo](https://github.com/NVIDIA/NeMo) [\[4\]](#ref-4). We recommend you install it after you've installed Cython and latest PyTorch version.
363
 
364
  ```bash
365
  apt-get update && apt-get install -y libsndfile1 ffmpeg
@@ -579,8 +574,29 @@ _Languages are ordered by accuracy (lowest WER first)._
579
 
580
  These **8 language-locales** are recognized by the tokenizer but are not tuned for production transcription out of the box: **Greek (el-GR), Hebrew (he-IL), Lithuanian (lt-LT), Slovenian (sl-SI), Latvian (lv-LV), Maltese (mt-MT), Thai (th-TH), and Norwegian Nynorsk (nn-NO)**. Fine-tuning on in-domain data is recommended to bring them to production quality.
581
 
582
- Check our [blog post](https://huggingface.co/blog/nvidia/fine-tuning-nemotron-35-asr) of **how to fine-tune Nemotron 3.5 ASR to improve these languages**, including before/after results.
 
 
 
 
 
 
 
 
 
 
 
 
583
 
 
 
 
 
 
 
 
 
 
584
 
585
  ---
586
 
 
199
  </style>
200
 
201
  <p align="center">
202
+ <a href="#model-architecture">
203
+ <img src="https://img.shields.io/badge/Model_Arch-FastConformer--CacheAware--RNNT-76b900?style=flat#model-badge" alt="Model architecture"/>
204
+ </a>
205
  &nbsp;
206
+ <a href="#model-architecture">
207
+ <img src="https://img.shields.io/badge/Params-600M-76b900?style=flat#model-badge" alt="Model size"/>
208
+ </a>
209
  &nbsp;
210
+ <a href="#supported-languages">
211
+ <img src="https://img.shields.io/badge/Language-Multilingual-76b900?style=flat#model-badge" alt="Language"/>
212
+ </a>
213
+ <a href="https://developer.nvidia.com/nemotron" target="_blank" style="margin: 2px;">
214
+ <img alt="Homepage" src="https://img.shields.io/badge/🏠Nemotron Developer Page-Learn More Here!-536af5?color=76B900&logoColor=white" style="display: inline-block; vertical-align: middle;"/>
215
+ </a>
216
+ <a href="https://discord.gg/9xpKQtVvrk" target="_blank" style="margin: 2px;">
217
+ <img alt="Discord" src="https://img.shields.io/badge/Discord-NVIDIA%20AI%20Developer-7289da?logo=discord&logoColor=white&color=7289da" style="display: inline-block; vertical-align: middle;"/>
218
+ </a>
219
  </p>
220
 
221
  <p align="center">
222
  <img src="model_overview.png" alt="Nemotron 3.5 ASR overview: multilingual audio across 40 language-locales is transcribed by a cache-aware FastConformer-RNNT model with language-ID prompting into punctuated text with an automatic language tag" width="900"/>
223
  </p>
224
 
225
+ <p align="center">
226
+ <img src="latency_vs_parallel.png" alt="Median final-token latency vs number of parallel requests on a single H100, Nemotron vs Parakeet RNNT across chunk sizes" width="900"/>
227
+ </p>
228
+
229
+
230
  > [!Note]
231
  > This model is the multilingual extension of [nvidia/nemotron-speech-streaming-en-0.6b](https://huggingface.co/nvidia/nemotron-speech-streaming-en-0.6b), adding language-ID prompt conditioning to support transcription across **40 language-locales** from a single model.
232
 
 
238
 
239
  This model is ready for commercial use.
240
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
241
  ## Release Date
242
 
243
  - Hugging Face [06/04/2026] via https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b
244
 
 
 
 
 
 
 
 
 
 
 
245
  ## Why Choose Nemotron 3.5 ASR?
246
 
247
  - 🌍 **Single Multilingual Model:** Transcribes 40 language-locales from one model through language-ID prompt conditioning, with optional automatic language detection.
 
250
  - 🎛️ **Dynamic Runtime Flexibility:** Choose the optimal operating point on the latency-accuracy Pareto curve at inference time. No re-training is required to adjust for different use-case requirements.
251
  - 📝 **Punctuation & Capitalization:** Built-in support for punctuation and capitalization in output text.
252
 
253
+ - 🔧 **Fine-tuning** Check our [blog post](https://huggingface.co/blog/nvidia/fine-tuning-nemotron-35-asr) of **how to fine-tune Nemotron 3.5 ASR to improve these languages**, including before/after results.
254
+
255
  ---
256
 
257
  ## Supported Languages
 
289
 
290
  The language-ID prompt is fused with the acoustic representation as follows:
291
 
292
+ - **FastConformer encoder** processes audio into an acoustic embedding of shape (D=1024, T).
293
  - **Language Encoding** expands a 128-dim one-hot language vector across the time axis → (K=128, T), broadcasting the language identity to every frame.
294
  - **Concatenation** along the feature axis → fused tensor (D + K, T).
295
  - **Projection layer** maps the fused features to the RNNT decoder.
 
354
 
355
  ## NVIDIA NeMo
356
 
357
+ To train, fine-tune or perform inference with this model, you will need to install [NVIDIA NeMo](https://github.com/NVIDIA/NeMo) [\[4\]](#ref-4). We recommend you install it after you've installed python >3.11, Cython and latest PyTorch version.
358
 
359
  ```bash
360
  apt-get update && apt-get install -y libsndfile1 ffmpeg
 
574
 
575
  These **8 language-locales** are recognized by the tokenizer but are not tuned for production transcription out of the box: **Greek (el-GR), Hebrew (he-IL), Lithuanian (lt-LT), Slovenian (sl-SI), Latvian (lv-LV), Maltese (mt-MT), Thai (th-TH), and Norwegian Nynorsk (nn-NO)**. Fine-tuning on in-domain data is recommended to bring them to production quality.
576
 
577
+ ---
578
+
579
+ ## License/Terms of Use
580
+
581
+ Governing Terms: Use of the model is governed by the [OpenMDW-1.1](https://openmdw.ai/license/1-1/) license.
582
+
583
+ ## Deployment Geography
584
+
585
+ Global
586
+
587
+ ## Use Case
588
+
589
+ This model is for transcription of multilingual audio.
590
 
591
+ ## References
592
+
593
+ <a id="ref-1"></a>[1] [Stateful Conformer with Cache-based Inference for Streaming Automatic Speech Recognition](https://arxiv.org/abs/2312.17279)
594
+
595
+ <a id="ref-2"></a>[2] [Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition](https://arxiv.org/abs/2305.05084)
596
+
597
+ <a id="ref-3"></a>[3] [NVIDIA Granary](https://huggingface.co/datasets/nvidia/Granary)
598
+
599
+ <a id="ref-4"></a>[4] [NVIDIA NeMo Framework](https://github.com/NVIDIA/NeMo)
600
 
601
  ---
602
 
model_overview.png CHANGED

Git LFS Details

  • SHA256: e4fa275d1d6cb0b01064df92c051dc0888dc63d18a8b3c8ae9b35268f2c89427
  • Pointer size: 131 Bytes
  • Size of remote file: 205 kB

Git LFS Details

  • SHA256: 075554d371b33de450cb7a13fa9886c3689855d4930b37edc1ecb954b01f33c5
  • Pointer size: 131 Bytes
  • Size of remote file: 114 kB