Question about reproducing the reported Nemotron 3.5 ASR benchmark results
I'm benchmarking Nemotron 3.5 ASR Streaming 0.6B and trying to reproduce the results reported on the model card.
Initially, I obtained a higher WER on FLEURS English, but I later discovered the issue on my side: I was comparing predictions against the raw_transcription field, while the normalization removes words inside parentheses. Using the correct reference field, I can now reproduce the reported results.
For evaluation, I'm using: whisper-normalizer && NeMo's WER implementation:
https://github.com/NVIDIA-NeMo/NeMo/blob/main/nemo/collections/asr/metrics/wer.py
I still have a couple of questions:
- Is there an official inference/evaluation pipeline or script that was used to rerun the inference?
- The model card reports the FLEURS results, but I couldn't find results on LibriSpeech or other English benchmarks. Are those results available somewhere, or do you plan to release them?
Please also follow HF WER calculation and normalization script.
For English WER
Clone:
git clone https://github.com/huggingface/open_asr_leaderboard.git
python /path/to/hf_multilingual_wer_eval_latest.py
-i /path/to/output -l en
--reference-key text --prediction-key pred_text
--open-asr-repo-root /path/to/open_asr_leaderboard
To calc CER:
python /path/to/hf_multilingual_cer_eval.py
-i /path/to/output.json
-l ja
--reference-key text
--prediction-key predicted
--remove-spaces-auto
To calc WER for other languages, use the same script as follows:
python /path/to/hf_multilingual_wer_eval_latest.py
-i /path/to/output.json
-l
--reference-key text
--prediction-key pred_text