bezzam HF Staff commited on
Commit
0b36164
·
verified ·
1 Parent(s): 593ce35

Update with Transformers usage

Browse files
Files changed (1) hide show
  1. README.md +123 -1
README.md CHANGED
@@ -43,6 +43,7 @@ tags:
43
  - pytorch
44
  - NeMo
45
  - hf-asr-leaderboard
 
46
  widget:
47
  - example_title: Librispeech sample 1
48
  src: https://cdn-media.huggingface.co/speech_samples/sample1.flac
@@ -804,6 +805,7 @@ This model is ready for commercial/non-commercial use.
804
 
805
  For full details on the model architecture, training methodology, datasets, and evaluation results, check out the **[Technical Report](https://arxiv.org/abs/2509.14128)**.
806
 
 
807
  ## <span style="color:#466f00;">License/Terms of Use:</span>
808
 
809
  GOVERNING TERMS: Use of this model is governed by the [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/legalcode.en) license.
@@ -881,6 +883,10 @@ pip install -U nemo_toolkit['asr']
881
  ```
882
  The model is available for use in the NeMo toolkit [5], and can be used as a pre-trained checkpoint for inference or for fine-tuning on another dataset.
883
 
 
 
 
 
884
  #### Automatically instantiate the model
885
 
886
  ```python
@@ -945,6 +951,122 @@ python NeMo/main/examples/asr/asr_chunked_inference/rnnt/speech_to_text_streamin
945
 
946
  NVIDIA NIM for v2 parakeet model is available at [https://build.nvidia.com/nvidia/parakeet-tdt-0_6b-v2](https://build.nvidia.com/nvidia/parakeet-tdt-0_6b-v2).
947
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
948
  ## <span style="color:#466f00;">Software Integration:</span>
949
 
950
  **Runtime Engine(s):**
@@ -963,7 +1085,7 @@ NVIDIA NIM for v2 parakeet model is available at [https://build.nvidia.com/nvidi
963
 
964
  **Hardware Specific Requirements:**
965
 
966
- Atleast 2GB RAM for model to load. The bigger the RAM, the larger audio input it supports.
967
 
968
  #### Model Version
969
 
 
43
  - pytorch
44
  - NeMo
45
  - hf-asr-leaderboard
46
+ - Transformers
47
  widget:
48
  - example_title: Librispeech sample 1
49
  src: https://cdn-media.huggingface.co/speech_samples/sample1.flac
 
805
 
806
  For full details on the model architecture, training methodology, datasets, and evaluation results, check out the **[Technical Report](https://arxiv.org/abs/2509.14128)**.
807
 
808
+
809
  ## <span style="color:#466f00;">License/Terms of Use:</span>
810
 
811
  GOVERNING TERMS: Use of this model is governed by the [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/legalcode.en) license.
 
883
  ```
884
  The model is available for use in the NeMo toolkit [5], and can be used as a pre-trained checkpoint for inference or for fine-tuning on another dataset.
885
 
886
+ You can also run Parakeet TDT with [Transformers](https://github.com/huggingface/transformers) 🤗 (more below).
887
+
888
+ ### 1) NeMo usage
889
+
890
  #### Automatically instantiate the model
891
 
892
  ```python
 
951
 
952
  NVIDIA NIM for v2 parakeet model is available at [https://build.nvidia.com/nvidia/parakeet-tdt-0_6b-v2](https://build.nvidia.com/nvidia/parakeet-tdt-0_6b-v2).
953
 
954
+
955
+ ### 2) [Transformers](https://github.com/huggingface/transformers) 🤗 usage
956
+
957
+
958
+ Until Parakeet TDT is part of an official Transformers release, you can use it by installing from source.
959
+
960
+ ```bash
961
+ pip install git+https://github.com/huggingface/transformers
962
+ ```
963
+
964
+ <details>
965
+ <summary>➡️ Pipeline usage</summary>
966
+
967
+ ```python
968
+ from transformers import pipeline
969
+
970
+ pipe = pipeline("automatic-speech-recognition", model="nvidia/parakeet-tdt-0.6b-v3")
971
+ out = pipe("https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/bcn_weather.mp3")
972
+ print(out)
973
+ ```
974
+ </details>
975
+
976
+ <details>
977
+ <summary>➡️ AutoModel</summary>
978
+
979
+ ```python
980
+ from transformers import AutoModelForTDT, AutoProcessor
981
+ from datasets import load_dataset, Audio
982
+ import torch
983
+
984
+ device = "cuda" if torch.cuda.is_available() else "cpu"
985
+ num_samples = 3
986
+
987
+ model_id = "nvidia/parakeet-tdt-0.6b-v3"
988
+ processor = AutoProcessor.from_pretrained(model_id)
989
+ model = AutoModelForTDT.from_pretrained(model_id, dtype="auto", device_map=device)
990
+
991
+ ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
992
+ ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))
993
+ speech_samples = [el["array"] for el in ds["audio"][:num_samples]]
994
+
995
+ inputs = processor(speech_samples, sampling_rate=processor.feature_extractor.sampling_rate)
996
+ inputs.to(model.device, dtype=model.dtype)
997
+ output = model.generate(**inputs, return_dict_in_generate=True)
998
+ print(processor.decode(output.sequences, skip_special_tokens=True))
999
+ ```
1000
+ </details>
1001
+
1002
+
1003
+ <details>
1004
+ <summary>➡️ Timestamping</summary>
1005
+
1006
+ ```python
1007
+ from datasets import Audio, load_dataset
1008
+ from transformers import AutoModelForTDT, AutoProcessor
1009
+ import torch
1010
+
1011
+ device = "cuda" if torch.cuda.is_available() else "cpu"
1012
+ num_samples = 3
1013
+
1014
+ model_id = "nvidia/parakeet-tdt-0.6b-v3"
1015
+ processor = AutoProcessor.from_pretrained(model_id)
1016
+ model = AutoModelForTDT.from_pretrained(model_id, dtype="auto", device_map=device)
1017
+
1018
+ ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
1019
+ ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))
1020
+ speech_samples = [el["array"] for el in ds["audio"][:num_samples]]
1021
+
1022
+ inputs = processor(speech_samples, sampling_rate=processor.feature_extractor.sampling_rate)
1023
+ inputs.to(model.device, dtype=model.dtype)
1024
+ output = model.generate(**inputs, return_dict_in_generate=True)
1025
+ decoded_output, decoded_timestamps = processor.decode(
1026
+ output.sequences,
1027
+ durations=output.durations,
1028
+ skip_special_tokens=True,
1029
+ )
1030
+ print("Transcription:", decoded_output)
1031
+ print("Timestamped tokens:", decoded_timestamps)
1032
+ ```
1033
+ </details>
1034
+
1035
+ <details>
1036
+ <summary>➡️ Training</summary>
1037
+
1038
+ ```python
1039
+ from transformers import AutoModelForTDT, AutoProcessor
1040
+ from datasets import load_dataset, Audio
1041
+ import torch
1042
+
1043
+ device = "cuda" if torch.cuda.is_available() else "cpu"
1044
+
1045
+ model_id = "nvidia/parakeet-tdt-0.6b-v3"
1046
+ NUM_SAMPLES = 4
1047
+
1048
+ processor = AutoProcessor.from_pretrained(model_id)
1049
+ model = AutoModelForTDT.from_pretrained(model_id, dtype=torch.bfloat16, device_map=device)
1050
+ model.train()
1051
+
1052
+ ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
1053
+ ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))
1054
+ speech_samples = [el["array"] for el in ds["audio"][:NUM_SAMPLES]]
1055
+ text_samples = ds["text"][:NUM_SAMPLES]
1056
+
1057
+ # passing `text` to the processor will prepare inputs' `labels` key
1058
+ inputs = processor(audio=speech_samples, text=text_samples, sampling_rate=processor.feature_extractor.sampling_rate)
1059
+ inputs.to(device=model.device, dtype=model.dtype)
1060
+
1061
+ outputs = model(**inputs)
1062
+ print("Loss:", outputs.loss.item())
1063
+ outputs.loss.backward()
1064
+ ```
1065
+ </details>
1066
+
1067
+ For more details about usage, please refer to the [Transformers' documentation](https://huggingface.co/docs/transformers/en/model_doc/parakeet).
1068
+
1069
+
1070
  ## <span style="color:#466f00;">Software Integration:</span>
1071
 
1072
  **Runtime Engine(s):**
 
1085
 
1086
  **Hardware Specific Requirements:**
1087
 
1088
+ At least 2GB RAM for model to load. The bigger the RAM, the larger audio input it supports.
1089
 
1090
  #### Model Version
1091