|
|
| --- |
| language: |
| - multilingual |
| - ar |
| - as |
| - br |
| - ca |
| - cnh |
| - cs |
| - cv |
| - cy |
| - de |
| - dv |
| - el |
| - en |
| - eo |
| - es |
| - et |
| - eu |
| - fa |
| - fi |
| - fr |
| - hi |
| - hsb |
| - hu |
| - ia |
| - id |
| - ja |
| - ka |
| - ky |
| - lg |
| - lt |
| - ly |
| - mn |
| - mt |
| - nl |
| - or |
| - pl |
| - pt |
| - ro |
| - ru |
| - sah |
| - sl |
| - ta |
| - th |
| - tr |
| - tt |
| - uk |
| - vi |
| license: apache-2.0 |
| tags: |
| - audio |
| - automatic-speech-recognition |
| - hf-asr-leaderboard |
| - robust-speech-event |
| - speech |
| - xlsr-fine-tuning-week |
| datasets: |
| - common_voice |
| language_bcp47: |
| - fy-NL |
| - ga-IE |
| - pa-IN |
| - rm-sursilv |
| - rm-vallader |
| - sy-SE |
| - zh-CN |
| - zh-HK |
| - zh-TW |
| model-index: |
| - name: XLSR Wav2Vec2 for 56 language by Voidful |
| results: |
| - task: |
| type: automatic-speech-recognition |
| name: Speech Recognition |
| dataset: |
| name: Common Voice |
| type: common_voice |
| metrics: |
| - type: cer |
| value: 23.21 |
| name: Test CER |
| --- |
| |
| # Model Card for wav2vec2-xlsr-multilingual-56 |
| |
| |
| # Model Details |
| |
| ## Model Description |
| |
| - **Developed by:** voidful |
| - **Shared by [Optional]:** Hugging Face |
| - **Model type:** automatic-speech-recognition |
| - **Language(s) (NLP):** multilingual (*56 language, 1 model Multilingual ASR*) |
| - **License:** Apache-2.0 |
| - **Related Models:** |
| - **Parent Model:** wav2vec |
| - **Resources for more information:** |
| - [GitHub Repo](https://github.com/voidful/wav2vec2-xlsr-multilingual-56) |
| - [Model Space](https://huggingface.co/spaces/Kamtera/Persian_Automatic_Speech_Recognition_and-more) |
| |
| |
| # Uses |
| |
| |
| ## Direct Use |
| |
| This model can be used for the task of automatic-speech-recognition |
| |
| ## Downstream Use [Optional] |
| |
| More information needed |
| |
| ## Out-of-Scope Use |
| |
| The model should not be used to intentionally create hostile or alienating environments for people. |
| |
| # Bias, Risks, and Limitations |
| |
| Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups. |
| |
| |
| ## Recommendations |
| |
| Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations. |
| |
| |
| # Training Details |
| |
| ## Training Data |
| |
| See the [common_voice dataset card](https://huggingface.co/datasets/common_voice) |
| Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on 56 language using the [Common Voice](https://huggingface.co/datasets/common_voice). |
| |
| ## Training Procedure |
| |
| |
| ### Preprocessing |
| |
| More information needed |
| |
| ### Speeds, Sizes, Times |
| |
| |
| When using this model, make sure that your speech input is sampled at 16kHz. |
| |
| |
| # Evaluation |
| |
| |
| ## Testing Data, Factors & Metrics |
| |
| ### Testing Data |
| |
| More information needed |
| |
| ### Factors |
| |
| |
| ### Metrics |
| |
| More information needed |
| ## Results |
| <details> |
| <summary> Click to expand </summary> |
| |
| | Common Voice Languages | Num. of data | Hour | WER | CER | |
| |------------------------|--------------|--------|--------|-------| |
| | ar | 21744 | 81.5 | 75.29 | 31.23 | |
| | as | 394 | 1.1 | 95.37 | 46.05 | |
| | br | 4777 | 7.4 | 93.79 | 41.16 | |
| | ca | 301308 | 692.8 | 24.80 | 10.39 | |
| | cnh | 1563 | 2.4 | 68.11 | 23.10 | |
| | cs | 9773 | 39.5 | 67.86 | 12.57 | |
| | cv | 1749 | 5.9 | 95.43 | 34.03 | |
| | cy | 11615 | 106.7 | 67.03 | 23.97 | |
| | de | 262113 | 822.8 | 27.03 | 6.50 | |
| | dv | 4757 | 18.6 | 92.16 | 30.15 | |
| | el | 3717 | 11.1 | 94.48 | 58.67 | |
| | en | 580501 | 1763.6 | 34.87 | 14.84 | |
| | eo | 28574 | 162.3 | 37.77 | 6.23 | |
| | es | 176902 | 337.7 | 19.63 | 5.41 | |
| | et | 5473 | 35.9 | 86.87 | 20.79 | |
| | eu | 12677 | 90.2 | 44.80 | 7.32 | |
| | fa | 12806 | 290.6 | 53.81 | 15.09 | |
| | fi | 875 | 2.6 | 93.78 | 27.57 | |
| | fr | 314745 | 664.1 | 33.16 | 13.94 | |
| | fy-NL | 6717 | 27.2 | 72.54 | 26.58 | |
| | ga-IE | 1038 | 3.5 | 92.57 | 51.02 | |
| | hi | 292 | 2.0 | 90.95 | 57.43 | |
| | hsb | 980 | 2.3 | 89.44 | 27.19 | |
| | hu | 4782 | 9.3 | 97.15 | 36.75 | |
| | ia | 5078 | 10.4 | 52.00 | 11.35 | |
| | id | 3965 | 9.9 | 82.50 | 22.82 | |
| | it | 70943 | 178.0 | 39.09 | 8.72 | |
| | ja | 1308 | 8.2 | 99.21 | 62.06 | |
| | ka | 1585 | 4.0 | 90.53 | 18.57 | |
| | ky | 3466 | 12.2 | 76.53 | 19.80 | |
| | lg | 1634 | 17.1 | 98.95 | 43.84 | |
| | lt | 1175 | 3.9 | 92.61 | 26.81 | |
| | lv | 4554 | 6.3 | 90.34 | 30.81 | |
| | mn | 4020 | 11.6 | 82.68 | 30.14 | |
| | mt | 3552 | 7.8 | 84.18 | 22.96 | |
| | nl | 14398 | 71.8 | 57.18 | 19.01 | |
| | or | 517 | 0.9 | 90.93 | 27.34 | |
| | pa-IN | 255 | 0.8 | 87.95 | 42.03 | |
| | pl | 12621 | 112.0 | 56.14 | 12.06 | |
| | pt | 11106 | 61.3 | 53.24 | 16.32 | |
| | rm-sursilv | 2589 | 5.9 | 78.17 | 23.31 | |
| | rm-vallader | 931 | 2.3 | 73.67 | 21.76 | |
| | ro | 4257 | 8.7 | 83.84 | 21.95 | |
| | ru | 23444 | 119.1 | 61.83 | 15.18 | |
| | sah | 1847 | 4.4 | 94.38 | 38.46 | |
| | sl | 2594 | 6.7 | 84.21 | 20.54 | |
| | sv-SE | 4350 | 20.8 | 83.68 | 30.79 | |
| | ta | 3788 | 18.4 | 84.19 | 21.60 | |
| | th | 4839 | 11.7 | 141.87 | 37.16 | |
| | tr | 3478 | 22.3 | 66.77 | 15.55 | |
| | tt | 13338 | 26.7 | 86.80 | 33.57 | |
| | uk | 7271 | 39.4 | 70.23 | 14.34 | |
| | vi | 421 | 1.7 | 96.06 | 66.25 | |
| | zh-CN | 27284 | 58.7 | 89.67 | 23.96 | |
| | zh-HK | 12678 | 92.1 | 81.77 | 18.82 | |
| | zh-TW | 6402 | 56.6 | 85.08 | 29.07 | |
| |
| </details> |
| # Model Examination |
| |
| More information needed |
| |
| # Environmental Impact |
| |
| |
| Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700). |
| |
| - **Hardware Type:** More information needed |
| - **Hours used:** More information needed |
| - **Cloud Provider:** More information needed |
| - **Compute Region:** More information needed |
| - **Carbon Emitted:** More information needed |
| |
| # Technical Specifications [optional] |
| |
| ## Model Architecture and Objective |
| |
| More information needed |
| |
| ## Compute Infrastructure |
| |
| More information needed |
| |
| ### Hardware |
| |
| More information needed |
| |
| ### Software |
| More information needed |
| |
| # Citation |
| |
| |
| **BibTeX:** |
| ``` |
| More information needed |
| ``` |
| |
| **APA:** |
| ``` |
| More information needed |
| ``` |
| |
| # Glossary [optional] |
| More information needed |
| |
| # More Information [optional] |
| |
| More information needed |
| |
| # Model Card Authors [optional] |
| |
| voidful in collaboration with Ezi Ozoani and the Hugging Face team |
| |
| # Model Card Contact |
| |
| More information needed |
| |
| # How to Get Started with the Model |
| |
| Use the code below to get started with the model. |
| |
| <details> |
| <summary> Click to expand </summary> |
| |
| |
| ## Env setup: |
| ``` |
| !pip install torchaudio |
| !pip install datasets transformers |
| !pip install asrp |
| !wget -O lang_ids.pk https://huggingface.co/voidful/wav2vec2-xlsr-multilingual-56/raw/main/lang_ids.pk |
| ``` |
| |
| ## Usage |
|
|
| ``` |
| import torchaudio |
| from datasets import load_dataset, load_metric |
| from transformers import ( |
| Wav2Vec2ForCTC, |
| Wav2Vec2Processor, |
| AutoTokenizer, |
| AutoModelWithLMHead |
| ) |
| import torch |
| import re |
| import sys |
| import soundfile as sf |
| model_name = "voidful/wav2vec2-xlsr-multilingual-56" |
| device = "cuda" |
| processor_name = "voidful/wav2vec2-xlsr-multilingual-56" |
| |
| import pickle |
| with open("lang_ids.pk", 'rb') as output: |
| lang_ids = pickle.load(output) |
| |
| model = Wav2Vec2ForCTC.from_pretrained(model_name).to(device) |
| processor = Wav2Vec2Processor.from_pretrained(processor_name) |
| |
| model.eval() |
| |
| def load_file_to_data(file,sampling_rate=16_000): |
| batch = {} |
| speech, _ = torchaudio.load(file) |
| if sampling_rate != '16_000' or sampling_rate != '16000': |
| resampler = torchaudio.transforms.Resample(orig_freq=sampling_rate, new_freq=16_000) |
| batch["speech"] = resampler.forward(speech.squeeze(0)).numpy() |
| batch["sampling_rate"] = resampler.new_freq |
| else: |
| batch["speech"] = speech.squeeze(0).numpy() |
| batch["sampling_rate"] = '16000' |
| return batch |
| |
| |
| def predict(data): |
| features = processor(data["speech"], sampling_rate=data["sampling_rate"], padding=True, return_tensors="pt") |
| input_values = features.input_values.to(device) |
| attention_mask = features.attention_mask.to(device) |
| with torch.no_grad(): |
| logits = model(input_values, attention_mask=attention_mask).logits |
| decoded_results = [] |
| for logit in logits: |
| pred_ids = torch.argmax(logit, dim=-1) |
| mask = pred_ids.ge(1).unsqueeze(-1).expand(logit.size()) |
| vocab_size = logit.size()[-1] |
| voice_prob = torch.nn.functional.softmax((torch.masked_select(logit, mask).view(-1,vocab_size)),dim=-1) |
| comb_pred_ids = torch.argmax(voice_prob, dim=-1) |
| decoded_results.append(processor.decode(comb_pred_ids)) |
| |
| return decoded_results |
| |
| def predict_lang_specific(data,lang_code): |
| features = processor(data["speech"], sampling_rate=data["sampling_rate"], padding=True, return_tensors="pt") |
| input_values = features.input_values.to(device) |
| attention_mask = features.attention_mask.to(device) |
| with torch.no_grad(): |
| logits = model(input_values, attention_mask=attention_mask).logits |
| decoded_results = [] |
| for logit in logits: |
| pred_ids = torch.argmax(logit, dim=-1) |
| mask = ~pred_ids.eq(processor.tokenizer.pad_token_id).unsqueeze(-1).expand(logit.size()) |
| vocab_size = logit.size()[-1] |
| voice_prob = torch.nn.functional.softmax((torch.masked_select(logit, mask).view(-1,vocab_size)),dim=-1) |
| filtered_input = pred_ids[pred_ids!=processor.tokenizer.pad_token_id].view(1,-1).to(device) |
| if len(filtered_input[0]) == 0: |
| decoded_results.append("") |
| else: |
| lang_mask = torch.empty(voice_prob.shape[-1]).fill_(0) |
| lang_index = torch.tensor(sorted(lang_ids[lang_code])) |
| lang_mask.index_fill_(0, lang_index, 1) |
| lang_mask = lang_mask.to(device) |
| comb_pred_ids = torch.argmax(lang_mask*voice_prob, dim=-1) |
| decoded_results.append(processor.decode(comb_pred_ids)) |
| |
| return decoded_results |
| |
| |
| predict(load_file_to_data('audio file path',sampling_rate=16_000)) # beware of the audio file sampling rate |
| |
| predict_lang_specific(load_file_to_data('audio file path',sampling_rate=16_000),'en') # beware of the audio file sampling rate |
| |
| ``` |
| |
| ```python |
| {{ get_started_code | default("More information needed", true)}} |
| ``` |
| </details> |
|
|
|
|
