| --- |
| license: mit |
| datasets: |
| - Isma/alffa_wolof |
| language: |
| - wo |
| metrics: |
| - wer |
| base_model: |
| - facebook/mms-1b |
| pipeline_tag: automatic-speech-recognition |
| --- |
| # wav2vec2-large-mms-1b-wolof |
|
|
| This model is a fine-tuned version of [facebook/mms-1b-all](https://huggingface.co/facebook/mms-1b-all) on the **Isma/alffa_wolof** dataset. It is designed to perform automatic speech recognition (ASR) in the Wolof language. |
| |
| ## Model description |
| |
| This model is based on the Wav2Vec 2.0 architecture, which has been fine-tuned for speech recognition tasks. The base model, **facebook/mms-1b-all**, was trained on a multilingual corpus for general-purpose ASR. This fine-tuned version has been specifically trained on the **Waxal Wolof** dataset, which contains audio recordings in the Wolof language. |
| |
| ## Training and evaluation data |
| |
| The model was trained on the **Isma/alffa_wolof** dataset, which contains audio samples in the Wolof language. This dataset is used to fine-tune the model to improve accuracy on the specific phonetic characteristics of Wolof speech. |
|
|
| ## Inference manually |
|
|
| ```python |
| ! pip install datasets |
| |
| # Load test dataset |
| from datasets import load_dataset, Audio |
| |
| dataset = load_dataset("perrynelson/waxal-wolof", trust_remote_code=True) |
| dataset |
| |
| # Display the first audio using Ipython |
| from IPython.display import Audio, display |
| |
| Audio(dataset['train'][322]['audio']['array'], rate=16000) |
| |
| from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor |
| import torch |
| |
| model_id = "bilalfaye/wav2vec2-large-mms-1b-wolof" |
| |
| device = torch.device("cuda" if torch.cuda.is_available() else "cpu") |
| |
| # Load the model on CPU first |
| model = Wav2Vec2ForCTC.from_pretrained(model_id, |
| target_lang="wol", |
| torch_dtype=torch.float16 # Use half-precision |
| ).to(device) |
| |
| |
| processor = Wav2Vec2Processor.from_pretrained(model_id) |
| processor.tokenizer.set_target_lang("wol") |
| |
| |
| # Process the audio |
| input_dict = processor( |
| dataset['train'][322]["audio"]["array"], |
| sampling_rate=16_000, |
| return_tensors="pt", |
| padding=True |
| ) |
| |
| # Move inputs to the appropriate device for the first processing layer |
| input_values = input_dict.input_values.to(device, dtype=torch.float16) |
| |
| # Perform inference |
| logits = model(input_values).logits |
| |
| # Decode predictions |
| pred_ids = torch.argmax(logits, dim=-1)[0] |
| |
| print("Prediction:") |
| print(processor.decode(pred_ids)) |
| |
| print("\nReference:") |
| print(dataset['train'][322]['transcription'].lower()) |
| ``` |
|
|
| ## Inference with pipeline |
|
|
| ```python |
| from transformers import pipeline |
| import torch |
| |
| # Model ID |
| model_id = "bilalfaye/wav2vec2-large-mms-1b-wolof" |
| |
| # Determine device (use GPU if available, otherwise fallback to CPU) |
| device = 0 if torch.cuda.is_available() else -1 |
| |
| # Use half precision (float16) for inference if GPU is available |
| torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32 |
| |
| # Set up the pipeline for automatic speech recognition |
| pipe = pipeline( |
| task="automatic-speech-recognition", |
| model=model_id, |
| processor=model_id, |
| device=device, # Specify the device (GPU if available, otherwise CPU) |
| torch_dtype=torch_dtype, # Set the precision (float16 for half precision, float32 otherwise) |
| framework="pt" # Use PyTorch as the framework |
| ) |
| |
| # Input audio processing |
| audio_array = dataset['train'][322]["audio"]["array"] # Fetching an audio sample |
| |
| # Run inference |
| result = pipe(audio_array) |
| |
| # Prediction |
| print("Prediction:") |
| print(result['text']) |
| |
| # Reference (for comparison) |
| print("\nReference:") |
| print(dataset['train'][322]['transcription'].lower()) |
| ``` |
|
|
| --- |
|
|
| ## Free memory |
|
|
| ```python |
| import gc |
| import torch |
| import psutil |
| |
| # Free up unused memory in CUDA (GPU) - only needed if you use a GPU |
| if torch.cuda.is_available(): |
| torch.cuda.empty_cache() # Clears GPU memory cache |
| torch.cuda.reset_peak_memory_stats() # Resets memory stats |
| |
| # Collect any unused memory in Python (CPU) |
| gc.collect() # Collect unused memory in Python's garbage collector |
| |
| # Optionally, check memory status after clearing |
| if torch.cuda.is_available(): |
| print(f"GPU Memory Allocated: {torch.cuda.memory_allocated()} bytes") |
| print(f"GPU Memory Cached: {torch.cuda.memory_reserved()} bytes") |
| else: |
| print(f"CPU Memory Usage: {psutil.virtual_memory().percent}%") |
| ``` |
|
|
| --- |
|
|
| ### Training hyperparameters |
|
|
| The following hyperparameters were used during training: |
| - learning_rate: 0.0001 |
| - train_batch_size: 16 |
| - eval_batch_size: 8 |
| - seed: 42 |
| - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 |
| - lr_scheduler_type: linear |
| - lr_scheduler_warmup_steps: 500 |
| - num_epochs: 20 |
| - mixed_precision_training: Native AMP |
| |
| ### Training results |
| |
| | Training Loss | Epoch | Step | Validation Loss | Wer | |
| |:-------------:|:-----:|:-----:|:---------------:|:------:| |
| | 0.3793 | 14.0 | 12250 | 0.1517 | 0.1888 | |
| | 0.3709 | 15.0 | 13125 | 0.1512 | 0.1882 | |
| | 0.3702 | 16.0 | 14000 | 0.1499 | 0.1858 | |
| | 0.367 | 17.0 | 14875 | 0.1492 | 0.1848 | |
| | 0.3656 | 18.0 | 15750 | 0.1493 | 0.1842 | |
| |
| |
| ### Framework versions |
| |
| - Transformers 4.41.2 |
| - Pytorch 2.4.0+cu121 |
| - Datasets 3.2.0 |
| - Tokenizers 0.19.1 |
| |
| --- |
| |
| ## Intended uses & limitations |
| |
| - **Intended uses**: This model is intended for speech-to-text tasks in Wolof. It can be used to transcribe audio recordings in Wolof into written text. |
| - **Limitations**: This model performs best with clean audio and may struggle with noisy or low-quality recordings. It is designed specifically for the Wolof language and may not work well with other languages. |
| |
| |
| |
| ### Author Information |
| |
| - **Author**: Bilal FAYE |
