---
language:
- vi
license: apache-2.0
base_model: openai/whisper-small
tags:
- whisper
- asr
- vietnamese
- peft
- lora
- audio
datasets:
- mozilla-foundation/common_voice_11_0
metrics:
- wer
pipeline_tag: automatic-speech-recognition
---

# Whisper Small — Vietnamese (LoRA Fine-tuned)

Fine-tuned version of [openai/whisper-small](https://huggingface.co/openai/whisper-small) on Vietnamese speech using LoRA adapters and the Mozilla Common Voice 11 dataset.

## Training Results

| Metric | Value |
|---|---|
| Training Loss | 0.9382 |
| Epochs | 5 |
| Global Steps | 470 |
| Samples/sec | 7.37 |
| Total FLOPs | 4.60e+18 |

## Model Details

- **Base model:** `openai/whisper-small` (244M params)
- **Method:** LoRA (Low-Rank Adaptation)
- **Trainable params:** ~13M (5.09% of base)
- **Target modules:** `q_proj`, `v_proj`, `k_proj`, `out_proj`, `fc1`, `fc2`
- **LoRA rank:** 32 · **alpha:** 64 · **dropout:** 0.05
- **Language:** Vietnamese (`vi`)
- **Task:** Transcription

## Training Details

- **Dataset:** Mozilla Common Voice 11.0 (`vi`)
- **Learning rate:** 1e-4 with linear warmup (500 steps)
- **Batch size:** 8 × 2 gradient accumulation = effective 16
- **Precision:** FP16
- **Framework:** 🤗 Transformers + PEFT

**Data augmentation applied:**
- Speed perturbation ±10% (p=0.3)
- Additive Gaussian noise (p=0.3)

## Usage

```python
from peft import PeftModel
from transformers import WhisperForConditionalGeneration, WhisperProcessor
import torch

base = WhisperForConditionalGeneration.from_pretrained("openai/whisper-small")
model = PeftModel.from_pretrained(base, "LakoreAI/whisper-small-vi-lora")
processor = WhisperProcessor.from_pretrained("LakoreAI/whisper-small-vi-lora")

# Optional: merge LoRA for faster inference
model = model.merge_and_unload()
model.eval()

# Inference
def transcribe(audio_array, sampling_rate=16000):
    inputs = processor(audio_array, sampling_rate=sampling_rate, return_tensors="pt")
    with torch.no_grad():
        ids = model.generate(
            inputs.input_features,
            language="vietnamese",
            task="transcribe",
            max_new_tokens=225,
        )
    return processor.tokenizer.decode(ids[0], skip_special_tokens=True)
```

## Limitations

- Optimized for Vietnamese only; other languages will degrade significantly
- Common Voice data skews toward read speech; spontaneous/accented speech may perform worse
- Short clips (<1s) or clipped audio may cause hallucinations