SpeakPay: Domain-Adaptive LoRA Fine-Tuning of Whisper for Low-Resource Nepali Financial Speech Recognition
Paper • 2609.01737 • Published
Model Card for Whisper Large-v2 + LoRA (Nepali Financial)
This is a domain-adapted version of Whisper large-v2 fine-tuned with LoRA on the NepFinSpeech-403 dataset for low-resource Nepali financial speech recognition. It is designed to recognize spoken commands for mobile wallet operations (send, load, balance) and handles Devanagari numerals.
The model significantly reduces word error rate (WER) on financial utterances compared to the zero-shot baseline, and is deployed in the SpeakPay voice-first e-wallet application.
Performance
Evaluation on the held-out test set of NepFinSpeech-403:
| Model | WER% (↓) | CER% (↓) | NumAcc% (↑) |
|---|---|---|---|
| Whisper large-v2 (zero-shot) | 129.95 | 92.32 | 0.0 |
| Whisper small (general Nepali FT) | 106.32 | 63.48 | 0.0 |
| Whisper large-v2 + LoRA (ours) | 42.58 | 16.95 | 73.9 |
Relative WER reduction of 67.2% over the zero-shot baseline, with improvements on 59/60 test utterances (sign test, p = 3.5×10⁻¹⁸). Transaction Success Rate improves from 1.67% to 33.33%.
Usage
This is a PEFT adapter and must be loaded together with the base model openai/whisper-large-v2. For usage examples and full training/inference pipeline, please refer to the SpeakPay GitHub repository.
Model Details
- Base model:
openai/whisper-large-v2 - Fine-tuning method: LoRA (rank 32, alpha 64)
- Task: Automatic speech recognition (Nepali financial domain)
- Language: Nepali
- License: MIT (code), CC-BY 4.0 (dataset)
Model Sources
- Repository: github.com/subedibiraj/speakpay
- Paper: SpeakPay: Domain-Adaptive LoRA Fine-Tuning of Whisper for Low-Resource Nepali Financial Speech Recognition
- Demo: speakpay.biraj-subedi.com.np
- Dataset: huggingface.co/datasets/birajsubedi/NepFinSpeech
Evaluation
The model was evaluated on the held‑out test set of NepFinSpeech, containing 60 utterances spanning all command types and numeral ranges. Metrics reported are word error rate (WER), character error rate (CER), and Devanagari numeral accuracy (NumAcc). Full analysis (including confusion patterns and data‑efficiency sweeps) is available in the paper and in the analysis/ directory of the repository.
Training Procedure
The LoRA adapter was fine-tuned using the training/ pipeline from the SpeakPay repository. Hyperparameters and details can be found in the repository documentation.
Framework versions
- PEFT 0.6.2
Citation
@misc{subedi2026speakpay,
title = {SpeakPay: Domain-Adaptive LoRA Fine-Tuning of Whisper for
Low-Resource Nepali Financial Speech Recognition},
author = {Subedi, Biraj},
year = {2026},
url = {https://github.com/subedibiraj/speakpay}
}
More Information
- Downloads last month
- 23
Model tree for birajsubedi/whisper-large-v2-nepali-financial
Base model
openai/whisper-large-v2