---
language:
- sl
license: apache-2.0
library_name: transformers
tags:
- automatic-speech-recognition
- speech
- whisper
- slovenian
- seq2seq
datasets:
- mozilla-foundation/common_voice_17_0
- yuriyvnv/synthetic_asr_et_sl
base_model: openai/whisper-large-v3
pipeline_tag: automatic-speech-recognition
model-index:
- name: whisper-large-v3-slovenian
  results:
  - task:
      type: automatic-speech-recognition
      name: Speech Recognition
    dataset:
      name: Common Voice 17.0 (sl) - Test
      type: mozilla-foundation/common_voice_17_0
      config: sl
      split: test
    metrics:
    - type: wer
      value: 15.65
      name: WER (raw)
    - type: wer
      value: 13.24
      name: WER (normalized)
    - type: cer
      value: 3.88
      name: CER (raw)
  - task:
      type: automatic-speech-recognition
      name: Speech Recognition
    dataset:
      name: FLEURS (sl) - Test
      type: google/fleurs
      config: sl_si
      split: test
    metrics:
    - type: wer
      value: 40.46
      name: WER (raw)
    - type: wer
      value: 24.52
      name: WER (normalized)
    - type: cer
      value: 11.98
      name: CER (raw)
---

# Whisper-large-v3 Slovenian

Fine-tuned [OpenAI Whisper-large-v3](https://huggingface.co/openai/whisper-large-v3) for Slovenian automatic speech recognition, augmented with TTS-generated synthetic data.

This model is part of the paper: *"Synthetic Speech Augmentation for Low-Resource Estonian and Slovenian ASR: Comparing Parakeet-TDT and Whisper"* (Interspeech 2026). Paper coming soon.

> **Note:** Fine-tuning Whisper on CommonVoice data degrades FLEURS performance for Slovenian. Zero-shot Whisper (37.02% raw / 19.18% normalized) outperforms all fine-tuned variants on FLEURS. This is because Whisper's pre-trained model handles FLEURS-style punctuation and casing, which fine-tuning on CV-style data erodes. On the in-domain CV17 test set, this model provides substantial improvements over both zero-shot and CV-only fine-tuning.

## Model Description

- **Architecture:** Encoder-decoder Transformer (Whisper)
- **Parameters:** 1.55B
- **Tokenizer:** 51,865-token byte-level BPE
- **Base model:** `openai/whisper-large-v3`
- **Fine-tuning data:** CommonVoice 17.0 Slovenian + ~5,850 synthetic sentences (paraphrase + domain expansion, without morphological category)
- **Training config:** CV + Synth No Morph (synthetic corpus excluding morphological augmentation)

## Evaluation Results

### Raw WER/CER (no text normalization)

| Test Set | WER | CER |
|----------|-----|-----|
| CommonVoice 17 Test | **15.65** | **3.88** |
| CommonVoice 17 Val | **15.08** | **3.60** |
| FLEURS Test | 40.46 | 11.98 |

### Normalized WER/CER (lowercase + punctuation removal)

| Test Set | WER | CER |
|----------|-----|-----|
| CommonVoice 17 Test | **13.24** | **3.48** |
| CommonVoice 17 Val | **12.98** | **3.24** |
| FLEURS Test | 24.52 | 8.32 |

### Improvement over baselines

| Comparison | CV17 Test (WER) | FLEURS Test (WER) |
|-----------|-----------------|-------------------|
| vs. Zero-shot | -5.55 pp | +3.44 pp* |
| vs. CV-only fine-tuning | -3.66 pp | -6.33 pp |

*FLEURS degrades vs. zero-shot due to domain mismatch (see note above), but improves substantially vs. CV-only fine-tuning.

All CV17 improvements are statistically significant (paired bootstrap, p < 0.001, n = 100,000).

## Usage

```python
from transformers import WhisperForConditionalGeneration, WhisperProcessor
import torch

# Load model and processor
model = WhisperForConditionalGeneration.from_pretrained("yuriyvnv/whisper-large-v3-slovenian")
processor = WhisperProcessor.from_pretrained("yuriyvnv/whisper-large-v3-slovenian")

# Load audio (16kHz mono)
import librosa
audio, sr = librosa.load("audio.wav", sr=16000)

# Transcribe
input_features = processor(audio, sampling_rate=16000, return_tensors="pt").input_features
with torch.no_grad():
    predicted_ids = model.generate(input_features, language="sl", task="transcribe")
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
print(transcription)
```

## Training Details

- **Optimizer:** AdamW fused (lr=5e-5)
- **Schedule:** Linear decay with 10% warmup
- **Effective batch size:** 128 (64 x 2 gradient accumulation)
- **Epochs:** 5
- **Best model:** selected by eval_loss
- **Precision:** bf16
- **Seed:** 42

## Synthetic Data Augmentation

The synthetic training data was generated using a three-stage pipeline:
1. **Text generation:** GPT-5-mini generates diverse sentences across paraphrase, domain expansion, and morphological categories
2. **LLM-as-judge validation:** Each sentence validated for grammaticality, naturalness, and language purity
3. **Speech synthesis:** OpenAI gpt-4o-mini-tts with 11-voice rotation

Dataset: [yuriyvnv/synthetic_asr_et_sl](https://huggingface.co/datasets/yuriyvnv/synthetic_asr_et_sl)

## Acknowledgments

- Base model: [OpenAI Whisper-large-v3](https://huggingface.co/openai/whisper-large-v3)
- Training data: [Mozilla Common Voice 17.0](https://commonvoice.mozilla.org/)
- Evaluation: [Google FLEURS](https://huggingface.co/datasets/google/fleurs)