---
language: en
license: mit
tags:
  - pronunciation-assessment
  - phoneme-scoring
  - wavlm
  - speech
  - children-speech
base_model: microsoft/wavlm-large
pipeline_tag: audio-classification
---

# WavLM Phoneme Scorer

Fine-tuned **WavLM-Large** backbone + MLP scoring head for phoneme-level English pronunciation assessment, trained on 11K children's speech samples.

Given a reference text and an audio recording, this model identifies which phonemes are pronounced correctly and which have errors.

## Quick Start

### Install

```bash
pip install torch torchaudio transformers g2p-en huggingface_hub
```

### Python API

```python
from pipeline_v2 import PronunciationAssessorV2

# Auto-download model from HuggingFace
assessor = PronunciationAssessorV2.from_pretrained()

result = assessor.assess("audio.mp3", "Hello, Peter.")

print(result["overall_score"])        # 85.2
print(result["n_errors"])             # 0
for word in result["words"]:
    print(f"{word['word']}: score={word['score']}")
    for ph in word["phonemes"]:
        err = " <- ERROR" if ph["error"] else ""
        print(f"  /{ph['phone']}/  score={ph['score']}  pherr={ph['pherr_prob']:.2f}{err}")
```

### CLI

```bash
# Model downloads automatically on first run
python pipeline_v2.py --audio audio.mp3 --text "Hello, Peter."
```

Output:
```
============================================================
Text: "Hello, Peter."
Overall Score: 85.2/100  (errors: 0/8)
============================================================

  ✓ Hello            score= 87.7  errors=0/4
      /hh  /  score= 98.6  GOP= -0.97  pherr=0.05
      /ah  /  score= 73.1  GOP= -7.40  pherr=0.43
      /l   /  score= 88.6  GOP= +4.00  pherr=0.29
      /ow  /  score= 90.7  GOP= -6.05  pherr=0.13

  ✓ Peter            score= 82.6  errors=0/4
      /p   /  score= 95.7  GOP= +5.40  pherr=0.08
      /iy  /  score= 90.2  GOP= +3.70  pherr=0.12
      /t   /  score= 72.5  GOP= +0.50  pherr=0.55
      /er  /  score= 71.8  GOP= -1.40  pherr=0.61
```

### Download Model Manually

```bash
# Via huggingface-cli
huggingface-cli download Jianshu001/wavlm-phoneme-scorer wavlm_finetuned.pt --local-dir .

# Via Python
from huggingface_hub import hf_hub_download
hf_hub_download(repo_id="Jianshu001/wavlm-phoneme-scorer", filename="wavlm_finetuned.pt", local_dir=".")

# Then use with local path
assessor = PronunciationAssessorV2(checkpoint_path="wavlm_finetuned.pt")
```

## Architecture

```
Reference Text ──→ G2P ──→ Expected phoneme sequence
                                    │
Audio ──→ CTC model ──→ Viterbi Forced Alignment ──→ Frame segments
  │                                                       │
  └──→ WavLM-Large (fine-tuned) ──→ Hidden states ──→ Pool per segment
                                                          │
                                                    + phone embedding (32d)
                                                    + GOP score (1d)
                                                    + n_frames (1d)
                                                          │
                                                    MLP (1058 → 512 → 512 → 256)
                                                    ├── score_head → phoneme score (0-100)
                                                    └── pherr_head → error probability (0-1)
```

- **Backbone**: WavLM-Large with top 6 transformer layers fine-tuned (76.6M trainable / 316.4M total)
- **Alignment**: `facebook/wav2vec2-xlsr-53-espeak-cv-ft` (frozen, CTC-based Viterbi forced alignment)
- **Scoring head**: MLP with BatchNorm, GELU, Dropout, multi-task (score regression + error classification)

## Performance

Evaluated on test set (8062 phonemes from 1727 audio files, children's speech):

| Metric | GOP Baseline (v1.0) | This Model |
|--------|-------------------|------------|
| Phoneme Error AUC-ROC | 0.738 | **0.870** |
| Phoneme Error F1 | 0.476 | **0.595** |
| Phoneme Error Precision | 0.379 | **0.592** |
| Phoneme Error Recall | 0.638 | **0.598** |
| Phone Score Pearson | 0.372 | **0.645** |
| Phone Score MAE | 27.44 | **16.47** |

## Training Data

- 11,601 audio recordings of English learners (children)
- 53,926 phonemes with professional human evaluation labels
- Labels include per-phoneme scores (0-100) and error flags (pherr 0/1)

## Training Details

- Fine-tuned top 6 of 24 WavLM transformer layers
- Differential learning rate: backbone 1e-5, head 5e-4
- AdamW optimizer, cosine annealing, early stopping (patience=8)
- Gradient accumulation (4 steps), gradient clipping (max_norm=1.0)
- Batch size 64, trained for 24 epochs (early stopped)
- Multi-task loss: MSE/100 (score) + BCE with pos_weight=5.2 (error)
- Train/Val/Test split by audio file: 40K/5K/8K phonemes

## Files

| File | Description | Size |
|------|-------------|------|
| `wavlm_finetuned.pt` | Full checkpoint (backbone + head state dict) | 1.2GB |
| `pipeline_v2.py` | Inference pipeline with `from_pretrained()` support | 18KB |
| `finetune_wavlm.py` | Training script (reproducing the fine-tuning) | 25KB |

## Full Repository

See the complete project (data, evaluation, all experiments) at: [github.com/Jianshu-She/Voice-correction](https://github.com/Jianshu-She/Voice-correction) (branch: `feature/wavlm-pipeline`)