Jianshu001/wavlm-word-v13c2

WavLM Word V13c2 — English Pronunciation Scoring

Single-word pronunciation scoring model trained on 860K real user evaluations (children English learning). Given an audio clip and the expected word, predicts a 0–100 score that approximates an English-evaluation engine's overall output.

Architecture

WavLM-Large (top 12 layers unfrozen) + phone embeddings as cross-attention queries + multi-task head:

audio (16 kHz mono)                   → WavLM-Large → audio_frames (B, T_a, 1024)
expected phone IDs (≤16 phones)       → phone_emb (50 → 256 → 1024)  → ph_q
ph_q × audio_frames (cross-attention) → per-phone repr (B, T_p, 1024)
per-phone repr → per-phone score head (regression, 0..100)
per-phone repr + audio attention pool → word score head (regression, 0..100)

Trained with stratified sampling over 5 score bands (0–20, 20–40, 40–60, 60–80, 80–100), Huber loss δ=5, L = 1.0·L_word + 0.8·L_phone.

Performance — held-out 0521 validation set (99,857 samples)

Metric	V9 baseline	V13c2
MAE (lower better)	37.98	12.63
Pearson corr	0.665	0.819
P(|err|<5)	6.1%	29.2%
P(|err|<10)	—	51.6%

Per-band P(|err|<5)

GT band	n	MAE	P<5
0–20	19,879	15.74	15.7%
20–40	20,095	12.41	32.4%
40–60	19,724	13.23	21.9%
60–80	19,926	11.58	30.2%
80–100	20,233	10.25	45.2%

Files

• wavlm_word_v13c2.pt — model checkpoint (state_dict + phone_vocab + training args + metrics)
• train_v13_multitask.py — training script with model class definition

Usage

import torch
import sys, os
# Make sure train_v13_multitask.py is on the path so the model class can be imported
sys.path.insert(0, "/path/to/checkpoint/directory")
from train_v13_multitask import V13Model

ckpt = torch.load("wavlm_word_v13c2.pt", map_location="cpu", weights_only=False)
phone_vocab = ckpt["phone_vocab"]
phone_to_id = {p: i for i, p in enumerate(phone_vocab)}

model = V13Model(n_phones=len(phone_vocab), unfreeze_top_n=12).eval().cuda()
model.load_state_dict(ckpt["model_state"])

# Prepare a single example:
import soundfile as sf, numpy as np, torch.nn.functional as F
wav, sr = sf.read("user_word.mp3", dtype="float32")
if wav.ndim > 1: wav = wav.mean(axis=1)
if sr != 16000:
    idxs = np.linspace(0, len(wav)-1, int(len(wav)*16000/sr)).astype(np.int64)
    wav = wav[idxs]
wav = wav[:40000]                        # truncate to 2.5s
wav = np.pad(wav, 4800, mode="constant") # 0.3s silence each side
wav = (wav - wav.mean()) / (wav.std() + 1e-7)
mask = np.ones_like(wav, dtype=np.float32)

# Expected phones for "street"  (use your phone tokenizer; vocab listed in ckpt['phone_vocab'])
phones = ["s", "tr", "iy", "t"]
phone_ids = [phone_to_id.get(p, 1) for p in phones]
# Pad to MAX_PHONES=16
phone_ids += [0] * (16 - len(phone_ids))
ph_mask = [1.0] * len(phones) + [0.0] * (16 - len(phones))

with torch.no_grad():
    word_pred, phone_pred = model(
        torch.FloatTensor(wav).unsqueeze(0).cuda(),
        torch.FloatTensor(mask).unsqueeze(0).cuda(),
        torch.LongTensor([phone_ids]).cuda(),
        torch.FloatTensor([ph_mask]).cuda(),
    )
print(f"word score: {float(word_pred):.1f}")
print(f"per-phone scores: {phone_pred[0][:len(phones)].tolist()}")

Phone vocabulary

48 IPA-derived phone symbols + 2 special tokens (<pad>, <unk>); full list is ckpt['phone_vocab'].

Training data

• 860,465 real user word-evaluation samples from a children English learning platform
• Sources: 0423 (100K), 0512 (500K), 0519 (~260K)
• Each sample = (audio mp3, expected word text, engine overall 0–100, per-phone scores)
• Score distribution: mean 60.9, std 25.6

Limitations

• The engine score we approximate has known noise on some labels; even an oracle predictor saturates around MAE 0.2.
• Current model plateaus at P<5 ≈ 29% — for the goal of "80% of samples within 5 points", architectural changes (e.g. TTS-reference Siamese, larger backbone, or per-phone classification) are needed.
• Mostly children's voices in training data; performance on adult speakers untested.

Citation

If you use this checkpoint:

@misc{she2026wavlmwordv13c2,
  title  = {WavLM Word V13c2: WavLM-Large + phone cross-attention for English pronunciation scoring},
  author = {She, Jianshu},
  year   = {2026},
  url    = {https://huggingface.co/Jianshu001/wavlm-word-v13c2},
}