README.md

---
license: mit
language:
- en
tags:
- automatic-speech-recognition
- whisper
- replication
- educational
- research-artifact
- not-for-production
datasets:
- librispeech_asr
metrics:
- wer
library_name: pytorch
pipeline_tag: automatic-speech-recognition
model-index:
- name: whispr-100h
  results:
  - task:
      type: automatic-speech-recognition
      name: Automatic Speech Recognition
    dataset:
      name: LibriSpeech dev-clean
      type: librispeech_asr
      split: dev-clean
    metrics:
    - type: wer
      value: 103.4
      name: WER
      verified: false
---

# whispr — a from-scratch Whisper replication (these models do not work)

> ### ⚠️ Read this before downloading
>
> **These models do not transcribe speech.** Word error rate is **103%** and
> **150%** — worse than outputting nothing. They emit fluent English that loops
> and largely ignores the audio.
>
> That is not a bug report, it is the result. They are published so the numbers
> in the accompanying write-up are checkable and so nobody has to repeat five
> hours of GPU time to reproduce them.
>
> **If you want to transcribe audio, use [`openai/whisper-tiny`](https://huggingface.co/openai/whisper-tiny)**
> — same architecture, 6,800× more training data, and it actually works.
>
> These are also **not `transformers`-compatible.** `AutoModel.from_pretrained`
> will not load them. See [Usage](#usage).

## What this is

A from-scratch replication of
[*Robust Speech Recognition via Large-Scale Weak Supervision*](https://arxiv.org/abs/2212.04356)
(Whisper), built and trained on an M1 MacBook Air as a way of learning audio ML
from the ground up.

Code, a written note explaining every step, and the full method:
**https://github.com/HitendraKawale/whispr_replication**

The architecture is verified faithful: OpenAI's real `whisper-tiny` checkpoint
loads into this implementation with `strict=True` and produces **bit-identical**
outputs (max difference `0.0`). The log-mel frontend matches
`whisper.log_mel_spectrogram` to `1.2e-7`. What differs is the training data —
100 hours instead of 680,000.

## The checkpoints

| File | Train audio | Steps | Best val loss | WER (unseen speakers) |
|---|---|---|---|---|
| `whispr-100h-step20000.pt` | 100.3 h `train-clean-100` | 20,000 (~6 epochs) | 4.028 | **103.4%** |
| `whispr-3.7h-step1250.pt` | 3.7 h `dev-clean` | 1,250 (~4.7 epochs) | 5.574 | **149.9%** |

Each `.pt` requires its matching `tokenizer-*.json`. The vocabulary is fitted
per-run on that run's training transcripts, so the token ids differ between
them — **pairing a checkpoint with the wrong tokenizer decodes to nonsense
silently**, without raising.

~18M parameters each (Whisper Tiny's shape: 4 layers, width 384, 6 heads, with a
2,048-token vocabulary instead of GPT-2's 50,257).

## What they actually output

On speakers they have never heard:

```
REF  SHE GOT UP ON HER KNEES AND WRUNG HER HANDS
HYP  I'LL NOT THINK IT'T YOU'T DO YOU'T                                  (3.7 h)

REF  I SAW THAT INTERVIEW IN THE PAPER YESTERDAY TELLING WHERE YOU WERE
HYP  AND THE OTHER TWO OF THE MEN WHO WERE IN THE SAME WAY THE OLD MAN   (100 h)
```

WER above 100% is possible because insertions are unbounded — the model emits
more words than the reference.

## Why they fail, which is the interesting part

The decoder learns two things at once: the **distribution of English text**, and
the **mapping from audio to text**. The first is cheap — a few thousand
transcripts teach you that "THE" is common and follows almost anything, and it
requires no encoder at all. The second is expensive.

So gradient descent does the cheap thing first. Cross-entropy falls from 7.6 to
~5.5 almost entirely by learning unigrams and bigrams, then plateaus, because
further progress needs the encoder to become useful.

At 3.7 hours the model never gets there and overfits — validation traced a
textbook U, bottoming at step 1,250 then climbing while training loss kept
falling. At 100 hours the U disappears: validation was still falling when the
schedule ended, with training loss tracking it closely. The problem changed from
**overfitting to underfitting**, which is a better problem — it says the next win
is more epochs and more data, not more regularisation.

Whisper used 680,000 hours. That gap is the paper's central claim, measured from
the wrong end.

## Usage

These use a custom model class, not `transformers`.

```bash
git clone https://github.com/HitendraKawale/whispr_replication
cd whispr_replication && uv sync
```

```python
import torch
from huggingface_hub import hf_hub_download

from whispr.audio import load_audio
from whispr.config import AudioConfig, Config, ModelConfig, TrainConfig
from whispr.decode import Decoder
from whispr.mel import log_mel_spectrogram
from whispr.model import build_model
from whispr.tokenizer import WhisprTokenizer

REPO = "HitendraKawale/whispr-replication"
ckpt_path = hf_hub_download(REPO, "whispr-100h-step20000.pt")
tok_path = hf_hub_download(REPO, "tokenizer-100h.json")   # must match!

ckpt = torch.load(ckpt_path, map_location="cpu", weights_only=False)
saved = ckpt["config"]
cfg = Config(
    audio=AudioConfig(**saved["audio"]),
    model=ModelConfig(**saved["model"]),
    train=TrainConfig(**saved["train"]),
)

model = build_model(cfg.model)
model.load_state_dict(ckpt["model"])
model.eval()

tokenizer = WhisprTokenizer.load(tok_path)
mel = log_mel_spectrogram(load_audio("clip.flac"), pad_to=cfg.audio.n_samples)
print(Decoder(model, tokenizer).greedy(mel)[0].text)
```

Each checkpoint carries its own config, so you don't have to remember that the
100 h model uses a 17 s window (850 encoder positions) and the 3.7 h model uses
15 s (750). Loading one with the other's config fails loudly.

### Resuming training

Both include optimizer state, so training continues with the LR schedule intact:

```bash
cp whispr-100h-step20000.pt checkpoints/run_100h/best.pt
uv run python scripts/07_train.py --corpus train-clean-100 --window 17 \
    --steps 20000 --mel-cache --resume best.pt
```

## Training details

Optimiser settings follow the paper's Table 17 exactly: AdamW, β=(0.9, 0.98),
ε=1e-6, weight decay 0.1, gradient-norm clip 1.0, linear warmup then linear decay
to zero, Gaussian fan-in initialisation.

Deviations, all forced by a single laptop and all documented with reasoning in
`whispr/config.py`:

| | Paper | Here |
|---|---|---|
| Training audio | 680,000 h, multilingual | 100.3 h English read speech |
| Updates | 1,048,576 @ batch 256 | 20,000 @ batch 8 |
| Vocabulary | 50,257 (GPT-2 BPE) | 2,048, fitted on our transcripts |
| Window | 30 s (1500 enc positions) | 17 s (850) |
| Augmentation | none | ±6 dB gain jitter |
| Tasks | transcribe + translate + langID + VAD | English transcription only |

Trained on an Apple M1 (16 GB) via MPS. Roughly 6 hours for the 100 h run.

## Evaluation

Speaker-disjoint throughout. The 100 h model trains on `train-clean-100` and
validates on `dev-clean` — LibriSpeech's own partition, so the number is
comparable to published results rather than only to itself. Greedy decoding, 400
held-out utterances, WER computed corpus-level (not averaged per utterance) after
Whisper-style text normalisation.

## Limitations

Everything. Specifically: English only, read audiobook speech only, no
punctuation or casing (LibriSpeech labels have none), no timestamps, no
translation, no robustness to noise or accents or spontaneous speech, and it does
not produce correct transcripts of anything.

## Reproducibility

MPS kernels are nondeterministic, so identical seeds do not give identical runs.
An earlier 100 h run of this same code reached val 3.876 / 110.2% WER using a
25,000-step schedule. Expect the qualitative findings to reproduce and the third
decimal not to.

## Citation

The paper being replicated:

```bibtex
@article{radford2022whisper,
  title   = {Robust Speech Recognition via Large-Scale Weak Supervision},
  author  = {Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg
             and McLeavey, Christine and Sutskever, Ilya},
  journal = {arXiv preprint arXiv:2212.04356},
  year    = {2022}
}
```

## Licence

MIT, matching the repository. LibriSpeech is CC BY 4.0.