cjpais commited on
Commit
7f907ba
·
verified ·
1 Parent(s): a1aa6ab

Update WER table from new test-clean sweep (transcribe.cpp 4d2270e)

Browse files
Files changed (1) hide show
  1. README.md +6 -6
README.md CHANGED
@@ -132,12 +132,12 @@ OpenAI Whisper tiny — converted to GGUF for transcribe.cpp. Multilingual trans
132
 
133
  | Quantization | Download | Size | WER (LibriSpeech test-clean) |
134
  | --- | --- | ---: | ---: |
135
- | F32 | [whisper-tiny-F32.gguf](https://huggingface.co/handy-computer/whisper-tiny-gguf/resolve/main/whisper-tiny-F32.gguf) | 146 MB | 7.49% |
136
- | F16 | [whisper-tiny-F16.gguf](https://huggingface.co/handy-computer/whisper-tiny-gguf/resolve/main/whisper-tiny-F16.gguf) | 76 MB | 7.48% |
137
- | Q8_0 | [whisper-tiny-Q8_0.gguf](https://huggingface.co/handy-computer/whisper-tiny-gguf/resolve/main/whisper-tiny-Q8_0.gguf) | 44 MB | 7.52% |
138
- | Q6_K | [whisper-tiny-Q6_K.gguf](https://huggingface.co/handy-computer/whisper-tiny-gguf/resolve/main/whisper-tiny-Q6_K.gguf) | 43 MB | 7.54% |
139
- | Q5_K_M | [whisper-tiny-Q5_K_M.gguf](https://huggingface.co/handy-computer/whisper-tiny-gguf/resolve/main/whisper-tiny-Q5_K_M.gguf) | 42 MB | 7.82% |
140
- | Q4_K_M | [whisper-tiny-Q4_K_M.gguf](https://huggingface.co/handy-computer/whisper-tiny-gguf/resolve/main/whisper-tiny-Q4_K_M.gguf) | 42 MB | 7.78% |
141
 
142
  WER measured on the full LibriSpeech test-clean split (2620 utterances) with the transcribe.cpp default decode (greedy, suppress_tokens, temperature fallback, segment timestamps enabled). OpenAI's self-reported number on the same split is 7.54%. We don't know upstream's exact eval config, but the most likely cause of any divergence is that OpenAI's `model.generate()` defaults to `<|notimestamps|>` while transcribe.cpp's pipeline runs with timestamps enabled.
143
 
 
132
 
133
  | Quantization | Download | Size | WER (LibriSpeech test-clean) |
134
  | --- | --- | ---: | ---: |
135
+ | F32 | [whisper-tiny-F32.gguf](https://huggingface.co/handy-computer/whisper-tiny-gguf/resolve/main/whisper-tiny-F32.gguf) | 146 MB | 7.54% |
136
+ | F16 | [whisper-tiny-F16.gguf](https://huggingface.co/handy-computer/whisper-tiny-gguf/resolve/main/whisper-tiny-F16.gguf) | 76 MB | 7.49% |
137
+ | Q8_0 | [whisper-tiny-Q8_0.gguf](https://huggingface.co/handy-computer/whisper-tiny-gguf/resolve/main/whisper-tiny-Q8_0.gguf) | 44 MB | 7.53% |
138
+ | Q6_K | [whisper-tiny-Q6_K.gguf](https://huggingface.co/handy-computer/whisper-tiny-gguf/resolve/main/whisper-tiny-Q6_K.gguf) | 43 MB | 7.63% |
139
+ | Q5_K_M | [whisper-tiny-Q5_K_M.gguf](https://huggingface.co/handy-computer/whisper-tiny-gguf/resolve/main/whisper-tiny-Q5_K_M.gguf) | 42 MB | 7.63% |
140
+ | Q4_K_M | [whisper-tiny-Q4_K_M.gguf](https://huggingface.co/handy-computer/whisper-tiny-gguf/resolve/main/whisper-tiny-Q4_K_M.gguf) | 42 MB | 7.76% |
141
 
142
  WER measured on the full LibriSpeech test-clean split (2620 utterances) with the transcribe.cpp default decode (greedy, suppress_tokens, temperature fallback, segment timestamps enabled). OpenAI's self-reported number on the same split is 7.54%. We don't know upstream's exact eval config, but the most likely cause of any divergence is that OpenAI's `model.generate()` defaults to `<|notimestamps|>` while transcribe.cpp's pipeline runs with timestamps enabled.
143