File size: 3,665 Bytes
2b48fb3 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 | ---
license: mit
library_name: whisperdrz
pipeline_tag: automatic-speech-recognition
tags:
- whisper
- speaker-diarization
- diarization
- speech-recognition
- asr
- word-timestamps
language:
- en
---
# WhisperDRZ
WhisperDRZ is a speaker-aware automatic speech recognition model. It transcribes
audio into text with **word-level timestamps**, **per-line speaker tags**, and
**non-speech event tags** ([laugh], [breath], ...). It is a Whisper-style
encoder-decoder model and handles long audio by chunking and stitching
internally.
- 🎙️ **Try it in the browser** — no install: **[fluxions.ai/transcribe](https://fluxions.ai/transcribe)**
- 💻 **Code (inference)**: <https://github.com/fluxions-ai/whisperdrz>
- 📖 **Full write-up** — approach, honest results, what didn't work: see the repo's `writeup.md`
## Usage
```bash
pip install git+https://github.com/fluxions-ai/whisperdrz
```
Requires Python 3.12+ and FFmpeg. Runs on a CUDA GPU or on CPU. Weights download
automatically from this repo on first use.
### Command line
```bash
whisperdrz audio.wav # defaults: this model, --lang en
whisperdrz audio.wav --output_format json > out.json
whisperdrz audio.wav --lang auto # auto-detect language
```
### Python
```python
import whisperdrz
from whisperdrz.audio import load_audio, SAMPLE_RATE
transcriber = whisperdrz.load_model("whisperdrz-large-v3.safetensors", lang="en")
audio, _ = load_audio("audio.wav", sample_rate=SAMPLE_RATE)
result = transcriber.transcribe(audio.mean(0)) # mono, 16 kHz
print(result.text) # speaker-tagged text with timestamps
print(result.segments) # list of {speaker, start, end, text}
```
You can also point `load_model` at this repo id (`fluxions/whisperdrz`) directly.
## Output format
Each line begins with a speaker tag. Timed words and tags are wrapped in a
start/end timestamp pair; the first and last word of each line are always timed:
```
[0] <|0.00|>Hello<|0.45|> there <|0.80|>world.<|1.10|>
[1] <|1.20|>Hi<|1.40|> <|1.45|>[laugh]<|1.60|> <|1.70|>there.<|1.95|>
```
- `[0]`, `[1]`, ... are speaker IDs; `[c]` marks crowd/ambient.
- `<|t|>` are timestamps in seconds (two decimals), always in a pair wrapping a word or tag.
- `[laugh]`, `[breath]`, and similar are non-speech event tags.
## Evaluation
Measured on this checkpoint:
| Benchmark | WER | DER (miss / FA / conf) | cpWER / tcpWER |
|---|---|---|---|
| ESB (English ASR, 1000 utts) | 9.6% macro / 5.9% micro | — | — |
| Internal conversational (26 clips) | 11.1% | — (WDER 33%) | 46% cpWER |
| VoxConverse dev (216, overlap-heavy) | — | **26.3%** (3.1 / 15.8 / 7.5) | — |
| CALLHOME eng (140, 2-spk telephone) | — | **38.6%** (5.9 / 14.0 / 18.7) | — |
| AMI test (16 meetings, Mix-Headset) | 22.8% | **50.6%** (10.6 / 28.9 / 11.1) | 72% / 84% |
DER is at the standard 0.25s collar, scoring overlapping speech. AMI
cpWER/tcpWER and the 22.8% WER use oracle speaker count.
**WhisperDRZ is an ASR-first model.** Transcription is strong across the board,
but diarization trails purpose-built systems (~10–25% DER) — speaker confusion
dominates on long, overlapping multi-party audio.
**No VAD.** It transcribes over silence and music, so false alarm is a large part
of DER. Gating to detected speech before scoring recovers much of it:
| Benchmark | DER (ungated) | + silero VAD | + oracle VAD |
|---|---|---|---|
| VoxConverse dev | 26.3% | 24.0% | 20.9% |
| CALLHOME eng | 38.6% | 34.2% | 28.6% |
| AMI test | 50.6% | 40.6% | 31.1% |
See the write-up in the code repo for full analysis (timing, non-speech events,
multilingual).
## License
MIT.
|