README.md

---
license: apache-2.0
license_name: mixed-per-model
license_link: LICENSE
library_name: onnx
pipeline_tag: audio-classification
language:
  - en
tags:
  - end-of-turn-detection
  - turn-taking
  - voice-agents
  - speech
  - onnx
  - from-scratch
datasets:
  - pipecat-ai/smart-turn-data-v3.2-train
  - Scicom-intl/semantic-vad-eot
  - li2017dailydialog/daily_dialog
metrics:
  - roc_auc
  - average_precision
model-index:
  - name: TurnWave
    results:
      - task:
          type: audio-classification
          name: End-of-turn detection
        dataset:
          type: livekit/eot-bench-data
          name: eot-bench (English)
          split: validation
        metrics:
          - type: roc_auc
            value: 0.77
            name: AUC
          - type: average_precision
            value: 0.602
            name: Average precision
          - type: false_cutoff_rate
            value: 42.1
            name: False cutoffs @300ms latency budget (%)
          - type: false_cutoff_rate
            value: 17.2
            name: False cutoffs @600ms latency budget (%)
---

# TurnWave — end-of-turn detection for voice agents

Decides whether a caller has **finished speaking** or is only pausing, so a voice
agent neither interrupts them nor leaves an awkward silence. It replaces the fixed
300–700 ms silence timeout most pipelines still use.

**Trained from scratch — no pretrained weights anywhere.** A causal transformer
(RoPE, RMSNorm, SwiGLU) over the transcript tail, and a CNN over log-mel
spectrograms for prosody. Even the log-mel front end is hand-built on `torch.stft`,
so there is no torchaudio or librosa dependency.

## Benchmark

Scored by [LiveKit's eot-bench](https://github.com/livekit/eot-bench) harness on
real human-to-agent conversation, using their code and published baselines. Lower
is better; **bold marks the best per column.**

| model | false cutoffs @300 ms ↓ | @600 ms ↓ | latency @5% cutoff ↓ |
|---|---|---|---|
| VAD baseline | 55.6% | 21.7% | 1600 ms |
| **TurnWave audio branch (this model)** | 42.1% | 17.2% | 1195 ms |
| SmartTurn v3.2 | 35.2% | 14.8% | 1051 ms |
| LiveKit Turn Detector v1 | **9.9%** | **4.5%** | **543 ms** |

TurnWave beats the VAD baseline on every metric the harness reports.

## Models in this repo

| file | licence | training data |
|---|---|---|
| `audio_eot_v2.onnx` | apache-2.0 | trained on smart-turn conversational clips |
| `audio_eot.onnx` | cc-by-4.0 | Phase 4; trained on semantic-vad-eot (CC BY 4.0) |
| `text_eot.int8.onnx` | cc-by-nc-sa-4.0 | trained on DailyDialog (CC BY-NC-SA 4.0) — non-commercial |
| `fusion_eot.onnx` | cc-by-nc-sa-4.0 | contains the text branch, so it inherits the same terms |

Each model's licence follows its training data, so they differ. `audio_eot_v2` is
the one the benchmark above measures and the one to use.

## Usage

```python
from huggingface_hub import hf_hub_download
from turnwave.infer import TurnDetector   # pip install git+https://github.com/Nikhils-G/turnwave

detector = TurnDetector(hf_hub_download("Nikhil-09/turnwave", "audio_eot_v2.onnx"))
if detector.predict(audio=wav_16k) > 0.5:
    respond()
```

16 kHz mono. The model reads the last 2 seconds ending at the decision point, which
sits 0.2 s into the pause — where a live agent decides, and where eot-bench scores.

| model | variant | CPU latency | size |
|---|---|---|---|
| audio_eot_v2 | fp32 | 4.81 ms | 14.0 MB |
| audio_eot | fp32 | 4.52 ms | 14.0 MB |
| text_eot | int8 | 5.02 ms | 7.2 MB |
| fusion_eot | fp32 | 9.35 ms | 42.4 MB |

INT8 is not applied blindly: dynamic quantization rewrites MatMul, so it speeds up
the transformer and *slows down* the conv-heavy branches. Each model ships whichever
variant measured faster.

## What this project found

The first version of this model scored **AP 0.945** on its own held-out test set and
**AUC 0.563** on eot-bench — barely above random. The policy sweep chose thresholds
of 0.0 and 1.0, meaning *ignore the model entirely*.

The cause was the training corpus, not the architecture. It derived from a dataset
whose own card declares `task_categories: [text-to-speech]` — read speech, whose
pauses are reading hesitations rather than conversational turn-yields. The model had
learned *"has this sentence finished being read aloud."*

Retraining on conversational data, changing nothing else, lifted AUC to **0.770**.
The in-domain score could never have revealed this; only a benchmark on data we did
not build could.

## Limitations

- **English only.** Other languages are in the training data but untested here.
- **Behind the production models**, and not a fair comparison: SmartTurn starts from
  a pretrained Whisper encoder, LiveKit's is a fine-tuned 0.5B LLM distilled from a
  7B teacher. This is 3.49M parameters from random initialisation.
- **The fusion model is stale.** It was trained on the read-speech corpus, which the
  benchmark showed to be the wrong task. The conversational corpus has no
  transcripts, so retraining fusion needs ASR first.
- **Non-commercial models included.** The text and fusion models derive from
  DailyDialog (CC BY-NC-SA 4.0). Only the audio branches are permissively licensed.

Code, training scripts, and the full write-up: **https://github.com/Nikhils-G/turnwave**