README.md

---
license: apache-2.0
library_name: transformers
pipeline_tag: audio-classification
language:
  - en
base_model:
  - kyutai/mimi
tags:
  - turn-taking
  - endpointing
  - end-of-turn
  - voice-activity-detection
  - voice-agents
  - conversation
  - speech
  - audio
  - mimi
  - dualturn
  - onnx
  - real-time
arxiv: 2603.08216
---

# DualTurn Endpointing

A small, **causal, audio-only** turn-taking model for voice agents. It listens to **both channels** of a
live conversation (user + agent) and emits, frame-by-frame at 12.5 Hz, the signals a voice agent needs to
decide when to speak — with **end-of-turn (endpointing)** as the primary output.

Adapted from the **DualTurn** paper ([arXiv:2603.08216](https://arxiv.org/abs/2603.08216)) and fine-tuned
for the **endpointing head**. It runs on frozen [Mimi](https://huggingface.co/kyutai/mimi) features, adds
only **~1.5M parameters** on top of the encoder, needs **no ASR and no cloud**, and is designed to run
**on-device / on CPU**.

## How it's meant to be used

The model **sits inside a live two-channel call** between a **user** (CH0) and the **agent** (CH1). It
continuously listens to **both** channels at the same time and, every 80 ms, emits per-frame predictions:
**VAD** and **FVAD** for *both* speakers, and the **user's end-of-turn (EOT)**. A voice agent consumes
these — chiefly the user's EOT — to decide *when it's the agent's turn to speak* (and VAD/FVAD to sense
who's talking and who's about to talk). The agent channel is not just context: hearing the agent's own
speech (and overlaps/interruptions) is what makes the user-EOT judgment reliable in a real conversation.

It can also run as a **single-stream (user-only) detector**: feed the user audio and **silence the agent
channel** (pass mono, or a zeros agent channel). You still get the user's EOT + user VAD/FVAD; you simply
lose the agent-context benefit. This is the setup used for benchmarking and for user-mic-only deployments.

| Output | Shape | Channels | Meaning |
|---|---|---|---|
| `eot_probs` | `[1, T, 1]` | **user** | P(user has finished their turn) — the endpointing signal |
| `vad_probs` | `[1, T, 2]` | user, agent | P(speaking) |
| `fvad_probs` | `[1, T, 2, 4]` | user, agent | P(channel speaks within a future window) — 4 horizons: 0–240 / 240–640 / 640–1200 / 1200–2000 ms |

Frames are **80 ms (12.5 Hz)**; `T ≈ audio_seconds × 12.5`. Streaming/causal — outputs at time *t* use
only audio up to *t*.

## Inference modes & latency

Four ready-to-use paths (all **fp32, same weights**). **Offline** processes a whole file at once;
**streaming** runs per 80 ms tick for live calls. ONNX paths need only `onnxruntime` (no PyTorch).

| # | mode | runtime | latency (CPU, 4 threads) | deps |
|---|---|---|---|---|
| 1 | Offline | PyTorch — `model(wav, sr)` | ~7.5 ms/frame (batch, ~11× real-time) | torch + transformers |
| 2 | Streaming | PyTorch — `model.streaming()` | **~63 ms / 80 ms tick** (real-time, flat) | torch + transformers |
| 3 | Offline | ONNX — `model.onnx` | ~7.3 ms/frame (~11× real-time) | onnxruntime |
| 4 | Streaming | ONNX — `stream_tick.onnx` | **~52 ms / 80 ms tick** (real-time, flat, tightest tail) | onnxruntime |

*Offline latency is amortized batch throughput (all frames returned after the file is processed); streaming
latency is the real per-tick cost in a live call — both well under the 80 ms budget. Streaming is causal +
bounded (transformer KV capped at 250 frames = 10 s), so per-tick latency stays flat over arbitrarily long
calls. Streaming outputs are decision-equivalent to offline — they match the batch model to within ~0.02–0.03
(fp32 op-order drift in the incremental path), which doesn't move threshold+sustain endpoint decisions.*

### Inference interval (compute ↔ latency)

Two streaming ONNX graphs are shipped — pick by how often you need to react vs how much CPU you want to
spend. **Both are strictly causal** (the transformer runs frame-by-frame *inside* the graph — no future
audio) and **decision-equivalent to offline** — they track the batch model to within ~0.02–0.03 (fp32
op-order drift in the streaming path; the two graphs are bit-identical to *each other*, Δ≈0). The 240 ms
graph runs one shared encoder pass over 3 frames instead of three separate passes, so it's ~1.8× cheaper
per second of audio:

- **`stream_tick.onnx`** — 1 frame per call (`[2, 1920]` = 80 ms).
- **`stream_tick_240.onnx`** — 3 frames per call (`[2, 5760]` = 240 ms), returns all 3 × 80 ms frames.

Measured on a real call (CPU, 4 threads, carried KV cache). **Grace** = idle time left in the interval after
inference (`interval − latency`); **CPU busy** = per-stream duty cycle, fraction of wall-clock the inference
runs to keep up with one live stream (`latency × calls/sec`):

| graph | infer every | latency / call | calls / audio-sec | grace / call | CPU busy / stream |
|---|---|---|---|---|---|
| `stream_tick.onnx` | **80 ms** | 52 ms | 12.5 | 28 ms | ~65% |
| `stream_tick_240.onnx` | **240 ms** | 85 ms | 4.17 | 155 ms | ~36% |

- **80 ms — snappiest, but heaviest.** Each call eats 52 ms → only **28 ms grace**, pool busy **~65%** per
  stream. Fastest reactions, least room for anything else.
- **240 ms — much cheaper, lots of slack.** Each call takes 85 ms (vs 3 × 52 = 156 ms for three 80 ms ticks)
  but leaves **155 ms grace** and sits at **~36% busy**.
- **The real win is at scale.** CPU per second of audio drops **~1.8×** (650 → 354 ms/s, ≈45% less), so **one
  machine holds roughly ~1.8× as many concurrent calls** — the per-stream % looks modest, but that factor
  multiplies across every simultaneous call in your fleet. *(`CPU busy %` is a single-stream duty cycle
  measured with 4 threads — read it as relative cost between graphs, not absolute single-core load.)*
- **Accuracy is unaffected** — the 240 ms graph is decision-equivalent to the 80 ms one (endpoint decisions
  match); the only cost is that you learn the turn ended at the next 240 ms boundary (up to one interval later).

## 1. Offline — PyTorch

```python
import torch, torchaudio
from transformers import AutoModel

model = AutoModel.from_pretrained("anyreach-ai/dualturn-endpointing", trust_remote_code=True).eval()

wav, sr = torchaudio.load("conversation.wav")   # [2, T]  CH0 = user, CH1 = agent
with torch.no_grad():
    out = model(wav, sr=sr)

out.eot_probs    # [1, T, 1]      P(user end-of-turn)
out.vad_probs    # [1, T, 2]      P(speaking) — (user, agent)
out.fvad_probs   # [1, T, 2, 4]   P(speaks soon) — (user, agent) × (0-240/240-640/640-1200/1200-2000 ms)
```

Turn the end-of-turn stream into a decision with a thin policy — e.g. fire when `eot_probs` stays above a
threshold for a short sustain window:

```python
eot = out.eot_probs[0, :, 0]
thr, sustain = 0.5, 3                       # 3 frames ≈ 240 ms
fired = (eot > thr).unfold(0, sustain, 1).all(-1).nonzero()   # frame indices of detected turn-ends
```

Mono input `[T]` or `[1, T]` is also accepted (treated as user with a silent agent — a single-stream
fallback). Any sample rate works (`sr=`); audio is internally resampled to **24 kHz** (Mimi's input rate).
The call above is the **offline** mode (whole file at once). For live calls, use the **streaming** mode below.

## 2. Streaming — PyTorch (real-time)

For live calls, don't re-run the whole file each tick — use the stateful streamer. It's **causal** (uses
only past audio), keeps **bounded state** (transformer KV window + LSTM state), and runs **~63 ms per
80 ms tick on CPU** (≥4 threads), **flat regardless of call length**.

```python
from transformers import AutoModel
model = AutoModel.from_pretrained("anyreach-ai/dualturn-endpointing", trust_remote_code=True).eval()
streamer = model.streaming()

# feed audio as it arrives, in 80 ms chunks: [2, 1920] @ 24 kHz  (CH0=user, CH1=agent)
for chunk in chunks_at_24kHz:            # mono [1920] is also accepted (silent agent)
    out = streamer.push(chunk)           # None until the next 12.5 Hz frame is ready
    if out is not None:
        p_eot = out.eot_probs[0, -1, 0].item()      # latest P(user end-of-turn)
        # fire your end-of-turn policy on p_eot (threshold + short sustain)

streamer.reset()                         # between calls
```

`push` returns a `DualTurnOutput` (same `eot_probs` / `vad_probs` / `fvad_probs`) for the new frame(s).
It matches the offline model to ~3e-4 mean H1. Feed **24 kHz** (resample the continuous stream once, not
per-chunk, to avoid chunk-edge artifacts).

## 3. Offline — ONNX (no PyTorch)

A self-contained `model.onnx` (Mimi encoder + readout in one graph) is included for CPU / on-device
deployment with just `onnxruntime`. Input is a **2-channel waveform at 24 kHz** `[2, T]` (CH0=user,
CH1=agent); resample first with any tool.

```python
import numpy as np, librosa, onnxruntime as ort
from huggingface_hub import hf_hub_download

onnx_path = hf_hub_download("anyreach-ai/dualturn-endpointing", "model.onnx")
sess = ort.InferenceSession(onnx_path, providers=["CPUExecutionProvider"])

wav, sr = librosa.load("conversation.wav", sr=24000, mono=False)   # [2, T] @ 24 kHz, CH0=user CH1=agent
eot, vad, fvad = sess.run(None, {"audio_24k": wav.astype("float32")})
# eot  [1, T', 1]      P(user end-of-turn)
# vad  [1, T', 2]      P(speaking) — (user, agent)
# fvad [1, T', 2, 4]   P(speaks soon) — (user, agent) × (0-240/240-640/640-1200/1200-2000 ms)

eot = eot[0, :, 0]
fired = np.where(np.convolve((eot > 0.5).astype(int), np.ones(3), "valid") == 3)[0]   # sustained ≥240 ms
```

Runs comfortably faster than real-time on CPU (≈1 s of compute per 10 s of audio, single-threaded). The
graph is fully self-contained — no `torch` or `transformers` needed at inference.

## 4. Streaming — ONNX (real-time, no PyTorch)

Torch-free real-time streaming via `stream_tick.onnx` + the `onnx_streaming.py` helper (`DualTurnONNXStreamer`).
Each 80 ms tick only computes the **new frame** — the graph carries all state (conv buffer, transformer **KV
window**, downsample history, LSTM state) in/out, so latency is low (**~52 ms/tick**) and **flat over long
calls**. Decision-equivalent to the PyTorch model.

```python
from onnx_streaming import DualTurnONNXStreamer     # download onnx_streaming.py + stream_tick.onnx from the repo

streamer = DualTurnONNXStreamer()                    # auto-downloads stream_tick.onnx (via huggingface_hub)
for chunk in chunks_24k:                             # each: [2, 1920] float32 @ 24 kHz (CH0=user, CH1=agent; mono OK)
    out = streamer.push(chunk)
    p_eot = out["eot"]                               # P(user end-of-turn); also out["vad"] [2], out["fvad"] [2,4]
    # fire your end-of-turn policy on p_eot (threshold + short sustain)
streamer.reset()                                     # between calls
```

Needs only `onnxruntime` + `numpy` (+ `huggingface_hub` for the auto-download). Feed **24 kHz** audio in
**exactly 1920-sample (80 ms)** chunks.

To spend ~1.8× less CPU (at the cost of reacting every 240 ms instead of 80 ms), use the **240 ms graph**
`stream_tick_240.onnx` via `DualTurnONNXStreamer240` — same helper file, `[2, 5760]` chunks, returns the 3
× 80 ms frames per call. Both graphs are strictly causal and decision-equivalent to offline; see [Inference interval](#inference-interval-compute--latency).

```python
from onnx_streaming import DualTurnONNXStreamer240      # downloads stream_tick_240.onnx

streamer = DualTurnONNXStreamer240()
for chunk in chunks_240ms:                               # each: [2, 5760] float32 @ 24 kHz
    for fr in streamer.push(chunk):                      # list of 3 frames: fr["eot"], fr["vad"], fr["fvad"]
        ...                                              # apply your end-of-turn policy on fr["eot"]
streamer.reset()
```

## Performance

Benchmarked on **LiveKit's [eot-bench](https://huggingface.co/datasets/livekit/eot-bench-data)** (a neutral
end-of-turn benchmark). Metric is a latency / false-cutoff Pareto: **FC@N ms** = % of mid-turn pauses
falsely fired on at an N-ms latency budget (**lower is better**); **Lat@X%** = ms of dead-air after a true
turn-end at an X% false-cutoff budget (**lower is better**). Our harness reproduces LiveKit's published
SmartTurn numbers exactly, so rows are directly comparable; DualTurn is scored single-stream
(agent = silence) — an honest handicap. English slice:

| # | Model | Params | FC@300 ↓ | FC@600 ↓ | Lat@5% ↓ | Lat@10% ↓ | Runs on |
|---|---|---|---:|---:|---:|---:|---|
| 1 | LiveKit Turn Detector v1 | undisclosed | 9.9% | 4.5% | 543 ms | 295 ms | ☁️ Cloud |
| 2 | Deepgram Flux | undisclosed | 12.9% | 9.9% | 1151 ms | 548 ms | ☁️ Cloud |
| **3** | **DualTurn Endpointing (this model)** | **~1.5M** | **21.8%** | **10.2%** | 1115 ms | 610 ms | ✅ **CPU / GPU** |
| 4 | ultraVAD (Ultravox / Llama-8B) | ~8B | 27.7% | 11.9% | 899 ms | 663 ms | 🖥️ GPU |
| 5 | LiveKit Turn Detector v1-mini | undisclosed | 27.8% | 12.1% | 1070 ms | 698 ms | ✅ CPU |
| 6 | SmartTurn v3.2 | ~8M | 35.2% | 14.8% | 1051 ms | 739 ms | ✅ CPU |
| 7 | AssemblyAI | undisclosed | 49.4% | 14.6% | 1049 ms | 713 ms | ☁️ Cloud |
| 8 | VAD baseline (Silero) | ~0.3–1.5M | 55.6% | 21.7% | 1600 ms | 1000 ms | ✅ CPU |

**DualTurn Endpointing is the best on-device / audio-only turn detector on the board** — beating LiveKit's
own on-device v1-mini, SmartTurn v3.2, and the open 8B ultraVAD, at ~1.5M params without transcription or
cloud. Only closed cloud end-of-turn services lead. It holds the same #1-on-device rank on **zero-shot
Spanish** (not trained on Spanish), evidence that turn-taking here is language-agnostic acoustics.

## How it works
Adapted from the DualTurn paper ([arXiv:2603.08216](https://arxiv.org/abs/2603.08216)) and fine-tuned for endpointing: both channels are encoded by a **frozen
Mimi** speech encoder into 12.5 Hz continuous features; a small **causal** readout (~1.5M params) produces
the per-frame turn-taking probabilities above. Audio-only (no ASR), streaming, and light enough for CPU /
on-device inference. The Mimi encoder is downloaded automatically from `kyutai/mimi` on first use.

## Training & evaluation
- **Method.** Built on the approach introduced in the [DualTurn paper](https://arxiv.org/abs/2603.08216) (frozen dual-channel Mimi
  features → small causal turn-taking readout), specialized here for user end-of-turn endpointing.
- **Training data.** The two public dual-channel (human–human) corpora used in the DualTurn paper —
  **Switchboard** and **OtoSpeech** — plus a large **private dual-channel dataset** of real AI-agent phone
  calls (the production domain this model targets).
- **Evaluation.** Measured on our **internal private test set** (held-out real agent calls; the recall /
  interval and latency numbers above) and, for an external neutral comparison, on **LiveKit's
  [eot-bench](https://huggingface.co/datasets/livekit/eot-bench-data)** (reported in
  [Performance](#performance) above; other systems' rows are LiveKit's published numbers).

## Files
- `model.safetensors` — the ~1.5M-param readout + input-standardization stats (PyTorch paths).
- `config.json` — with `auto_map` for `trust_remote_code`.
- `modeling_dualturn.py` — PyTorch model + both PyTorch modes: **offline** (`model(wav, sr)`) and **streaming** (`model.streaming()`).
- `model.onnx` — self-contained **offline** ONNX graph (Mimi encoder + readout).
- `stream_tick.onnx` — self-contained **streaming** ONNX graph (one 80 ms tick, state in/out).
- `stream_tick_240.onnx` — 240 ms streaming graph (3 frames/tick, ~1.8× cheaper/sec of audio, strictly causal).
- `onnx_streaming.py` — `DualTurnONNXStreamer` (80 ms) + `DualTurnONNXStreamer240` (240 ms) helpers (pure `onnxruntime` + `numpy`).

## Notes & limitations
- This is a **perception** model (it emits turn-taking *signals*, not agent actions) — pair it with a thin
  decision policy.
- **`eot_probs` is the user's end-of-turn** (this is an endpointing model). `vad_probs` and `fvad_probs`
  cover **both** channels (user, agent).
- Best in conversational voice-agent (telephone) audio, the domain it was trained on.
- **Deps by path:** PyTorch modes (1, 2) need `torch` + `torchaudio` + `transformers`; ONNX modes (3, 4)
  need only `onnxruntime` + `numpy` (+ `librosa`/`soundfile` for loading, `huggingface_hub` for download).

## Author
[Shangeth Rajaa](https://github.com/shangeth) — Senior ML Research Scientist, Anyreach AI.

## Citation
```bibtex
@misc{rajaa2026dualturnlearningturntakingdualchannel,
      title={DualTurn: Learning Turn-Taking from Dual-Channel Generative Speech Pretraining},
      author={Shangeth Rajaa},
      year={2026},
      eprint={2603.08216},
      archivePrefix={arXiv},
      primaryClass={eess.AS},
      url={https://arxiv.org/abs/2603.08216},
}
```