---
language:
- en
tags:
- turn-taking
- voice-activity-detection
- speech
- conversation
- audio
- dualturn
- endpointing
base_model:
- anyreach-ai/dualturn-qwen2.5-mimi-0.5B
license: apache-2.0
---

# anyreach-ai/dualturn-endpointing

Real-time speech endpoint detector for two-channel (user + agent) audio.

Built on top of [DualTurn-Qwen2.5-Mimi-0.5B](https://huggingface.co/anyreach-ai/dualturn-qwen2.5-mimi-0.5B) with a trained endpoint classifier that answers one question at every VAD offset:

> **Has the user finished speaking?**  
> `endpoint=True` — yes, agent should respond now  
> `endpoint=False` — no, user just paused mid-sentence, keep waiting

| Output | Shape | Description |
|--------|-------|-------------|
| `vad_probs` | `[B, T, 2]` | P(speaking now) — `[:,0]`=user `[:,1]`=agent |
| `eot_probs` | `[B, T, 2]` | P(end of turn) per channel |
| `bot_probs` | `[B, T, 2]` | P(beginning of turn) per channel |
| `fvad_probs` | `[B, T, 4]` | Fast VAD (Silero) — user_short, user_long, agent_short, agent_long |
| `endpoint_probs` | `[B, T, 1]` | P(user finished) at VAD-offset frames, 0 elsewhere |
| `endpoints` | `list[dict]` | Sparse decisions at VAD-offset anchors with `endpoint` bool and `p_endpoint` |

Frame rate: **12.5 Hz** (80 ms per frame). Audio resampled to 24 kHz internally.

---

## Inference

```bash
pip install transformers torch torchaudio joblib scikit-learn silero-vad huggingface_hub
```

```python
import torch, torchaudio
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "anyreach-ai/dualturn-endpointing",
    trust_remote_code=True,
)
model.eval()

wav, sr = torchaudio.load("conversation.wav")  # [2, T]  CH0=user  CH1=agent

with torch.no_grad():
    out = model(wav, sr=sr)

# Per-frame signals at 12.5 Hz
print(out.vad_probs.shape)    # [1, T, 2]   P(speaking) — (user, agent)
print(out.eot_probs.shape)    # [1, T, 2]   P(end-of-turn)
print(out.bot_probs.shape)    # [1, T, 2]   P(begin-of-turn)
print(out.fvad_probs.shape)   # [1, T, 4]   fast VAD

# Sparse decisions — one per VAD offset (user stops speaking)
for ep in out.endpoints:
    print(f"t={ep['t_s']:.2f}s  endpoint={ep['endpoint']}  P={ep['p_endpoint']:.3f}")
    # ep["endpoint"]   →  True  = user finished, agent should respond
    #                     False = user paused mid-sentence, keep waiting
    # ep["p_endpoint"] →  P(user is done)  threshold = 0.30
```

---

## Endpoint decision logic

```
Dual-channel audio (24 kHz stereo)
      │
      ▼  Mimi encoder + DualTurn backbone  (every 80 ms)
Per-frame signals:
  vad_user, vad_agent, eot_user, eot_agent,
  bot_user, bot_agent, fvad_*_short, fvad_*_long
      │
      ▼  watch vad_user crossing 0.5
      │
  VAD offset detected (user stopped) + agent silent?
      │
      ▼  endpoint_clf.predict_proba(10 signal values)
      P(endpoint) >= 0.30  →  endpoint=True   agent should respond now
      P(endpoint) <  0.30  →  endpoint=False  user paused mid-sentence, wait
```

Threshold **0.30** tuned to maximise endpoint recall (99% recall on held-out test set).

---

## Files

| File | Description |
|------|-------------|
| `best.pt` | DualTurn backbone weights (two-stream transformer, 34M params) |
| `endpoint_clf.pkl` | Endpoint classifier — sklearn bundle with trained model + recommended threshold |
| `modeling_dualturn.py` | `DualTurnModel(PreTrainedModel)` — AutoModel-compatible wrapper |
| `configuration_dualturn.py` | `DualTurnConfig` |
| `config.json` | `auto_map` for AutoModel/AutoConfig |
| `endpointing.py` | Higher-level `DualTurnEndpointing` class with streaming support |
| `src/` | Bundled dualturn + evaluation source code |

---

## Streaming (real-time)

```python
from endpointing import DualTurnEndpointing

model  = DualTurnEndpointing.from_pretrained("anyreach-ai/dualturn-endpointing")
stream = model.stream(user_channel=0, agent_channel=1)

# Feed 80 ms of 24 kHz stereo PCM float32 every tick
# chunk shape: (2, 1920)
for chunk in audio_source():
    result = stream.push(chunk)
    if result:
        if result["endpoint"]:
            agent.start_responding()
        # result["p_endpoint"] — P(user finished)  ∈ [0, 1]
        # result["signals"]    — all 10 signal values at the anchor
```

---

## Authors

* Shangeth Rajaa — Senior ML Research Scientist, Anyreach AI

---

## Citation

**Paper:** DualTurn: Learning Turn-Taking from Dual-Channel Generative Speech Pretraining

```bibtex
@misc{rajaa2026dualturnlearningturntakingdualchannel,
      title={DualTurn: Learning Turn-Taking from Dual-Channel Generative Speech Pretraining},
      author={Shangeth Rajaa},
      year={2026},
      eprint={2603.08216},
      archivePrefix={arXiv},
      primaryClass={eess.AS},
      url={https://arxiv.org/abs/2603.08216},
}
```