# dualturn-endpointing

Speech endpoint detection for two-channel (user + agent) audio.

Answers one question in real-time every 80 ms:

> **Has the user finished speaking?** → `ST` (start talking — agent responds) or `CL` (continue listening — user is mid-sentence)

---

## Models

| File | Size | Description |
|------|------|-------------|
| `best.pt` | 534 MB | Dualturn transformer backbone — predicts VAD/EOT/BOT per channel |
| `endpoint_clf.pkl` | 370 KB | Logistic regression endpoint classifier — P(ST) from 10 signals |

---

## How it works

```
Dual-channel audio (24 kHz stereo)
      │
      ▼  every 80 ms  (Mimi encoder → dualturn transformer)
Per-frame signals:
  vad_user, vad_agent          ← is each speaker currently talking?
  eot_user, eot_agent          ← near end of their turn?
  bot_user, bot_agent          ← beginning a new turn?
  fvad_user_short/long         ← fast VAD (Silero), two smoothing windows
  fvad_agent_short/long
      │
      ▼  VAD edge detector  (watches vad_user crossing 0.5)
      │
      └── VAD offset detected (user stopped) + agent silent?
              │
              ▼  endpoint_clf.predict_proba([10 signal values])
              P(ST) >= 0.30  →  ST  ✓  agent should respond
              P(ST) <  0.30  →  CL     wait, user paused mid-sentence
```

**Threshold = 0.30** — tuned on held-out test set to maximise ST recall (99% recall, 90% precision on test set).

---

## Install

```bash
pip install torch torchaudio joblib scikit-learn huggingface_hub silero-vad
```

---

## Usage

### From HuggingFace (recommended)

```python
from endpointing import DualTurnEndpointing

model = DualTurnEndpointing.from_pretrained("anyreach/dualturn-endpointing")

# Offline: process a stereo WAV file
frames, endpoints = model.process_file(
    "call.wav",
    user_channel=0,    # which stereo channel is the user
    agent_channel=1,
)

for ep in endpoints:
    print(f"t={ep['t_s']:.2f}s  action={ep['action']}  P(ST)={ep['p_st']:.3f}")
```

### From local files

```python
model = DualTurnEndpointing(
    backbone_path   = "best.pt",
    classifier_path = "endpoint_clf.pkl",
    device          = "cuda",
)
```

### Streaming (real-time, 80 ms chunks)

```python
model  = DualTurnEndpointing.from_pretrained("anyreach/dualturn-endpointing")
stream = model.stream(user_channel=0, agent_channel=1)

for chunk in audio_source():          # chunk: np.ndarray (2, 1920) float32
    result = stream.push(chunk)
    if result:
        if result["action"] == "ST":
            agent.start_responding()
        else:
            agent.keep_waiting()
```

### CLI

```bash
# From HuggingFace
python endpointing.py --audio call.wav --from-hf anyreach/dualturn-endpointing

# From local files
python endpointing.py \
    --audio call.wav \
    --backbone best.pt \
    --classifier endpoint_clf.pkl \
    --user-channel 0 \
    --agent-channel 1 \
    --out-json results.json
```

---

## Output format

Each endpoint decision:
```json
{
  "t_s":    9.70,
  "action": "ST",
  "p_st":   0.984,
  "signals": {
    "vad_user": 0.02, "vad_agent": 0.01,
    "eot_user": 0.91, "eot_agent": 0.03,
    "bot_user": 0.01, "bot_agent": 0.05,
    "fvad_user_short": 0.03, "fvad_user_long": 0.12,
    "fvad_agent_short": 0.01, "fvad_agent_long": 0.02
  }
}
```

---

## Performance (held-out test set, 15 calls)

| Metric | Value |
|--------|-------|
| ST recall | 99% |
| ST precision | 90% |
| CL recall | 6% |
| AUC | 0.853 |
| Threshold | 0.30 |

The model is tuned for **high ST recall** — it almost never misses a real turn end. CL recall is low by design (we'd rather respond slightly early than make the user repeat themselves).