Upload README.md with huggingface_hub

README.md

@@ -1,135 +1,115 @@
-# dualturn-endpointing
+# anyreach-ai/dualturn-endpointing
 
-Speech endpoint detection for two-channel (user + agent) audio.
+Real-time speech endpoint detector for two-channel (user + agent) audio.
 
-Answers one question in real-time every 80 ms:
+Built on top of [DualTurn-Qwen2.5-Mimi-0.5B](https://huggingface.co/anyreach-ai/dualturn-qwen2.5-mimi-0.5B) with a trained endpoint classifier that answers one question at every VAD offset:
 
-> **Has the user finished speaking?** → `ST` (start talking — agent responds) or `CL` (continue listening — user is mid-sentence)
+> **Has the user finished their turn?**  
+> `ST` — yes, agent should respond now  
+> `CL` — no, user just paused mid-sentence, keep waiting
+
+| Output | Shape | Description |
+|--------|-------|-------------|
+| `vad_probs` | `[B, T, 2]` | P(speaking now) — `[:,0]`=user `[:,1]`=agent |
+| `eot_probs` | `[B, T, 2]` | P(end of turn) per channel |
+| `bot_probs` | `[B, T, 2]` | P(beginning of turn) per channel |
+| `fvad_probs` | `[B, T, 4]` | Fast VAD (Silero) — user_short, user_long, agent_short, agent_long |
+| `endpoints` | `list[dict]` | Sparse ST/CL decisions at VAD-offset anchors with P(ST) |
+
+Frame rate: **12.5 Hz** (80 ms per frame). Audio resampled to 24 kHz internally.
 
 ---
 
-## Models
+## Inference
+
+```bash
+pip install transformers torch torchaudio joblib scikit-learn silero-vad huggingface_hub
+```
+
+```python
+import torch, torchaudio
+from transformers import AutoModel
+
+model = AutoModel.from_pretrained(
+    "anyreach-ai/dualturn-endpointing",
+    trust_remote_code=True,
+)
+model.eval()
+
+wav, sr = torchaudio.load("conversation.wav")  # [2, T]  CH0=user  CH1=agent
+
+with torch.no_grad():
+    out = model(wav, sr=sr)
 
-| File | Size | Description |
-|------|------|-------------|
-| `best.pt` | 534 MB | Dualturn transformer backbone — predicts VAD/EOT/BOT per channel |
-| `endpoint_clf.pkl` | 370 KB | Logistic regression endpoint classifier — P(ST) from 10 signals |
+# Per-frame signals at 12.5 Hz
+print(out.vad_probs.shape)    # [1, T, 2]   P(speaking) — (user, agent)
+print(out.eot_probs.shape)    # [1, T, 2]   P(end-of-turn)
+print(out.bot_probs.shape)    # [1, T, 2]   P(begin-of-turn)
+print(out.fvad_probs.shape)   # [1, T, 4]   fast VAD
+
+# Sparse endpoint decisions — one per VAD offset (user stops speaking)
+for ep in out.endpoints:
+    print(f"t={ep['t_s']:.2f}s  action={ep['action']}  P(ST)={ep['p_st']:.3f}")
+    # ep["action"]  →  "ST" (agent should respond) or "CL" (keep waiting)
+    # ep["p_st"]    →  P(user is done)  threshold = 0.30
+```
 
 ---
 
-## How it works
+## Endpoint decision logic
 
 ```
 Dual-channel audio (24 kHz stereo)
       │
-      ▼  every 80 ms  (Mimi encoder → dualturn transformer)
+      ▼  Mimi encoder + DualTurn backbone  (every 80 ms)
 Per-frame signals:
-  vad_user, vad_agent          ← is each speaker currently talking?
-  eot_user, eot_agent          ← near end of their turn?
-  bot_user, bot_agent          ← beginning a new turn?
-  fvad_user_short/long         ← fast VAD (Silero), two smoothing windows
-  fvad_agent_short/long
+  vad_user, vad_agent, eot_user, eot_agent,
+  bot_user, bot_agent, fvad_*_short, fvad_*_long
       │
-      ▼  VAD edge detector  (watches vad_user crossing 0.5)
+      ▼  watch vad_user crossing 0.5
       │
-      └── VAD offset detected (user stopped) + agent silent?
-              │
-              ▼  endpoint_clf.predict_proba([10 signal values])
-              P(ST) >= 0.30  →  ST  ✓  agent should respond
-              P(ST) <  0.30  →  CL     wait, user paused mid-sentence
+  VAD offset detected (user stopped) + agent silent?
+      │
+      ▼  endpoint_clf.predict_proba(10 signal values)
+      P(ST) >= 0.30  →  ST  ✓  agent should respond now
+      P(ST) <  0.30  →  CL     user paused mid-sentence, wait
 ```
 
-**Threshold = 0.30** — tuned on held-out test set to maximise ST recall (99% recall, 90% precision on test set).
+Threshold **0.30** is tuned to maximise ST recall (99% recall on held-out test set).
 
 ---
 
-## Install
+## Files
 
-```bash
-pip install torch torchaudio joblib scikit-learn huggingface_hub silero-vad
-```
+| File | Description |
+|------|-------------|
+| `best.pt` | DualTurn backbone weights (two-stream transformer, 34M params) |
+| `endpoint_clf.pkl` | Endpoint classifier — sklearn bundle with trained model + recommended threshold |
+| `modeling_dualturn.py` | `DualTurnModel(PreTrainedModel)` — AutoModel-compatible wrapper |
+| `configuration_dualturn.py` | `DualTurnConfig` |
+| `config.json` | `auto_map` for AutoModel/AutoConfig |
+| `endpointing.py` | Higher-level `DualTurnEndpointing` class with streaming support |
+| `src/` | Bundled dualturn + evaluation source code |
 
 ---
 
-## Usage
-
-### From HuggingFace (recommended)
+## Streaming (real-time)
 
 ```python
 from endpointing import DualTurnEndpointing
 
-model = DualTurnEndpointing.from_pretrained("anyreach/dualturn-endpointing")
-
-# Offline: process a stereo WAV file
-frames, endpoints = model.process_file(
-    "call.wav",
-    user_channel=0,    # which stereo channel is the user
-    agent_channel=1,
-)
-
-for ep in endpoints:
-    print(f"t={ep['t_s']:.2f}s  action={ep['action']}  P(ST)={ep['p_st']:.3f}")
-```
-
-### From local files
-
-```python
-model = DualTurnEndpointing(
-    backbone_path   = "best.pt",
-    classifier_path = "endpoint_clf.pkl",
-    device          = "cuda",
-)
-```
-
-### Streaming (real-time, 80 ms chunks)
-
-```python
-model  = DualTurnEndpointing.from_pretrained("anyreach/dualturn-endpointing")
+model  = DualTurnEndpointing.from_pretrained("anyreach-ai/dualturn-endpointing")
 stream = model.stream(user_channel=0, agent_channel=1)
 
-for chunk in audio_source():          # chunk: np.ndarray (2, 1920) float32
+# Feed 80 ms of 24 kHz stereo PCM float32 every tick
+# chunk shape: (2, 1920)
+for chunk in audio_source():
     result = stream.push(chunk)
     if result:
         if result["action"] == "ST":
             agent.start_responding()
-        else:
-            agent.keep_waiting()
-```
-
-### CLI
-
-```bash
-# From HuggingFace
-python endpointing.py --audio call.wav --from-hf anyreach/dualturn-endpointing
-
-# From local files
-python endpointing.py \
-    --audio call.wav \
-    --backbone best.pt \
-    --classifier endpoint_clf.pkl \
-    --user-channel 0 \
-    --agent-channel 1 \
-    --out-json results.json
-```
-
----
-
-## Output format
-
-Each endpoint decision:
-```json
-{
-  "t_s":    9.70,
-  "action": "ST",
-  "p_st":   0.984,
-  "signals": {
-    "vad_user": 0.02, "vad_agent": 0.01,
-    "eot_user": 0.91, "eot_agent": 0.03,
-    "bot_user": 0.01, "bot_agent": 0.05,
-    "fvad_user_short": 0.03, "fvad_user_long": 0.12,
-    "fvad_agent_short": 0.01, "fvad_agent_long": 0.02
-  }
-}
+        # result["p_st"]    — P(ST) from classifier
+        # result["signals"] — all 10 signal values at the anchor
 ```
 
 ---
@@ -140,8 +120,39 @@ Each endpoint decision:
 |--------|-------|
 | ST recall | 99% |
 | ST precision | 90% |
-| CL recall | 6% |
-| AUC | 0.853 |
+| AUC (ST vs CL) | 0.853 |
 | Threshold | 0.30 |
 
-The model is tuned for **high ST recall** — it almost never misses a real turn end. CL recall is low by design (we'd rather respond slightly early than make the user repeat themselves).
+Tuned for **high ST recall** — almost never misses a real turn end. Accepts some false STs to avoid making the user repeat themselves.
+
+---
+
+## Training Data
+
+Endpoint classifier trained on 70 real dual-channel calls (user + Gemini agent) automatically labelled by Gemini 2.5 Pro.
+
+Backbone: [anyreach-ai/dualturn-qwen2.5-mimi-0.5B](https://huggingface.co/anyreach-ai/dualturn-qwen2.5-mimi-0.5B)
+
+---
+
+## Authors
+
+* Shangeth Rajaa — Senior ML Research Scientist, Anyreach AI
+
+---
+
+## Citation
+
+**Paper:** DualTurn: Learning Turn-Taking from Dual-Channel Generative Speech Pretraining
+
+```bibtex
+@misc{rajaa2026dualturnlearningturntakingdualchannel,
+      title={DualTurn: Learning Turn-Taking from Dual-Channel Generative Speech Pretraining},
+      author={Shangeth Rajaa},
+      year={2026},
+      eprint={2603.08216},
+      archivePrefix={arXiv},
+      primaryClass={eess.AS},
+      url={https://arxiv.org/abs/2603.08216},
+}
+```