pandyamarut/osmosis-eot-lfm2-350m-clf-head3-onnx

Osmosis EOT — LFM2.5-350M (head-sliced INT8 ONNX)

End-of-turn (EOT) / turn-detection classifier for voice AI. Fine-tuned from LiquidAI/LFM2.5-350M, exported to INT8 ONNX, and head-sliced for low CPU latency.

Given a short conversation history and the latest user utterance, it predicts one of:

• finished — the user has completed their thought; the assistant may respond
• unfinished — the user is still mid-thought or trailing off; keep waiting
• wait — the user explicitly asked the assistant to wait / hold on

The model outputs 3 logits directly ([1, 3] = finished, unfinished, wait). Use softmax(...) and take p[0] as p_finished — the score to fuse with an audio EOT model.

Why "head-sliced"

The original CausalLM export projected the final hidden state to the full 65,536-token vocabulary in FP32 (~268 MB streamed per inference) just to read 3 numbers. We replaced that output projection with the 3 class rows only ([3 × 1024]), so it emits [1, 3] directly. This is bit-identical to reading those 3 vocab logits from the full model (verified max-abs-diff = 0.0) — no accuracy change — but removes an O(seq × 65536) FP32 matmul.

Performance (LiveKit EOT-bench, English validation, 1250 spans)

Composite = fusion of this text model with SmartTurn v3.2 (audio).

Config	text AUC	composite AUC	false-cutoff	latency*
Full-vocab model @ seq 256	0.7976	0.8724	19.8%	~128 ms
Head-sliced @ seq 256 (lossless)	0.7976	0.8724	19.8%	~94 ms
Head-sliced @ seq 128 (recommended)	0.7894	0.8740	17.9%	~48 ms

*CPU latency, 4 threads, worst case at full seq_len; typical prompts are shorter → faster.

The recommended config (seq_len 128) runs at ~48 ms on CPU while keeping composite AUC at 0.8740 — above the LiveKit baseline (0.8479).

Usage

from inference_example import TurnDetector

td = TurnDetector()   # loads ./model_q8_head3.onnx + ./tokenizer, seq_len=128
out = td.classify(
    history=[{"role": "assistant", "content": "Is now a good time to review your quote?"}],
    utterance="Yeah, that works for me, go ahead.",
)
print(out["label"], out["p_finished"])   # -> finished 0.99

Key inference details (must match to reproduce accuracy)

• System prompt: use the exact 3-class prompt in inference_example.py (verbatim).
• User prompt: Conversation history: (last 6 turns as User: / Agent:) + the latest utterance + Reply with only one word: finished, unfinished, or wait.
• Chat template: ChatML, with add_generation_prompt=True.
• add_special_tokens=True at tokenization — prepends BOS (<|startoftext|>), required.
• truncation_side="left", max_length=128.
• Output is [1, 3] in order (finished, unfinished, wait); p_finished = softmax(logits)[0].

Files

• model_q8_head3.onnx — head-sliced INT8 ONNX model ([1, 3] output)
• tokenizer/ — tokenizer for LiquidAI/LFM2.5-350M
• inference_example.py — self-contained CPU inference wrapper