Update README.md

README.md

@@ -16,41 +16,32 @@ tags:
 
 # Audio8 ASR Infinite
 
-**A low-latency streaming speech recognition model with unlimited-length transcription, a selectable 80/120/160 ms audio clock, and a per-session transcription delay. Chinese and English.**
-
+[![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Audio8--ASR--Infinite-yellow?style=for-the-badge)](https://huggingface.co/Edge0/Audio8-ASR-Infinite)
 [![GitHub](https://img.shields.io/badge/GitHub-Audio8--ASR--Infinite-black?style=for-the-badge&logo=github)](https://github.com/Edge0-AI/Audio8-ASR-Infinite)
+[![arXiv](https://img.shields.io/badge/arXiv-coming%20soon-b31b1b?style=for-the-badge&logo=arxiv&logoColor=white)](https://github.com/Edge0-AI/Audio8-ASR-Infinite)
 [![License](https://img.shields.io/badge/License-Apache%202.0-blue?style=for-the-badge)](https://github.com/Edge0-AI/Audio8-ASR-Infinite/blob/main/LICENSE)
 
 </div>
 
-**Audio8 ASR Infinite** is a low-latency streaming speech recognition model that
-emits one text token per selectable 80/120/160 ms clock step, transcribes audio
-of unlimited length without drifting, and supports Chinese and English with a
-per-session transcription delay (`target_delay_ms`).
-
-## TL;DR
-
-- **Low-latency streaming ASR** — text streams out one token per selectable
-  80/120/160 ms clock step.
-- **Unlimited-length transcription** — a 30 s rolling KV window keeps memory and
-  latency bounded no matter how long the audio runs.
-- **Deployable as-is** — merged checkpoint with embedded remote code,
-  per-session `target_delay_ms`, Chinese and English.
+**Audio8 ASR Infinite** is a native streaming speech recognition model built to be
+as responsive as possible. It offers a selectable audio clock (80/120/160 ms) and
+a transcription delay (240–560 ms).
+With our adapted vLLM build it transcribes unlimited-length audio **24/7** without drifting.
 
 ## Highlights
 
-- **Unlimited-length transcription** — a self-managed 30 s rolling KV window
-  (with exact RoPE re-basing) keeps memory and latency bounded regardless of
-  audio duration; validated end-to-end on 10-minute continuous streams with
-  zero transcription drift.
-- **Selectable 80/120/160 ms streaming clock** — text tokens are emitted one per
-  clock step (12.5 / 8.3 / 6.25 decisions per second), trading perception
-  granularity for compute; partial results stream out in real time.
-- **Configurable transcription delay** — pick the latency/quality trade-off at
-  session start (`target_delay_ms`), per session.
+- **Super responsive** — the native streaming architecture decodes 12.5 times per second.
+- **Unlimited-length transcription** — a rolling KV cache keeps memory and
+  latency bounded, even in **24/7 operation**.
+- **Selectable streaming clock** — one text token per clock step
+  (12.5 / 8.3 / 6.25 decisions per second), balancing perception granularity and resource cost.
+- **Configurable transcription delay** — set how much delay to trade for accuracy.
+- **Semantic VAD** — distinguishes thinking pauses, stuttering and real end of turn, where traditional acoustic VAD fails.
 - **Bilingual** — Chinese and English.
 
-## Supported operation points
+## Optimized operation points
+
+The following combinations of frame length and delay are post-trained. Other combinations can be used but performance may not be optimum.
 
 | audio clock | `frame_len` | `streaming_n_left_pad_tokens` | selectable `target_delay_ms` |
 | --- | --- | --- | --- |
@@ -63,36 +54,56 @@ delays stay available at every clock even when they are not listed above.
 
 ## Architecture
 
+Inherits the Voxtral realtime audio architecture and DSM-style streaming.
+
+| Component | Initial weights | Trained |
+| --- | --- | --- |
+| Causal Audio Tower | Voxtral Realtime 4B | ✅ |
+| Audio Projector  | random initialisation | ✅ |
+| Frame Length Embedding | random initialisation | ✅ |
+| Decoder | Qwen2.5-3B-Instruct | ✅ |
+| LM Head | Qwen2.5-3B-Instruct | ✅ |
+
+Checkpoint specification:
+
 | | |
 | --- | --- |
 | audio tower | 32 layers, hidden 1280, 128 mel bins, sliding window 750 |
 | text decoder | 36 layers, hidden 2048, 16 query heads / 2 KV heads |
 | projector | max frame len 8 → projection size 10240, gelu |
 | frame-length conditioning | enabled (`use_frame_len_embedding: true`) |
+| semantic VAD heads | `semantic_vad_heads.safetensors`, 8 classes, horizons 0.5 / 1.0 / 2.0 / 3.0 s |
 | vocab size | 151936 |
 | dtype | bfloat16 |
-| weights | single 8.17 GB `model.safetensors` |
+| weights | 8.17 GB `model.safetensors` (+ `semantic_vad_heads.safetensors`) |
 
-## Inference paths
+## Roadmap
 
-This repository holds **weights only**. The two inference paths live in the
-GitHub repository:
+This is the **preview release**: it delivers the transcription base. Realtime
+semantic perception is being built on the same frame grid and the same acoustic
+forward pass.
 
-| Path | Entry point | Notes |
+| Stage | Status | Scope |
 | --- | --- | --- |
-| torch (simulated streaming) | `audio8_asr_infinite/streaming_inference.py` | Definition-aligned simulated-streaming greedy decode: per-window feature extraction with a streaming audio/language KV cache; evaluation behavior matches deployment |
-| vLLM (real streaming service) | `audio8_asr_infinite/vllm_serve/` | Official vLLM 0.27.1 plugin points (zero patches, no site-packages overwrite); `/v1/realtime` WebSocket service + 30 s rolling KV |
+| **Preview — ASR base** | ✅ done | Streaming Chinese/English transcription: selectable 80/120/160 ms clock, configurable `target_delay_ms`, unlimited-length rolling KV window |
+| **Formal release** | 🏃in progress | Frame-level semantic perception on the same grid, beyond transcription |
 
-## Installation
+## Evaluation
 
-Version requirements: `transformers==5.13.0` (must include
-`models.voxtral_realtime`), `torch 2.13`, Python 3.11/3.12. The vLLM path also
-needs `vllm==0.27.1`.
+### 480 ms Delay, 80ms frame length
+
+| test set | metric | Audio8 ASR Infinite | Voxtral-Mini-4B-Realtime-2602 | nemotron-3.5-asr-streaming-0.6b |
+| --- | --- | --- | --- | --- |
+| aishell1/test | CER | **1.750** | 16.795 | 12.927@560ms |
+| aishell4/test | CER | **2.893** | 16.456 | 14.677@560ms |
+| librispeech test.clean | WER | 3.042 | **2.210** | 3.353@560ms |
+| librispeech test.other | WER | 6.808 | **5.552** | 7.140@560ms |
+| **average** | | **3.623** | 10.253 (2 sets) | 9.524 |
+
+Greedy decode with EOS suppressed, at the 80 ms audio clock with
+`target_delay_ms = 480` (6 delay tokens). Error rates in percent. No repetition
+loops and no dropped trailing words.
 
-```bash
-pip install -e .                 # torch path
-pip install -e ".[vllm]"         # vllm==0.27.1 + pybase64
-```
 
 ## Usage
 
@@ -143,94 +154,40 @@ print(results[0]["final_text"])
 Only a full merged weight directory is supported (this repository as-is);
 adapter-style or partially converted weights are not.
 
-Real streaming service — docker compose is the canonical deployment path (it
-also serves the web demo):
+## 24/7 inference with vLLM
+
+Docker compose is the canonical deployment path; it also serves the web demo:
 
 ```bash
 cd docker
-AUDIO8_MODEL_DIR=/path/to/merged-v2-checkpoint docker compose up -d
+AUDIO8_MODEL_DIR=/path/to/checkpoint docker compose up -d
 ```
 
-Verify with real audio using the bundled client:
+Verify with the web client shipped in the same stack:
 
-```bash
-python -m audio8_asr_infinite.examples.vllm_realtime_client \
-    --audio sample.wav --ws-url ws://127.0.0.1:18190/v1/realtime \
-    --language zh --target-delay-ms 480 --pace
 ```
-
-`docker/docker-compose.yml` in the repository is the deployment source of
-truth: GPU and port selection, the three plugin-injection switches
-(`PYTHONPATH` + `VLLM_PLUGINS` + `--scheduler-cls`), health checks, and the
-startup log markers that prove the plugins loaded.
-
-## Decoding note — EOS is suppressed unconditionally
-
-This checkpoint was trained **without EOS supervision**: after speech ends, every
-frame is labelled `[STREAMING_PAD]`, so when to stop is decided by the timeline
-length rather than by the model. The released decoder therefore masks the EOS
-logit at every step, and that behaviour is not configurable.
-
-Do not pair this checkpoint with a decoder that stops on EOS: re-enabling EOS
-decoding throws away the property this checkpoint was trained for. Conversely,
-this checkpoint — not an EOS-trained one — is the one that expects hard EOS
-suppression.
-
-## Evaluation
-
-Greedy decode with EOS suppressed, at the 80 ms audio clock with
-`target_delay_ms = 480` (6 delay tokens).
-
-| test set | rows | metric | score |
-| --- | --- | --- | --- |
-| aishell1/test | 7176 | CER | **1.75** |
-| librispeech test.clean | 2620 | WER | **3.04** |
-| librispeech test.other | 2939 | WER | **6.81** |
-
-Error rates in percent. No repetition loops and no dropped trailing words.
-
-## Files
-
-```
-config.json                              config + auto_map remote code
-configuration_audio8_asr_infinite.py     remote code: config class
-modeling_audio8_asr_infinite.py          remote code: model class
-model.safetensors                        8.17 GB, bf16, single file
-generation_config.json
-preprocessor_config.json                 16 kHz, 128 mel bins
-tokenizer.json / tokenizer_config.json / chat_template.jinja
+http://localhost:8080/     # plain HTTP
+https://localhost:8443/    # TLS proxy; accept the self-signed certificate
 ```
 
-## Repository layout
+The same socket can be driven from a terminal:
 
+```bash
+python -m audio8_asr_infinite.examples.vllm_realtime_client \
+    --ws-url ws://127.0.0.1:18191/v1/realtime \
+    --audio sample.wav --language zh --target-delay-ms 480 --pace
 ```
-audio8_asr_infinite/
-├── modeling/                      # HF-style config + modeling (Audio8ASRInfinite*)
-├── streaming_inference.py         # torch simulated-streaming decode (canonical inference entry)
-├── simulated_streaming_audio.py   # shared audio-window / streaming-KV helpers
-├── vllm_serve/                    # vLLM 0.27.1 plugins (model adapter + rolling scheduler + realtime endpoint)
-├── examples/                      # torch decode example + vLLM WebSocket client
-├── client/                        # browser realtime transcription demo (static)
-├── docker/                        # Dockerfile / docker-compose / TLS proxy
-└── hf_demo/                       # HuggingFace Spaces Gradio demo
-```
-
-## Roadmap
-
-This is the **preview checkpoint**, and it delivers the transcription base.
-Realtime semantic perception is being built on the same frame grid and the same
-acoustic forward pass, so the formal release extends this stack rather than
-replacing it.
 
-| Stage | Scope |
-| --- | --- |
-| **Preview — ASR base (this checkpoint)** | Streaming Chinese/English transcription: selectable 80/120/160 ms clock, per-session `target_delay_ms`, unlimited-length rolling KV window |
-| **Formal release (in progress)** | Frame-level semantic perception on the same grid, beyond transcription: semantic VAD, voice commands, multi-speaker handling |
+`18191` is the host port published by `docker/docker-compose.yml`; the service
+itself listens on `18190` inside the compose network. The rolling KV window is
+30 s with exact RoPE re-basing, which is what keeps memory and latency bounded
+over 24/7 operation.
 
-Nothing beyond transcription is claimed for this checkpoint.
+## Torch inference (simulated streaming decode)
 
-## License
+```bash
+python -m audio8_asr_infinite.examples.torch_streaming_decode \
+    --checkpoint /path/to/checkpoint \
+    --audio sample.wav --language zh --transcription-delay-ms 480
+```
 
-Apache-2.0. The vendored vLLM realtime files under
-`vllm_serve/vllm0271/realtime/` keep their upstream Apache-2.0 copyright
-headers; the licenses are compatible.