Add fail-closed stable speaker inference

README.md

@@ -20,7 +20,8 @@ models:
 
 ## 模式
 
-- **內建語者**：使用模型發佈內附的 speaker centroid。
+- **內建語者**：使用模型發佈內附的 speaker centroid；預設為已重新稽核的
+  `female_voice`（介面中的內建語者 B）。
 - **參考音色**：從至少 3 秒的授權參考音檔抽取 ECAPA speaker embedding；參考內容不需要逐字稿。
 - **穩定長文**：使用和短句相同的 speaker embedding，切段後輸出一個完整音檔。
 
@@ -33,35 +34,104 @@ Demo 預設採用目前通過長文穩定性評估的推論設定：
 
 模型 snapshot 固定在 `aaf1a0878e37875382bb0e5c8a3a2ba43be67297`；
 speaker-reference 模式使用的 ECAPA encoder 固定在
-`0f99f2d0ebe89ac095bcc5903c4dd8f72b367286`，TTS runtime package 亦固定 commit，
+`0f99f2d0ebe89ac095bcc5903c4dd8f72b367286`；TTS runtime 以
+`ce384c8cc54efea1aaba7b9f1d7ded6c1c99aa9a` 的已驗證原始碼隨 Space 保存，
+Barbet 另固定在 `6fcd7ce4aa37f2250a3242995bef0fbc3b026ba8`，
 避免 Space 重啟後在沒有程式版本變更的情況下改變模型或 speaker embedding 空間。
+候選語意驗證固定使用 Whisper large-v3-turbo revision
+`41f01f3fe87f28c78e2fbf8b568835947dd65ed9`。
 
 | 設定 | 值 |
 |---|---:|
 | CFG | 2.0 |
 | NFE steps | 10 |
 | Target pace | 4.0 speech units/sec |
-| Generation safety pace | 5.2 CJK / 4.6 ASCII-mixed units/sec + 1 latent step |
+| Candidate 0 generation policy | base: 5.2 CJK / 4.6 ASCII-mixed units/sec + 1 latent step |
+| Candidate 1–9 generation policy | safe duration: 4.6 CJK / 4.0 ASCII-mixed units/sec + 1 latent step |
 | Short-text guidance | minimum CFG 3.0 below 6 speech units |
 | Stop policy | 0.50 → 0.05 from 75% to 95% predicted progress, 1 hit |
 | Endpoint cue | append terminal punctuation for model input when missing, except very short text |
 | Hard stop | native-pace target steps, independent of playback pace |
-| Pace correction | pitch-preserving stretch after completion; onset split only at punctuation |
+| Pace correction | whole completed chunk, pitch-preserving; active-region stretcher disabled |
+| Pace / speaker eligibility duration | union of active 25 ms RMS frames at 10 ms hop; internal pauses excluded |
+| Semantic verification | whole / first 6 / last 6 units, no lexical tail; fail closed |
+| Speaker verification | frozen request centroid + begin/end directional drift |
+| Early accept speaker preference | similarity ≥ 0.25 and directional drop ≤ 0.05 for gated chunks |
+| Final hard speaker gate | similarity ≥ 0.10 and directional drop ≤ 0.10 |
+| Long speaker evidence | one ECAPA batch over up to four 3-second windows + begin/end |
+| Candidate cascade | same-seed whole trajectory first; then 1→5→10 sequence fallback within budget |
+| Whole-trajectory qualification | assemble with production RMS/fade/pause/crossfade/speed, then whole-output gate |
+| Sequence fallback | up to 3 distinct k-best paths from gate-passed chunks; each exact-assembled and whole-gated |
+| Final output gate | re-verify joined/faded/RMS-matched/speed-adjusted whole waveform; fail closed |
+| Runtime budget | at most 20 generated TTS chunks per request; at most 10 candidates |
 | Maximum chunk | 80 chars |
 | Minimum chunk | 12 chars |
 | Crossfade / internal edge fade | 80 ms / 80 ms |
 | Chunk RMS adjustment | at most 4 dB |
 | Generated continuation context | 0 sec |
-| Retry / rerank | disabled |
+| Generated-audio retry | disabled |
+| Quality selection | enabled; no unverified fallback |
 
-每個請求會取得新的隨機 seed；同一請求內的所有長文切段會重用該 seed。切段保留標點，
-並依逗號、分號或句末標點插入不同長度的停頓。輸出最後會套用保守的 RMS floor 與 peak limit。
+每個請求會取得新的隨機 root seed；每個候選 trajectory 的所有長文切段會重用同一 seed。
+`candidate 0` 使用 base policy（5.2 CJK / 4.6 ASCII），`candidate 1–9` 使用實證回收用的
+safe-duration policy（4.6 CJK / 4.0 ASCII）；兩者都只改 native-duration endpoint estimate，
+margin 固定 +1 latent step、`min_len` 固定為 2，不會為了播放目標語速硬撐 generation loop。
+選中完整 trajectory 或 sequence-DP 混合路徑後，log 會記錄每個 chunk 的 candidate、seed 與 policy。
+內部 hosted evaluator 可注入 `[0, 2^31)` 的固定 root seed 以重現結果；UI 不暴露這個參數，
+一般請求仍只在未注入 seed 時使用系統亂數。
+完整 same-seed trajectory 永遠優先；只有一個 stage 沒有完整 trajectory 時，才允許 sequence DP
+從該 stage 已通過 hard gate 的 chunks 混合 seed。切段保留標點，並依逗號、分號或句末標點
+插入不同長度的停頓。輸出最後會套用保守的 RMS floor 與 peak limit。
+完成 RMS matching、edge fade、pause、crossfade 與使用者 speed 後，服務會再對最終整段 waveform
+執行一次 normalized target 的 prefix/whole/suffix/tail、pace 與 speaker anchor/boundary gate；這個
+post-join gate 失敗時不會回傳先前已通過的 chunk 音訊。
+此外，same-seed trajectory 只有在每個 local chunk 都通過後，才會先用同一個
+`_assemble_trajectory_audio` 組成實際播放版本並跑整段 semantic/pace/speaker gate；joined gate
+失敗會取消該 whole trajectory 的資格，但保留已通過的 local evidence 供 sequence DP 使用。
+這個額外 whole-ASR 只對 local 全數通過的候選執行，DP 選定後仍會保留上述最終 post-join gate。
+若整個 candidate budget 都沒有 joined-safe whole trajectory，多 chunk request 才會進入 bounded
+k-best DP。Local score 加上相鄰 ECAPA/RMS transition cost 會產生最多 3 條不同完整路徑，依
+`(total cost, candidate-index path)` 穩定排序；每條都用同一個 production assembler 和整段
+semantic/pace/speaker gate 驗證，第一條通過才回傳。三條都失敗、callback 異常或沒有完整
+finite path 時一律 fail closed。這不增加 TTS generation 數量，也不套用於單 chunk request。
+被拒的 joined/final log 只記 normalized CER、prefix/suffix CER、tail units 與 reasons，不記逐字
+transcript。
+
+日期、24 小時制時間、百分比、常見單位與大寫 acronym/model code 會先轉成保守的
+zh-TW spoken form，例如 `2026/07/16`、`15:30`、`12.5%` 與 `10 km`。ASR 會先統一
+繁簡字形再評分，避免把正確的台灣華語輸出誤判為內容錯誤。
+ASR 評分另只將實證可互換的同音代詞 `她/它/牠/祂` 視為 `他`、`妳` 視為 `你`；
+這項 canonicalization 不會修改送入模型的文字，也不包含 `的/地/得` 或 `在/再`。
 
 自然 stop 在目前 checkpoint 上仍可能過快或錯過句尾。Demo 不再用播放目標語速決定生成長度；
 模型使用原生語速的安全上限，完成後才做保音高語速校正。缺少句末標點時只在模型輸入補上句號，
 少於 6 個 speech units 的短句會使用最低 CFG 3.0；stop threshold 只在預估進度 75% 後逐步降低，並於 95% 進度降至 0.05，以捕捉句末弱 stop 訊號而不影響前段內容。
 首段不會在無標點處硬切。進階參數可供研究比較，但 release
 profile 是 CFG 2.0、NFE 10、後處理語速 1.0。
+Online pace gate 與 1.5 秒 speaker eligibility 使用和獨立 hosted evaluator 相同的 active-frame
+interval union（25 ms frame、10 ms hop、peak -35 dB、absolute RMS floor 1e-4），因此句內或插入的
+pause 不會稀釋 CPS。逐 active interval 的 stretcher 曾在固定 root ablation 造成長文 availability
+回退，目前 production profile 不啟用；這個 union duration 只用於正確量測與 fail-closed gate。
+
+第一個候選同時通過 local 與 joined whole gate，且符合 early-accept speaker preference 時才會
+立即回傳；若 generated-chunk budget
+允許，短 request 才會依序擴展到 5、10 個候選。每個 stage 先找完整 same-seed trajectory；
+整個 budget 完全找不到 joined-safe whole trajectory 時，才用逐 chunk semantic/pace/speaker
+hard gate 的 local score，以及相鄰 chunks 的 ECAPA cosine 與 active-RMS dB 差做 bounded
+k-best dynamic programming。多 chunk request 的候選上限是
+`min(10, floor(20 / chunk count))`：1/2 chunks 可到 K10、3 chunks K6、4 chunks K5、
+5 chunks K4、6 chunks K3，因此不會無限制地把 K10 套到 360 字長文。
+非 final stage 的 speaker-gated chunks 只有在 similarity 至少 0.25、directional boundary drop
+不超過 0.05 時才能 early accept。只通過 0.10/0.10 hard gate 的 marginal candidate 會保留並
+繼續下一個可用 stage；到了 request budget 的 final stage 仍可作為 verified fallback，避免降低
+availability。未套 speaker hard gate 的極短句不受 preferred threshold 阻擋。
+若所有候選仍有漏字、多說字、句尾截斷、語速過快或可量測的音色漂移，服務會拒絕輸出，
+不會回傳任意 fallback。極短音檔的分段 ECAPA 不可靠，因此只採嚴格語意 gate，並在
+release audit 中獨立標記為 speaker-unverified；這不代表模型的短句音色已被根治。
+
+目前部署路徑不使用 reference-style distribution score；sequence DP 已使用 speaker/RMS
+transition，但尚未啟用 normalized median-F0 transition。Style 與 F0 helper 仍只供離線實驗，
+不能把它們列為目前 Space 已提供的穩定性保證。
 
 請只使用已取得授權的參考音檔。合成語音僅供研究與評估展示，正式使用前請人工檢視。
 

app.py

@@ -28,6 +28,7 @@ from production import (
     finish_audio,
     join_audio_chunks,
     match_chunk_rms,
+    normalize_spoken_forms,
     normalize_tts_text,
     punctuation_pause_seconds,
     select_generation_cps,
@@ -36,6 +37,29 @@ from production import (
     split_text_for_tts,
     target_pace_speed,
 )
+from quality_runtime import (
+    BASE_GENERATION_POLICY,
+    SAFE_DURATION_GENERATION_POLICY,
+    WHISPER_MODEL_ID,
+    WHISPER_REVISION,
+    CandidateObservation,
+    ChunkCandidateArtifact,
+    FinalOutputRejectedError,
+    GenerationPolicy,
+    NoQualifiedCandidateError,
+    active_voiced_duration_seconds,
+    active_audio_rms_db,
+    candidate_limit_for_chunk_budget,
+    generation_policy_for_candidate_offset,
+    prepare_candidate_audio,
+    qualify_trajectory_with_joined_output,
+    require_verified_final_output,
+    resolve_request_seed,
+    run_adaptive_cascade,
+    speaker_evidence_from_audio,
+    transcribe_whisper,
+    verify_trajectory,
+)
 
 
 try:
@@ -57,8 +81,6 @@ ECAPA_REVISION = "0f99f2d0ebe89ac095bcc5903c4dd8f72b367286"
 DEFAULT_CFG = 2.0
 DEFAULT_STEPS = 10
 TARGET_CPS = 4.0
-GENERATION_CPS = 5.2
-ASCII_GENERATION_CPS = 4.6
 MIN_ENDPOINT_CUE_UNITS = 6
 SHORT_TEXT_CFG_MIN = 3.0
 SHORT_TEXT_CFG_UNITS = 6
@@ -73,13 +95,27 @@ STOP_LATE_THRESHOLD = 0.05
 STOP_LATE_START_RATIO = 0.75
 STOP_LATE_FULL_RATIO = 0.95
 STOP_CONSECUTIVE = 1
-HARD_STOP_MARGIN_STEPS = 1
 MIN_PACE_SPEED = 0.80
 MAX_TEXT_CHARS = 360
+QUALITY_MAX_CANDIDATES = 10
+QUALITY_MAX_GENERATED_CHUNKS = 20
+QUALITY_FINAL_ASR_MAX_NEW_TOKENS = 440
+QUALITY_MAX_CER = 0.20
+QUALITY_MAX_PACE_CPS = 4.30
+QUALITY_PREFIX_SUFFIX_UNITS = 6
+QUALITY_MIN_SPEAKER_SIMILARITY = 0.10
+QUALITY_MAX_BOUNDARY_SPEAKER_DROP = 0.10
+QUALITY_PREFERRED_MIN_SPEAKER_SIMILARITY = 0.25
+QUALITY_PREFERRED_MAX_BOUNDARY_SPEAKER_DROP = 0.05
+SHORT_AUDIO_SPEAKER_GATE_SECONDS = 1.50
 
 
 print(f"[BlueMagpie] downloading model from {REPO_ID}@{MODEL_REVISION} ...")
 MODEL_DIR = snapshot_download(REPO_ID, revision=MODEL_REVISION)
+print(f"[BlueMagpie] caching speaker encoder from {ECAPA_REPO_ID}@{ECAPA_REVISION} ...")
+ECAPA_DIR = snapshot_download(ECAPA_REPO_ID, revision=ECAPA_REVISION)
+print(f"[BlueMagpie] caching quality ASR from {WHISPER_MODEL_ID}@{WHISPER_REVISION} ...")
+ASR_DIR = snapshot_download(WHISPER_MODEL_ID, revision=WHISPER_REVISION)
 tokenizer = PreTrainedTokenizerFast(tokenizer_file=os.path.join(MODEL_DIR, "tokenizer.json"))
 print(f"[BlueMagpie] loading model on device={DEVICE} ...")
 model = BlueMagpieModel.from_local(MODEL_DIR, tokenizer=tokenizer, training=False, device=DEVICE)
@@ -110,6 +146,8 @@ def _load_speakers() -> tuple[dict[str, torch.Tensor], str]:
         if isinstance(METADATA.get("recommended_generation_defaults"), dict)
         else None
     )
+    if requested_id not in speaker_ids and "female_voice" in speaker_ids:
+        requested_id = "female_voice"
     default_index = speaker_ids.index(requested_id) if requested_id in speaker_ids else 0
     return labels, f"內建語者 {chr(65 + default_index)}"
 
@@ -157,8 +195,11 @@ def _get_ecapa_encoder():
 
             cache_dir = os.path.join(os.environ.get("HF_HOME", "/tmp"), "speechbrain", "ecapa")
             _ECAPA_ENCODER = EncoderClassifier.from_hparams(
-                source=ECAPA_REPO_ID,
-                revision=ECAPA_REVISION,
+                source=ECAPA_DIR,
+                # The upstream hyperparams otherwise points back to the repo
+                # and SpeechBrain 1.0.3 uses a removed Hub keyword.  Keep every
+                # weight fetch inside the already pinned local snapshot.
+                overrides={"pretrained_path": ECAPA_DIR},
                 savedir=cache_dir,
                 run_opts={"device": "cpu"},
             )
@@ -179,17 +220,18 @@ def _generate_chunk(
     cfg: float,
     steps: int,
     request_seed: int,
+    policy: GenerationPolicy,
 ) -> np.ndarray:
     generation_cps = select_generation_cps(
         text,
-        cjk_cps=GENERATION_CPS,
-        ascii_cps=ASCII_GENERATION_CPS,
+        cjk_cps=policy.cjk_cps,
+        ascii_cps=policy.ascii_cps,
     )
     model_text, expected_steps, hard_stop_steps = endpoint_generation_plan(
         text,
         generation_cps=generation_cps,
         step_seconds=STEP_SECONDS,
-        margin_steps=HARD_STOP_MARGIN_STEPS,
+        margin_steps=policy.hard_stop_margin_steps,
         add_terminal_punctuation=count_speech_units(text) >= MIN_ENDPOINT_CUE_UNITS,
     )
     # Do not hold generation open to enforce pace. The model can finish the
@@ -237,6 +279,11 @@ def _generate_chunk(
             f"generated_steps={_STOP_CONTROLLER.last_generated_steps} "
             f"reason={_STOP_CONTROLLER.last_stop_reason} cfg={generation_cfg:.2f}"
         )
+    print(
+        "[BlueMagpie] generation policy "
+        f"name={policy.name} seed={request_seed} generation_cps={generation_cps:.2f} "
+        f"expected_steps={expected_steps} hard_stop_steps={hard_stop_steps} min_len={min_len}"
+    )
     audio = audio.detach().float().cpu().numpy().reshape(-1)
     pace_speed = target_pace_speed(
         audio.size,
@@ -248,6 +295,273 @@ def _generate_chunk(
     return _apply_speed(audio, pace_speed)
 
 
+def _speaker_anchor_array(centroid: torch.Tensor) -> np.ndarray:
+    anchor = torch.as_tensor(centroid).detach().float().cpu().numpy().reshape(-1)
+    if anchor.size == 0 or not np.isfinite(anchor).all():
+        raise ValueError("speaker anchor is invalid")
+    norm = float(np.linalg.norm(anchor))
+    if not np.isfinite(norm) or norm <= 1.0e-8:
+        raise ValueError("speaker anchor has zero norm")
+    return np.asarray(anchor / norm, dtype=np.float32)
+
+
+def _generate_trajectory(
+    chunks: tuple[str, ...],
+    centroid: torch.Tensor,
+    *,
+    cfg: float,
+    steps: int,
+    request_seed: int,
+    policy: GenerationPolicy,
+) -> tuple[np.ndarray, ...]:
+    return tuple(
+        _generate_chunk(
+            chunk,
+            centroid,
+            cfg=cfg,
+            steps=steps,
+            request_seed=request_seed,
+            policy=policy,
+        )
+        for chunk in chunks
+    )
+
+
+def _verify_trajectory_audio(
+    trajectory: tuple[np.ndarray, ...],
+    chunks: tuple[str, ...],
+    anchor: np.ndarray,
+    playback_speed: float,
+    asr_max_new_tokens: int = 128,
+):
+    if len(trajectory) != len(chunks):
+        return verify_trajectory(())
+    encoder = None
+    observations: list[CandidateObservation] = []
+    artifacts: list[ChunkCandidateArtifact] = []
+    for chunk, audio in zip(chunks, trajectory, strict=True):
+        prepared = prepare_candidate_audio(
+            audio,
+            SR,
+            transcriber=lambda waveform, sample_rate: transcribe_whisper(
+                waveform,
+                sample_rate,
+                max_new_tokens=asr_max_new_tokens,
+            ),
+        )
+        if prepared is None:
+            observations.append(
+                CandidateObservation(
+                    target_text=chunk,
+                    transcript_text="",
+                    audio_duration_seconds=0.0,
+                    pace_cps=None,
+                )
+            )
+            artifacts.append(ChunkCandidateArtifact())
+            continue
+        waveform = prepared.waveform
+        try:
+            duration = active_voiced_duration_seconds(waveform, SR)
+        except ValueError:
+            duration = 0.0
+        transcript = prepared.transcript_text
+        speaker_similarity = None
+        begin_similarity = None
+        end_similarity = None
+        speaker_embedding = None
+        rms_db = None
+        if duration >= SHORT_AUDIO_SPEAKER_GATE_SECONDS:
+            try:
+                if encoder is None:
+                    encoder = _get_ecapa_encoder()
+                evidence = speaker_evidence_from_audio(
+                    waveform,
+                    SR,
+                    encoder,
+                    anchor,
+                    device="cpu",
+                )
+                duration = evidence.active_duration_seconds
+                speaker_similarity = evidence.similarity
+                begin_similarity = evidence.begin_similarity
+                end_similarity = evidence.end_similarity
+                speaker_embedding = evidence.speaker_embedding
+                rms_db = evidence.active_rms_db
+            except ValueError:
+                # A malformed/empty speaker measurement remains missing and is
+                # rejected by the fail-closed gate for non-short candidates.
+                pass
+        if rms_db is None:
+            try:
+                rms_db = active_audio_rms_db(waveform)
+            except ValueError:
+                pass
+        observations.append(
+            CandidateObservation(
+                target_text=chunk,
+                transcript_text=transcript,
+                audio_duration_seconds=duration,
+                speaker_similarity=speaker_similarity,
+                begin_speaker_similarity=begin_similarity,
+                end_speaker_similarity=end_similarity,
+                pace_cps=(
+                    count_speech_units(chunk) / duration * float(playback_speed)
+                    if duration > 0.0
+                    else None
+                ),
+            )
+        )
+        artifacts.append(
+            ChunkCandidateArtifact(
+                speaker_embedding=speaker_embedding,
+                rms_db=rms_db,
+            )
+        )
+    return verify_trajectory(
+        observations,
+        chunk_artifacts=artifacts,
+        short_text_units=6,
+        short_text_max_cer=0.0,
+        max_cer=QUALITY_MAX_CER,
+        prefix_units=QUALITY_PREFIX_SUFFIX_UNITS,
+        suffix_units=QUALITY_PREFIX_SUFFIX_UNITS,
+        max_prefix_cer=0.0,
+        max_suffix_cer=0.0,
+        max_extra_tail_units=0,
+        short_audio_seconds=SHORT_AUDIO_SPEAKER_GATE_SECONDS,
+        min_speaker_similarity=QUALITY_MIN_SPEAKER_SIMILARITY,
+        max_boundary_speaker_drop=QUALITY_MAX_BOUNDARY_SPEAKER_DROP,
+        max_pace_cps=QUALITY_MAX_PACE_CPS,
+    )
+
+
+def _assemble_trajectory_audio(
+    trajectory: tuple[np.ndarray, ...],
+    chunks: tuple[str, ...],
+    playback_speed: float,
+) -> np.ndarray:
+    """Assemble chunks exactly as they will be returned to the listener."""
+
+    if not trajectory or len(trajectory) != len(chunks):
+        raise ValueError("trajectory and text chunks must be non-empty and aligned")
+    audio_chunks = [np.asarray(audio, dtype=np.float32).copy() for audio in trajectory]
+    pauses: list[int] = []
+    for index, chunk in enumerate(chunks):
+        if index > 0:
+            audio_chunks[index] = match_chunk_rms(
+                audio_chunks[0],
+                audio_chunks[index],
+                max_adjust_db=CHUNK_RMS_MATCH_DB,
+            )
+        if index + 1 < len(chunks):
+            pauses.append(int(round(punctuation_pause_seconds(chunk) * SR)))
+
+    audio_chunks = fade_internal_edges(audio_chunks, SR, fade_ms=CHUNK_EDGE_FADE_MS)
+    waveform = join_audio_chunks(
+        audio_chunks,
+        pauses,
+        crossfade_samples=int(round(CROSSFADE_MS * SR / 1000.0)),
+    )
+    waveform = apply_loudness_floor(
+        waveform,
+        min_rms=0.07,
+        peak_limit=0.95,
+        max_gain=3.0,
+    )
+    waveform = _apply_speed(waveform, playback_speed)
+    return finish_audio(waveform, SR)
+
+
+def _verification_metric_log_fields(verification) -> str:
+    """Format normalized semantic metrics without logging transcript content."""
+
+    if len(verification.candidate_results) != 1:
+        return (
+            "cer=nan prefix_cer=nan suffix_cer=nan tail_units=nan "
+            f"reasons={verification.rejection_reasons}"
+        )
+    comparison = verification.candidate_results[0].comparison
+    return (
+        f"cer={comparison.cer:.6f} prefix_cer={comparison.prefix_cer:.6f} "
+        f"suffix_cer={comparison.suffix_cer:.6f} "
+        f"tail_units={comparison.extra_tail_units} "
+        f"reasons={verification.rejection_reasons}"
+    )
+
+
+def _qualify_candidate_trajectory_audio(
+    trajectory: tuple[np.ndarray, ...],
+    chunks: tuple[str, ...],
+    whole_target_text: str,
+    anchor: np.ndarray,
+    playback_speed: float,
+    *,
+    candidate_seed: int,
+):
+    """Run whole-output qualification only after every local chunk passes."""
+
+    local_verification = _verify_trajectory_audio(
+        trajectory,
+        chunks,
+        anchor,
+        playback_speed,
+    )
+    if not local_verification.passed:
+        return local_verification
+
+    waveform = _assemble_trajectory_audio(trajectory, chunks, playback_speed)
+    joined_verification = _verify_trajectory_audio(
+        (waveform,),
+        (whole_target_text,),
+        anchor,
+        1.0,
+        QUALITY_FINAL_ASR_MAX_NEW_TOKENS,
+    )
+    qualified = qualify_trajectory_with_joined_output(
+        local_verification,
+        joined_verification,
+    )
+    if not qualified.passed:
+        print(
+            "[BlueMagpie] candidate joined output rejected "
+            f"seed={candidate_seed} "
+            f"{_verification_metric_log_fields(joined_verification)}"
+        )
+    return qualified
+
+
+def _verify_sequence_trajectory_audio(
+    sequence_result,
+    chunks: tuple[str, ...],
+    whole_target_text: str,
+    anchor: np.ndarray,
+    playback_speed: float,
+):
+    """Verify one ranked DP path after exact production assembly."""
+
+    waveform = _assemble_trajectory_audio(
+        sequence_result.trajectory,
+        chunks,
+        playback_speed,
+    )
+    verification = _verify_trajectory_audio(
+        (waveform,),
+        (whole_target_text,),
+        anchor,
+        1.0,
+        QUALITY_FINAL_ASR_MAX_NEW_TOKENS,
+    )
+    status = "verified" if verification.passed else "rejected"
+    print(
+        f"[BlueMagpie] sequence path {status} "
+        f"rank={sequence_result.sequence_path_rank} "
+        f"chunk_candidates={sequence_result.chunk_candidate_indices} "
+        f"{_verification_metric_log_fields(verification)}"
+    )
+    return verification
+
+
 def _synthesize(
     text: str,
     centroid: torch.Tensor,
@@ -255,12 +569,15 @@ def _synthesize(
     cfg: float,
     steps: int,
     speed: float,
+    request_seed: int | None = None,
 ) -> tuple[int, np.ndarray]:
-    text = normalize_tts_text(text)
+    text = normalize_spoken_forms(text, locale="zh-TW")
     if not text:
         raise gr.Error("請先輸入要合成的文字。")
     if len(text) > MAX_TEXT_CHARS:
         raise gr.Error(f"單次最多 {MAX_TEXT_CHARS} 個字元，請分段合成。")
+    if not np.isfinite(float(speed)) or not 0.85 <= float(speed) <= 1.05:
+        raise gr.Error("後處理語速必須介於 0.85 與 1.05。")
 
     chunks = split_text_for_tts(text, max_chars=CHUNK_CHARS, min_chunk_chars=MIN_CHUNK_CHARS)
     if chunks:
@@ -270,33 +587,96 @@ def _synthesize(
             min_chunk_chars=MIN_CHUNK_CHARS,
         )
         chunks = onset_chunks + chunks[1:]
-    request_seed = secrets.randbelow(2**31)
-    audio_chunks: list[np.ndarray] = []
-    pauses: list[int] = []
-    with _GENERATION_LOCK:
-        for index, chunk in enumerate(chunks):
-            audio = _generate_chunk(
-                chunk,
-                centroid,
-                cfg=cfg,
-                steps=steps,
-                request_seed=request_seed,
+    request_seed = resolve_request_seed(request_seed, secrets.randbelow)
+    max_candidates = candidate_limit_for_chunk_budget(
+        len(chunks),
+        max_candidates=QUALITY_MAX_CANDIDATES,
+        max_generated_chunks=QUALITY_MAX_GENERATED_CHUNKS,
+    )
+    anchor = _speaker_anchor_array(centroid)
+    try:
+        with _GENERATION_LOCK:
+            cascade = run_adaptive_cascade(
+                chunks,
+                request_seed,
+                lambda candidate_chunks, seed: _generate_trajectory(
+                    candidate_chunks,
+                    centroid,
+                    cfg=cfg,
+                    steps=steps,
+                    request_seed=seed,
+                    policy=generation_policy_for_candidate_offset(seed - request_seed),
+                ),
+                lambda trajectory, candidate_chunks, seed: _qualify_candidate_trajectory_audio(
+                    trajectory,
+                    candidate_chunks,
+                    text,
+                    anchor,
+                    speed,
+                    candidate_seed=seed,
+                ),
+                max_candidates=max_candidates,
+                preferred_min_speaker_similarity=(
+                    QUALITY_PREFERRED_MIN_SPEAKER_SIMILARITY
+                ),
+                preferred_max_boundary_speaker_drop=(
+                    QUALITY_PREFERRED_MAX_BOUNDARY_SPEAKER_DROP
+                ),
+                sequence_final_verifier=lambda sequence_result, candidate_chunks: (
+                    _verify_sequence_trajectory_audio(
+                        sequence_result,
+                        candidate_chunks,
+                        text,
+                        anchor,
+                        speed,
+                    )
+                ),
+                max_sequence_paths=3,
             )
-            if audio_chunks:
-                audio = match_chunk_rms(audio_chunks[0], audio, max_adjust_db=CHUNK_RMS_MATCH_DB)
-            audio_chunks.append(audio)
-            if index + 1 < len(chunks):
-                pauses.append(int(round(punctuation_pause_seconds(chunk) * SR)))
-
-    audio_chunks = fade_internal_edges(audio_chunks, SR, fade_ms=CHUNK_EDGE_FADE_MS)
-    waveform = join_audio_chunks(
-        audio_chunks,
-        pauses,
-        crossfade_samples=int(round(CROSSFADE_MS * SR / 1000.0)),
+    except NoQualifiedCandidateError as error:
+        raise gr.Error("目前沒有候選通過內容與音色驗證，請稍後重試或調整文字。") from error
+    except (RuntimeError, ValueError) as error:
+        raise gr.Error("品質驗證暫時無法完成，未回傳未驗證的語音。") from error
+
+    selected_policies = tuple(
+        generation_policy_for_candidate_offset(index).name
+        for index in cascade.chunk_candidate_indices
+    )
+    attempted_policies = tuple(
+        generation_policy_for_candidate_offset(seed - request_seed).name
+        for seed in cascade.attempted_seeds
+    )
+    print(
+        "[BlueMagpie] quality cascade "
+        f"candidate_index={cascade.candidate_index} attempts={len(cascade.attempted_seeds)} "
+        f"candidate_limit={max_candidates} selection={cascade.selection_mode} "
+        f"chunk_candidates={cascade.chunk_candidate_indices} "
+        f"chunk_seeds={cascade.chunk_seeds} score={cascade.verification.score:.6f}"
+        f" chunk_policies={selected_policies} attempted_policies={attempted_policies}"
+        f" sequence_rank={cascade.sequence_path_rank}"
+        f" sequence_paths_checked={cascade.sequence_paths_checked}"
+    )
+    waveform = _assemble_trajectory_audio(cascade.trajectory, chunks, speed)
+    final_verification = _verify_trajectory_audio(
+        (waveform,),
+        (text,),
+        anchor,
+        1.0,
+        QUALITY_FINAL_ASR_MAX_NEW_TOKENS,
+    )
+    try:
+        require_verified_final_output(final_verification)
+    except FinalOutputRejectedError as error:
+        print(
+            "[BlueMagpie] final output rejected "
+            f"{_verification_metric_log_fields(final_verification)}"
+        )
+        raise gr.Error("最終合成結果未通過整段內容、語速與音色驗證，未回傳音訊。") from error
+    print(
+        "[BlueMagpie] final output verified "
+        f"score={final_verification.score:.6f}"
     )
-    waveform = apply_loudness_floor(waveform, min_rms=0.07, peak_limit=0.95, max_gain=3.0)
-    waveform = _apply_speed(waveform, speed)
-    return SR, finish_audio(waveform, SR)
+    return SR, waveform
 
 
 @gpu
@@ -380,12 +760,19 @@ HEADER = f"""
 
 台灣華語與中英混合文字轉語音。模型版本：`{CHECKPOINT}`。
 
-執行環境固定於已驗證的 model revision `{MODEL_REVISION[:8]}` 與 ECAPA revision
-`{ECAPA_REVISION[:8]}`，避免服務重啟時無聲變更權重或 speaker embedding 空間。
+執行環境固定於已驗證的 model revision `{MODEL_REVISION[:8]}`、ECAPA revision
+`{ECAPA_REVISION[:8]}` 與 Whisper revision `{WHISPER_REVISION[:8]}`，避免服務重啟時
+無聲變更權重、speaker embedding 或語意驗證空間。
 
 目前預設採用穩定推論設定：`CFG 2.0`（極短句最低 `3.0`）、`NFE 10`、目標語速 `4.0 字/秒`、
-生成上限依模型原生語速計算、生成完成後校正至目標語速、補齊句末提示、尾端 weak-stop 保護、
-只在自然標點切開首段、每 80 字切段，且不使用 retry 或 rerank。
+candidate 0 使用 base duration estimate（{BASE_GENERATION_POLICY.cjk_cps:.1f} CJK /
+{BASE_GENERATION_POLICY.ascii_cps:.1f} ASCII），後續候選使用 safe duration estimate
+（{SAFE_DURATION_GENERATION_POLICY.cjk_cps:.1f} CJK /
+{SAFE_DURATION_GENERATION_POLICY.ascii_cps:.1f} ASCII）；兩者只調整生成上限，生成完成後才校正至
+目標語速。另補齊句末提示、套用尾端 weak-stop 保護、
+只在自然標點切開首段、每 80 字切段；先選完整 same-seed trajectory，失敗時才以 speaker/RMS
+transition 做逐 chunk DP fallback。短句最多擴展到 1→5→10，長文依 chunk 數縮小候選上限，
+確保每個 request 最多生成 20 個 TTS chunks。
 """
 
 
@@ -396,7 +783,7 @@ with gr.Blocks(title="BlueMagpie-TTS Demo", theme=gr.themes.Soft()) as demo:
         with gr.Row():
             cfg_input = gr.Slider(1.0, 4.0, value=DEFAULT_CFG, step=0.1, label="CFG")
             steps_input = gr.Slider(4, 20, value=DEFAULT_STEPS, step=1, label="NFE steps")
-            speed_input = gr.Slider(0.85, 1.1, value=1.0, step=0.05, label="後處理語速")
+            speed_input = gr.Slider(0.85, 1.05, value=1.0, step=0.05, label="後處理語速")
 
     with gr.Tab("內建語者"):
         with gr.Row():

bluemagpie/LICENSE.upstream

@@ -0,0 +1,201 @@
+Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2026 OpenFormosa
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

bluemagpie/UPSTREAM_RUNTIME.md

@@ -0,0 +1,12 @@
+# Vendored BlueMagpie runtime
+
+The Python runtime in this directory is copied byte-for-byte (apart from
+excluded optional `mlx`, `serving`, and cache directories) from:
+
+- Repository: `https://github.com/OpenFormosa/BlueMagpie-TTS`
+- Commit: `ce384c8cc54efea1aaba7b9f1d7ded6c1c99aa9a`
+
+It is vendored so a Hugging Face Space rebuild cannot silently resolve a
+different BlueMagpie or transitive Barbet revision. The upstream Apache-2.0
+license is preserved in `LICENSE.upstream`; the embedded VoxCPM attribution is
+preserved under `_vendor/voxcpm/`.

bluemagpie/__init__.py

@@ -0,0 +1,29 @@
+"""BlueMagpie-TTS: Barbet TSLM on top of VoxCPM2's acoustic stack."""
+
+from .adapter import ProjectionAdapter
+from .centroid import extract_speaker_centroid
+from .conditioning import SpeakerProjector
+from .config import AdapterConfig, BlueMagpieConfig
+from .loading import (
+    build_config_from_voxcpm2,
+    build_from_pretrained,
+    load_voxcpm2_teacher,
+    set_training_stage,
+)
+from .model import BlueMagpieModel
+from .tslm import BarbetStepState, BarbetTSLM
+
+__all__ = [
+    "AdapterConfig",
+    "BlueMagpieConfig",
+    "BlueMagpieModel",
+    "BarbetTSLM",
+    "BarbetStepState",
+    "ProjectionAdapter",
+    "SpeakerProjector",
+    "extract_speaker_centroid",
+    "build_config_from_voxcpm2",
+    "build_from_pretrained",
+    "load_voxcpm2_teacher",
+    "set_training_stage",
+]

bluemagpie/_vendor/__init__.py

@@ -0,0 +1,6 @@
+"""Vendored third-party code bundled with BlueMagpie-TTS.
+
+Subpackages here are copied from external projects so that BlueMagpie-TTS has no
+external git dependency on them. See each subpackage's ``PROVENANCE.md`` /
+``LICENSE`` for source and licensing.
+"""

bluemagpie/_vendor/voxcpm/LICENSE

@@ -0,0 +1,201 @@
+Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright OpenBMB
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

bluemagpie/_vendor/voxcpm/PROVENANCE.md

@@ -0,0 +1,35 @@
+# Vendored VoxCPM (inference subset)
+
+This directory contains a **partial copy** of [VoxCPM](https://github.com/OpenBMB/VoxCPM),
+bundled into BlueMagpie-TTS so the package has no external git dependency on
+`voxcpm`.
+
+- **Source:** https://github.com/OpenBMB/VoxCPM.git
+- **Commit:** `856d2fc2a853656e324e491706d1e8a6bfac361c`
+- **License:** Apache-2.0 (see `LICENSE` in this directory), Copyright OpenBMB.
+
+## What was copied
+
+Only the modules BlueMagpie-TTS uses at inference time:
+
+- `model/utils.py` — runtime helpers (`get_dtype`, `next_and_close`,
+  `pick_runtime_dtype`, `resolve_runtime_device`).
+- `model/voxcpm2.py` — VoxCPM2 config classes (`VoxCPMConfig`,
+  `VoxCPMDitConfig`, `VoxCPMEncoderConfig`) and the `VoxCPM2Model` reference
+  implementation.
+- `modules/audiovae/`, `modules/layers/`, `modules/locdit/`, `modules/locenc/`,
+  `modules/minicpm4/` — the AudioVAE / LocEnc / RALM / LocDiT acoustic stack.
+
+## What was changed
+
+- The package `__init__.py` files (`voxcpm/__init__.py`, `voxcpm/model/__init__.py`)
+  were replaced with empty stubs so importing this subset does **not** pull in
+  the upstream `core` pipeline or the v1 `VoxCPMModel`.
+- Nothing else was modified; all module source is upstream-verbatim, and all
+  internal imports are relative, so the subtree is self-contained.
+
+## Updating
+
+To refresh against a newer VoxCPM release, re-copy the files listed above from
+the upstream commit, keep the two stub `__init__.py` files, and re-run the test
+suite.

bluemagpie/_vendor/voxcpm/__init__.py

@@ -0,0 +1,17 @@
+"""Vendored subset of VoxCPM (https://github.com/OpenBMB/VoxCPM), Apache-2.0.
+
+Only the modules BlueMagpie-TTS needs at inference time are bundled here:
+VoxCPM2's acoustic stack (AudioVAE / LocEnc / RALM / LocDiT) and its config /
+runtime helpers. The original top-level ``voxcpm`` package eagerly imported the
+full ``VoxCPM`` pipeline (``core``), which pulls in dependencies BlueMagpie-TTS
+does not use, so this ``__init__`` is intentionally empty.
+
+Modified from upstream: the original ``voxcpm/__init__.py`` imported
+``from .core import VoxCPM``; that line was removed and replaced with this stub.
+
+Import the concrete submodules directly, e.g.::
+
+    from bluemagpie._vendor.voxcpm.modules.audiovae import AudioVAEV2
+
+See ``LICENSE`` and ``PROVENANCE.md`` in this directory for details.
+"""

bluemagpie/_vendor/voxcpm/model/__init__.py

@@ -0,0 +1,14 @@
+"""VoxCPM2 model package (vendored subset).
+
+The upstream ``voxcpm.model`` package also exposed the v1 ``VoxCPMModel``;
+BlueMagpie-TTS only uses VoxCPM2, so this ``__init__`` is empty and callers
+import the concrete submodules directly.
+
+Modified from upstream: the original ``model/__init__.py`` imported
+``from .voxcpm import VoxCPMModel`` and ``from .voxcpm2 import VoxCPM2Model``;
+those lines were removed and replaced with this stub (the v1 ``voxcpm.py`` is
+not vendored). Example::
+
+    from bluemagpie._vendor.voxcpm.model.voxcpm2 import VoxCPMConfig
+    from bluemagpie._vendor.voxcpm.model.utils import get_dtype
+"""

bluemagpie/_vendor/voxcpm/model/utils.py

@@ -0,0 +1,231 @@
+import os
+from typing import List, Optional
+import torch
+from transformers import PreTrainedTokenizer
+
+_LOW_PRECISION_DTYPES = {"bfloat16", "bf16", "float16", "fp16"}
+_VALID_DTYPE_OVERRIDES = {
+    "bfloat16", "bf16",
+    "float16", "fp16",
+    "float32", "fp32",
+}
+
+
+# Ref: https://github.com/OpenBMB/VoxCPM/issues/256#issuecomment-4235252732
+# Explicitly close partially-consumed generators so inference_mode cleanup
+# does not get deferred to Python's GC/finalizer path.
+def next_and_close(gen):
+    try:
+        return next(gen)
+    finally:
+        gen.close()
+
+
+def mask_multichar_chinese_tokens(tokenizer: PreTrainedTokenizer):
+    """Create a tokenizer wrapper that converts multi-character Chinese tokens to single characters.
+
+    This function creates a wrapper around the provided tokenizer that automatically
+    splits multi-character Chinese tokens into individual characters. This is useful
+    for ensuring consistent tokenization of Chinese text.
+
+    Args:
+        tokenizer: The base tokenizer to wrap
+
+    Returns:
+        A CharTokenizerWrapper instance that handles multi-character Chinese tokens
+
+    Example:
+        >>> from transformers import LlamaTokenizerFast
+        >>> tokenizer = LlamaTokenizerFast.from_pretrained("path/to/tokenizer")
+        >>> wrapped_tokenizer = mask_multichar_chinese_tokens(tokenizer)
+        >>> tokens = wrapped_tokenizer("你好世界")
+    """
+    # Pre-compute multi-character tokens (length >= 2, pure Chinese characters)
+    multichar_tokens = {
+        token for token in tokenizer.vocab.keys() if len(token) >= 2 and all("\u4e00" <= c <= "\u9fff" for c in token)
+    }
+
+    class CharTokenizerWrapper:
+        """Wrapper class for tokenizers that handles multi-character Chinese tokens.
+
+        This wrapper automatically splits multi-character Chinese tokens into
+        individual characters while preserving the original tokenizer's interface.
+        """
+
+        def __init__(self, base_tokenizer: PreTrainedTokenizer) -> None:
+            """Initialize the wrapper with a base tokenizer.
+
+            Args:
+                base_tokenizer: The tokenizer to wrap
+            """
+            self.tokenizer = base_tokenizer
+            self.multichar_tokens = multichar_tokens
+
+        def tokenize(self, text: str, **kwargs) -> List[str]:
+            """Tokenize text and split multi-character Chinese tokens into single characters.
+
+            Args:
+                text: Input text to tokenize
+                **kwargs: Additional arguments passed to the base tokenizer
+
+            Returns:
+                List of processed tokens with multi-character Chinese tokens split
+
+            Example:
+                >>> wrapper = CharTokenizerWrapper(tokenizer)
+                >>> tokens = wrapper.tokenize("你好世界")
+                >>> # Returns ["你", "好", "世", "界"] instead of ["你好", "世界"]
+            """
+            if not isinstance(text, str):
+                raise TypeError(f"Expected string input, got {type(text)}")
+
+            tokens = self.tokenizer.tokenize(text, **kwargs)
+            processed = []
+
+            for token in tokens:
+                # Remove possible subword prefix
+                clean_token = token.replace("▁", "")
+
+                if clean_token in self.multichar_tokens:
+                    # Split multi-character token into single characters
+                    chars = list(clean_token)
+                    processed.extend(chars)
+                else:
+                    processed.append(token)
+
+            return processed
+
+        def __call__(self, text: str, **kwargs) -> List[int]:
+            """Call the tokenizer and return token IDs.
+
+            This method provides the same interface as the original tokenizer
+            but with multi-character Chinese token handling.
+
+            Args:
+                text: Input text to tokenize
+                **kwargs: Additional arguments passed to the base tokenizer
+
+            Returns:
+                List of token IDs
+
+            Raises:
+                TypeError: If input is not a string
+                ValueError: If tokenization fails
+            """
+            try:
+                tokens = self.tokenize(text, **kwargs)
+                result = self.tokenizer.convert_tokens_to_ids(tokens)
+                return result
+            except Exception as e:
+                raise ValueError(f"Tokenization failed: {str(e)}") from e
+
+    return CharTokenizerWrapper(tokenizer)
+
+
+def get_dtype(dtype: str):
+    if dtype == "bfloat16":
+        return torch.bfloat16
+    elif dtype == "bf16":
+        return torch.bfloat16
+    elif dtype == "float16":
+        return torch.float16
+    elif dtype == "fp16":
+        return torch.float16
+    elif dtype == "float32":
+        return torch.float32
+    elif dtype == "fp32":
+        return torch.float32
+    else:
+        raise ValueError(f"Unsupported dtype: {dtype}")
+
+
+def _has_mps() -> bool:
+    return hasattr(torch.backends, "mps") and torch.backends.mps.is_available()
+
+
+def pick_runtime_dtype(device: str, configured_dtype: str) -> str:
+    """Pick a safe runtime dtype for the resolved device.
+
+    On Apple Silicon (MPS), bfloat16/float16 produce enough numerical drift
+    in the diffusion AR loop that the output is glitched and the model's
+    badcase detector triggers infinite retries. float32 is the only stable
+    option today. CUDA and CPU keep whatever the checkpoint was trained with.
+
+    Users can override with ``VOXCPM_MPS_DTYPE`` (e.g. ``bfloat16``) when
+    they want to test future MPS improvements.
+    """
+    if device != "mps":
+        return configured_dtype
+
+    override = os.environ.get("VOXCPM_MPS_DTYPE", "").strip().lower()
+    if override:
+        if override not in _VALID_DTYPE_OVERRIDES:
+            raise ValueError(
+                f"VOXCPM_MPS_DTYPE='{override}' is not one of "
+                f"{sorted(_VALID_DTYPE_OVERRIDES)}"
+            )
+        return override
+
+    if (configured_dtype or "").lower() in _LOW_PRECISION_DTYPES:
+        return "float32"
+    return configured_dtype
+
+
+def auto_select_device(preferred_device: Optional[str] = "cuda") -> str:
+    """
+    Choose a runtime device automatically.
+
+    Preference order:
+    - if the preferred device is available, use it
+    - otherwise fall back to CUDA -> MPS -> CPU
+    """
+    preferred = (preferred_device or "cuda").strip().lower()
+
+    if preferred.startswith("cuda") and torch.cuda.is_available():
+        return preferred
+    if preferred == "mps" and _has_mps():
+        return "mps"
+    if preferred == "cpu":
+        return "cpu"
+
+    if torch.cuda.is_available():
+        return "cuda"
+    if _has_mps():
+        return "mps"
+    return "cpu"
+
+
+def resolve_runtime_device(device: Optional[str], configured_device: str = "cuda") -> str:
+    """
+    Resolve the actual runtime device.
+
+    Semantics:
+    - ``device`` is ``None`` or ``"auto"``: use automatic fallback selection
+    - otherwise: treat it as an explicit user choice and validate availability
+    """
+    explicit = None if device is None else device.strip().lower()
+
+    if explicit is None or explicit == "auto":
+        return auto_select_device(configured_device)
+
+    if explicit.startswith("cuda"):
+        if not torch.cuda.is_available():
+            raise ValueError(
+                f"Requested device '{device}', but CUDA is not available. "
+                "Use device='auto' for automatic fallback."
+            )
+        return explicit
+    if explicit == "mps":
+        if not _has_mps():
+            raise ValueError(
+                "Requested device 'mps', but MPS is not available. "
+                "Use device='auto' for automatic fallback."
+            )
+        return "mps"
+    if explicit == "cpu":
+        return "cpu"
+
+    raise ValueError(
+        f"Unsupported device '{device}'. Supported values are 'auto', 'cpu', 'mps', "
+        "'cuda', or indexed CUDA devices like 'cuda:0'."
+    )

bluemagpie/_vendor/voxcpm/model/voxcpm2.py

@@ -0,0 +1,1256 @@
+"""
+VoxCPM: A Tokenizer-free speech generation model
+
+This module contains the main VoxCPM model implementation, including configuration classes
+and the core VoxCPMModel for text-to-speech generation.
+
+Copyright 2026 OpenBMB
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+"""
+
+import os
+import sys
+from typing import Tuple, Union, Generator, List, Optional
+
+import torch
+import torch.nn as nn
+import warnings
+import librosa
+import numpy as np
+from einops import rearrange
+from pydantic import BaseModel
+
+try:
+    from safetensors.torch import load_file
+
+    SAFETENSORS_AVAILABLE = True
+except ImportError:
+    SAFETENSORS_AVAILABLE = False
+from tqdm import tqdm
+from transformers import LlamaTokenizerFast
+
+from ..modules.audiovae import AudioVAEV2, AudioVAEConfigV2
+from ..modules.layers import ScalarQuantizationLayer
+from ..modules.layers.lora import apply_lora_to_named_linear_modules
+from ..modules.locdit import CfmConfig, UnifiedCFM, VoxCPMLocDiTV2
+from ..modules.locenc import VoxCPMLocEnc
+from ..modules.minicpm4 import MiniCPM4Config, MiniCPMModel
+from .utils import (
+    get_dtype,
+    mask_multichar_chinese_tokens,
+    next_and_close,
+    pick_runtime_dtype,
+    resolve_runtime_device,
+)
+
+
+# A simple function to trim audio silence using VAD, not used default
+def _trim_audio_silence_vad(audio: torch.Tensor, sample_rate: int, max_silence_ms: float = 200.0, top_db: float = 35.0) -> torch.Tensor:
+    if audio.numel() == 0:
+        return audio
+    y = audio.squeeze(0).numpy()
+    n = len(y)
+    frame_length = 2048
+    hop_length = 512
+    ref = np.max(np.abs(y))
+    if ref <= 0:
+        return audio
+    threshold = ref * (10.0 ** (-top_db / 20.0))
+
+    try:
+        _, (start, end) = librosa.effects.trim(
+            y, top_db=top_db, ref=np.max, frame_length=frame_length, hop_length=hop_length
+        )
+    except Exception:
+        start, end = 0, n
+
+    # Find the last frame with continuous energy, trim the long pseudo-silence at the end (low energy background noise, etc.)
+    n_frames = max(0, (n - frame_length) // hop_length + 1)
+    last_voice_frame = -1
+    for j in range(n_frames):
+        idx = j * hop_length
+        if idx + frame_length > n:
+            break
+        rms = np.sqrt(np.mean(y[idx : idx + frame_length] ** 2))
+        if rms >= threshold:
+            last_voice_frame = j
+    if last_voice_frame >= 0:
+        end_by_vad = min(n, (last_voice_frame + 1) * hop_length + (frame_length - hop_length))
+        end = min(end, end_by_vad)
+
+    max_silence_samples = int(max_silence_ms * sample_rate / 1000.0)
+    new_start = max(0, start - max_silence_samples)
+    new_end = min(n, end + max_silence_samples)
+    return audio[:, new_start:new_end]
+
+
+class VoxCPMEncoderConfig(BaseModel):
+    hidden_dim: int = 1024
+    ffn_dim: int = 4096
+    num_heads: int = 16
+    num_layers: int = 4
+    kv_channels: int = None
+
+
+class VoxCPMDitConfig(BaseModel):
+    hidden_dim: int = 1024
+    ffn_dim: int = 4096
+    num_heads: int = 16
+    num_layers: int = 4
+    kv_channels: int = None
+    dit_mean_mode: bool = False
+
+    cfm_config: CfmConfig
+
+
+class VoxCPMConfig(BaseModel):
+    lm_config: MiniCPM4Config
+    patch_size: int = 4
+    feat_dim: int = 64
+    residual_lm_num_layers: int = 8
+    residual_lm_no_rope: bool = False
+    scalar_quantization_latent_dim: int = 512
+    scalar_quantization_scale: int = 9
+
+    encoder_config: VoxCPMEncoderConfig
+    dit_config: VoxCPMDitConfig
+    audio_vae_config: Optional[AudioVAEConfigV2] = None
+
+    max_length: int = 8192
+    device: str = "cuda"
+    dtype: str = "bfloat16"
+
+
+class LoRAConfig(BaseModel):
+    enable_lm: bool = False  # Apply LoRA to base_lm + residual_lm
+    enable_dit: bool = False  # Apply LoRA to VoxCPMLocDiT
+    enable_proj: bool = False  # Apply LoRA to projection Linear layers
+
+    r: int = 8
+    alpha: int = 16
+    dropout: float = 0.0
+
+    # Target linear layer names for LM & DiT (matched by attribute name)
+    target_modules_lm: list[str] = ["q_proj", "v_proj", "k_proj", "o_proj"]
+    target_modules_dit: list[str] = ["q_proj", "v_proj", "k_proj", "o_proj"]
+    # Projection layer attribute names to find on VoxCPM2Model
+    target_proj_modules: list[str] = ["enc_to_lm_proj", "lm_to_dit_proj", "res_to_dit_proj", "fusion_concat_proj"]
+
+
+VoxCPMConfig.model_rebuild()
+
+
+class VoxCPM2Model(nn.Module):
+    def __init__(
+        self,
+        config: VoxCPMConfig,
+        tokenizer: LlamaTokenizerFast,
+        audio_vae: AudioVAEV2,
+        lora_config: LoRAConfig = None,
+        device: str | None = None,
+    ):
+        super().__init__()
+        self.config = config
+        self.lora_config = lora_config
+        self.feat_dim = config.feat_dim
+        self.patch_size = config.patch_size
+        self.device = resolve_runtime_device(device, config.device)
+        self.config.device = self.device
+        resolved_dtype = pick_runtime_dtype(self.device, self.config.dtype)
+        if resolved_dtype != self.config.dtype:
+            print(
+                f"[voxcpm2] adjusted dtype {self.config.dtype} -> {resolved_dtype} for device {self.device}",
+                file=sys.stderr,
+            )
+            self.config.dtype = resolved_dtype
+        print(f"Running on device: {self.device}, dtype: {self.config.dtype}", file=sys.stderr)
+
+        # Text-Semantic LM
+        self.base_lm = MiniCPMModel(config.lm_config)
+        self.base_lm.setup_cache(1, config.max_length, self.device, get_dtype(self.config.dtype))
+
+        self.text_tokenizer = mask_multichar_chinese_tokens(tokenizer)
+        self.audio_start_token = 101
+        self.audio_end_token = 102
+        self.ref_audio_start_token = 103
+        self.ref_audio_end_token = 104
+
+        # Residual Acoustic LM
+        residual_lm_config = config.lm_config.model_copy(deep=True)
+        residual_lm_config.num_hidden_layers = config.residual_lm_num_layers
+        residual_lm_config.vocab_size = 0
+        residual_lm_config.no_rope = config.residual_lm_no_rope
+        self.residual_lm = MiniCPMModel(residual_lm_config)
+        self.residual_lm.setup_cache(1, config.max_length, self.device, get_dtype(self.config.dtype))
+
+        # Local Encoder
+        encoder_config = config.lm_config.model_copy(deep=True)
+        encoder_config.hidden_size = config.encoder_config.hidden_dim
+        encoder_config.intermediate_size = config.encoder_config.ffn_dim
+        encoder_config.num_attention_heads = config.encoder_config.num_heads
+        encoder_config.num_hidden_layers = config.encoder_config.num_layers
+        encoder_config.kv_channels = config.encoder_config.kv_channels
+        encoder_config.vocab_size = 0
+        self.feat_encoder = VoxCPMLocEnc(encoder_config, input_dim=config.feat_dim)
+
+        # Local DiT
+        decoder_config = config.lm_config.model_copy(deep=True)
+        decoder_config.hidden_size = config.dit_config.hidden_dim
+        decoder_config.intermediate_size = config.dit_config.ffn_dim
+        decoder_config.num_attention_heads = config.dit_config.num_heads
+        decoder_config.num_hidden_layers = config.dit_config.num_layers
+        decoder_config.kv_channels = config.dit_config.kv_channels
+        decoder_config.vocab_size = 0
+        self.feat_decoder = UnifiedCFM(
+            in_channels=config.feat_dim,
+            cfm_params=config.dit_config.cfm_config,
+            estimator=VoxCPMLocDiTV2(decoder_config, in_channels=config.feat_dim),
+            mean_mode=config.dit_config.dit_mean_mode,
+        )
+
+        # Projection layers
+        self.fsq_layer = ScalarQuantizationLayer(
+            config.lm_config.hidden_size,
+            config.lm_config.hidden_size,
+            config.scalar_quantization_latent_dim,
+            config.scalar_quantization_scale,
+        )
+        self.enc_to_lm_proj = nn.Linear(config.encoder_config.hidden_dim, config.lm_config.hidden_size)
+        self.lm_to_dit_proj = nn.Linear(config.lm_config.hidden_size, config.dit_config.hidden_dim)
+        self.res_to_dit_proj = nn.Linear(config.lm_config.hidden_size, config.dit_config.hidden_dim)
+        self.fusion_concat_proj = nn.Linear(config.lm_config.hidden_size * 2, config.lm_config.hidden_size)
+
+        # Stop Predictor
+        self.stop_proj = nn.Linear(config.lm_config.hidden_size, config.lm_config.hidden_size)
+        self.stop_actn = nn.SiLU()
+        self.stop_head = nn.Linear(config.lm_config.hidden_size, 2, bias=False)
+        self.stop_loss = nn.CrossEntropyLoss(reduction="none")
+
+        # Audio VAE
+        self.audio_vae = audio_vae
+        self.chunk_size = audio_vae.chunk_size
+        self._decode_chunk_size = getattr(audio_vae, "decode_chunk_size", audio_vae.chunk_size)
+        self._encode_sample_rate = audio_vae.sample_rate
+        self.sample_rate = getattr(audio_vae, "out_sample_rate", audio_vae.sample_rate)
+
+        if self.lora_config is not None:
+            self._apply_lora()
+
+    def _apply_lora(self):
+        """注入 LoRA 到 LM / DiT / 投影层"""
+        cfg = self.lora_config
+        lora_kwargs = dict(r=cfg.r, alpha=cfg.alpha, dropout=cfg.dropout)
+
+        # LM: base_lm + residual_lm
+        if cfg.enable_lm:
+            for lm in [self.base_lm, self.residual_lm]:
+                apply_lora_to_named_linear_modules(lm, target_submodule_names=cfg.target_modules_lm, **lora_kwargs)
+
+        # DiT: feat_decoder.estimator
+        if cfg.enable_dit:
+            apply_lora_to_named_linear_modules(
+                self.feat_decoder.estimator, target_submodule_names=cfg.target_modules_dit, **lora_kwargs
+            )
+
+        # 投影层
+        if cfg.enable_proj:
+            from ..modules.layers.lora import LoRALinear
+
+            for attr_name in cfg.target_proj_modules:
+                module = getattr(self, attr_name, None)
+                if isinstance(module, nn.Linear):
+                    setattr(self, attr_name, LoRALinear(base=module, **lora_kwargs))
+
+    def optimize(self, disable: bool = False):
+        if disable:
+            return self
+        try:
+            if self.device != "cuda":
+                raise ValueError("VoxCPMModel can only be optimized on CUDA device")
+            try:
+                import triton  # noqa: F401
+            except ImportError:
+                raise ValueError("triton is not installed")
+            self.base_lm.forward_step = torch.compile(self.base_lm.forward_step, mode="reduce-overhead", fullgraph=True)
+            self.residual_lm.forward_step = torch.compile(
+                self.residual_lm.forward_step, mode="reduce-overhead", fullgraph=True
+            )
+            self._feat_encoder_raw = self.feat_encoder
+            self.feat_encoder = torch.compile(self.feat_encoder, mode="reduce-overhead", fullgraph=True)
+            self.feat_decoder.estimator = torch.compile(
+                self.feat_decoder.estimator, mode="reduce-overhead", fullgraph=True
+            )
+        except Exception as e:
+            print(f"Warning: torch.compile disabled - {e}", file=sys.stderr)
+        return self
+
+    def forward(
+        self,
+        text_tokens: torch.Tensor,
+        text_mask: torch.Tensor,
+        audio_feats: torch.Tensor,
+        audio_mask: torch.Tensor,
+        loss_mask: torch.Tensor,
+        position_ids: torch.Tensor,
+        labels: torch.Tensor,
+        *,
+        progress: float = 0.0,
+        sample_generate: bool = False,
+    ):
+        del position_ids  # not used yet
+
+        text_tokens = text_tokens.to(self.device, dtype=torch.long)
+        text_mask = text_mask.to(self.device, dtype=self._dtype())
+        audio_feats = audio_feats.to(self.device, dtype=self._dtype())
+        audio_mask = audio_mask.to(self.device, dtype=self._dtype())
+        loss_mask = loss_mask.to(self.device, dtype=self._dtype())
+        labels = labels.to(self.device, dtype=torch.long)
+
+        B, T, P, D = audio_feats.shape
+        feat_embed = self.feat_encoder(audio_feats)
+        feat_embed = self.enc_to_lm_proj(feat_embed)
+
+        scale_emb = getattr(self.config.lm_config, "scale_emb", 1.0)
+        if not getattr(self.config.lm_config, "use_mup", False):
+            scale_emb = 1.0
+        text_embed = self.base_lm.embed_tokens(text_tokens) * scale_emb
+        combined_embed = text_mask.unsqueeze(-1) * text_embed + audio_mask.unsqueeze(-1) * feat_embed
+
+        enc_outputs, _ = self.base_lm(inputs_embeds=combined_embed, is_causal=True)
+        enc_outputs = enc_outputs.to(self._dtype())
+        enc_outputs = self.fsq_layer(enc_outputs) * audio_mask.unsqueeze(-1) + enc_outputs * text_mask.unsqueeze(-1)
+        lm_hidden = torch.cat((torch.zeros_like(enc_outputs[:, 0:1, :]), enc_outputs[:, :-1, :]), dim=1)
+
+        residual_inputs = self.fusion_concat_proj(
+            torch.cat((enc_outputs, audio_mask.unsqueeze(-1) * feat_embed), dim=-1)
+        )
+        residual_outputs, _ = self.residual_lm(inputs_embeds=residual_inputs, is_causal=True)
+        residual_outputs = residual_outputs.to(self._dtype())
+        residual_hidden = torch.cat(
+            (torch.zeros_like(residual_outputs[:, 0:1, :]), residual_outputs[:, :-1, :]),
+            dim=1,
+        )
+
+        dit_hidden = torch.cat((self.lm_to_dit_proj(lm_hidden), self.res_to_dit_proj(residual_hidden)), dim=-1)
+        dit_hidden = rearrange(dit_hidden, "b t c -> (b t) c")
+
+        # Keep diffusion inputs in the same dtype as the model (e.g., bfloat16)
+        target_dtype = self._dtype()
+
+        feat_gt = rearrange(audio_feats.to(target_dtype), "b t p d -> (b t) p d")
+        feat_cond = torch.cat(
+            (torch.zeros_like(audio_feats[:, 0:1, ...]), audio_feats[:, :-1, ...]),
+            dim=1,
+        )
+        feat_cond = rearrange(feat_cond.to(target_dtype), "b t p d -> (b t) p d")
+
+        loss_seq_mask = loss_mask.unsqueeze(-1).repeat(1, 1, self.patch_size)
+        loss_seq_mask = rearrange(loss_seq_mask, "b t p -> (b t) p 1").to(target_dtype)
+
+        diff_loss = self.feat_decoder.compute_loss(
+            feat_gt.transpose(1, 2).contiguous(),
+            dit_hidden,
+            cond=feat_cond.transpose(1, 2).contiguous(),
+            tgt_mask=loss_seq_mask.transpose(1, 2).contiguous(),
+            progress=progress,
+        )
+
+        stop_logits = self.stop_head(self.stop_actn(self.stop_proj(lm_hidden)))
+        stop_losses = self.stop_loss(stop_logits.transpose(1, 2), labels)
+        denom = torch.clamp(loss_mask.sum(), min=1.0)
+        stop_loss = (stop_losses * loss_mask).sum() / denom
+
+        feat_pred = None
+        if sample_generate:
+            feat_cond_for_sample = feat_cond.transpose(1, 2).contiguous()
+            feat_pred_seq = self.feat_decoder(
+                mu=dit_hidden,
+                patch_size=self.patch_size,
+                cond=feat_cond_for_sample,
+                n_timesteps=10,
+            )
+            feat_pred = rearrange(feat_pred_seq.transpose(1, 2), "(b t) d p -> b d (t p)", b=B, p=self.patch_size)
+
+        feat_gt_tensor = rearrange(feat_gt, "(b t) p d -> b d (t p)", b=B, p=self.patch_size)
+
+        return {
+            "loss/diff": diff_loss,
+            "loss/stop": stop_loss,
+            "feat_gt": feat_gt_tensor,
+            "feat_pred": feat_pred,
+        }
+
+    def _dtype(self):
+        return get_dtype(self.config.dtype)
+
+    def _encode_wav(
+        self,
+        wav_path: str,
+        padding_mode: str = "right",
+        trim_silence_vad: bool = False,
+    ) -> torch.Tensor:
+        """Load, trim, pad and VAE-encode an audio file.
+
+        Args:
+            wav_path: path to the audio file.
+            padding_mode: "right" (default) or "left" padding for alignment.
+            trim_silence_vad: whether to apply VAD-based silence trimming.
+
+        Returns:
+            audio_feat: (T, P, D) tensor of latent patches.
+        """
+        audio, _ = librosa.load(wav_path, sr=self._encode_sample_rate, mono=True)
+        audio = torch.from_numpy(audio).unsqueeze(0)
+        if trim_silence_vad:
+            audio = _trim_audio_silence_vad(audio, self._encode_sample_rate, max_silence_ms=200.0)
+        patch_len = self.patch_size * self.chunk_size
+        if audio.size(1) % patch_len != 0:
+            padding_size = patch_len - audio.size(1) % patch_len
+            pad = (padding_size, 0) if padding_mode == "left" else (0, padding_size)
+            audio = torch.nn.functional.pad(audio, pad)
+        feat = self.audio_vae.encode(audio.to(self.device), self._encode_sample_rate).cpu()
+        return feat.view(self.audio_vae.latent_dim, -1, self.patch_size).permute(1, 2, 0)
+
+    def _make_ref_prefix(self, ref_feat: torch.Tensor, device: torch.device):
+        """Build the [ref_start ref_audio ref_end] prefix segments.
+
+        Returns:
+            tokens, feats, text_mask, audio_mask  (all 1-D / 2-D tensors)
+        """
+        ref_len = ref_feat.size(0)
+        z1 = torch.zeros((1, self.patch_size, self.audio_vae.latent_dim), dtype=torch.float32, device=device)
+        tokens = torch.cat(
+            [
+                torch.tensor([self.ref_audio_start_token], dtype=torch.int32, device=device),
+                torch.zeros(ref_len, dtype=torch.int32, device=device),
+                torch.tensor([self.ref_audio_end_token], dtype=torch.int32, device=device),
+            ]
+        )
+        feats = torch.cat([z1, ref_feat, z1], dim=0)
+        t_mask = torch.cat(
+            [
+                torch.tensor([1], dtype=torch.int32),
+                torch.zeros(ref_len, dtype=torch.int32),
+                torch.tensor([1], dtype=torch.int32),
+            ]
+        ).to(device)
+        a_mask = torch.cat(
+            [
+                torch.tensor([0], dtype=torch.int32),
+                torch.ones(ref_len, dtype=torch.int32),
+                torch.tensor([0], dtype=torch.int32),
+            ]
+        ).to(device)
+        return tokens, feats, t_mask, a_mask
+
+    def generate(self, *args, **kwargs) -> torch.Tensor:
+        return next_and_close(self._generate(*args, streaming=False, **kwargs))
+
+    def generate_streaming(self, *args, **kwargs) -> Generator[torch.Tensor, None, None]:
+        return self._generate(*args, streaming=True, **kwargs)
+
+    @torch.inference_mode()
+    def _generate(
+        self,
+        target_text: str,
+        prompt_text: str = "",
+        prompt_wav_path: str = "",
+        reference_wav_path: str = "",
+        min_len: int = 2,
+        max_len: int = 2000,
+        inference_timesteps: int = 10,
+        cfg_value: float = 2.0,
+        retry_badcase: bool = False,
+        retry_badcase_max_times: int = 3,
+        retry_badcase_ratio_threshold: float = 6.0,
+        trim_silence_vad: bool = False,
+        streaming: bool = False,
+        streaming_prefix_len: int = 4,
+    ) -> Generator[torch.Tensor, None, None]:
+        if retry_badcase and streaming:
+            warnings.warn("Retry on bad cases is not supported in streaming mode, setting retry_badcase=False.")
+            retry_badcase = False
+
+        if reference_wav_path and prompt_wav_path:
+            # Combined mode: reference isolation prefix + continuation suffix
+            text = prompt_text + target_text
+            text_token = torch.LongTensor(self.text_tokenizer(text))
+            text_token = torch.cat(
+                [
+                    text_token,
+                    torch.tensor([self.audio_start_token], dtype=torch.int32, device=text_token.device),
+                ],
+                dim=-1,
+            )
+            text_length = text_token.shape[0]
+
+            ref_feat = self._encode_wav(
+                reference_wav_path,
+                padding_mode="right",
+                trim_silence_vad=trim_silence_vad,
+            )
+            prompt_feat = self._encode_wav(prompt_wav_path, padding_mode="left", trim_silence_vad=trim_silence_vad)
+            prompt_audio_length = prompt_feat.size(0)
+
+            ref_tokens, ref_feats, ref_t_mask, ref_a_mask = self._make_ref_prefix(ref_feat, text_token.device)
+
+            prompt_pad_token = torch.zeros(prompt_audio_length, dtype=torch.int32, device=text_token.device)
+            text_pad_feat = torch.zeros(
+                (text_length, self.patch_size, self.audio_vae.latent_dim),
+                dtype=torch.float32,
+                device=text_token.device,
+            )
+
+            text_token = torch.cat([ref_tokens, text_token, prompt_pad_token])
+            audio_feat = torch.cat([ref_feats, text_pad_feat, prompt_feat], dim=0)
+            text_mask = torch.cat(
+                [
+                    ref_t_mask,
+                    torch.ones(text_length, dtype=torch.int32).to(text_token.device),
+                    torch.zeros(prompt_audio_length, dtype=torch.int32).to(text_token.device),
+                ]
+            )
+            audio_mask = torch.cat(
+                [
+                    ref_a_mask,
+                    torch.zeros(text_length, dtype=torch.int32).to(text_token.device),
+                    torch.ones(prompt_audio_length, dtype=torch.int32).to(text_token.device),
+                ]
+            )
+
+        elif reference_wav_path:
+            # Reference-only mode (prompt isolation)
+            text = target_text
+            text_token = torch.LongTensor(self.text_tokenizer(text))
+            text_token = torch.cat(
+                [
+                    text_token,
+                    torch.tensor([self.audio_start_token], dtype=torch.int32, device=text_token.device),
+                ],
+                dim=-1,
+            )
+            text_length = text_token.shape[0]
+
+            ref_feat = self._encode_wav(
+                reference_wav_path,
+                padding_mode="right",
+                trim_silence_vad=trim_silence_vad,
+            )
+            ref_tokens, ref_feats, ref_t_mask, ref_a_mask = self._make_ref_prefix(ref_feat, text_token.device)
+
+            text_pad_feat = torch.zeros(
+                (text_length, self.patch_size, self.audio_vae.latent_dim),
+                dtype=torch.float32,
+                device=text_token.device,
+            )
+            text_token = torch.cat([ref_tokens, text_token])
+            audio_feat = torch.cat([ref_feats, text_pad_feat], dim=0)
+            text_mask = torch.cat(
+                [
+                    ref_t_mask,
+                    torch.ones(text_length, dtype=torch.int32).to(text_token.device),
+                ]
+            )
+            audio_mask = torch.cat(
+                [
+                    ref_a_mask,
+                    torch.zeros(text_length, dtype=torch.int32).to(text_token.device),
+                ]
+            )
+
+        elif len(prompt_wav_path) == 0:
+            # Zero-shot mode
+            text = target_text
+            text_token = torch.LongTensor(self.text_tokenizer(text))
+            text_token = torch.cat(
+                [
+                    text_token,
+                    torch.tensor([self.audio_start_token], dtype=torch.int32, device=text_token.device),
+                ],
+                dim=-1,
+            )
+            text_length = text_token.shape[0]
+
+            audio_feat = torch.zeros(
+                (text_length, self.patch_size, self.audio_vae.latent_dim),
+                dtype=torch.float32,
+                device=text_token.device,
+            )
+            text_mask = torch.ones(text_length, dtype=torch.int32).to(text_token.device)
+            audio_mask = torch.zeros(text_length, dtype=torch.int32).to(text_token.device)
+
+        else:
+            # Continuation-only mode
+            text = prompt_text + target_text
+            text_token = torch.LongTensor(self.text_tokenizer(text))
+            text_token = torch.cat(
+                [
+                    text_token,
+                    torch.tensor([self.audio_start_token], dtype=torch.int32, device=text_token.device),
+                ],
+                dim=-1,
+            )
+            text_length = text_token.shape[0]
+
+            prompt_feat = self._encode_wav(prompt_wav_path, padding_mode="left", trim_silence_vad=trim_silence_vad)
+            prompt_audio_length = prompt_feat.size(0)
+            prompt_pad_token = torch.zeros(prompt_audio_length, dtype=torch.int32, device=text_token.device)
+            text_pad_feat = torch.zeros(
+                (text_length, self.patch_size, self.audio_vae.latent_dim),
+                dtype=torch.float32,
+                device=text_token.device,
+            )
+            text_token = torch.cat([text_token, prompt_pad_token])
+            audio_feat = torch.cat([text_pad_feat, prompt_feat], dim=0)
+            text_mask = torch.cat(
+                [
+                    torch.ones(text_length, dtype=torch.int32),
+                    torch.zeros(prompt_audio_length, dtype=torch.int32),
+                ]
+            ).to(text_token.device)
+            audio_mask = torch.cat(
+                [
+                    torch.zeros(text_length, dtype=torch.int32),
+                    torch.ones(prompt_audio_length, dtype=torch.int32),
+                ]
+            ).to(text_token.device)
+
+        text_token = text_token.unsqueeze(0).to(self.device)
+        text_mask = text_mask.unsqueeze(0).to(self.device)
+        audio_feat = audio_feat.unsqueeze(0).to(self.device).to(get_dtype(self.config.dtype))
+        audio_mask = audio_mask.unsqueeze(0).to(self.device)
+
+        target_text_length = len(self.text_tokenizer(target_text))
+
+        retry_badcase_times = 0
+        while retry_badcase_times < retry_badcase_max_times:
+            inference_result = self._inference(
+                text_token,
+                text_mask,
+                audio_feat,
+                audio_mask,
+                min_len=min_len,
+                max_len=min(int(target_text_length * retry_badcase_ratio_threshold + 10), max_len),
+                inference_timesteps=inference_timesteps,
+                cfg_value=cfg_value,
+                streaming=streaming,
+                streaming_prefix_len=streaming_prefix_len,
+            )
+            if streaming:
+                with self.audio_vae.streaming_decode() as vae_dec:
+                    for latent_pred, _, _ctx in inference_result:
+                        decode_audio = vae_dec.decode_chunk(latent_pred.to(torch.float32))
+                        decode_audio = decode_audio.squeeze(1).cpu()
+                        yield decode_audio
+                break
+            else:
+                latent_pred, pred_audio_feat, context_len = next_and_close(inference_result)
+                if retry_badcase:
+                    if pred_audio_feat.shape[0] >= target_text_length * retry_badcase_ratio_threshold:
+                        print(
+                            f"  Badcase detected, audio_text_ratio={pred_audio_feat.shape[0] / target_text_length}, retrying...",
+                            file=sys.stderr,
+                        )
+                        retry_badcase_times += 1
+                        continue
+                    else:
+                        break
+                else:
+                    break
+
+        if not streaming:
+            decode_audio = self.audio_vae.decode(latent_pred.to(torch.float32))
+            decode_patch_len = self.patch_size * self._decode_chunk_size
+            if context_len > 0:
+                decode_audio = decode_audio[..., decode_patch_len * context_len:].squeeze(1).cpu()
+            else:
+                decode_audio = decode_audio.squeeze(1).cpu()
+            yield decode_audio
+
+    @torch.inference_mode()
+    def build_prompt_cache(
+        self,
+        prompt_text: str = None,
+        prompt_wav_path: str = None,
+        reference_wav_path: str = None,
+        trim_silence_vad: bool = False,
+    ):
+        """
+        Build prompt cache for subsequent generation.
+
+        Supports the same parameter combinations as ``generate()``:
+        - ``reference_wav_path`` only -> reference mode (voice cloning, isolated)
+        - ``prompt_text`` + ``prompt_wav_path`` -> continuation mode
+        - all three -> combined ref + continuation mode
+
+        Args:
+            prompt_text: prompt text for continuation mode.
+                Must be paired with ``prompt_wav_path``.
+            prompt_wav_path: prompt audio path for continuation mode.
+                Must be paired with ``prompt_text``.
+            reference_wav_path: reference audio path for voice cloning
+                (structurally isolated via ref_audio tokens).
+            trim_silence_vad: whether to apply VAD-based silence trimming
+                before encoding prompt/reference audio.
+
+        Returns:
+            prompt_cache: dict used by ``_generate_with_prompt_cache``.
+        """
+        if (prompt_wav_path is None) != (prompt_text is None):
+            raise ValueError("prompt_wav_path and prompt_text must both be provided or both be None")
+        if prompt_wav_path is None and reference_wav_path is None:
+            raise ValueError("At least one of prompt_wav_path or reference_wav_path must be provided")
+
+        cache = {}
+
+        if reference_wav_path:
+            cache["ref_audio_feat"] = self._encode_wav(
+                reference_wav_path,
+                padding_mode="right",
+                trim_silence_vad=trim_silence_vad,
+            )
+
+        if prompt_wav_path and prompt_text is not None:
+            cache["prompt_text"] = prompt_text
+            cache["audio_feat"] = self._encode_wav(
+                prompt_wav_path,
+                padding_mode="left",
+                trim_silence_vad=trim_silence_vad,
+            )
+
+        has_ref = "ref_audio_feat" in cache
+        has_prompt = "audio_feat" in cache
+        if has_ref and has_prompt:
+            cache["mode"] = "ref_continuation"
+        elif has_ref:
+            cache["mode"] = "reference"
+        else:
+            cache["mode"] = "continuation"
+
+        return cache
+
+    def merge_prompt_cache(
+        self,
+        original_cache: dict,
+        new_text: str,
+        new_audio_feat: torch.Tensor,
+    ):
+        """
+        Merge original prompt cache with newly generated content to stabilize voice.
+
+        Args:
+            original_cache: original prompt cache (any mode)
+            new_text: newly generated text
+            new_audio_feat: newly generated audio features
+
+        Returns:
+            merged_cache: merged cache with prompt_text and audio_feat
+        """
+        if original_cache is None:
+            return {
+                "prompt_text": new_text,
+                "audio_feat": new_audio_feat,
+                "mode": "continuation",
+            }
+        merged = {}
+        if "ref_audio_feat" in original_cache:
+            merged["ref_audio_feat"] = original_cache["ref_audio_feat"]
+        merged["prompt_text"] = original_cache.get("prompt_text", "") + new_text
+        old_feat = original_cache.get("audio_feat", new_audio_feat.new_empty(0, *new_audio_feat.shape[1:]))
+        merged["audio_feat"] = torch.cat([old_feat, new_audio_feat], dim=0)
+        merged["mode"] = "ref_continuation" if "ref_audio_feat" in merged else "continuation"
+        return merged
+
+    def generate_with_prompt_cache(self, *args, **kwargs) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
+        return next_and_close(self._generate_with_prompt_cache(*args, streaming=False, **kwargs))
+
+    def generate_with_prompt_cache_streaming(
+        self, *args, **kwargs
+    ) -> Generator[Tuple[torch.Tensor, torch.Tensor, List[torch.Tensor]], None, None]:
+        return self._generate_with_prompt_cache(*args, streaming=True, **kwargs)
+
+    @torch.inference_mode()
+    def _generate_with_prompt_cache(
+        self,
+        target_text: str,
+        prompt_cache: dict,
+        min_len: int = 2,
+        max_len: int = 2000,
+        inference_timesteps: int = 10,
+        cfg_value: float = 2.0,
+        retry_badcase: bool = False,
+        retry_badcase_max_times: int = 3,
+        retry_badcase_ratio_threshold: float = 6.0,
+        streaming: bool = False,
+        streaming_prefix_len: int = 4,
+    ) -> Generator[Tuple[torch.Tensor, torch.Tensor, Union[torch.Tensor, List[torch.Tensor]]], None, None]:
+        """
+        Generate audio using pre-built prompt cache.
+
+        Args:
+            target_text: Text to convert to speech
+            prompt_cache: Cache built by ``build_prompt_cache()``. Can be None
+                for zero-shot generation.
+            min_len: Minimum audio length to avoid very short audio
+            max_len: Maximum audio length
+            inference_timesteps: Number of diffusion sampling steps
+            cfg_value: Classifier-free guidance value
+            retry_badcase: Whether to retry on bad cases
+            retry_badcase_max_times: Maximum retry attempts
+            retry_badcase_ratio_threshold: Threshold for audio-to-text ratio
+            streaming: Whether to return a generator of audio chunks
+            streaming_prefix_len: Number of prefix audio patches to use for streaming mode
+
+        Returns:
+            Generator of Tuple containing:
+                - Decoded audio tensor for the current step if ``streaming=True``, else final decoded audio tensor
+                - Tensor of new text tokens
+                - New audio features up to the current step as a List if ``streaming=True``, else as a concatenated Tensor
+        """
+        if retry_badcase and streaming:
+            warnings.warn("Retry on bad cases is not supported in streaming mode, setting retry_badcase=False.")
+            retry_badcase = False
+
+        # Determine mode from cache
+        if prompt_cache is None:
+            mode = "zero_shot"
+            text = target_text
+        else:
+            mode = prompt_cache.get("mode", "continuation")
+            if mode in ("continuation", "ref_continuation"):
+                prompt_text = prompt_cache.get("prompt_text", "")
+                text = prompt_text + target_text
+            else:
+                text = target_text
+
+        text_token = torch.LongTensor(self.text_tokenizer(text))
+        text_token = torch.cat(
+            [
+                text_token,
+                torch.tensor([self.audio_start_token], dtype=torch.int32, device=text_token.device),
+            ],
+            dim=-1,
+        )
+
+        target_text_token = torch.LongTensor(self.text_tokenizer(target_text))
+        text_length = text_token.shape[0]
+
+        if mode in ("zero_shot", "continuation"):
+            prompt_audio_feat = (
+                prompt_cache["audio_feat"]
+                if prompt_cache
+                else torch.empty((0, self.patch_size, self.audio_vae.latent_dim), dtype=torch.float32)
+            )
+            audio_length = prompt_audio_feat.size(0)
+            text_pad_token = torch.zeros(audio_length, dtype=torch.int32, device=text_token.device)
+            text_pad_feat = torch.zeros(
+                (text_length, self.patch_size, self.audio_vae.latent_dim),
+                dtype=torch.float32,
+                device=text_token.device,
+            )
+            text_token = torch.cat([text_token, text_pad_token])
+            audio_feat = torch.cat([text_pad_feat, prompt_audio_feat], dim=0)
+            text_mask = torch.cat(
+                [torch.ones(text_length, dtype=torch.int32), torch.zeros(audio_length, dtype=torch.int32)]
+            ).to(text_token.device)
+            audio_mask = torch.cat(
+                [torch.zeros(text_length, dtype=torch.int32), torch.ones(audio_length, dtype=torch.int32)]
+            ).to(text_token.device)
+
+        elif mode == "reference":
+            ref_audio_feat = prompt_cache["ref_audio_feat"]
+            ref_tokens, ref_feats, ref_t_mask, ref_a_mask = self._make_ref_prefix(ref_audio_feat, text_token.device)
+            text_pad_feat = torch.zeros(
+                (text_length, self.patch_size, self.audio_vae.latent_dim),
+                dtype=torch.float32,
+                device=text_token.device,
+            )
+            text_token = torch.cat([ref_tokens, text_token])
+            audio_feat = torch.cat([ref_feats, text_pad_feat], dim=0)
+            text_mask = torch.cat([ref_t_mask, torch.ones(text_length, dtype=torch.int32).to(text_token.device)])
+            audio_mask = torch.cat([ref_a_mask, torch.zeros(text_length, dtype=torch.int32).to(text_token.device)])
+
+        else:
+            # ref_continuation mode
+            ref_audio_feat = prompt_cache["ref_audio_feat"]
+            prompt_audio_feat = prompt_cache["audio_feat"]
+            prompt_audio_length = prompt_audio_feat.size(0)
+
+            ref_tokens, ref_feats, ref_t_mask, ref_a_mask = self._make_ref_prefix(ref_audio_feat, text_token.device)
+
+            prompt_pad_token = torch.zeros(prompt_audio_length, dtype=torch.int32, device=text_token.device)
+            text_pad_feat = torch.zeros(
+                (text_length, self.patch_size, self.audio_vae.latent_dim),
+                dtype=torch.float32,
+                device=text_token.device,
+            )
+
+            text_token = torch.cat([ref_tokens, text_token, prompt_pad_token])
+            audio_feat = torch.cat([ref_feats, text_pad_feat, prompt_audio_feat], dim=0)
+            text_mask = torch.cat(
+                [
+                    ref_t_mask,
+                    torch.ones(text_length, dtype=torch.int32).to(text_token.device),
+                    torch.zeros(prompt_audio_length, dtype=torch.int32).to(text_token.device),
+                ]
+            )
+            audio_mask = torch.cat(
+                [
+                    ref_a_mask,
+                    torch.zeros(text_length, dtype=torch.int32).to(text_token.device),
+                    torch.ones(prompt_audio_length, dtype=torch.int32).to(text_token.device),
+                ]
+            )
+
+        text_token = text_token.unsqueeze(0).to(self.device)
+        text_mask = text_mask.unsqueeze(0).to(self.device)
+        audio_feat = audio_feat.unsqueeze(0).to(self.device).to(get_dtype(self.config.dtype))
+        audio_mask = audio_mask.unsqueeze(0).to(self.device)
+
+        # run inference
+        target_text_length = len(self.text_tokenizer(target_text))
+        retry_badcase_times = 0
+        while retry_badcase_times < retry_badcase_max_times:
+            inference_result = self._inference(
+                text_token,
+                text_mask,
+                audio_feat,
+                audio_mask,
+                min_len=min_len,
+                max_len=min(int(target_text_length * retry_badcase_ratio_threshold + 10), max_len),
+                inference_timesteps=inference_timesteps,
+                cfg_value=cfg_value,
+                streaming=streaming,
+                streaming_prefix_len=streaming_prefix_len,
+            )
+            if streaming:
+                with self.audio_vae.streaming_decode() as vae_dec:
+                    for latent_pred, pred_audio_feat, _ctx in inference_result:
+                        decode_audio = vae_dec.decode_chunk(latent_pred.to(torch.float32))
+                        decode_audio = decode_audio.squeeze(1).cpu()
+                        yield (decode_audio, target_text_token, pred_audio_feat)
+                break
+            else:
+                latent_pred, pred_audio_feat, context_len = next_and_close(inference_result)
+                if retry_badcase:
+                    if pred_audio_feat.shape[0] >= target_text_length * retry_badcase_ratio_threshold:
+                        print(
+                            f"  Badcase detected, audio_text_ratio={pred_audio_feat.shape[0] / target_text_length}, retrying...",
+                            file=sys.stderr,
+                        )
+                        retry_badcase_times += 1
+                        continue
+                    else:
+                        break
+                else:
+                    break
+        if not streaming:
+            decode_audio = self.audio_vae.decode(latent_pred.to(torch.float32))
+            decode_patch_len = self.patch_size * self._decode_chunk_size
+            if context_len > 0:
+                decode_audio = decode_audio[..., decode_patch_len * context_len:].squeeze(1).cpu()
+            else:
+                decode_audio = decode_audio.squeeze(1).cpu()
+            yield (decode_audio, target_text_token, pred_audio_feat)
+
+    def inference(self, *args, **kwargs) -> Tuple[torch.Tensor, torch.Tensor]:
+        feat_pred, generated_feat, _ = next_and_close(self._inference(*args, streaming=False, **kwargs))
+        return feat_pred, generated_feat
+
+    def inference_streaming(self, *args, **kwargs) -> Generator[Tuple[torch.Tensor, List[torch.Tensor]], None, None]:
+        for feat_pred, pred_feat_seq, _ in self._inference(*args, streaming=True, **kwargs):
+            yield feat_pred, pred_feat_seq
+
+    @torch.inference_mode()
+    def _inference(
+        self,
+        text: torch.Tensor,
+        text_mask: torch.Tensor,
+        feat: torch.Tensor,
+        feat_mask: torch.Tensor,
+        min_len: int = 2,
+        max_len: int = 2000,
+        inference_timesteps: int = 10,
+        cfg_value: float = 2.0,
+        streaming: bool = False,
+        streaming_prefix_len: int = 4,
+    ) -> Generator[Tuple[torch.Tensor, Union[torch.Tensor, List[torch.Tensor]]], None, None]:
+        """Core inference method for audio generation.
+
+        This is the main inference loop that generates audio features
+        using the language model and diffusion transformer.
+
+        Args:
+            text: Input text tokens
+            text_mask: Mask for text tokens
+            feat: Input audio features
+            feat_mask: Mask for audio features
+            min_len: Minimum generation length
+            max_len: Maximum generation length
+            inference_timesteps: Number of diffusion steps
+            cfg_value: Classifier-free guidance value
+            streaming: Whether to yield each step latent feature or just the final result
+
+        Returns:
+            Generator of Tuple containing:
+                - Predicted latent feature at the current step if ``streaming=True``, else final latent features
+                - Predicted audio feature sequence so far as a List if ``streaming=True``, else as a concatenated Tensor
+        """
+        B, T, P, D = feat.shape
+
+        prefill_encoder = getattr(self, "_feat_encoder_raw", self.feat_encoder)
+        feat_embed = prefill_encoder(feat)  # [b, t, h_feat]
+        feat_embed = self.enc_to_lm_proj(feat_embed)
+
+        if self.config.lm_config.use_mup:
+            scale_emb = self.config.lm_config.scale_emb
+        else:
+            scale_emb = 1.0
+
+        text_embed = self.base_lm.embed_tokens(text) * scale_emb
+        combined_embed = text_mask.unsqueeze(-1) * text_embed + feat_mask.unsqueeze(-1) * feat_embed
+
+        prefix_feat_cond = feat[:, -1, ...]  # b, p, d
+        pred_feat_seq = []  # b, t, p, d
+        curr_embed = None
+
+        # Prepare prompt context patches for streaming mode
+        # - Continuation modes (feat_mask ends with 1): use the last (streaming_prefix_len - 1)
+        #   trailing audio patches as initial context so the VAE can decode smoothly.
+        # - Reference-only / zero-shot (feat_mask ends with 0): start from scratch.
+        has_continuation_audio = feat_mask[0, -1].item() == 1
+        context_len = 0
+        if has_continuation_audio:
+            audio_indices = feat_mask.squeeze(0).nonzero(as_tuple=True)[0]
+            context_len = min(streaming_prefix_len - 1, len(audio_indices))
+            last_audio_indices = audio_indices[-context_len:]
+            pred_feat_seq = list(feat[:, last_audio_indices, :, :].split(1, dim=1))
+        else:
+            pred_feat_seq = []
+
+        enc_outputs, kv_cache_tuple = self.base_lm(
+            inputs_embeds=combined_embed,
+            is_causal=True,
+        )
+        self.base_lm.kv_cache.fill_caches(kv_cache_tuple)
+
+        enc_outputs = self.fsq_layer(enc_outputs) * feat_mask.unsqueeze(-1) + enc_outputs * text_mask.unsqueeze(-1)
+        lm_hidden = enc_outputs[:, -1, :]
+
+        residual_enc_inputs = self.fusion_concat_proj(
+            torch.cat((enc_outputs, feat_mask.unsqueeze(-1) * feat_embed), dim=-1)
+        )
+        residual_enc_outputs, residual_kv_cache_tuple = self.residual_lm(
+            inputs_embeds=residual_enc_inputs,
+            is_causal=True,
+        )
+        self.residual_lm.kv_cache.fill_caches(residual_kv_cache_tuple)
+        residual_hidden = residual_enc_outputs[:, -1, :]
+
+        for i in tqdm(range(max_len)):
+            dit_hidden_1 = self.lm_to_dit_proj(lm_hidden)  # [b, h_dit]
+            dit_hidden_2 = self.res_to_dit_proj(residual_hidden)  # [b, h_dit]
+            dit_hidden = torch.cat((dit_hidden_1, dit_hidden_2), dim=-1)
+
+            pred_feat = self.feat_decoder(
+                mu=dit_hidden,
+                patch_size=self.patch_size,
+                cond=prefix_feat_cond.transpose(1, 2).contiguous(),
+                n_timesteps=inference_timesteps,
+                cfg_value=cfg_value,
+            ).transpose(
+                1, 2
+            )  # [b, p, d]
+
+            curr_embed = self.feat_encoder(pred_feat.unsqueeze(1))  # b, 1, c
+            curr_embed = self.enc_to_lm_proj(curr_embed)
+
+            pred_feat_seq.append(pred_feat.unsqueeze(1))  # b, 1, p, d
+            prefix_feat_cond = pred_feat
+
+            if streaming:
+                # Yield only the newest patch latent for stateful VAE decode
+                feat_pred = rearrange(pred_feat.unsqueeze(1), "b t p d -> b d (t p)", b=B, p=self.patch_size)
+
+                yield feat_pred, pred_feat_seq, context_len
+
+                if len(pred_feat_seq) > streaming_prefix_len:
+                    pred_feat_seq = pred_feat_seq[-streaming_prefix_len:]
+
+            stop_flag = self.stop_head(self.stop_actn(self.stop_proj(lm_hidden))).argmax(dim=-1)[0].cpu().item()
+            if i > min_len and stop_flag == 1:
+                break
+
+            lm_hidden = self.base_lm.forward_step(
+                curr_embed[:, 0, :], torch.tensor([self.base_lm.kv_cache.step()], device=curr_embed.device)
+            ).clone()
+
+            lm_hidden = self.fsq_layer(lm_hidden)
+            curr_residual_input = self.fusion_concat_proj(torch.cat((lm_hidden, curr_embed[:, 0, :]), dim=-1))
+            residual_hidden = self.residual_lm.forward_step(
+                curr_residual_input, torch.tensor([self.residual_lm.kv_cache.step()], device=curr_embed.device)
+            ).clone()
+
+        if not streaming:
+            pred_feat_seq = torch.cat(pred_feat_seq, dim=1)  # b, t, p, d
+            feat_pred = rearrange(pred_feat_seq, "b t p d -> b d (t p)", b=B, p=self.patch_size)
+            generated_feat = pred_feat_seq[:, context_len:, :, :].squeeze(0).cpu()
+            yield feat_pred, generated_feat, context_len
+
+    @classmethod
+    def from_local(
+        cls,
+        path: str,
+        optimize: bool = True,
+        training: bool = False,
+        device: str | None = None,
+        lora_config: LoRAConfig = None,
+    ):
+        with open(os.path.join(path, "config.json"), "r", encoding="utf-8") as _cfg_f:
+            config = VoxCPMConfig.model_validate_json(_cfg_f.read())
+        tokenizer = LlamaTokenizerFast.from_pretrained(path)
+        audio_vae_config = getattr(config, "audio_vae_config", None)
+        audio_vae = AudioVAEV2(config=audio_vae_config) if audio_vae_config else AudioVAEV2()
+        # Try to load AudioVAE from safetensors first, fallback to pytorch
+        audiovae_safetensors_path = os.path.join(path, "audiovae.safetensors")
+        audiovae_pth_path = os.path.join(path, "audiovae.pth")
+        if os.path.exists(audiovae_safetensors_path) and SAFETENSORS_AVAILABLE:
+            print(f"Loading AudioVAE from safetensors: {audiovae_safetensors_path}", file=sys.stderr)
+            vae_state_dict = load_file(audiovae_safetensors_path, device="cpu")
+        elif os.path.exists(audiovae_pth_path):
+            print(f"Loading AudioVAE from pytorch: {audiovae_pth_path}", file=sys.stderr)
+            checkpoint = torch.load(
+                audiovae_pth_path,
+                map_location="cpu",
+                weights_only=True,
+            )
+            vae_state_dict = checkpoint.get("state_dict", checkpoint)
+        else:
+            raise FileNotFoundError(
+                f"AudioVAE checkpoint not found. Expected either {audiovae_safetensors_path} or {audiovae_pth_path}"
+            )
+        model = cls(config, tokenizer, audio_vae, lora_config, device=device)
+        if not training:
+            lm_dtype = get_dtype(model.config.dtype)
+            model = model.to(lm_dtype)
+        else:  # training mode
+            for name, param in model.named_parameters():
+                if "audio_vae" in name:  # freeze VAE weights
+                    param.requires_grad = False
+                    continue
+                if lora_config is not None:
+                    if "lora" not in name:  # freeze non-LoRA weights
+                        param.requires_grad = False
+        model.audio_vae = model.audio_vae.to(torch.float32)
+
+        # Try to load from safetensors first, fallback to pytorch_model.bin
+        safetensors_path = os.path.join(path, "model.safetensors")
+        pytorch_model_path = os.path.join(path, "pytorch_model.bin")
+
+        if os.path.exists(safetensors_path) and SAFETENSORS_AVAILABLE:
+            print(f"Loading model from safetensors: {safetensors_path}", file=sys.stderr)
+            model_state_dict = load_file(safetensors_path)
+        elif os.path.exists(pytorch_model_path):
+            print(f"Loading model from pytorch_model.bin: {pytorch_model_path}", file=sys.stderr)
+            checkpoint = torch.load(
+                pytorch_model_path,
+                map_location="cpu",
+                weights_only=True,
+            )
+            model_state_dict = checkpoint.get("state_dict", checkpoint)
+        else:
+            raise FileNotFoundError(f"Model file not found. Expected either {safetensors_path} or {pytorch_model_path}")
+
+        for kw, val in vae_state_dict.items():
+            model_state_dict[f"audio_vae.{kw}"] = val
+
+        # LoRALinear keeps weight/bias compatible with nn.Linear but adds
+        # lora_A/lora_B, which are absent from base pretrained checkpoints.
+        model.load_state_dict(model_state_dict, strict=False)
+        if training:
+            return model
+        return model.to(model.device).eval().optimize(disable=not optimize)
+
+    # ------------------------------------------------------------------ #
+    # LoRA Weight Management
+    # ------------------------------------------------------------------ #
+    def _iter_lora_modules(self):
+        """Iterate over all LoRA modules."""
+        from ..modules.layers.lora import LoRALinear
+
+        for module in self.modules():
+            if isinstance(module, LoRALinear):
+                yield module
+
+    def load_lora_weights(self, lora_path: str, device: str = None):
+        """
+        Load LoRA weights from file, supports calling after torch.compile.
+        Uses named_parameters() to handle compile's _orig_mod wrapper.
+        Supports both safetensors and pytorch formats.
+
+        Args:
+            lora_path: Checkpoint path (directory or .safetensors/.ckpt file)
+            device: Target device, defaults to model's current device
+        Returns:
+            tuple: (loaded_keys, skipped_keys)
+        """
+        from pathlib import Path
+
+        device = device or self.device
+        lora_p = Path(lora_path)
+
+        # Try safetensors first, then fallback to .ckpt
+        if lora_p.is_dir():
+            safetensors_file = lora_p / "lora_weights.safetensors"
+            ckpt_file = lora_p / "lora_weights.ckpt"
+        else:
+            safetensors_file = lora_p if lora_p.suffix == ".safetensors" else None
+            ckpt_file = lora_p if lora_p.suffix in [".ckpt", ".pth"] else None
+
+        # Load from safetensors if available
+        if safetensors_file and safetensors_file.exists() and SAFETENSORS_AVAILABLE:
+            state_dict = load_file(str(safetensors_file), device=device)
+        elif ckpt_file and ckpt_file.exists():
+            ckpt = torch.load(ckpt_file, map_location=device, weights_only=True)
+            state_dict = ckpt.get("state_dict", ckpt)
+        else:
+            raise FileNotFoundError(f"LoRA checkpoint not found. Expected either {safetensors_file} or {ckpt_file}")
+
+        # Build param mapping (handle torch.compile's _orig_mod prefix)
+        model_params = dict(self.named_parameters())
+        key_mapping = {k.replace("._orig_mod.", "."): k for k in model_params if "._orig_mod." in k}
+
+        loaded_keys, skipped_keys = [], []
+        for key, value in state_dict.items():
+            target_key = key if key in model_params else key_mapping.get(key)
+            if target_key:
+                model_params[target_key].data.copy_(value.to(device))
+                loaded_keys.append(key)
+            else:
+                skipped_keys.append(key)
+
+        return loaded_keys, skipped_keys
+
+    def set_lora_enabled(self, enabled: bool):
+        """Enable/disable all LoRA layers."""
+        for module in self._iter_lora_modules():
+            module.set_enabled(enabled)
+
+    def reset_lora_weights(self):
+        """Reset all LoRA weights (A: kaiming, B: zeros), effectively unloading LoRA."""
+        for module in self._iter_lora_modules():
+            module.reset_lora_parameters()
+
+    def get_lora_state_dict(self) -> dict:
+        """Get all LoRA parameters (lora_A/lora_B)."""
+        return {name: param.data.clone() for name, param in self.named_parameters() if "lora_" in name}

bluemagpie/_vendor/voxcpm/modules/audiovae/__init__.py

@@ -0,0 +1,2 @@
+from .audio_vae import AudioVAE, AudioVAEConfig
+from .audio_vae_v2 import AudioVAE as AudioVAEV2, AudioVAEConfig as AudioVAEConfigV2

bluemagpie/_vendor/voxcpm/modules/audiovae/audio_vae.py

@@ -0,0 +1,377 @@
+import math
+from typing import List
+
+import numpy as np
+import torch
+from torch import nn
+import torch.nn.functional as F
+from torch.nn.utils import weight_norm
+from pydantic import BaseModel
+
+
+def WNConv1d(*args, **kwargs):
+    return weight_norm(nn.Conv1d(*args, **kwargs))
+
+
+def WNConvTranspose1d(*args, **kwargs):
+    return weight_norm(nn.ConvTranspose1d(*args, **kwargs))
+
+
+class CausalConv1d(nn.Conv1d):
+    def __init__(self, *args, padding: int = 0, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.__padding = padding
+
+    def forward(self, x):
+        x_pad = F.pad(x, (self.__padding * 2, 0))
+        return super().forward(x_pad)
+
+
+class CausalTransposeConv1d(nn.ConvTranspose1d):
+    def __init__(self, *args, padding: int = 0, output_padding: int = 0, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.__padding = padding
+        self.__output_padding = output_padding
+
+    def forward(self, x):
+        return super().forward(x)[..., : -(self.__padding * 2 - self.__output_padding)]
+
+
+def WNCausalConv1d(*args, **kwargs):
+    return weight_norm(CausalConv1d(*args, **kwargs))
+
+
+def WNCausalTransposeConv1d(*args, **kwargs):
+    return weight_norm(CausalTransposeConv1d(*args, **kwargs))
+
+
+# Scripting this brings model speed up 1.4x
+@torch.jit.script
+def snake(x, alpha):
+    shape = x.shape
+    x = x.reshape(shape[0], shape[1], -1)
+    x = x + (alpha + 1e-9).reciprocal() * torch.sin(alpha * x).pow(2)
+    x = x.reshape(shape)
+    return x
+
+
+class Snake1d(nn.Module):
+    def __init__(self, channels):
+        super().__init__()
+        self.alpha = nn.Parameter(torch.ones(1, channels, 1))
+
+    def forward(self, x):
+        return snake(x, self.alpha)
+
+
+def init_weights(m):
+    if isinstance(m, nn.Conv1d):
+        nn.init.trunc_normal_(m.weight, std=0.02)
+        if m.bias is not None:
+            nn.init.constant_(m.bias, 0)
+
+
+class CausalResidualUnit(nn.Module):
+    def __init__(self, dim: int = 16, dilation: int = 1, kernel: int = 7, groups: int = 1):
+        super().__init__()
+        pad = ((7 - 1) * dilation) // 2
+        self.block = nn.Sequential(
+            Snake1d(dim),
+            WNCausalConv1d(
+                dim,
+                dim,
+                kernel_size=kernel,
+                dilation=dilation,
+                padding=pad,
+                groups=groups,
+            ),
+            Snake1d(dim),
+            WNCausalConv1d(dim, dim, kernel_size=1),
+        )
+
+    def forward(self, x):
+        y = self.block(x)
+        pad = (x.shape[-1] - y.shape[-1]) // 2
+        assert pad == 0
+        if pad > 0:
+            x = x[..., pad:-pad]
+        return x + y
+
+
+class CausalEncoderBlock(nn.Module):
+    def __init__(self, output_dim: int = 16, input_dim=None, stride: int = 1, groups=1):
+        super().__init__()
+        input_dim = input_dim or output_dim // 2
+        self.block = nn.Sequential(
+            CausalResidualUnit(input_dim, dilation=1, groups=groups),
+            CausalResidualUnit(input_dim, dilation=3, groups=groups),
+            CausalResidualUnit(input_dim, dilation=9, groups=groups),
+            Snake1d(input_dim),
+            WNCausalConv1d(
+                input_dim,
+                output_dim,
+                kernel_size=2 * stride,
+                stride=stride,
+                padding=math.ceil(stride / 2),
+            ),
+        )
+
+    def forward(self, x):
+        return self.block(x)
+
+
+class CausalEncoder(nn.Module):
+    def __init__(
+        self,
+        d_model: int = 64,
+        latent_dim: int = 32,
+        strides: list = [2, 4, 8, 8],
+        depthwise: bool = False,
+    ):
+        super().__init__()
+        # Create first convolution
+        self.block = [WNCausalConv1d(1, d_model, kernel_size=7, padding=3)]
+
+        # Create EncoderBlocks that double channels as they downsample by `stride`
+        for stride in strides:
+            d_model *= 2
+            groups = d_model // 2 if depthwise else 1
+            self.block += [CausalEncoderBlock(output_dim=d_model, stride=stride, groups=groups)]
+
+        groups = d_model if depthwise else 1
+
+        # Create two convolution, for mu and logvar
+        self.fc_mu = WNCausalConv1d(d_model, latent_dim, kernel_size=3, padding=1)
+        self.fc_logvar = WNCausalConv1d(d_model, latent_dim, kernel_size=3, padding=1)
+
+        # Wrap black into nn.Sequential
+        self.block = nn.Sequential(*self.block)
+        self.enc_dim = d_model
+
+    def forward(self, x):
+        hidden_state = self.block(x)
+        return {
+            "hidden_state": hidden_state,
+            "mu": self.fc_mu(hidden_state),
+            "logvar": self.fc_logvar(hidden_state),
+        }
+
+
+class NoiseBlock(nn.Module):
+    def __init__(self, dim):
+        super().__init__()
+        self.linear = WNCausalConv1d(dim, dim, kernel_size=1, bias=False)
+
+    def forward(self, x):
+        B, C, T = x.shape
+        noise = torch.randn((B, 1, T), device=x.device, dtype=x.dtype)
+        h = self.linear(x)
+        n = noise * h
+        x = x + n
+        return x
+
+
+class CausalDecoderBlock(nn.Module):
+    def __init__(
+        self,
+        input_dim: int = 16,
+        output_dim: int = 8,
+        stride: int = 1,
+        groups=1,
+        use_noise_block: bool = False,
+    ):
+        super().__init__()
+        layers = [
+            Snake1d(input_dim),
+            WNCausalTransposeConv1d(
+                input_dim,
+                output_dim,
+                kernel_size=2 * stride,
+                stride=stride,
+                padding=math.ceil(stride / 2),
+                output_padding=stride % 2,
+            ),
+        ]
+        if use_noise_block:
+            layers.append(NoiseBlock(output_dim))
+        layers.extend(
+            [
+                CausalResidualUnit(output_dim, dilation=1, groups=groups),
+                CausalResidualUnit(output_dim, dilation=3, groups=groups),
+                CausalResidualUnit(output_dim, dilation=9, groups=groups),
+            ]
+        )
+        self.block = nn.Sequential(*layers)
+
+    def forward(self, x):
+        return self.block(x)
+
+
+class TransposeLastTwoDim(torch.nn.Module):
+    def forward(self, x):
+        return torch.transpose(x, -1, -2)
+
+
+class CausalDecoder(nn.Module):
+    def __init__(
+        self,
+        input_channel,
+        channels,
+        rates,
+        depthwise: bool = False,
+        d_out: int = 1,
+        use_noise_block: bool = False,
+    ):
+        super().__init__()
+
+        # Add first conv layer
+        if depthwise:
+            layers = [
+                WNCausalConv1d(
+                    input_channel,
+                    input_channel,
+                    kernel_size=7,
+                    padding=3,
+                    groups=input_channel,
+                ),
+                WNCausalConv1d(input_channel, channels, kernel_size=1),
+            ]
+        else:
+            layers = [WNCausalConv1d(input_channel, channels, kernel_size=7, padding=3)]
+
+        # Add upsampling + MRF blocks
+        for i, stride in enumerate(rates):
+            input_dim = channels // 2**i
+            output_dim = channels // 2 ** (i + 1)
+            groups = output_dim if depthwise else 1
+            layers += [
+                CausalDecoderBlock(
+                    input_dim,
+                    output_dim,
+                    stride,
+                    groups=groups,
+                    use_noise_block=use_noise_block,
+                )
+            ]
+
+        # Add final conv layer
+        layers += [
+            Snake1d(output_dim),
+            WNCausalConv1d(output_dim, d_out, kernel_size=7, padding=3),
+            nn.Tanh(),
+        ]
+
+        self.model = nn.Sequential(*layers)
+
+    def forward(self, x):
+        return self.model(x)
+
+
+class AudioVAEConfig(BaseModel):
+    encoder_dim: int = 128
+    encoder_rates: List[int] = [2, 5, 8, 8]
+    latent_dim: int = 64
+    decoder_dim: int = 1536
+    decoder_rates: List[int] = [8, 8, 5, 2]
+    depthwise: bool = True
+    sample_rate: int = 16000
+    use_noise_block: bool = False
+
+
+class AudioVAE(nn.Module):
+    """
+    Args:
+    """
+
+    def __init__(
+        self,
+        config: AudioVAEConfig = None,
+    ):
+        # 如果没有传入config，使用默认配置
+        if config is None:
+            config = AudioVAEConfig()
+
+        super().__init__()
+
+        encoder_dim = config.encoder_dim
+        encoder_rates = config.encoder_rates
+        latent_dim = config.latent_dim
+        decoder_dim = config.decoder_dim
+        decoder_rates = config.decoder_rates
+        depthwise = config.depthwise
+        sample_rate = config.sample_rate
+        use_noise_block = config.use_noise_block
+
+        self.encoder_dim = encoder_dim
+        self.encoder_rates = encoder_rates
+        self.decoder_dim = decoder_dim
+        self.decoder_rates = decoder_rates
+        self.depthwise = depthwise
+
+        self.use_noise_block = use_noise_block
+
+        if latent_dim is None:
+            latent_dim = encoder_dim * (2 ** len(encoder_rates))
+
+        self.latent_dim = latent_dim
+        self.hop_length = np.prod(encoder_rates)
+        self.encoder = CausalEncoder(
+            encoder_dim,
+            latent_dim,
+            encoder_rates,
+            depthwise=depthwise,
+        )
+
+        self.decoder = CausalDecoder(
+            latent_dim,
+            decoder_dim,
+            decoder_rates,
+            depthwise=depthwise,
+            use_noise_block=use_noise_block,
+        )
+        self.sample_rate = sample_rate
+        self.chunk_size = math.prod(encoder_rates)
+
+    def preprocess(self, audio_data, sample_rate):
+        if sample_rate is None:
+            sample_rate = self.sample_rate
+        assert sample_rate == self.sample_rate
+        pad_to = self.hop_length
+        length = audio_data.shape[-1]
+        right_pad = math.ceil(length / pad_to) * pad_to - length
+        audio_data = nn.functional.pad(audio_data, (0, right_pad))
+
+        return audio_data
+
+    def decode(self, z: torch.Tensor):
+        """Decode given latent codes and return audio data
+
+        Parameters
+        ----------
+        z : Tensor[B x D x T]
+            Quantized continuous representation of input
+        length : int, optional
+            Number of samples in output audio, by default None
+
+        Returns
+        -------
+        dict
+            A dictionary with the following keys:
+            "audio" : Tensor[B x 1 x length]
+                Decoded audio data.
+        """
+        return self.decoder(z)
+
+    def encode(self, audio_data: torch.Tensor, sample_rate: int):
+        """
+        Args:
+            audio_data: Tensor[B x 1 x T]
+            sample_rate: int
+        Returns:
+            z: Tensor[B x D x T]
+        """
+        if audio_data.ndim == 2:
+            audio_data = audio_data.unsqueeze(1)
+
+        audio_data = self.preprocess(audio_data, sample_rate)
+        return self.encoder(audio_data)["mu"]

bluemagpie/_vendor/voxcpm/modules/audiovae/audio_vae_v2.py

@@ -0,0 +1,580 @@
+import math
+from typing import List, Optional
+
+import numpy as np
+import torch
+from torch import nn
+import torch.nn.functional as F
+from torch.nn.utils import weight_norm
+from pydantic import BaseModel
+
+
+def WNConv1d(*args, **kwargs):
+    return weight_norm(nn.Conv1d(*args, **kwargs))
+
+
+def WNConvTranspose1d(*args, **kwargs):
+    return weight_norm(nn.ConvTranspose1d(*args, **kwargs))
+
+
+class CausalConv1d(nn.Conv1d):
+    def __init__(self, *args, padding: int = 0, output_padding: int = 0, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.__padding = padding
+        self.__output_padding = output_padding
+
+    def forward(self, x):
+        x_pad = F.pad(x, (self.__padding * 2 - self.__output_padding, 0))
+        return super().forward(x_pad)
+
+
+class CausalTransposeConv1d(nn.ConvTranspose1d):
+    def __init__(self, *args, padding: int = 0, output_padding: int = 0, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.__padding = padding
+        self.__output_padding = output_padding
+
+    def forward(self, x):
+        return super().forward(x)[..., : -(self.__padding * 2 - self.__output_padding)]
+
+
+def WNCausalConv1d(*args, **kwargs):
+    return weight_norm(CausalConv1d(*args, **kwargs))
+
+
+def WNCausalTransposeConv1d(*args, **kwargs):
+    return weight_norm(CausalTransposeConv1d(*args, **kwargs))
+
+
+# Scripting this brings model speed up 1.4x
+@torch.jit.script
+def snake(x, alpha):
+    shape = x.shape
+    x = x.reshape(shape[0], shape[1], -1)
+    x = x + (alpha + 1e-9).reciprocal() * torch.sin(alpha * x).pow(2)
+    x = x.reshape(shape)
+    return x
+
+
+class Snake1d(nn.Module):
+    def __init__(self, channels):
+        super().__init__()
+        self.alpha = nn.Parameter(torch.ones(1, channels, 1))
+
+    def forward(self, x):
+        return snake(x, self.alpha)
+
+
+def init_weights(m):
+    if isinstance(m, nn.Conv1d):
+        nn.init.trunc_normal_(m.weight, std=0.02)
+        if m.bias is not None:
+            nn.init.constant_(m.bias, 0)
+
+
+class CausalResidualUnit(nn.Module):
+    def __init__(self, dim: int = 16, dilation: int = 1, kernel: int = 7, groups: int = 1):
+        super().__init__()
+        pad = ((7 - 1) * dilation) // 2
+        self.block = nn.Sequential(
+            Snake1d(dim),
+            WNCausalConv1d(
+                dim,
+                dim,
+                kernel_size=kernel,
+                dilation=dilation,
+                padding=pad,
+                groups=groups,
+            ),
+            Snake1d(dim),
+            WNCausalConv1d(dim, dim, kernel_size=1),
+        )
+
+    def forward(self, x):
+        y = self.block(x)
+        pad = (x.shape[-1] - y.shape[-1]) // 2
+        assert pad == 0
+        if pad > 0:
+            x = x[..., pad:-pad]
+        return x + y
+
+
+class CausalEncoderBlock(nn.Module):
+    def __init__(self, output_dim: int = 16, input_dim=None, stride: int = 1, groups=1):
+        super().__init__()
+        input_dim = input_dim or output_dim // 2
+        self.block = nn.Sequential(
+            CausalResidualUnit(input_dim, dilation=1, groups=groups),
+            CausalResidualUnit(input_dim, dilation=3, groups=groups),
+            CausalResidualUnit(input_dim, dilation=9, groups=groups),
+            Snake1d(input_dim),
+            WNCausalConv1d(
+                input_dim,
+                output_dim,
+                kernel_size=2 * stride,
+                stride=stride,
+                padding=math.ceil(stride / 2),
+                output_padding=stride % 2,
+            ),
+        )
+
+    def forward(self, x):
+        return self.block(x)
+
+
+class CausalEncoder(nn.Module):
+    def __init__(
+        self,
+        d_model: int = 64,
+        latent_dim: int = 32,
+        strides: list = [2, 4, 8, 8],
+        depthwise: bool = False,
+    ):
+        super().__init__()
+        # Create first convolution
+        self.block = [WNCausalConv1d(1, d_model, kernel_size=7, padding=3)]
+
+        # Create EncoderBlocks that double channels as they downsample by `stride`
+        for stride in strides:
+            d_model *= 2
+            groups = d_model // 2 if depthwise else 1
+            self.block += [CausalEncoderBlock(output_dim=d_model, stride=stride, groups=groups)]
+
+        groups = d_model if depthwise else 1
+
+        # Create two convolution, for mu and logvar
+        self.fc_mu = WNCausalConv1d(d_model, latent_dim, kernel_size=3, padding=1)
+        self.fc_logvar = WNCausalConv1d(d_model, latent_dim, kernel_size=3, padding=1)
+
+        # Wrap black into nn.Sequential
+        self.block = nn.Sequential(*self.block)
+        self.enc_dim = d_model
+
+    def forward(self, x):
+        hidden_state = self.block(x)
+        return {
+            "hidden_state": hidden_state,
+            "mu": self.fc_mu(hidden_state),
+            "logvar": self.fc_logvar(hidden_state),
+        }
+
+
+class NoiseBlock(nn.Module):
+    def __init__(self, dim):
+        super().__init__()
+        self.linear = WNCausalConv1d(dim, dim, kernel_size=1, bias=False)
+
+    def forward(self, x):
+        B, C, T = x.shape
+        noise = torch.randn((B, 1, T), device=x.device, dtype=x.dtype)
+        h = self.linear(x)
+        n = noise * h
+        x = x + n
+        return x
+
+
+class CausalDecoderBlock(nn.Module):
+    def __init__(
+        self,
+        input_dim: int = 16,
+        output_dim: int = 8,
+        stride: int = 1,
+        groups=1,
+        use_noise_block: bool = False,
+    ):
+        super().__init__()
+        layers = [
+            Snake1d(input_dim),
+            WNCausalTransposeConv1d(
+                input_dim,
+                output_dim,
+                kernel_size=2 * stride,
+                stride=stride,
+                padding=math.ceil(stride / 2),
+                output_padding=stride % 2,
+            ),
+        ]
+        if use_noise_block:
+            layers.append(NoiseBlock(output_dim))
+        layers.extend(
+            [
+                CausalResidualUnit(output_dim, dilation=1, groups=groups),
+                CausalResidualUnit(output_dim, dilation=3, groups=groups),
+                CausalResidualUnit(output_dim, dilation=9, groups=groups),
+            ]
+        )
+        self.block = nn.Sequential(*layers)
+        self.input_channels = input_dim
+
+    def forward(self, x):
+        return self.block(x)
+
+
+class TransposeLastTwoDim(torch.nn.Module):
+    def forward(self, x):
+        return torch.transpose(x, -1, -2)
+
+
+class SampleRateConditionLayer(nn.Module):
+    def __init__(
+        self,
+        input_dim: int,
+        sr_bin_buckets: int = None,
+        cond_type: str = "scale_bias",
+        cond_dim: int = 128,
+        out_layer: bool = False,
+    ):
+        super().__init__()
+
+        self.cond_type, out_layer_in_dim = cond_type, input_dim
+
+        if cond_type == "scale_bias":
+            self.scale_embed = nn.Embedding(sr_bin_buckets, input_dim)
+            self.bias_embed = nn.Embedding(sr_bin_buckets, input_dim)
+            nn.init.ones_(self.scale_embed.weight)
+            nn.init.zeros_(self.bias_embed.weight)
+        elif cond_type == "scale_bias_init":
+            self.scale_embed = nn.Embedding(sr_bin_buckets, input_dim)
+            self.bias_embed = nn.Embedding(sr_bin_buckets, input_dim)
+            nn.init.normal_(self.scale_embed.weight, mean=1)
+            nn.init.normal_(self.bias_embed.weight)
+        elif cond_type == "add":
+            self.cond_embed = nn.Embedding(sr_bin_buckets, input_dim)
+            nn.init.normal_(self.cond_embed.weight)
+        elif cond_type == "concat":
+            self.cond_embed = nn.Embedding(sr_bin_buckets, cond_dim)
+            assert out_layer, "out_layer must be True for concat cond_type"
+            out_layer_in_dim = input_dim + cond_dim
+        else:
+            raise ValueError(f"Invalid cond_type: {cond_type}")
+
+        if out_layer:
+            self.out_layer = nn.Sequential(
+                Snake1d(out_layer_in_dim),
+                WNCausalConv1d(out_layer_in_dim, input_dim, kernel_size=1),
+            )
+        else:
+            self.out_layer = nn.Identity()
+
+    def forward(self, x, sr_cond):
+        if self.cond_type == "scale_bias" or self.cond_type == "scale_bias_init":
+            x = x * self.scale_embed(sr_cond).unsqueeze(-1) + self.bias_embed(sr_cond).unsqueeze(-1)
+        elif self.cond_type == "add":
+            x = x + self.cond_embed(sr_cond).unsqueeze(-1)
+        elif self.cond_type == "concat":
+            x = torch.cat([x, self.cond_embed(sr_cond).unsqueeze(-1).repeat(1, 1, x.shape[-1])], dim=1)
+
+        return self.out_layer(x)
+
+
+class CausalDecoder(nn.Module):
+    def __init__(
+        self,
+        input_channel,
+        channels,
+        rates,
+        depthwise: bool = False,
+        d_out: int = 1,
+        use_noise_block: bool = False,
+        sr_bin_boundaries: List[int] = None,
+        cond_type: str = "scale_bias",
+        cond_dim: int = 128,
+        cond_out_layer: bool = False,
+    ):
+        super().__init__()
+
+        # Add first conv layer
+        if depthwise:
+            layers = [
+                WNCausalConv1d(input_channel, input_channel, kernel_size=7, padding=3, groups=input_channel),
+                WNCausalConv1d(input_channel, channels, kernel_size=1),
+            ]
+        else:
+            layers = [WNCausalConv1d(input_channel, channels, kernel_size=7, padding=3)]
+
+        # Add upsampling + MRF blocks
+        for i, stride in enumerate(rates):
+            input_dim = channels // 2**i
+            output_dim = channels // 2 ** (i + 1)
+            groups = output_dim if depthwise else 1
+            layers += [
+                CausalDecoderBlock(
+                    input_dim,
+                    output_dim,
+                    stride,
+                    groups=groups,
+                    use_noise_block=use_noise_block,
+                )
+            ]
+
+        # Add final conv layer
+        layers += [
+            Snake1d(output_dim),
+            WNCausalConv1d(output_dim, d_out, kernel_size=7, padding=3),
+            nn.Tanh(),
+        ]
+
+        if sr_bin_boundaries is None:
+            self.model = nn.Sequential(*layers)
+            self.sr_bin_boundaries = None
+        else:
+            self.model = nn.ModuleList(layers)
+
+            self.register_buffer("sr_bin_boundaries", torch.tensor(sr_bin_boundaries, dtype=torch.int32))
+            self.sr_bin_buckets = len(sr_bin_boundaries) + 1
+
+            cond_layers = []
+            for layer in self.model:
+                if layer.__class__.__name__ == "CausalDecoderBlock":
+                    cond_layers.append(
+                        SampleRateConditionLayer(
+                            input_dim=layer.input_channels,
+                            sr_bin_buckets=self.sr_bin_buckets,
+                            cond_type=cond_type,
+                            cond_dim=cond_dim,
+                            out_layer=cond_out_layer,
+                        )
+                    )
+                else:
+                    cond_layers.append(None)
+            self.sr_cond_model = nn.ModuleList(cond_layers)
+
+    def get_sr_idx(self, sr):
+        return torch.bucketize(sr, self.sr_bin_boundaries)
+
+    def forward(self, x, sr_cond=None):
+        if self.sr_bin_boundaries is not None:
+            # assert sr_cond is not None
+            sr_cond = self.get_sr_idx(sr_cond)
+
+            for layer, sr_cond_layer in zip(self.model, self.sr_cond_model):
+                if sr_cond_layer is not None:
+                    x = sr_cond_layer(x, sr_cond)
+                x = layer(x)
+            return x
+        else:
+            return self.model(x)
+
+
+class AudioVAEConfig(BaseModel):
+    encoder_dim: int = 128
+    encoder_rates: List[int] = [2, 5, 8, 8]
+    latent_dim: int = 64
+    decoder_dim: int = 2048
+    decoder_rates: List[int] = [8, 6, 5, 2, 2, 2]
+    depthwise: bool = True
+    sample_rate: int = 16000
+    out_sample_rate: int = 48000
+    use_noise_block: bool = False
+    sr_bin_boundaries: Optional[List[int]] = [20000, 30000, 40000]
+    cond_type: str = "scale_bias"
+    cond_dim: int = 128
+    cond_out_layer: bool = False
+
+
+class AudioVAE(nn.Module):
+    """
+    Args:
+    """
+
+    def __init__(
+        self,
+        config: AudioVAEConfig = None,
+    ):
+        # 如果没有传入config，使用默认配置
+        if config is None:
+            config = AudioVAEConfig()
+
+        super().__init__()
+
+        encoder_dim = config.encoder_dim
+        encoder_rates = config.encoder_rates
+        latent_dim = config.latent_dim
+        decoder_dim = config.decoder_dim
+        decoder_rates = config.decoder_rates
+        depthwise = config.depthwise
+        sample_rate = config.sample_rate
+        out_sample_rate = config.out_sample_rate
+        use_noise_block = config.use_noise_block
+        sr_bin_boundaries = config.sr_bin_boundaries
+        cond_type = config.cond_type
+        cond_dim = config.cond_dim
+        cond_out_layer = config.cond_out_layer
+
+        self.encoder_dim = encoder_dim
+        self.encoder_rates = encoder_rates
+        self.decoder_dim = decoder_dim
+        self.decoder_rates = decoder_rates
+        self.depthwise = depthwise
+
+        self.use_noise_block = use_noise_block
+
+        if latent_dim is None:
+            latent_dim = encoder_dim * (2 ** len(encoder_rates))
+
+        self.latent_dim = latent_dim
+        self.hop_length = np.prod(encoder_rates)
+        self.encoder = CausalEncoder(
+            encoder_dim,
+            latent_dim,
+            encoder_rates,
+            depthwise=depthwise,
+        )
+
+        self.decoder = CausalDecoder(
+            latent_dim,
+            decoder_dim,
+            decoder_rates,
+            depthwise=depthwise,
+            use_noise_block=use_noise_block,
+            sr_bin_boundaries=sr_bin_boundaries,
+            cond_type=cond_type,
+            cond_dim=cond_dim,
+            cond_out_layer=cond_out_layer,
+        )
+        self.sample_rate = sample_rate
+        self.out_sample_rate = out_sample_rate
+        self.sr_bin_boundaries = sr_bin_boundaries
+        self.chunk_size = math.prod(encoder_rates)
+        self.decode_chunk_size = math.prod(decoder_rates)
+
+    def preprocess(self, audio_data, sample_rate):
+        if sample_rate is None:
+            sample_rate = self.sample_rate
+        assert sample_rate == self.sample_rate
+        pad_to = self.hop_length
+        length = audio_data.shape[-1]
+        right_pad = math.ceil(length / pad_to) * pad_to - length
+        audio_data = nn.functional.pad(audio_data, (0, right_pad))
+
+        return audio_data
+
+    def decode(self, z: torch.Tensor, sr_cond: torch.Tensor = None):
+        """Decode given latent codes and return audio data
+
+        Parameters
+        ----------
+        z : Tensor[B x D x T]
+            Quantized continuous representation of input
+        length : int, optional
+            Number of samples in output audio, by default None
+
+        Returns
+        -------
+        dict
+            A dictionary with the following keys:
+            "audio" : Tensor[B x 1 x length]
+                Decoded audio data.
+        """
+        if self.sr_bin_boundaries is not None:
+            # use default output sample rate
+            if sr_cond is None:
+                sr_cond = torch.tensor([self.out_sample_rate], device=z.device, dtype=torch.int32)
+        return self.decoder(z, sr_cond)
+
+    def streaming_decode(self):
+        """Return a ``StreamingVAEDecoder`` context manager for stateful
+        chunk-by-chunk decoding.  Each call to ``decode_chunk`` processes only
+        the new latent patch and carries causal-conv state internally, avoiding
+        the redundant overlap decode used previously.
+
+        Usage::
+
+            with vae.streaming_decode() as dec:
+                for patch in patches:
+                    audio_chunk = dec.decode_chunk(patch)
+        """
+        return StreamingVAEDecoder(self)
+
+    def encode(self, audio_data: torch.Tensor, sample_rate: int):
+        """
+        Args:
+            audio_data: Tensor[B x 1 x T]
+            sample_rate: int
+        Returns:
+            z: Tensor[B x D x T]
+        """
+        if audio_data.ndim == 2:
+            audio_data = audio_data.unsqueeze(1)
+
+        audio_data = self.preprocess(audio_data, sample_rate)
+        return self.encoder(audio_data)["mu"]
+
+
+class StreamingVAEDecoder:
+    """Stateful streaming wrapper for :class:`AudioVAE`.
+
+    Carries causal-convolution padding buffers between calls so that each
+    ``decode_chunk`` processes only the new latent patch — no overlap needed.
+    """
+
+    def __init__(self, vae: AudioVAE):
+        self._vae = vae
+        self._states: dict = {}
+        self._originals: list = []
+
+    # -- context manager --------------------------------------------------
+    def __enter__(self):
+        self._states.clear()
+        self._install()
+        return self
+
+    def __exit__(self, *exc):
+        self._restore()
+        self._states.clear()
+
+    # -- public API --------------------------------------------------------
+    def decode_chunk(self, z_chunk: torch.Tensor) -> torch.Tensor:
+        """Decode a single latent chunk and return the audio waveform."""
+        return self._vae.decode(z_chunk)
+
+    # -- internals ---------------------------------------------------------
+    def _install(self):
+        for name, mod in self._vae.decoder.named_modules():
+            if isinstance(mod, CausalConv1d):
+                pad = mod._CausalConv1d__padding * 2 - mod._CausalConv1d__output_padding
+                if pad > 0:
+                    self._patch_causal_conv(mod, pad)
+            elif isinstance(mod, CausalTransposeConv1d):
+                trim = mod._CausalTransposeConv1d__padding * 2 - mod._CausalTransposeConv1d__output_padding
+                ctx = (mod.kernel_size[0] - 1) // mod.stride[0]
+                if ctx > 0:
+                    self._patch_transpose_conv(mod, ctx, trim)
+
+    def _patch_causal_conv(self, mod, pad_size):
+        states = self._states
+        key = id(mod)
+        orig = mod.forward
+
+        def fwd(x, _k=key, _p=pad_size, _m=mod):
+            x_pad = torch.cat([states[_k], x], dim=-1) if _k in states else F.pad(x, (_p, 0))
+            if x.shape[-1] >= _p:
+                states[_k] = x[:, :, -_p:].detach()
+            else:
+                prev = states.get(_k, torch.zeros(x.shape[0], x.shape[1], _p,
+                                                  device=x.device, dtype=x.dtype))
+                states[_k] = torch.cat([prev, x], dim=-1)[:, :, -_p:].detach()
+            return nn.Conv1d.forward(_m, x_pad)
+
+        mod.forward = fwd
+        self._originals.append((mod, orig))
+
+    def _patch_transpose_conv(self, mod, ctx, trim):
+        states = self._states
+        key = id(mod)
+        orig = mod.forward
+
+        def fwd(x, _k=key, _c=ctx, _t=trim, _m=mod):
+            x_full = torch.cat([states[_k], x], dim=-1) if _k in states else F.pad(x, (_c, 0))
+            states[_k] = x[:, :, -_c:].detach()
+            out = nn.ConvTranspose1d.forward(_m, x_full)
+            left = _c * _m.stride[0]
+            return out[..., left:-_t] if _t > 0 else out[..., left:]
+
+        mod.forward = fwd
+        self._originals.append((mod, orig))
+
+    def _restore(self):
+        for mod, orig in self._originals:
+            mod.forward = orig
+        self._originals.clear()

bluemagpie/_vendor/voxcpm/modules/layers/__init__.py

@@ -0,0 +1 @@
+from .scalar_quantization_layer import ScalarQuantizationLayer

bluemagpie/_vendor/voxcpm/modules/layers/lora.py

@@ -0,0 +1,130 @@
+import math
+from typing import Optional
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+
+class LoRALinear(nn.Module):
+    """
+    LoRA 线性层：直接持有 weight/bias，保持与 nn.Linear 相同的 state_dict key 结构。
+
+    state_dict 结构：
+        - weight: 原始权重（与 nn.Linear 一致）
+        - bias: 原始偏置（与 nn.Linear 一致）
+        - lora_A: LoRA 低秩矩阵 A
+        - lora_B: LoRA 低秩矩阵 B
+
+    这样设计的好处：加载预训练权重时无需做 key 转换。
+    """
+
+    def __init__(
+        self,
+        base: nn.Linear,
+        r: int,
+        alpha: float = 1.0,
+        dropout: float = 0.0,
+    ):
+        super().__init__()
+        assert isinstance(base, nn.Linear), "LoRALinear only supports wrapping nn.Linear."
+
+        self.in_features = base.in_features
+        self.out_features = base.out_features
+        self.r = r
+        self.alpha = alpha
+        self._base_scaling = alpha / r if r > 0 else 0.0
+
+        # 使用 buffer 存储 scaling，这样修改值不会触发 torch.compile 重编译
+        # persistent=False 表示不保存到 state_dict，避免加载时 missing key
+        self.register_buffer("scaling", torch.tensor(self._base_scaling), persistent=False)
+
+        # 直接持有 weight 和 bias（从原始 Linear 转移过来）
+        self.weight = base.weight
+        self.bias = base.bias  # 可能是 None
+
+        # LoRA 参数
+        if r > 0:
+            self.lora_A = nn.Parameter(torch.zeros(r, self.in_features))
+            self.lora_B = nn.Parameter(torch.zeros(self.out_features, r))
+            nn.init.kaiming_uniform_(self.lora_A, a=math.sqrt(5))
+            nn.init.zeros_(self.lora_B)
+        else:
+            self.register_parameter("lora_A", None)
+            self.register_parameter("lora_B", None)
+
+        self.dropout = nn.Dropout(dropout) if dropout > 0 else nn.Identity()
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        # 基础 Linear 计算
+        result = F.linear(x, self.weight, self.bias)
+        if self.r <= 0 or self.lora_A is None:
+            return result
+        # LoRA: result + dropout(x @ A^T @ B^T) * scaling
+        lora_out = F.linear(F.linear(x, self.lora_A), self.lora_B)
+        return result + self.dropout(lora_out) * self.scaling
+
+    def reset_lora_parameters(self):
+        """重置 LoRA 参数到初始状态"""
+        if self.r > 0 and self.lora_A is not None:
+            nn.init.kaiming_uniform_(self.lora_A, a=math.sqrt(5))
+            nn.init.zeros_(self.lora_B)
+
+    def set_enabled(self, enabled: bool):
+        """启用/禁用 LoRA（通过 scaling 控制，兼容 torch.compile）"""
+        # 使用 fill_ 原地修改 buffer 值，不会触发重编译
+        self.scaling.fill_(self._base_scaling if enabled else 0.0)
+
+    @property
+    def enabled(self) -> bool:
+        return self.scaling.item() != 0.0
+
+
+def _get_parent_module(root: nn.Module, name: str) -> Optional[nn.Module]:
+    """
+    根据类似 'layers.0.self_attn.q_proj' 的全名，返回 parent module（即 q_proj 的上一级）。
+    """
+    parts = name.split(".")
+    if len(parts) == 1:
+        return root
+    parent = root
+    for p in parts[:-1]:
+        if not hasattr(parent, p):
+            return None
+        parent = getattr(parent, p)
+    return parent
+
+
+def apply_lora_to_named_linear_modules(
+    root: nn.Module,
+    *,
+    target_submodule_names: list[str],
+    r: int,
+    alpha: float,
+    dropout: float,
+) -> None:
+    """
+    在给定模块及其子模块中，对名字以 target_submodule_names 结尾的 Linear 层注入 LoRA。
+
+    例如 target_submodule_names=["q_proj", "v_proj"] 时，
+    会在所有名为 *.q_proj / *.v_proj 的 nn.Linear 上替换为 LoRALinear。
+    """
+    for full_name, module in list(root.named_modules()):
+        if not isinstance(module, nn.Linear):
+            continue
+        short_name = full_name.split(".")[-1]
+        if short_name not in target_submodule_names:
+            continue
+
+        parent = _get_parent_module(root, full_name)
+        if parent is None:
+            continue
+
+        # 用 LoRALinear 替换原始 Linear
+        lora_layer = LoRALinear(
+            base=module,
+            r=r,
+            alpha=alpha,
+            dropout=dropout,
+        )
+        setattr(parent, short_name, lora_layer)

bluemagpie/_vendor/voxcpm/modules/layers/scalar_quantization_layer.py

@@ -0,0 +1,26 @@
+import torch
+import torch.nn as nn
+
+
+class ScalarQuantizationLayer(nn.Module):
+    def __init__(self, in_dim, out_dim, latent_dim: int = 64, scale: int = 9):
+        super().__init__()
+        self.in_dim = in_dim
+        self.out_dim = out_dim
+        self.latent_dim = latent_dim
+        self.scale = scale
+
+        self.in_proj = nn.Linear(in_dim, latent_dim)
+        self.out_proj = nn.Linear(latent_dim, out_dim)
+
+    def forward(self, hidden):
+        hidden = self.in_proj(hidden)
+        hidden = torch.tanh(hidden)
+
+        if self.training:
+            quantized = torch.round(hidden * self.scale) / self.scale
+            hidden = hidden + (quantized - hidden).detach()
+        else:
+            hidden = torch.round(hidden * self.scale) / self.scale
+
+        return self.out_proj(hidden)

bluemagpie/_vendor/voxcpm/modules/locdit/__init__.py

@@ -0,0 +1,3 @@
+from .unified_cfm import UnifiedCFM, CfmConfig
+from .local_dit import VoxCPMLocDiT
+from .local_dit_v2 import VoxCPMLocDiT as VoxCPMLocDiTV2

bluemagpie/_vendor/voxcpm/modules/locdit/local_dit.py

@@ -0,0 +1,114 @@
+import torch
+from ..minicpm4 import MiniCPMModel, MiniCPM4Config
+import torch.nn as nn
+import math
+
+
+class SinusoidalPosEmb(torch.nn.Module):
+    def __init__(self, dim):
+        super().__init__()
+        self.dim = dim
+        assert self.dim % 2 == 0, "SinusoidalPosEmb requires dim to be even"
+
+    def forward(self, x, scale=1000):
+        if x.ndim < 1:
+            x = x.unsqueeze(0)
+        device = x.device
+        half_dim = self.dim // 2
+        emb = math.log(10000) / (half_dim - 1)
+        emb = torch.exp(torch.arange(half_dim, dtype=x.dtype, device=device) * -emb)
+        emb = scale * x.unsqueeze(1) * emb.unsqueeze(0)
+        emb = torch.cat((emb.sin(), emb.cos()), dim=-1)
+        return emb
+
+
+class TimestepEmbedding(nn.Module):
+    def __init__(
+        self,
+        in_channels: int,
+        time_embed_dim: int,
+        out_dim: int = None,
+    ):
+        super().__init__()
+
+        self.linear_1 = nn.Linear(in_channels, time_embed_dim, bias=True)
+        self.act = nn.SiLU()
+        if out_dim is not None:
+            time_embed_dim_out = out_dim
+        else:
+            time_embed_dim_out = time_embed_dim
+
+        self.linear_2 = nn.Linear(time_embed_dim, time_embed_dim_out, bias=True)
+
+    def forward(self, sample):
+        sample = self.linear_1(sample)
+        sample = self.act(sample)
+        sample = self.linear_2(sample)
+        return sample
+
+
+class VoxCPMLocDiT(nn.Module):
+    """
+    Diffusion model with a Transformer backbone.
+    """
+
+    def __init__(
+        self,
+        config: MiniCPM4Config,
+        in_channels: int = 64,
+    ):
+        super().__init__()
+        self.in_channels = in_channels
+        self.out_channels = in_channels
+        self.config = config
+
+        self.in_proj = nn.Linear(in_channels, config.hidden_size, bias=True)
+        self.cond_proj = nn.Linear(in_channels, config.hidden_size, bias=True)
+        self.out_proj = nn.Linear(config.hidden_size, self.out_channels, bias=True)
+
+        self.time_embeddings = SinusoidalPosEmb(config.hidden_size)
+        self.time_mlp = TimestepEmbedding(
+            in_channels=config.hidden_size,
+            time_embed_dim=config.hidden_size,
+        )
+        self.delta_time_mlp = TimestepEmbedding(
+            in_channels=config.hidden_size,
+            time_embed_dim=config.hidden_size,
+        )
+
+        assert config.vocab_size == 0, "vocab_size must be 0 for local DiT"
+        self.decoder = MiniCPMModel(config)
+
+    def forward(
+        self,
+        x: torch.Tensor,
+        mu: torch.Tensor,
+        t: torch.Tensor,
+        cond: torch.Tensor,
+        dt: torch.Tensor,
+    ):
+        """
+        Forward pass of DiT.
+        x: (N, C, T) tensor of inputs
+        mu: (N, C) tensor of hidden embedding
+        t: (N,) tensor of diffusion timesteps
+        cond: (N, C, T') tensor of prefix conditions
+        dt: (N,) used for mean velocity (may be supported in the future...)
+        """
+        x = self.in_proj(x.transpose(1, 2).contiguous())
+
+        cond = self.cond_proj(cond.transpose(1, 2).contiguous())
+        prefix = cond.size(1)
+
+        t = self.time_embeddings(t).to(x.dtype)
+        t = self.time_mlp(t)
+        dt = self.time_embeddings(dt).to(x.dtype)
+        dt = self.delta_time_mlp(dt)
+        t = t + dt
+
+        x = torch.cat([(mu + t).unsqueeze(1), cond, x], dim=1)
+        hidden, _ = self.decoder(x, is_causal=False)
+        hidden = hidden[:, prefix + 1 :, :]
+        hidden = self.out_proj(hidden)
+
+        return hidden.transpose(1, 2).contiguous()

bluemagpie/_vendor/voxcpm/modules/locdit/local_dit_v2.py

@@ -0,0 +1,116 @@
+import torch
+from ..minicpm4 import MiniCPMModel, MiniCPM4Config
+import torch.nn as nn
+import math
+
+
+class SinusoidalPosEmb(torch.nn.Module):
+    def __init__(self, dim):
+        super().__init__()
+        self.dim = dim
+        assert self.dim % 2 == 0, "SinusoidalPosEmb requires dim to be even"
+
+    def forward(self, x, scale=1000):
+        if x.ndim < 1:
+            x = x.unsqueeze(0)
+        device = x.device
+        half_dim = self.dim // 2
+        emb = math.log(10000) / (half_dim - 1)
+        emb = torch.exp(torch.arange(half_dim, dtype=x.dtype, device=device) * -emb)
+        emb = scale * x.unsqueeze(1) * emb.unsqueeze(0)
+        emb = torch.cat((emb.sin(), emb.cos()), dim=-1)
+        return emb
+
+
+class TimestepEmbedding(nn.Module):
+    def __init__(
+        self,
+        in_channels: int,
+        time_embed_dim: int,
+        out_dim: int = None,
+    ):
+        super().__init__()
+
+        self.linear_1 = nn.Linear(in_channels, time_embed_dim, bias=True)
+        self.act = nn.SiLU()
+        if out_dim is not None:
+            time_embed_dim_out = out_dim
+        else:
+            time_embed_dim_out = time_embed_dim
+
+        self.linear_2 = nn.Linear(time_embed_dim, time_embed_dim_out, bias=True)
+
+    def forward(self, sample):
+        sample = self.linear_1(sample)
+        sample = self.act(sample)
+        sample = self.linear_2(sample)
+        return sample
+
+
+class VoxCPMLocDiT(nn.Module):
+    """
+    Diffusion model with a Transformer backbone.
+    """
+
+    def __init__(
+        self,
+        config: MiniCPM4Config,
+        in_channels: int = 64,
+    ):
+        super().__init__()
+        self.in_channels = in_channels
+        self.out_channels = in_channels
+        self.config = config
+
+        self.in_proj = nn.Linear(in_channels, config.hidden_size, bias=True)
+        self.cond_proj = nn.Linear(in_channels, config.hidden_size, bias=True)
+        self.out_proj = nn.Linear(config.hidden_size, self.out_channels, bias=True)
+
+        self.time_embeddings = SinusoidalPosEmb(config.hidden_size)
+        self.time_mlp = TimestepEmbedding(
+            in_channels=config.hidden_size,
+            time_embed_dim=config.hidden_size,
+        )
+        self.delta_time_mlp = TimestepEmbedding(
+            in_channels=config.hidden_size,
+            time_embed_dim=config.hidden_size,
+        )
+
+        assert config.vocab_size == 0, "vocab_size must be 0 for local DiT"
+        self.decoder = MiniCPMModel(config)
+
+    def forward(
+        self,
+        x: torch.Tensor,
+        mu: torch.Tensor,
+        t: torch.Tensor,
+        cond: torch.Tensor,
+        dt: torch.Tensor,
+    ):
+        """
+        Forward pass of DiT.
+        x: (N, C, T) tensor of inputs
+        mu: (N, C) tensor of hidden embedding
+        t: (N,) tensor of diffusion timesteps
+        cond: (N, C, T') tensor of prefix conditions
+        dt: (N,) used for mean velocity (may be supported in the future...)
+        """
+        x = self.in_proj(x.transpose(1, 2).contiguous())
+
+        cond = self.cond_proj(cond.transpose(1, 2).contiguous())
+        prefix = cond.size(1)
+
+        t = self.time_embeddings(t).to(x.dtype)
+        t = self.time_mlp(t)
+        dt = self.time_embeddings(dt).to(x.dtype)
+        dt = self.delta_time_mlp(dt)
+        t = t + dt
+
+        mu = mu.view(x.size(0), -1, x.size(-1))
+        x = torch.cat([mu, (t).unsqueeze(1), cond, x], dim=1)
+
+        hidden, _ = self.decoder(x, is_causal=False)
+        hidden = hidden[:, prefix + mu.size(1) + 1 :, :]
+        hidden = self.out_proj(hidden)
+
+        return hidden.transpose(1, 2).contiguous()

bluemagpie/_vendor/voxcpm/modules/locdit/unified_cfm.py

@@ -0,0 +1,232 @@
+from typing import Tuple
+
+import torch
+import torch.nn.functional as F
+from torch.func import jvp
+from pydantic import BaseModel
+
+from .local_dit import VoxCPMLocDiT
+
+
+class CfmConfig(BaseModel):
+    sigma_min: float = 1e-6
+    solver: str = "euler"
+    t_scheduler: str = "log-norm"
+    training_cfg_rate: float = 0.1
+    inference_cfg_rate: float = 1.0
+    reg_loss_type: str = "l1"
+    ratio_r_neq_t_range: Tuple[float, float] = (0.25, 0.75)
+    noise_cond_prob_range: Tuple[float, float] = (0.0, 0.0)
+    noise_cond_scale: float = 0.0
+
+
+class UnifiedCFM(torch.nn.Module):
+    def __init__(
+        self,
+        in_channels: int,
+        cfm_params: CfmConfig,
+        estimator: VoxCPMLocDiT,
+        mean_mode: bool = False,
+    ):
+        super().__init__()
+        self.solver = cfm_params.solver
+        self.sigma_min = cfm_params.sigma_min
+        self.t_scheduler = cfm_params.t_scheduler
+        self.training_cfg_rate = cfm_params.training_cfg_rate
+        self.inference_cfg_rate = cfm_params.inference_cfg_rate
+        self.reg_loss_type = cfm_params.reg_loss_type
+        self.ratio_r_neq_t_range = cfm_params.ratio_r_neq_t_range
+        self.noise_cond_prob_range = cfm_params.noise_cond_prob_range
+        self.noise_cond_scale = cfm_params.noise_cond_scale
+
+        self.in_channels = in_channels
+        self.mean_mode = mean_mode
+
+        self.estimator = estimator
+
+    # ------------------------------------------------------------------ #
+    # Inference
+    # ------------------------------------------------------------------ #
+    @torch.inference_mode()
+    def forward(
+        self,
+        mu: torch.Tensor,
+        n_timesteps: int,
+        patch_size: int,
+        cond: torch.Tensor,
+        temperature: float = 1.0,
+        cfg_value: float = 1.0,
+        sway_sampling_coef: float = 1.0,
+        use_cfg_zero_star: bool = True,
+    ):
+        b, _ = mu.shape
+        t = patch_size
+        z = torch.randn((b, self.in_channels, t), device=mu.device, dtype=mu.dtype) * temperature
+
+        t_span = torch.linspace(1, 0, n_timesteps + 1, device=mu.device, dtype=mu.dtype)
+        t_span = t_span + sway_sampling_coef * (torch.cos(torch.pi / 2 * t_span) - 1 + t_span)
+
+        return self.solve_euler(
+            x=z,
+            t_span=t_span,
+            mu=mu,
+            cond=cond,
+            cfg_value=cfg_value,
+            use_cfg_zero_star=use_cfg_zero_star,
+        )
+
+    def optimized_scale(self, positive_flat: torch.Tensor, negative_flat: torch.Tensor):
+        dot_product = torch.sum(positive_flat * negative_flat, dim=1, keepdim=True)
+        squared_norm = torch.sum(negative_flat**2, dim=1, keepdim=True) + 1e-8
+        st_star = dot_product / squared_norm
+        return st_star
+
+    def solve_euler(
+        self,
+        x: torch.Tensor,
+        t_span: torch.Tensor,
+        mu: torch.Tensor,
+        cond: torch.Tensor,
+        cfg_value: float = 1.0,
+        use_cfg_zero_star: bool = True,
+    ):
+        t, _, dt = t_span[0], t_span[-1], t_span[0] - t_span[1]
+
+        sol = []
+        zero_init_steps = max(1, int(len(t_span) * 0.04))
+        for step in range(1, len(t_span)):
+            if use_cfg_zero_star and step <= zero_init_steps:
+                dphi_dt = torch.zeros_like(x)
+            else:
+                # Classifier-Free Guidance inference introduced in VoiceBox
+                b = x.size(0)
+                x_in = torch.zeros([2 * b, self.in_channels, x.size(2)], device=x.device, dtype=x.dtype)
+                mu_in = torch.zeros([2 * b, mu.size(1)], device=x.device, dtype=x.dtype)
+                t_in = torch.zeros([2 * b], device=x.device, dtype=x.dtype)
+                dt_in = torch.zeros([2 * b], device=x.device, dtype=x.dtype)
+                cond_in = torch.zeros([2 * b, self.in_channels, cond.size(2)], device=x.device, dtype=x.dtype)
+                x_in[:b], x_in[b:] = x, x
+                mu_in[:b] = mu
+                t_in[:b], t_in[b:] = t.unsqueeze(0), t.unsqueeze(0)
+                dt_in[:b], dt_in[b:] = dt.unsqueeze(0), dt.unsqueeze(0)
+                # not used now
+                if not self.mean_mode:
+                    dt_in = torch.zeros_like(dt_in)
+                cond_in[:b], cond_in[b:] = cond, cond
+
+                dphi_dt = self.estimator(x_in, mu_in, t_in, cond_in, dt_in)
+                dphi_dt, cfg_dphi_dt = torch.split(dphi_dt, [x.size(0), x.size(0)], dim=0)
+
+                if use_cfg_zero_star:
+                    positive_flat = dphi_dt.view(b, -1)
+                    negative_flat = cfg_dphi_dt.view(b, -1)
+                    st_star = self.optimized_scale(positive_flat, negative_flat)
+                    st_star = st_star.view(b, *([1] * (len(dphi_dt.shape) - 1)))
+                else:
+                    st_star = 1.0
+
+                dphi_dt = cfg_dphi_dt * st_star + cfg_value * (dphi_dt - cfg_dphi_dt * st_star)
+
+            x = x - dt * dphi_dt
+            t = t - dt
+            sol.append(x)
+            if step < len(t_span) - 1:
+                dt = t - t_span[step + 1]
+
+        return sol[-1]
+
+    # ------------------------------------------------------------------ #
+    # Training loss
+    # ------------------------------------------------------------------ #
+    def adaptive_loss_weighting(
+        self, losses: torch.Tensor, mask: torch.Tensor | None = None, p: float = 0.0, epsilon: float = 1e-3
+    ):
+        weights = 1.0 / ((losses + epsilon).pow(p))
+        if mask is not None:
+            weights = weights * mask
+        return weights.detach()
+
+    def sample_r_t(self, x: torch.Tensor, mu: float = -0.4, sigma: float = 1.0, ratio_r_neq_t: float = 0.0):
+        batch_size = x.shape[0]
+        if self.t_scheduler == "log-norm":
+            s_r = torch.randn(batch_size, device=x.device, dtype=x.dtype) * sigma + mu
+            s_t = torch.randn(batch_size, device=x.device, dtype=x.dtype) * sigma + mu
+            r = torch.sigmoid(s_r)
+            t = torch.sigmoid(s_t)
+        elif self.t_scheduler == "uniform":
+            r = torch.rand(batch_size, device=x.device, dtype=x.dtype)
+            t = torch.rand(batch_size, device=x.device, dtype=x.dtype)
+        else:
+            raise ValueError(f"Unsupported t_scheduler: {self.t_scheduler}")
+
+        mask = torch.rand(batch_size, device=x.device, dtype=x.dtype) < ratio_r_neq_t
+        r, t = torch.where(
+            mask,
+            torch.stack([torch.min(r, t), torch.max(r, t)], dim=0),
+            torch.stack([t, t], dim=0),
+        )
+
+        return r.squeeze(), t.squeeze()
+
+    def compute_loss(
+        self,
+        x1: torch.Tensor,
+        mu: torch.Tensor,
+        cond: torch.Tensor | None = None,
+        tgt_mask: torch.Tensor | None = None,
+        progress: float = 0.0,
+    ):
+        b, _, _ = x1.shape
+
+        if self.training_cfg_rate > 0:
+            cfg_mask = torch.rand(b, device=x1.device) > self.training_cfg_rate
+            mu = mu * cfg_mask.view(-1, 1)
+
+        if cond is None:
+            cond = torch.zeros_like(x1)
+
+        noisy_mask = torch.rand(b, device=x1.device) > (
+            1.0
+            - (
+                self.noise_cond_prob_range[0]
+                + progress * (self.noise_cond_prob_range[1] - self.noise_cond_prob_range[0])
+            )
+        )
+        cond = cond + noisy_mask.view(-1, 1, 1) * torch.randn_like(cond) * self.noise_cond_scale
+
+        ratio_r_neq_t = (
+            self.ratio_r_neq_t_range[0] + progress * (self.ratio_r_neq_t_range[1] - self.ratio_r_neq_t_range[0])
+            if self.mean_mode
+            else 0.0
+        )
+
+        r, t = self.sample_r_t(x1, ratio_r_neq_t=ratio_r_neq_t)
+        r_ = r.detach().clone()
+        t_ = t.detach().clone()
+        z = torch.randn_like(x1)
+        y = (1 - t_.view(-1, 1, 1)) * x1 + t_.view(-1, 1, 1) * z
+        v = z - x1
+
+        def model_fn(z_sample, r_sample, t_sample):
+            return self.estimator(z_sample, mu, t_sample, cond, dt=t_sample - r_sample)
+
+        if self.mean_mode:
+            v_r = torch.zeros_like(r)
+            v_t = torch.ones_like(t)
+            from torch.backends.cuda import sdp_kernel
+
+            with sdp_kernel(enable_flash=False, enable_mem_efficient=False):
+                u_pred, dudt = jvp(model_fn, (y, r, t), (v, v_r, v_t))
+            u_tgt = v - (t_ - r_).view(-1, 1, 1) * dudt
+        else:
+            u_pred = model_fn(y, r, t)
+            u_tgt = v
+
+        losses = F.mse_loss(u_pred, u_tgt.detach(), reduction="none").mean(dim=1)
+        if tgt_mask is not None:
+            weights = self.adaptive_loss_weighting(losses, tgt_mask.squeeze(1))
+            loss = (weights * losses).sum() / torch.clamp(torch.sum(tgt_mask), min=1.0)
+        else:
+            loss = losses.mean()
+
+        return loss

bluemagpie/_vendor/voxcpm/modules/locenc/__init__.py

@@ -0,0 +1 @@
+from .local_encoder import VoxCPMLocEnc

bluemagpie/_vendor/voxcpm/modules/locenc/local_encoder.py

@@ -0,0 +1,30 @@
+import torch
+import torch.nn as nn
+from ..minicpm4 import MiniCPMModel, MiniCPM4Config
+from einops import rearrange
+
+
+class VoxCPMLocEnc(nn.Module):
+    def __init__(self, config: MiniCPM4Config, input_dim: int = 64):
+        super().__init__()
+        self.config = config
+        self.special_token = nn.Parameter(torch.randn(1, 1, 1, config.hidden_size))
+        self.in_proj = nn.Linear(input_dim, config.hidden_size, bias=True)
+
+        assert config.vocab_size == 0, "vocab_size must be 0 for local encoder"
+        self.encoder = MiniCPMModel(config)
+
+    def forward(self, x):
+        """
+        x: [B, T, P, D]
+        """
+        B, T, P, D = x.shape
+
+        x = self.in_proj(x)
+        special_tokens = self.special_token.expand(B, T, 1, -1)
+        x = torch.cat([special_tokens, x], dim=2)
+        x = rearrange(x, "b t p c -> (b t) p c")
+        outputs, _ = self.encoder(x, is_causal=False)
+        cls_output = outputs[:, 0, :]
+
+        return rearrange(cls_output, "(b t) c -> b t c", b=B)

bluemagpie/_vendor/voxcpm/modules/minicpm4/__init__.py

@@ -0,0 +1,3 @@
+from .config import MiniCPM4Config
+from .model import MiniCPMModel
+from .cache import StaticKVCache

bluemagpie/_vendor/voxcpm/modules/minicpm4/cache.py

@@ -0,0 +1,47 @@
+from typing import List, Tuple
+import torch
+
+
+class StaticKVCache:
+    def __init__(
+        self,
+        num_layers: int,
+        num_kv_heads: int,
+        dim_kv_head: int,
+        batch_size: int,
+        device: torch.device,
+        dtype: torch.dtype,
+        max_length: int = 8192,
+    ):
+        self.max_length = max_length
+        self.num_layers = num_layers
+
+        self.kv_cache = torch.zeros(
+            2,
+            num_layers,
+            batch_size,
+            num_kv_heads,
+            max_length,
+            dim_kv_head,
+            device=device,
+            dtype=dtype,
+        )
+        self.current_length = 0
+
+    def get_layer_cache(self, layer_idx: int) -> Tuple[torch.Tensor, torch.Tensor]:
+        return self.kv_cache[0, layer_idx], self.kv_cache[1, layer_idx]
+
+    def step(self) -> int:
+        if self.current_length >= self.max_length:
+            raise ValueError("KV cache is full")
+
+        ret = self.current_length
+        self.current_length += 1
+        return ret
+
+    def fill_caches(self, kv_caches: List[Tuple[torch.Tensor, torch.Tensor]]):
+        self.current_length = kv_caches[0][0].size(2)
+        self.kv_cache.zero_()
+        for i in range(self.num_layers):
+            self.kv_cache[0, i, :, :, : self.current_length, :] = kv_caches[i][0]
+            self.kv_cache[1, i, :, :, : self.current_length, :] = kv_caches[i][1]

bluemagpie/_vendor/voxcpm/modules/minicpm4/config.py

@@ -0,0 +1,30 @@
+from pydantic import BaseModel
+from typing import List
+
+
+class RopeScalingConfig(BaseModel):
+    type: str
+    long_factor: List[float]
+    short_factor: List[float]
+    original_max_position_embeddings: int
+
+
+class MiniCPM4Config(BaseModel):
+    bos_token_id: int
+    eos_token_id: int
+    hidden_size: int
+    intermediate_size: int
+    max_position_embeddings: int
+    num_attention_heads: int
+    num_hidden_layers: int
+    num_key_value_heads: int
+    rms_norm_eps: float
+    rope_scaling: RopeScalingConfig
+    vocab_size: int
+    use_mup: bool = True
+    scale_emb: float
+    dim_model_base: int
+    scale_depth: float
+    rope_theta: float
+    kv_channels: int = None
+    no_rope: bool = False

bluemagpie/_vendor/voxcpm/modules/minicpm4/model.py

@@ -0,0 +1,431 @@
+from .config import MiniCPM4Config
+import torch
+import torch.nn as nn
+from typing import List, Tuple
+import math
+from .cache import StaticKVCache
+
+
+def rms_layernorm(hidden: torch.Tensor, weight: torch.Tensor, eps: float):
+    old_dtype = hidden.dtype
+    variance = hidden.to(torch.float32).pow(2).mean(dim=-1, keepdim=True)
+    hidden = (hidden * torch.rsqrt(variance + eps)).to(old_dtype)
+    return hidden * weight
+
+
+class MiniCPMRMSNorm(nn.Module):
+    def __init__(self, hidden_size, eps=1e-6):
+        """
+        MiniCPMRMSNorm is equivalent to T5LayerNorm
+        """
+        super().__init__()
+        self.weight = nn.Parameter(torch.ones(hidden_size))
+        self.variance_epsilon = eps
+
+    def forward(self, hidden_states):
+        return rms_layernorm(hidden_states, self.weight, self.variance_epsilon)
+
+
+def rotate_half(x):
+    """Rotates half the hidden dims of the input."""
+    x1, x2 = x.chunk(2, dim=-1)
+    return torch.cat((-x2, x1), dim=-1)
+
+
+def apply_rotary_pos_emb(q: torch.Tensor, k: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor):
+    """
+    Args:
+        q: Tensor(batch_size, num_heads, seq_len, head_dim)
+        k: Tensor(batch_size, num_key_value_heads, seq_len, head_dim)
+        cos: Tensor(seq_len, head_dim)
+        sin: Tensor(seq_len, head_dim)
+    Returns:
+        Tensor(batch_size, num_heads, seq_len, head_dim), Tensor(batch_size, num_key_value_heads, seq_len, head_dim)
+    """
+    orig_dtype = q.dtype
+    q = q.to(torch.float32)
+    k = k.to(torch.float32)
+    q_embed = (q * cos) + (rotate_half(q) * sin)
+    k_embed = (k * cos) + (rotate_half(k) * sin)
+    return q_embed.to(orig_dtype), k_embed.to(orig_dtype)
+
+
+class MiniCPMLongRoPE(nn.Module):
+    """MiniCPMRotaryEmbedding extended with Dynamic NTK scaling. Credits to the Reddit users /u/bloc97 and /u/emozilla"""
+
+    def __init__(self, config: MiniCPM4Config):
+        super().__init__()
+        self.config = config
+        self.dim = config.kv_channels if config.kv_channels else config.hidden_size // config.num_attention_heads
+        self.base = config.rope_theta
+        self.max_position_embeddings = config.max_position_embeddings
+
+        self.short_factor = config.rope_scaling.short_factor
+        self.long_factor = config.rope_scaling.long_factor
+        self.original_max_position_embeddings = config.rope_scaling.original_max_position_embeddings
+
+        scale = self.max_position_embeddings / self.original_max_position_embeddings
+        self.scaling_factor = math.sqrt(1 + math.log(scale) / math.log(self.original_max_position_embeddings))
+        inv_freq = 1.0 / (self.base ** (torch.arange(0, self.dim, 2).float() / self.dim))
+        self.register_buffer("inv_freq", inv_freq, persistent=False)
+
+        self.max_seq_len_cached = 0
+
+        self.register_buffer("cos_cached", torch.empty(0), persistent=False)
+        self.register_buffer("sin_cached", torch.empty(0), persistent=False)
+
+        self._set_cos_sin_cache(seq_len=self.max_position_embeddings, device=self.inv_freq.device, dtype=torch.float32)
+
+    def _set_cos_sin_cache(self, seq_len, device, dtype):
+        """设置cos和sin缓存"""
+        self.max_seq_len_cached = seq_len
+        t = torch.arange(self.max_seq_len_cached, device=device, dtype=self.inv_freq.dtype)
+
+        if seq_len > self.original_max_position_embeddings:
+            ext_factors = torch.tensor(self.long_factor, dtype=torch.float32, device=device)
+        else:
+            ext_factors = torch.tensor(self.short_factor, dtype=torch.float32, device=device)
+
+        freqs = torch.mul(
+            torch.outer(t, 1.0 / ext_factors).to(device=device), self.inv_freq.to(device=device).to(dtype)
+        )
+
+        # 创建embeddings
+        emb = torch.cat((freqs, freqs), dim=-1)
+
+        self.cos_cached = emb.cos().to(dtype) * self.scaling_factor
+        self.sin_cached = emb.sin().to(dtype) * self.scaling_factor
+
+    def forward(self, position_ids: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
+        """
+        Args:
+            position_ids: Tensor(seq_len) 或 Tensor(batch_size, seq_len)
+        Returns:
+            Tensor(seq_len, head_dim), Tensor(seq_len, head_dim)
+        """
+        cos = self.cos_cached[position_ids]
+        sin = self.sin_cached[position_ids]
+
+        return cos, sin
+
+
+class MiniCPMAttention(nn.Module):
+    def __init__(self, config: MiniCPM4Config, layer_idx: int):
+        super().__init__()
+        self.config = config
+        self.layer_idx = layer_idx
+        self.hidden_size = config.hidden_size
+        self.num_heads = config.num_attention_heads
+        self.head_dim = (
+            config.hidden_size // config.num_attention_heads if config.kv_channels is None else config.kv_channels
+        )
+        self.num_key_value_heads = config.num_key_value_heads
+        self.num_key_value_groups = self.num_heads // self.num_key_value_heads
+        self.max_position_embeddings = config.max_position_embeddings
+        self.rope_theta = 10000.0
+
+        self.q_proj = nn.Linear(self.hidden_size, self.num_heads * self.head_dim, bias=False)
+        self.k_proj = nn.Linear(self.hidden_size, self.num_key_value_heads * self.head_dim, bias=False)
+        self.v_proj = nn.Linear(self.hidden_size, self.num_key_value_heads * self.head_dim, bias=False)
+        self.o_proj = nn.Linear(self.num_heads * self.head_dim, self.hidden_size, bias=False)
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        position_emb: Tuple[torch.Tensor, torch.Tensor],
+        is_causal: bool,
+    ) -> Tuple[torch.Tensor, Tuple[torch.Tensor, torch.Tensor]]:
+        bsz, q_len, _ = hidden_states.size()
+
+        query_states = self.q_proj(hidden_states)
+        key_states = self.k_proj(hidden_states)
+        value_states = self.v_proj(hidden_states)
+
+        query_states = query_states.view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)
+        key_states = key_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
+        value_states = value_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
+
+        if position_emb is not None:
+            cos, sin = position_emb
+            query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)
+
+        # ref: https://github.com/pytorch/pytorch/issues/163597
+        # there is a bug in MPS for non-contiguous tensors, so we need to make them contiguous
+        query_states = query_states.contiguous()
+        key_states = key_states.contiguous()
+        value_states = value_states.contiguous()
+        attn_output = torch.nn.functional.scaled_dot_product_attention(
+            query_states,
+            key_states,
+            value_states,
+            is_causal=is_causal,
+            enable_gqa=True,
+        )
+
+        attn_output = attn_output.transpose(1, 2).contiguous()
+        attn_output = attn_output.reshape(bsz, q_len, self.num_heads * self.head_dim)
+
+        attn_output = self.o_proj(attn_output)
+
+        past_key_value = (key_states, value_states)
+        return attn_output, past_key_value
+
+    def forward_step(
+        self,
+        hidden_states: torch.Tensor,
+        position_emb: Tuple[torch.Tensor, torch.Tensor],
+        position_id: int,
+        kv_cache: Tuple[torch.Tensor, torch.Tensor],
+    ) -> torch.Tensor:
+        bsz, _ = hidden_states.size()
+
+        query_states = self.q_proj(hidden_states)
+        key_states = self.k_proj(hidden_states)
+        value_states = self.v_proj(hidden_states)
+
+        query_states = query_states.view(bsz, 1, self.num_heads, self.head_dim).transpose(1, 2)
+        key_states = key_states.view(bsz, 1, self.num_key_value_heads, self.head_dim).transpose(1, 2)
+        value_states = value_states.view(bsz, 1, self.num_key_value_heads, self.head_dim).transpose(1, 2)
+
+        if position_emb is not None:
+            cos, sin = position_emb
+            query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)
+
+        key_cache, value_cache = kv_cache
+
+        key_cache[:, :, position_id, :] = key_states
+        value_cache[:, :, position_id, :] = value_states
+
+        # Use an explicit broadcastable mask shape for SDPA. A 1D mask can
+        # trigger a CPU-side dimension bug in some PyTorch versions.
+        attn_mask = (torch.arange(key_cache.size(2), device=key_cache.device) <= position_id).view(1, 1, 1, -1)
+
+        # ref: https://github.com/pytorch/pytorch/issues/163597
+        # there is a bug in MPS for non-contiguous tensors, so we need to make them contiguous
+        query_states = query_states.contiguous()
+        key_cache = key_cache.contiguous()
+        value_cache = value_cache.contiguous()
+        attn_output = torch.nn.functional.scaled_dot_product_attention(
+            query_states,
+            key_cache,
+            value_cache,
+            attn_mask=attn_mask,
+            enable_gqa=True,
+        )
+
+        attn_output = attn_output.transpose(1, 2).contiguous()
+        attn_output = attn_output.reshape(bsz, self.num_heads * self.head_dim)
+        attn_output = self.o_proj(attn_output)
+
+        return attn_output
+
+
+class MiniCPMMLP(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.config = config
+        self.hidden_size = config.hidden_size
+        self.intermediate_size = config.intermediate_size
+        self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
+        self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
+        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=False)
+        self.act_fn = nn.SiLU()
+
+    def forward(self, x):
+        return self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
+
+
+class MiniCPMDecoderLayer(nn.Module):
+    def __init__(self, config: MiniCPM4Config, layer_idx: int):
+        super().__init__()
+        self.hidden_size = config.hidden_size
+        self.self_attn = MiniCPMAttention(config=config, layer_idx=layer_idx)
+
+        self.mlp = MiniCPMMLP(config)
+        self.input_layernorm = MiniCPMRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.post_attention_layernorm = MiniCPMRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+
+        self.scale_depth = config.scale_depth
+        self.num_hidden_layers = config.num_hidden_layers
+        self.use_mup = config.use_mup
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        position_emb: Tuple[torch.Tensor, torch.Tensor],
+        is_causal: bool,
+    ) -> Tuple[torch.Tensor, Tuple[torch.Tensor, torch.Tensor]]:
+        """
+        Args:
+            hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)`
+            position_ids (`torch.LongTensor`): position ids of shape `(batch_size, seq_len)`
+            is_causal (`bool`): whether the attention mask is causal
+        """
+        residual = hidden_states
+        hidden_states = self.input_layernorm(hidden_states)
+        # Self Attention
+        hidden_states, present_key_value = self.self_attn(
+            hidden_states=hidden_states,
+            position_emb=position_emb,
+            is_causal=is_causal,
+        )
+
+        if self.use_mup:
+            hidden_states = residual + hidden_states * (self.scale_depth / math.sqrt(self.num_hidden_layers))
+        else:
+            hidden_states = residual + hidden_states
+
+        # Fully Connected
+        residual = hidden_states
+        hidden_states = self.post_attention_layernorm(hidden_states)
+
+        hidden_states = self.mlp(hidden_states)
+        if self.use_mup:
+            hidden_states = residual + hidden_states * (self.scale_depth / math.sqrt(self.num_hidden_layers))
+        else:
+            hidden_states = residual + hidden_states
+
+        return hidden_states, present_key_value
+
+    def forward_step(
+        self,
+        hidden_states: torch.Tensor,
+        position_emb: Tuple[torch.Tensor, torch.Tensor],
+        position_id: torch.Tensor,
+        kv_cache: Tuple[torch.Tensor, torch.Tensor],
+    ) -> torch.Tensor:
+        residual = hidden_states
+        hidden_states = self.input_layernorm(hidden_states)
+        # Self Attention
+        hidden_states = self.self_attn.forward_step(
+            hidden_states=hidden_states,
+            position_emb=position_emb,
+            position_id=position_id,
+            kv_cache=kv_cache,
+        )
+
+        if self.use_mup:
+            hidden_states = residual + hidden_states * (self.scale_depth / math.sqrt(self.num_hidden_layers))
+        else:
+            hidden_states = residual + hidden_states
+
+        # Fully Connected
+        residual = hidden_states
+        hidden_states = self.post_attention_layernorm(hidden_states)
+
+        hidden_states = self.mlp(hidden_states)
+        if self.use_mup:
+            hidden_states = residual + hidden_states * (self.scale_depth / math.sqrt(self.num_hidden_layers))
+        else:
+            hidden_states = residual + hidden_states
+
+        return hidden_states
+
+
+class MiniCPMModel(nn.Module):
+    """
+    Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`MiniCPMDecoderLayer`]
+
+    Args:
+        config: MiniCPMConfig
+    """
+
+    def __init__(self, config: MiniCPM4Config):
+        super().__init__()
+        self.vocab_size = config.vocab_size
+        self.config = config
+
+        if config.vocab_size > 0:
+            self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size)
+        else:
+            self.embed_tokens = nn.Identity()
+
+        self.layers = nn.ModuleList(
+            [MiniCPMDecoderLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)]
+        )
+
+        self.norm = MiniCPMRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        if config.no_rope:
+            self.rope_emb = None
+        else:
+            self.rope_emb = MiniCPMLongRoPE(config)
+
+        self.kv_cache = None
+
+    def forward(
+        self,
+        inputs_embeds: torch.Tensor,
+        is_causal: bool = True,
+    ) -> Tuple[torch.Tensor, List[Tuple[torch.Tensor, torch.Tensor]]]:
+        """
+        Args:
+            inputs_embeds: Tensor(batch_size, seq_length, hidden_size)
+            is_causal: bool, whether the attention mask is causal
+        Returns:
+            hidden_states: Tensor(batch_size, seq_length, hidden_size)
+            next_decoder_cache: List[(batch_size, num_heads, seq_length, head_dim), (batch_size, num_heads, seq_length, head_dim)]
+        """
+        if self.rope_emb is not None:
+            position_ids = torch.arange(0, inputs_embeds.size(1), dtype=torch.long, device=inputs_embeds.device)
+            position_emb = self.rope_emb(position_ids)
+        else:
+            position_emb = None
+        hidden_states = inputs_embeds
+
+        next_decoder_cache = []
+
+        for decoder_layer in self.layers:
+
+            hidden_states, this_cache = decoder_layer(
+                hidden_states,
+                position_emb,
+                is_causal,
+            )
+            next_decoder_cache.append(this_cache)
+        hidden_states = self.norm(hidden_states)
+        return hidden_states, next_decoder_cache
+
+    def forward_step(
+        self,
+        inputs_embeds: torch.Tensor,
+        position_id: torch.Tensor,
+    ) -> torch.Tensor:
+        """
+        Args:
+            inputs_embeds: Tensor(batch_size, hidden_size)
+        Returns:
+            hidden_states: Tensor(batch_size, hidden_size)
+        """
+        assert self.kv_cache is not None, "KV cache is not setup"
+
+        if self.rope_emb is not None:
+            position_emb = self.rope_emb(position_id)
+        else:
+            position_emb = None
+        hidden_states = inputs_embeds
+
+        for i, decoder_layer in enumerate(self.layers):
+            hidden_states = decoder_layer.forward_step(
+                hidden_states,
+                position_emb,
+                position_id,
+                self.kv_cache.get_layer_cache(i),
+            )
+
+        hidden_states = self.norm(hidden_states)
+        return hidden_states
+
+    def setup_cache(self, batch_size: int, max_length: int, device, dtype: torch.dtype):
+        self.kv_cache = StaticKVCache(
+            num_layers=self.config.num_hidden_layers,
+            num_kv_heads=self.config.num_key_value_heads,
+            dim_kv_head=(
+                self.config.hidden_size // self.config.num_attention_heads
+                if self.config.kv_channels is None
+                else self.config.kv_channels
+            ),
+            batch_size=batch_size,
+            device=device,
+            dtype=dtype,
+            max_length=max_length,
+        )

bluemagpie/adapter.py

@@ -0,0 +1,68 @@
+"""Projection adapter between Barbet hidden space and VoxCPM2 LM hidden space.
+
+Barbet's hidden states are not drop-in compatible with the space VoxCPM2's
+pretrained downstream modules (FSQ layer, RALM fusion, DiT projections, stop
+head) were trained in. The adapter is the explicit bridge:
+
+    barbet_hidden (H_b) --RMSNorm--Linear--> H_v --[zero-init residual MLP]*N--> H_v
+
+The residual MLP blocks have their output projections zero-initialized, so at
+init the adapter is exactly RMSNorm + Linear. Training can then grow extra
+capacity without perturbing the warm-start behaviour.
+"""
+
+from __future__ import annotations
+
+import torch
+import torch.nn.functional as F
+from torch import nn
+
+from .config import AdapterConfig
+
+
+class RMSNorm(nn.Module):
+    def __init__(self, hidden_size: int, eps: float = 1.0e-6) -> None:
+        super().__init__()
+        self.weight = nn.Parameter(torch.ones(hidden_size))
+        self.eps = eps
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        variance = x.float().pow(2).mean(dim=-1, keepdim=True)
+        x = x.float() * torch.rsqrt(variance + self.eps)
+        return x.to(dtype=self.weight.dtype) * self.weight
+
+
+class ResidualSwiGLUBlock(nn.Module):
+    def __init__(self, dim: int, ffn_mult: float, eps: float) -> None:
+        super().__init__()
+        inner = int(dim * ffn_mult)
+        self.norm = RMSNorm(dim, eps)
+        self.gate_proj = nn.Linear(dim, inner, bias=False)
+        self.up_proj = nn.Linear(dim, inner, bias=False)
+        self.down_proj = nn.Linear(inner, dim, bias=False)
+        nn.init.zeros_(self.down_proj.weight)
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        h = self.norm(x)
+        return x + self.down_proj(F.silu(self.gate_proj(h)) * self.up_proj(h))
+
+
+class ProjectionAdapter(nn.Module):
+    """Maps Barbet hidden states into the VoxCPM2 semantic LM space."""
+
+    def __init__(self, in_dim: int, out_dim: int, config: AdapterConfig) -> None:
+        super().__init__()
+        self.norm = RMSNorm(in_dim, config.rms_norm_eps)
+        self.proj = nn.Linear(in_dim, out_dim)
+        self.blocks = nn.ModuleList(
+            [
+                ResidualSwiGLUBlock(out_dim, config.ffn_mult, config.rms_norm_eps)
+                for _ in range(config.num_residual_blocks)
+            ]
+        )
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        x = self.proj(self.norm(x))
+        for block in self.blocks:
+            x = block(x)
+        return x

bluemagpie/centroid.py

@@ -0,0 +1,67 @@
+"""Extract a speaker centroid from reference audio, for voice cloning.
+
+Computes an ECAPA-TDNN centroid in the same space as the model's training
+centroids — the same mechanism the bundled ``hung_yi_lee`` example uses — ready
+to pass as ``model.generate(..., speaker_centroid=centroid)``.
+
+Requires ``speechbrain`` (an optional extra): ``pip install -e ".[clone]"``.
+"""
+
+from __future__ import annotations
+
+from typing import Sequence, Union
+
+import numpy as np
+import torch
+
+DEFAULT_ECAPA = "speechbrain/spkrec-ecapa-voxceleb"
+_SR = 16000
+
+
+def _load_encoder(model_id: str, device: str):
+    try:
+        from speechbrain.inference.speaker import EncoderClassifier
+    except Exception as e:  # pragma: no cover - heavy optional dep
+        raise ImportError(
+            f"speechbrain is required to extract speaker centroids ({e}). "
+            'Install with: pip install -e ".[clone]"  (or: pip install speechbrain)'
+        ) from e
+    return EncoderClassifier.from_hparams(source=model_id, run_opts={"device": device})
+
+
+@torch.no_grad()
+def extract_speaker_centroid(
+    audio: Union[str, Sequence[str]],
+    *,
+    ecapa_model: str = DEFAULT_ECAPA,
+    device: str = "cpu",
+    window_s: float = 6.0,
+    encoder=None,
+) -> torch.Tensor:
+    """One or more reference clips (same speaker) -> a ``[192]`` L2-normalized centroid.
+
+    Matches ``scripts/build_speaker_centroids.py``: each ~``window_s`` chunk is
+    ECAPA-embedded and L2-normalized, all chunks are averaged, and the mean is
+    L2-normalized — so the result lives in the model's training-centroid space.
+    Pass a single path or a list of paths. Provide a loaded ``encoder`` to reuse it.
+    """
+    import librosa
+
+    paths = [audio] if isinstance(audio, str) else list(audio)
+    clf = encoder if encoder is not None else _load_encoder(ecapa_model, device)
+    win = int(window_s * _SR)
+    embs = []
+    for path in paths:
+        wav, _ = librosa.load(path, sr=_SR, mono=True)
+        x = torch.from_numpy(np.ascontiguousarray(wav)).float()
+        chunks = [x[i : i + win] for i in range(0, x.numel(), win)]
+        chunks = [c for c in chunks if c.numel() >= _SR]
+        if not chunks and x.numel() > 0:
+            chunks = [x]  # shorter than 1 s: use as-is
+        for c in chunks:
+            e = clf.encode_batch(c.unsqueeze(0).to(device)).reshape(-1)  # [192]
+            embs.append(torch.nn.functional.normalize(e, dim=0).cpu())
+    if not embs:
+        raise ValueError("no usable audio; provide ~3-10 s of clean single-speaker speech")
+    mean = torch.stack(embs).mean(0)
+    return torch.nn.functional.normalize(mean, dim=0)

bluemagpie/conditioning.py

@@ -0,0 +1,37 @@
+"""Speaker conditioning for BlueMagpie-TTS (v1).
+
+v1 conditions on a *speaker centroid* — a per-speaker ECAPA-TDNN embedding
+averaged over several clips, computed offline. The centroid carries the constant part
+of a voice (identity, timbre, baseline pitch); using a denoised per-speaker
+average rather than a per-utterance embedding prevents the model from leaking
+that clip's prosody/content as a shortcut.
+
+``SpeakerProjector`` maps the centroid into the Barbet hidden space; the
+resulting vector is placed at the ``[spk]`` token slot in the input sequence
+(see ``BlueMagpieModel``), so Barbet's causal attention propagates the speaker
+identity into every downstream position.
+"""
+
+from __future__ import annotations
+
+import torch
+from torch import nn
+
+from .adapter import RMSNorm
+
+
+class SpeakerProjector(nn.Module):
+    """Project an L2-normalized speaker centroid into the Barbet hidden space."""
+
+    def __init__(self, in_dim: int, out_dim: int, eps: float = 1.0e-6) -> None:
+        super().__init__()
+        self.norm = RMSNorm(in_dim, eps)
+        self.proj = nn.Linear(in_dim, out_dim)
+        # Small init so a cold-started speaker vector does not destabilize the
+        # warm-started (pretrained) Barbet backbone early in training.
+        nn.init.normal_(self.proj.weight, mean=0.0, std=0.02)
+        nn.init.zeros_(self.proj.bias)
+
+    def forward(self, centroid: torch.Tensor) -> torch.Tensor:
+        """``centroid``: [B, in_dim] -> [B, out_dim]."""
+        return self.proj(self.norm(centroid))

bluemagpie/config.py

@@ -0,0 +1,135 @@
+"""Configuration for BlueMagpie-TTS.
+
+BlueMagpie = Barbet (TSLM) + VoxCPM2 acoustic stack (LocEnc / RALM / LocDiT / AudioVAE).
+
+The config keeps two "hidden spaces" explicit:
+
+- ``barbet_config``         -> Barbet's hidden space  (H_b, e.g. 1024 for 300M, 1536 for 1B)
+- ``vox_lm_config``         -> VoxCPM2's semantic LM hidden space (H_v, 2048 for openbmb/VoxCPM2)
+
+``vox_lm_config`` is the MiniCPM4 config of the *original* VoxCPM2 TSLM. The full
+28-layer TSLM is never instantiated here — the config object is kept because it is
+the template from which VoxCPM2 derives its RALM / LocEnc / LocDiT submodule
+configs, and because ``hidden_size`` defines the space all pretrained projection
+layers (fsq, fusion_concat_proj, lm_to_dit_proj, ...) live in.
+"""
+
+from __future__ import annotations
+
+from typing import Optional
+
+from pydantic import BaseModel
+from bluemagpie._vendor.voxcpm.model.voxcpm2 import VoxCPMDitConfig, VoxCPMEncoderConfig
+from bluemagpie._vendor.voxcpm.modules.audiovae import AudioVAEConfigV2
+from bluemagpie._vendor.voxcpm.modules.minicpm4 import MiniCPM4Config
+
+
+class AdapterConfig(BaseModel):
+    """Projection adapter bridging Barbet hidden space into VoxCPM2 LM space."""
+
+    num_residual_blocks: int = 1
+    ffn_mult: float = 2.0
+    rms_norm_eps: float = 1.0e-6
+
+
+class BlueMagpieConfig(BaseModel):
+    # Raw kwargs for barbet.BarbetConfig (kept as dict so the pydantic model
+    # stays serializable independently of transformers).
+    barbet_config: dict
+
+    # Semantic-space template (the original VoxCPM2 TSLM config).
+    vox_lm_config: MiniCPM4Config
+
+    patch_size: int = 4
+    feat_dim: int = 64
+    residual_lm_num_layers: int = 8
+    residual_lm_no_rope: bool = False
+    scalar_quantization_latent_dim: int = 512
+    scalar_quantization_scale: int = 9
+
+    encoder_config: VoxCPMEncoderConfig
+    dit_config: VoxCPMDitConfig
+    audio_vae_config: Optional[AudioVAEConfigV2] = None
+
+    adapter_config: AdapterConfig = AdapterConfig()
+
+    # Dimensionality of the speaker centroid (ECAPA-TDNN default = 192). The
+    # SpeakerProjector maps this into the Barbet hidden space at the [spk] slot.
+    speaker_embed_dim: int = 192
+
+    # Special token ids in *Barbet* vocab space. -1 means "allocate the first
+    # free id after the effective vocab" (resolved by resolve_barbet_config()).
+    audio_start_token: int = -1
+    audio_end_token: int = -1
+    ref_audio_start_token: int = -1
+    ref_audio_end_token: int = -1
+    # Placeholder slot whose embedding row is overwritten by the projected
+    # speaker centroid (and serves as the learned "null speaker" when dropped).
+    spk_token: int = -1
+
+    # Number of ids actually used by the tokenizer. Barbet R2 checkpoints pad
+    # the embedding to a multiple of 128 for Megatron (vocab_size 114944 vs
+    # effective 114822 for PangolinTokenizer), so auto-allocated special
+    # tokens can live in the padding region without growing the embedding.
+    # None -> auto-detect from the Pangolin contract, else fall back to
+    # vocab_size (append-and-grow).
+    barbet_effective_vocab_size: Optional[int] = None
+
+    max_length: int = 8192
+    device: str = "cuda"
+    dtype: str = "bfloat16"
+
+    def _effective_vocab_size(self, barbet_cfg) -> int:
+        """First id available for auto-allocated special tokens."""
+        if self.barbet_effective_vocab_size is not None:
+            return self.barbet_effective_vocab_size
+        try:
+            from barbet.configuration_barbet import EFFECTIVE_VOCAB_SIZE, MEGATRON_PADDED_VOCAB_SIZE
+
+            # Pangolin contract: ids [EFFECTIVE, PADDED) are Megatron padding
+            # rows the tokenizer never produces — free for our special tokens.
+            if barbet_cfg.vocab_size == MEGATRON_PADDED_VOCAB_SIZE:
+                return EFFECTIVE_VOCAB_SIZE
+        except ImportError:  # pre-R2 barbet without the contract constants
+            pass
+        return barbet_cfg.vocab_size
+
+    def resolve_barbet_config(self):
+        """Build the BarbetConfig and resolve special-token ids.
+
+        Returns:
+            (barbet_config, token_ids) where token_ids is a dict with keys
+            audio_start/audio_end/ref_audio_start/ref_audio_end/spk. Ids set to
+            -1 are allocated from the first free id after the effective vocab —
+            inside the Megatron padding region for R2 checkpoints (no
+            embedding growth), or appended after the vocab otherwise. The
+            vocab only grows when an id falls beyond the current size.
+        """
+        from barbet import BarbetConfig
+
+        kwargs = dict(self.barbet_config)
+        barbet_cfg = BarbetConfig(**kwargs)
+
+        names = ["audio_start", "audio_end", "ref_audio_start", "ref_audio_end", "spk"]
+        requested = [
+            self.audio_start_token,
+            self.audio_end_token,
+            self.ref_audio_start_token,
+            self.ref_audio_end_token,
+            self.spk_token,
+        ]
+        token_ids = {}
+        next_id = self._effective_vocab_size(barbet_cfg)
+        for name, tok in zip(names, requested):
+            if tok is None or tok < 0:
+                token_ids[name] = next_id
+                next_id += 1
+            else:
+                token_ids[name] = tok
+        needed = max(token_ids.values()) + 1
+        if needed > barbet_cfg.vocab_size:
+            barbet_cfg.vocab_size = needed
+        return barbet_cfg, token_ids
+
+
+BlueMagpieConfig.model_rebuild()

bluemagpie/loading.py

@@ -0,0 +1,284 @@
+"""Checkpoint assembly: pretrained VoxCPM2 acoustic stack + Barbet TSLM.
+
+The hybrid is built by weight surgery:
+
+- from the VoxCPM2 checkpoint we keep every module *except* the TSLM
+  (``base_lm.*`` keys are dropped): LocEnc, RALM, LocDiT, FSQ, the projection
+  layers, the stop head and the AudioVAE.
+- Barbet weights are loaded into ``base_lm.backbone`` (with the 4 appended
+  special-token embedding rows freshly initialized).
+- the two bridge modules (``enc_to_tslm_proj``, ``tslm_adapter``) are always
+  freshly initialized — they are what training stage 0/1 is for.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import sys
+from typing import Optional
+
+import torch
+
+from bluemagpie._vendor.voxcpm.model.voxcpm2 import VoxCPMConfig
+from bluemagpie._vendor.voxcpm.modules.audiovae import AudioVAEV2
+
+from .config import AdapterConfig, BlueMagpieConfig
+from .model import BlueMagpieModel
+
+try:
+    from safetensors.torch import load_file
+
+    SAFETENSORS_AVAILABLE = True
+except ImportError:  # pragma: no cover
+    SAFETENSORS_AVAILABLE = False
+
+# VoxCPM2 modules reused verbatim (state-dict key prefixes).
+VOXCPM2_REUSED_PREFIXES = (
+    "feat_encoder.",
+    "residual_lm.",
+    "feat_decoder.",
+    "fsq_layer.",
+    "enc_to_lm_proj.",
+    "lm_to_dit_proj.",
+    "res_to_dit_proj.",
+    "fusion_concat_proj.",
+    "stop_proj.",
+    "stop_head.",
+)
+
+
+def _load_state_dict(path_dir: str, stem: str) -> dict:
+    st_path = os.path.join(path_dir, f"{stem}.safetensors")
+    pt_candidates = [os.path.join(path_dir, f"{stem}.pth"), os.path.join(path_dir, f"{stem}.bin")]
+    if stem == "model":
+        pt_candidates.insert(0, os.path.join(path_dir, "pytorch_model.bin"))
+    if os.path.exists(st_path) and SAFETENSORS_AVAILABLE:
+        return load_file(st_path, device="cpu")
+    for p in pt_candidates:
+        if os.path.exists(p):
+            ckpt = torch.load(p, map_location="cpu", weights_only=True)
+            return ckpt.get("state_dict", ckpt)
+    raise FileNotFoundError(f"No checkpoint found for '{stem}' under {path_dir}")
+
+
+def build_config_from_voxcpm2(
+    voxcpm2_path: str,
+    barbet_config_kwargs: dict,
+    adapter_config: Optional[AdapterConfig] = None,
+    **overrides,
+) -> BlueMagpieConfig:
+    """Derive a BlueMagpieConfig from a local VoxCPM2 checkpoint directory."""
+    with open(os.path.join(voxcpm2_path, "config.json"), "r", encoding="utf-8") as f:
+        vox_cfg = VoxCPMConfig.model_validate_json(f.read())
+
+    cfg = BlueMagpieConfig(
+        barbet_config=barbet_config_kwargs,
+        vox_lm_config=vox_cfg.lm_config,
+        patch_size=vox_cfg.patch_size,
+        feat_dim=vox_cfg.feat_dim,
+        residual_lm_num_layers=vox_cfg.residual_lm_num_layers,
+        residual_lm_no_rope=vox_cfg.residual_lm_no_rope,
+        scalar_quantization_latent_dim=vox_cfg.scalar_quantization_latent_dim,
+        scalar_quantization_scale=vox_cfg.scalar_quantization_scale,
+        encoder_config=vox_cfg.encoder_config,
+        dit_config=vox_cfg.dit_config,
+        audio_vae_config=vox_cfg.audio_vae_config,
+        adapter_config=adapter_config or AdapterConfig(),
+        max_length=vox_cfg.max_length,
+        device=vox_cfg.device,
+        dtype=vox_cfg.dtype,
+        **overrides,
+    )
+    return cfg
+
+
+def load_barbet_config_kwargs(barbet_path: str) -> dict:
+    """Read a Barbet HF checkpoint's config.json into plain kwargs."""
+    with open(os.path.join(barbet_path, "config.json"), "r", encoding="utf-8") as f:
+        kwargs = json.load(f)
+    for k in ("architectures", "auto_map", "model_type", "transformers_version", "torch_dtype"):
+        kwargs.pop(k, None)
+    return kwargs
+
+
+def build_from_pretrained(
+    voxcpm2_path: str,
+    barbet_path: Optional[str] = None,
+    barbet_config_kwargs: Optional[dict] = None,
+    tokenizer=None,
+    adapter_config: Optional[AdapterConfig] = None,
+    device: str | None = None,
+    training: bool = True,
+    **config_overrides,
+) -> BlueMagpieModel:
+    """Assemble a BlueMagpieModel from pretrained VoxCPM2 + Barbet checkpoints.
+
+    Args:
+        voxcpm2_path: local directory of the VoxCPM2 checkpoint
+            (e.g. ``huggingface_hub.snapshot_download("openbmb/VoxCPM2")``).
+        barbet_path: local directory of a Barbet checkpoint. If None, the TSLM
+            is randomly initialized from ``barbet_config_kwargs``.
+        barbet_config_kwargs: Barbet config kwargs; defaults to the checkpoint's
+            config.json (if ``barbet_path`` given) or BarbetConfig defaults.
+        tokenizer: HF tokenizer matching Barbet's vocab. Defaults to
+            ``AutoTokenizer.from_pretrained(barbet_path)`` when available.
+        adapter_config: projection adapter hyperparameters.
+        training: if True, returns a float32 trainable model (VAE frozen);
+            if False, casts to the configured inference dtype and eval().
+    """
+    if barbet_config_kwargs is None:
+        if barbet_path is not None:
+            barbet_config_kwargs = load_barbet_config_kwargs(barbet_path)
+        else:
+            barbet_config_kwargs = {}
+
+    config = build_config_from_voxcpm2(
+        voxcpm2_path, barbet_config_kwargs, adapter_config=adapter_config, **config_overrides
+    )
+
+    if tokenizer is None and barbet_path is not None:
+        try:
+            from transformers import AutoTokenizer
+
+            tokenizer = AutoTokenizer.from_pretrained(barbet_path)
+        except Exception:
+            print("[bluemagpie] no tokenizer found at barbet_path; attach one manually", file=sys.stderr)
+
+    # Audio VAE from the VoxCPM2 checkpoint
+    audio_vae = AudioVAEV2(config=config.audio_vae_config) if config.audio_vae_config else AudioVAEV2()
+    vae_state = _load_state_dict(voxcpm2_path, "audiovae")
+    audio_vae.load_state_dict(vae_state)
+
+    model = BlueMagpieModel(config, tokenizer, audio_vae, device=device)
+
+    # ---- VoxCPM2 weight surgery (drop base_lm.*) ---- #
+    vox_state = _load_state_dict(voxcpm2_path, "model")
+    kept = {k: v for k, v in vox_state.items() if k.startswith(VOXCPM2_REUSED_PREFIXES)}
+    dropped = len(vox_state) - len(kept)
+    missing, unexpected = model.load_state_dict(kept, strict=False)
+    print(
+        f"[bluemagpie] VoxCPM2 surgery: kept {len(kept)} tensors, dropped {dropped} (base_lm.* etc.), "
+        f"unexpected {len(unexpected)}",
+        file=sys.stderr,
+    )
+
+    # ---- Barbet TSLM weights ---- #
+    if barbet_path is not None:
+        load_barbet_weights(model, barbet_path)
+
+    model.audio_vae = model.audio_vae.to(torch.float32)
+    for name, param in model.named_parameters():
+        if name.startswith("audio_vae."):
+            param.requires_grad = False
+
+    if not training:
+        from bluemagpie._vendor.voxcpm.model.utils import get_dtype
+
+        non_vae = [m for n, m in model.named_children() if n != "audio_vae"]
+        for m in non_vae:
+            m.to(get_dtype(model.config.dtype))
+        model = model.to(model.device).eval()
+    else:
+        model = model.to(model.device)
+    return model
+
+
+def load_barbet_weights(model: BlueMagpieModel, barbet_path: str) -> None:
+    """Load Barbet checkpoint into model.base_lm.backbone.
+
+    Accepts both BarbetModel and BarbetForCausalLM state dicts (the ``model.``
+    prefix is stripped; ``lm_head``/``mtp`` keys are dropped — R2 checkpoints
+    tie lm_head to the embeddings anyway). The embedding rows of the resolved
+    TTS special tokens are always re-initialized from N(0, initializer_range):
+    for R2 checkpoints those ids sit in the Megatron padding region, whose
+    checkpoint rows are typically all-zero and would otherwise make
+    audio_start/end/ref markers indistinguishable.
+    """
+    state = _load_state_dict(barbet_path, "model")
+    cleaned = {}
+    for k, v in state.items():
+        if k.startswith("lm_head.") or k.startswith("mtp."):
+            continue
+        cleaned[k.removeprefix("model.")] = v
+
+    backbone = model.base_lm.backbone
+    target_embed = backbone.embed_tokens.weight
+    src_embed = cleaned.get("embed_tokens.weight")
+    if src_embed is not None and src_embed.shape[0] != target_embed.shape[0]:
+        n_src = src_embed.shape[0]
+        grown = target_embed.detach().clone()
+        grown[:n_src] = src_embed
+        cleaned["embed_tokens.weight"] = grown
+        print(
+            f"[bluemagpie] resized Barbet embeddings {n_src} -> {target_embed.shape[0]}",
+            file=sys.stderr,
+        )
+
+    missing, unexpected = backbone.load_state_dict(cleaned, strict=False)
+    print(
+        f"[bluemagpie] Barbet load: missing {len(missing)}, unexpected {len(unexpected)}",
+        file=sys.stderr,
+    )
+
+    special_ids = [
+        model.audio_start_token,
+        model.audio_end_token,
+        model.ref_audio_start_token,
+        model.ref_audio_end_token,
+        model.spk_token,
+    ]
+    std = model.barbet_config.initializer_range
+    with torch.no_grad():
+        fresh = torch.empty(len(special_ids), target_embed.shape[1], dtype=target_embed.dtype)
+        fresh.normal_(mean=0.0, std=std)
+        backbone.embed_tokens.weight[special_ids] = fresh.to(target_embed.device)
+    print(f"[bluemagpie] re-initialized special token rows {special_ids}", file=sys.stderr)
+
+
+# ---------------------------------------------------------------------- #
+# Stage-wise freezing
+# ---------------------------------------------------------------------- #
+STAGES = ("bridge", "tslm", "full")
+
+
+def set_training_stage(model: BlueMagpieModel, stage: str) -> None:
+    """Configure requires_grad for the staged training recipe.
+
+    - ``bridge``: only the new bridge modules train (enc_to_tslm_proj +
+      tslm_adapter). Use with hidden-space distillation against the original
+      VoxCPM2 TSLM, or with the diffusion loss directly.
+    - ``tslm``:   bridge + Barbet + speaker projector train; the VoxCPM2
+      acoustic stack stays frozen (RALM / LocDiT keep handling acoustic detail
+      unchanged). This is where v1 speaker conditioning is learned.
+    - ``full``:   everything trains except the AudioVAE.
+    """
+    if stage not in STAGES:
+        raise ValueError(f"stage must be one of {STAGES}, got {stage!r}")
+
+    bridge_prefixes = ("enc_to_tslm_proj.", "tslm_adapter.")
+    tslm_prefixes = bridge_prefixes + ("base_lm.", "speaker_projector.")
+
+    for name, param in model.named_parameters():
+        if name.startswith("audio_vae."):
+            param.requires_grad = False
+        elif stage == "bridge":
+            param.requires_grad = name.startswith(bridge_prefixes)
+        elif stage == "tslm":
+            param.requires_grad = name.startswith(tslm_prefixes)
+        else:  # full
+            param.requires_grad = True
+
+
+def load_voxcpm2_teacher(voxcpm2_path: str, device: str | None = None):
+    """Load the original VoxCPM2 model as a frozen distillation teacher.
+
+    Its pre-FSQ TSLM hidden states (``enc_outputs`` before the fsq/text mix)
+    are the target for aligning ``tslm_adapter`` outputs in stage ``bridge``.
+    """
+    from bluemagpie._vendor.voxcpm.model.voxcpm2 import VoxCPM2Model
+
+    teacher = VoxCPM2Model.from_local(voxcpm2_path, optimize=False, training=False, device=device)
+    for p in teacher.parameters():
+        p.requires_grad = False
+    return teacher

bluemagpie/model.py

@@ -0,0 +1,717 @@
+"""BlueMagpie-TTS hybrid model.
+
+Architecture = VoxCPM2 with its Text-Semantic LM (MiniCPM4) swapped for Barbet:
+
+    text tokens ──Barbet.embed_tokens──────────────┐
+                                                   ├─ interleave ─> Barbet backbone ─> H_b
+    audio latents ─LocEnc─┬─ enc_to_tslm_proj(H_b)─┘                    │
+                          └─ enc_to_lm_proj(H_v)──────────┐      tslm_adapter (H_b -> H_v)
+                                                          │             │
+                                                          │       FSQ (audio positions)
+                                                          │             │ enc_outputs (H_v)
+                                  fusion_concat_proj(cat) ┴─────────────┤
+                                            │                           │
+                                       RALM (MiniCPM4 8L)         lm_to_dit_proj
+                                            │ res_to_dit_proj           │
+                                            └───────── concat ──────────┘
+                                                          │ mu (2 prefix tokens)
+                                                     LocDiT (CFM)
+                                                          │ latent patch
+                                                      AudioVAE
+
+Division of labour (matching VoxCPM2's design intent):
+- Barbet (TSLM): what to say, prosody planning, pacing, emphasis, control text.
+- RALM + LocDiT: fine-grained acoustic detail, kept verbatim from VoxCPM2 so
+  pretrained weights load unchanged.
+- ``tslm_adapter`` is the explicit bridge between the two incompatible hidden
+  spaces; ``enc_to_tslm_proj`` is its input-side counterpart feeding LocEnc
+  features into Barbet.
+
+Everything outside the TSLM block mirrors
+``voxcpm/model/voxcpm2.py`` (Apache-2.0, Copyright 2026 OpenBMB) so that
+VoxCPM2 checkpoints remain loadable.
+"""
+
+from __future__ import annotations
+
+import os
+import sys
+import warnings
+from typing import Generator, List, Optional, Tuple, Union
+
+import librosa
+import torch
+import torch.nn as nn
+from einops import rearrange
+from tqdm import tqdm
+
+from bluemagpie._vendor.voxcpm.model.utils import get_dtype, next_and_close, pick_runtime_dtype, resolve_runtime_device
+from bluemagpie._vendor.voxcpm.modules.audiovae import AudioVAEV2
+from bluemagpie._vendor.voxcpm.modules.layers import ScalarQuantizationLayer
+from bluemagpie._vendor.voxcpm.modules.locdit import UnifiedCFM, VoxCPMLocDiTV2
+from bluemagpie._vendor.voxcpm.modules.locenc import VoxCPMLocEnc
+from bluemagpie._vendor.voxcpm.modules.minicpm4 import MiniCPMModel
+
+from .adapter import ProjectionAdapter
+from .conditioning import SpeakerProjector
+from .config import BlueMagpieConfig
+from .tslm import BarbetTSLM
+
+
+class BlueMagpieModel(nn.Module):
+    def __init__(
+        self,
+        config: BlueMagpieConfig,
+        tokenizer=None,
+        audio_vae: AudioVAEV2 = None,
+        device: str | None = None,
+    ):
+        super().__init__()
+        self.config = config
+        self.feat_dim = config.feat_dim
+        self.patch_size = config.patch_size
+        self.device = resolve_runtime_device(device, config.device)
+        self.config.device = self.device
+        resolved_dtype = pick_runtime_dtype(self.device, self.config.dtype)
+        if resolved_dtype != self.config.dtype:
+            print(
+                f"[bluemagpie] adjusted dtype {self.config.dtype} -> {resolved_dtype} for device {self.device}",
+                file=sys.stderr,
+            )
+            self.config.dtype = resolved_dtype
+
+        vox_lm = config.vox_lm_config
+        h_vox = vox_lm.hidden_size
+
+        # ---------------- Text-Semantic LM: Barbet ---------------- #
+        barbet_cfg, token_ids = config.resolve_barbet_config()
+        self.barbet_config = barbet_cfg
+        self.base_lm = BarbetTSLM(barbet_cfg)
+        h_barbet = barbet_cfg.hidden_size
+
+        self.text_tokenizer = tokenizer
+        self.audio_start_token = token_ids["audio_start"]
+        self.audio_end_token = token_ids["audio_end"]
+        self.ref_audio_start_token = token_ids["ref_audio_start"]
+        self.ref_audio_end_token = token_ids["ref_audio_end"]
+        self.spk_token = token_ids["spk"]
+
+        # ---------------- Residual Acoustic LM (VoxCPM2) ---------------- #
+        residual_lm_config = vox_lm.model_copy(deep=True)
+        residual_lm_config.num_hidden_layers = config.residual_lm_num_layers
+        residual_lm_config.vocab_size = 0
+        residual_lm_config.no_rope = config.residual_lm_no_rope
+        self.residual_lm = MiniCPMModel(residual_lm_config)
+        self.residual_lm.setup_cache(1, config.max_length, self.device, get_dtype(self.config.dtype))
+
+        # ---------------- Local Encoder (VoxCPM2) ---------------- #
+        encoder_config = vox_lm.model_copy(deep=True)
+        encoder_config.hidden_size = config.encoder_config.hidden_dim
+        encoder_config.intermediate_size = config.encoder_config.ffn_dim
+        encoder_config.num_attention_heads = config.encoder_config.num_heads
+        encoder_config.num_hidden_layers = config.encoder_config.num_layers
+        encoder_config.kv_channels = config.encoder_config.kv_channels
+        encoder_config.vocab_size = 0
+        self.feat_encoder = VoxCPMLocEnc(encoder_config, input_dim=config.feat_dim)
+
+        # ---------------- Local DiT (VoxCPM2) ---------------- #
+        decoder_config = vox_lm.model_copy(deep=True)
+        decoder_config.hidden_size = config.dit_config.hidden_dim
+        decoder_config.intermediate_size = config.dit_config.ffn_dim
+        decoder_config.num_attention_heads = config.dit_config.num_heads
+        decoder_config.num_hidden_layers = config.dit_config.num_layers
+        decoder_config.kv_channels = config.dit_config.kv_channels
+        decoder_config.vocab_size = 0
+        self.feat_decoder = UnifiedCFM(
+            in_channels=config.feat_dim,
+            cfm_params=config.dit_config.cfm_config,
+            estimator=VoxCPMLocDiTV2(decoder_config, in_channels=config.feat_dim),
+            mean_mode=config.dit_config.dit_mean_mode,
+        )
+
+        # ---------------- VoxCPM2 projections (semantic space H_v) ---------------- #
+        self.fsq_layer = ScalarQuantizationLayer(
+            h_vox, h_vox, config.scalar_quantization_latent_dim, config.scalar_quantization_scale
+        )
+        self.enc_to_lm_proj = nn.Linear(config.encoder_config.hidden_dim, h_vox)
+        self.lm_to_dit_proj = nn.Linear(h_vox, config.dit_config.hidden_dim)
+        self.res_to_dit_proj = nn.Linear(h_vox, config.dit_config.hidden_dim)
+        self.fusion_concat_proj = nn.Linear(h_vox * 2, h_vox)
+
+        # Stop Predictor (semantic space)
+        self.stop_proj = nn.Linear(h_vox, h_vox)
+        self.stop_actn = nn.SiLU()
+        self.stop_head = nn.Linear(h_vox, 2, bias=False)
+        self.stop_loss = nn.CrossEntropyLoss(reduction="none")
+
+        # ---------------- Barbet <-> VoxCPM2 bridges (new, trained from scratch) -------- #
+        self.enc_to_tslm_proj = nn.Linear(config.encoder_config.hidden_dim, h_barbet)
+        self.tslm_adapter = ProjectionAdapter(h_barbet, h_vox, config.adapter_config)
+
+        # ---------------- Speaker conditioning (v1: centroid -> [spk] slot) ------------ #
+        self.speaker_embed_dim = config.speaker_embed_dim
+        self.speaker_projector = SpeakerProjector(config.speaker_embed_dim, h_barbet)
+
+        # ---------------- Audio VAE ---------------- #
+        self.audio_vae = audio_vae
+        if audio_vae is not None:
+            self.chunk_size = audio_vae.chunk_size
+            self._decode_chunk_size = getattr(audio_vae, "decode_chunk_size", audio_vae.chunk_size)
+            self._encode_sample_rate = audio_vae.sample_rate
+            self.sample_rate = getattr(audio_vae, "out_sample_rate", audio_vae.sample_rate)
+
+    def _dtype(self):
+        return get_dtype(self.config.dtype)
+
+    def _runtime_dtype(self) -> torch.dtype:
+        """Return the dtype the non-AudioVAE network is currently using."""
+        return next(self.parameters()).dtype
+
+    def _runtime_device(self) -> torch.device:
+        """Return the device the module is actually placed on.
+
+        Distributed wrappers may move parameters after construction. Keep the
+        cached device string in sync so Triton-backed kernels launch on the
+        same CUDA device as their inputs.
+        """
+        device = next(self.parameters()).device
+        if device.type == "cuda" and device.index is not None and torch.cuda.current_device() != device.index:
+            torch.cuda.set_device(device)
+        device_str = str(device)
+        if device_str != str(self.device):
+            self.device = device_str
+            self.config.device = device_str
+        return device
+
+    def _inject_speaker(self, combined_embed, speaker_centroids, spk_mask):
+        """Add the projected speaker centroid at the [spk] position.
+
+        ``combined_embed``: [B, T, H_b]. ``speaker_centroids``: [B, D] (the
+        per-speaker ECAPA centroid). ``spk_mask``: [B, T], 1 only at the [spk]
+        slot of samples whose speaker is present (not dropped). Where
+        ``spk_mask`` is 0 the slot keeps its [spk] token embedding, which acts
+        as the learned null speaker.
+        """
+        if speaker_centroids is None or spk_mask is None:
+            return combined_embed
+        dtype = combined_embed.dtype
+        spk_vec = self.speaker_projector(speaker_centroids.to(combined_embed.device, dtype=dtype))  # [B, H_b]
+        spk_mask = spk_mask.to(combined_embed.device, dtype=dtype)
+        return combined_embed + spk_mask.unsqueeze(-1) * spk_vec.unsqueeze(1)
+
+    def _tokenize(self, text: str) -> List[int]:
+        if self.text_tokenizer is None:
+            raise ValueError("No tokenizer attached to BlueMagpieModel")
+        if hasattr(self.text_tokenizer, "encode"):
+            return self.text_tokenizer.encode(text, add_special_tokens=False)
+        return self.text_tokenizer(text)
+
+    # ------------------------------------------------------------------ #
+    # Training forward
+    # ------------------------------------------------------------------ #
+    def forward(
+        self,
+        text_tokens: torch.Tensor,
+        text_mask: torch.Tensor,
+        audio_feats: torch.Tensor,
+        audio_mask: torch.Tensor,
+        loss_mask: torch.Tensor,
+        position_ids: torch.Tensor,
+        labels: torch.Tensor,
+        *,
+        speaker_centroids: Optional[torch.Tensor] = None,
+        spk_mask: Optional[torch.Tensor] = None,
+        progress: float = 0.0,
+        sample_generate: bool = False,
+        sample_generate_timesteps: int = 10,
+    ):
+        del position_ids  # not used (parity with VoxCPM2)
+
+        device = self._runtime_device()
+        text_tokens = text_tokens.to(device, dtype=torch.long)
+        text_mask = text_mask.to(device, dtype=self._dtype())
+        audio_feats = audio_feats.to(device, dtype=self._dtype())
+        audio_mask = audio_mask.to(device, dtype=self._dtype())
+        loss_mask = loss_mask.to(device, dtype=self._dtype())
+        labels = labels.to(device, dtype=torch.long)
+
+        B, T, P, D = audio_feats.shape
+        feat_locenc = self.feat_encoder(audio_feats)
+        feat_embed_tslm = self.enc_to_tslm_proj(feat_locenc)  # Barbet input space
+        feat_embed_lm = self.enc_to_lm_proj(feat_locenc)  # VoxCPM2 semantic space (RALM fusion)
+
+        text_embed = self.base_lm.embed_tokens(text_tokens)
+        combined_embed = text_mask.unsqueeze(-1) * text_embed + audio_mask.unsqueeze(-1) * feat_embed_tslm
+        combined_embed = self._inject_speaker(combined_embed, speaker_centroids, spk_mask)
+
+        barbet_hidden = self.base_lm(inputs_embeds=combined_embed)
+        tslm_hidden = self.tslm_adapter(barbet_hidden).to(self._dtype())
+
+        enc_outputs = self.fsq_layer(tslm_hidden) * audio_mask.unsqueeze(-1) + tslm_hidden * text_mask.unsqueeze(-1)
+        lm_hidden = torch.cat((torch.zeros_like(enc_outputs[:, 0:1, :]), enc_outputs[:, :-1, :]), dim=1)
+
+        residual_inputs = self.fusion_concat_proj(
+            torch.cat((enc_outputs, audio_mask.unsqueeze(-1) * feat_embed_lm), dim=-1)
+        )
+        residual_outputs, _ = self.residual_lm(inputs_embeds=residual_inputs, is_causal=True)
+        residual_outputs = residual_outputs.to(self._dtype())
+        residual_hidden = torch.cat(
+            (torch.zeros_like(residual_outputs[:, 0:1, :]), residual_outputs[:, :-1, :]),
+            dim=1,
+        )
+
+        dit_hidden = torch.cat((self.lm_to_dit_proj(lm_hidden), self.res_to_dit_proj(residual_hidden)), dim=-1)
+        dit_hidden = rearrange(dit_hidden, "b t c -> (b t) c")
+
+        target_dtype = self._dtype()
+        feat_gt = rearrange(audio_feats.to(target_dtype), "b t p d -> (b t) p d")
+        feat_cond = torch.cat(
+            (torch.zeros_like(audio_feats[:, 0:1, ...]), audio_feats[:, :-1, ...]),
+            dim=1,
+        )
+        feat_cond = rearrange(feat_cond.to(target_dtype), "b t p d -> (b t) p d")
+
+        loss_seq_mask = loss_mask.unsqueeze(-1).repeat(1, 1, self.patch_size)
+        loss_seq_mask = rearrange(loss_seq_mask, "b t p -> (b t) p 1").to(target_dtype)
+
+        diff_loss = self.feat_decoder.compute_loss(
+            feat_gt.transpose(1, 2).contiguous(),
+            dit_hidden,
+            cond=feat_cond.transpose(1, 2).contiguous(),
+            tgt_mask=loss_seq_mask.transpose(1, 2).contiguous(),
+            progress=progress,
+        )
+
+        stop_logits = self.stop_head(self.stop_actn(self.stop_proj(lm_hidden)))
+        stop_losses = self.stop_loss(stop_logits.transpose(1, 2), labels)
+        denom = torch.clamp(loss_mask.sum(), min=1.0)
+        stop_loss = (stop_losses * loss_mask).sum() / denom
+
+        feat_pred = None
+        if sample_generate:
+            feat_cond_for_sample = feat_cond.transpose(1, 2).contiguous()
+            feat_pred_seq = self.feat_decoder(
+                mu=dit_hidden,
+                patch_size=self.patch_size,
+                cond=feat_cond_for_sample,
+                n_timesteps=max(int(sample_generate_timesteps), 1),
+            )
+            feat_pred = rearrange(feat_pred_seq.transpose(1, 2), "(b t) p d -> b d (t p)", b=B, p=self.patch_size)
+
+        feat_gt_tensor = rearrange(feat_gt, "(b t) p d -> b d (t p)", b=B, p=self.patch_size)
+
+        return {
+            "loss/diff": diff_loss,
+            "loss/stop": stop_loss,
+            "feat_gt": feat_gt_tensor,
+            "feat_pred": feat_pred,
+            "stop_logits": stop_logits,
+            # Pre-FSQ adapted hidden states, for hidden-space distillation
+            # against the original VoxCPM2 TSLM (see README, stage 0).
+            "tslm_hidden": tslm_hidden,
+        }
+
+    # ------------------------------------------------------------------ #
+    # Input assembly (the four VoxCPM2 prompting modes)
+    # ------------------------------------------------------------------ #
+    def _encode_wav(
+        self,
+        wav_path: str,
+        padding_mode: str = "right",
+    ) -> torch.Tensor:
+        audio, _ = librosa.load(wav_path, sr=self._encode_sample_rate, mono=True)
+        audio = torch.from_numpy(audio).unsqueeze(0)
+        patch_len = self.patch_size * self.chunk_size
+        if audio.size(1) % patch_len != 0:
+            padding_size = patch_len - audio.size(1) % patch_len
+            pad = (padding_size, 0) if padding_mode == "left" else (0, padding_size)
+            audio = torch.nn.functional.pad(audio, pad)
+        device = self._runtime_device()
+        feat = self.audio_vae.encode(audio.to(device), self._encode_sample_rate).cpu()
+        return feat.view(self.audio_vae.latent_dim, -1, self.patch_size).permute(1, 2, 0)
+
+    def _make_ref_prefix(self, ref_feat: torch.Tensor, device: torch.device):
+        ref_len = ref_feat.size(0)
+        z1 = torch.zeros((1, self.patch_size, self.audio_vae.latent_dim), dtype=torch.float32, device=device)
+        tokens = torch.cat(
+            [
+                torch.tensor([self.ref_audio_start_token], dtype=torch.int32, device=device),
+                torch.zeros(ref_len, dtype=torch.int32, device=device),
+                torch.tensor([self.ref_audio_end_token], dtype=torch.int32, device=device),
+            ]
+        )
+        feats = torch.cat([z1, ref_feat, z1], dim=0)
+        t_mask = torch.cat(
+            [
+                torch.tensor([1], dtype=torch.int32),
+                torch.zeros(ref_len, dtype=torch.int32),
+                torch.tensor([1], dtype=torch.int32),
+            ]
+        ).to(device)
+        a_mask = torch.cat(
+            [
+                torch.tensor([0], dtype=torch.int32),
+                torch.ones(ref_len, dtype=torch.int32),
+                torch.tensor([0], dtype=torch.int32),
+            ]
+        ).to(device)
+        return tokens, feats, t_mask, a_mask
+
+    def _build_inputs(
+        self,
+        text: str,
+        ref_feat: Optional[torch.Tensor] = None,
+        prompt_feat: Optional[torch.Tensor] = None,
+        speaker_slot: str = "none",
+    ):
+        """Assemble (text_token, audio_feat, text_mask, audio_mask, spk_mask).
+
+        Layout: [spk?] [ref prefix?] [text + audio_start] [prompt audio?]
+
+        ``speaker_slot`` is "none", "null", or "centroid". The null slot keeps
+        the learned [spk] token embedding, matching speaker-dropout training.
+        The centroid slot is filled by the projected speaker vector.
+        """
+        if speaker_slot not in {"none", "null", "centroid"}:
+            raise ValueError(f"speaker_slot must be one of none/null/centroid, got {speaker_slot!r}")
+
+        text_token = torch.LongTensor(self._tokenize(text))
+        text_token = torch.cat(
+            [
+                text_token,
+                torch.tensor([self.audio_start_token], dtype=torch.long, device=text_token.device),
+            ],
+            dim=-1,
+        )
+        text_length = text_token.shape[0]
+        device = text_token.device
+
+        text_pad_feat = torch.zeros(
+            (text_length, self.patch_size, self.audio_vae.latent_dim),
+            dtype=torch.float32,
+            device=device,
+        )
+
+        tokens = [text_token]
+        feats = [text_pad_feat]
+        t_masks = [torch.ones(text_length, dtype=torch.int32, device=device)]
+        a_masks = [torch.zeros(text_length, dtype=torch.int32, device=device)]
+        s_masks = [torch.zeros(text_length, dtype=torch.int32, device=device)]
+
+        if ref_feat is not None:
+            ref_tokens, ref_feats, ref_t_mask, ref_a_mask = self._make_ref_prefix(ref_feat, device)
+            tokens.insert(0, ref_tokens.long())
+            feats.insert(0, ref_feats)
+            t_masks.insert(0, ref_t_mask)
+            a_masks.insert(0, ref_a_mask)
+            s_masks.insert(0, torch.zeros(ref_tokens.shape[0], dtype=torch.int32, device=device))
+
+        if speaker_slot != "none":
+            one_feat = torch.zeros((1, self.patch_size, self.audio_vae.latent_dim), dtype=torch.float32, device=device)
+            tokens.insert(0, torch.tensor([self.spk_token], dtype=torch.long, device=device))
+            feats.insert(0, one_feat)
+            t_masks.insert(
+                0,
+                torch.ones(1, dtype=torch.int32, device=device)
+                if speaker_slot == "null"
+                else torch.zeros(1, dtype=torch.int32, device=device),
+            )
+            a_masks.insert(0, torch.zeros(1, dtype=torch.int32, device=device))
+            s_masks.insert(
+                0,
+                torch.ones(1, dtype=torch.int32, device=device)
+                if speaker_slot == "centroid"
+                else torch.zeros(1, dtype=torch.int32, device=device),
+            )
+
+        if prompt_feat is not None:
+            prompt_len = prompt_feat.size(0)
+            tokens.append(torch.zeros(prompt_len, dtype=torch.long, device=device))
+            feats.append(prompt_feat)
+            t_masks.append(torch.zeros(prompt_len, dtype=torch.int32, device=device))
+            a_masks.append(torch.ones(prompt_len, dtype=torch.int32, device=device))
+            s_masks.append(torch.zeros(prompt_len, dtype=torch.int32, device=device))
+
+        return (
+            torch.cat(tokens),
+            torch.cat(feats, dim=0),
+            torch.cat(t_masks),
+            torch.cat(a_masks),
+            torch.cat(s_masks),
+        )
+
+    # ------------------------------------------------------------------ #
+    # Generation
+    # ------------------------------------------------------------------ #
+    def generate(self, *args, **kwargs) -> torch.Tensor:
+        return next_and_close(self._generate(*args, streaming=False, **kwargs))
+
+    def generate_streaming(self, *args, **kwargs) -> Generator[torch.Tensor, None, None]:
+        return self._generate(*args, streaming=True, **kwargs)
+
+    @torch.inference_mode()
+    def _generate(
+        self,
+        target_text: str,
+        prompt_text: str = "",
+        prompt_wav_path: str = "",
+        reference_wav_path: str = "",
+        speaker_centroid: Optional[torch.Tensor] = None,
+        min_len: int = 2,
+        max_len: int = 2000,
+        inference_timesteps: int = 10,
+        cfg_value: float = 2.0,
+        retry_badcase: bool = False,
+        retry_badcase_max_times: int = 3,
+        retry_badcase_ratio_threshold: float = 6.0,
+        use_null_speaker: bool = True,
+        streaming: bool = False,
+        streaming_prefix_len: int = 4,
+    ) -> Generator[torch.Tensor, None, None]:
+        if retry_badcase and streaming:
+            warnings.warn("Retry on bad cases is not supported in streaming mode, setting retry_badcase=False.")
+            retry_badcase = False
+
+        device = self._runtime_device()
+        ref_feat = self._encode_wav(reference_wav_path, padding_mode="right") if reference_wav_path else None
+        prompt_feat = self._encode_wav(prompt_wav_path, padding_mode="left") if prompt_wav_path else None
+        text = (prompt_text + target_text) if prompt_feat is not None else target_text
+
+        speaker_centroids = None
+        if speaker_centroid is not None:
+            speaker_centroids = speaker_centroid.reshape(1, -1).to(device, dtype=self._runtime_dtype())
+
+        speaker_slot = "centroid" if speaker_centroids is not None else ("null" if use_null_speaker else "none")
+        text_token, audio_feat, text_mask, audio_mask, spk_mask = self._build_inputs(
+            text, ref_feat, prompt_feat, speaker_slot=speaker_slot
+        )
+
+        text_token = text_token.unsqueeze(0).to(device)
+        text_mask = text_mask.unsqueeze(0).to(device, dtype=self._runtime_dtype())
+        audio_feat = audio_feat.unsqueeze(0).to(device, dtype=self._runtime_dtype())
+        audio_mask = audio_mask.unsqueeze(0).to(device, dtype=self._runtime_dtype())
+        spk_mask = spk_mask.unsqueeze(0).to(device, dtype=self._runtime_dtype())
+
+        target_text_length = len(self._tokenize(target_text))
+
+        retry_badcase_times = 0
+        while retry_badcase_times < retry_badcase_max_times:
+            inference_result = self._inference(
+                text_token,
+                text_mask,
+                audio_feat,
+                audio_mask,
+                min_len=min_len,
+                max_len=min(int(target_text_length * retry_badcase_ratio_threshold + 10), max_len),
+                inference_timesteps=inference_timesteps,
+                cfg_value=cfg_value,
+                speaker_centroids=speaker_centroids,
+                spk_mask=spk_mask,
+                streaming=streaming,
+                streaming_prefix_len=streaming_prefix_len,
+            )
+            if streaming:
+                with self.audio_vae.streaming_decode() as vae_dec:
+                    for latent_pred, _, _ctx in inference_result:
+                        decode_audio = vae_dec.decode_chunk(latent_pred.to(torch.float32))
+                        yield decode_audio.squeeze(1).cpu()
+                break
+            else:
+                latent_pred, pred_audio_feat, context_len = next_and_close(inference_result)
+                if retry_badcase and pred_audio_feat.shape[0] >= target_text_length * retry_badcase_ratio_threshold:
+                    print(
+                        f"  Badcase detected, audio_text_ratio={pred_audio_feat.shape[0] / target_text_length}, retrying...",
+                        file=sys.stderr,
+                    )
+                    retry_badcase_times += 1
+                    continue
+                break
+
+        if not streaming:
+            decode_audio = self.audio_vae.decode(latent_pred.to(torch.float32))
+            decode_patch_len = self.patch_size * self._decode_chunk_size
+            if context_len > 0:
+                decode_audio = decode_audio[..., decode_patch_len * context_len :].squeeze(1).cpu()
+            else:
+                decode_audio = decode_audio.squeeze(1).cpu()
+            yield decode_audio
+
+    # ------------------------------------------------------------------ #
+    # Core AR loop
+    # ------------------------------------------------------------------ #
+    @torch.inference_mode()
+    def _inference(
+        self,
+        text: torch.Tensor,
+        text_mask: torch.Tensor,
+        feat: torch.Tensor,
+        feat_mask: torch.Tensor,
+        min_len: int = 2,
+        max_len: int = 2000,
+        inference_timesteps: int = 10,
+        cfg_value: float = 2.0,
+        speaker_centroids: Optional[torch.Tensor] = None,
+        spk_mask: Optional[torch.Tensor] = None,
+        streaming: bool = False,
+        streaming_prefix_len: int = 4,
+    ) -> Generator[Tuple[torch.Tensor, Union[torch.Tensor, List[torch.Tensor]], int], None, None]:
+        device = self._runtime_device()
+        dtype = self._runtime_dtype()
+        text = text.to(device)
+        text_mask = text_mask.to(device, dtype=dtype)
+        feat = feat.to(device, dtype=dtype)
+        feat_mask = feat_mask.to(device, dtype=dtype)
+        if speaker_centroids is not None:
+            speaker_centroids = speaker_centroids.to(device, dtype=dtype)
+        if spk_mask is not None:
+            spk_mask = spk_mask.to(device, dtype=dtype)
+        B, T, P, D = feat.shape
+
+        feat_locenc = self.feat_encoder(feat)  # [b, t, h_enc]
+        feat_embed_tslm = self.enc_to_tslm_proj(feat_locenc)
+        feat_embed_lm = self.enc_to_lm_proj(feat_locenc)
+
+        text_embed = self.base_lm.embed_tokens(text)
+        combined_embed = text_mask.unsqueeze(-1) * text_embed + feat_mask.unsqueeze(-1) * feat_embed_tslm
+        combined_embed = self._inject_speaker(combined_embed, speaker_centroids, spk_mask)
+
+        prefix_feat_cond = feat[:, -1, ...]  # b, p, d
+        curr_embed = None
+
+        # Streaming context patches (continuation modes only)
+        has_continuation_audio = feat_mask[0, -1].item() == 1
+        context_len = 0
+        if has_continuation_audio:
+            audio_indices = feat_mask.squeeze(0).nonzero(as_tuple=True)[0]
+            context_len = min(streaming_prefix_len - 1, len(audio_indices))
+            last_audio_indices = audio_indices[-context_len:]
+            pred_feat_seq = list(feat[:, last_audio_indices, :, :].split(1, dim=1))
+        else:
+            pred_feat_seq = []
+
+        # --- TSLM prefill (Barbet stepwise cache) --- #
+        barbet_hidden_seq, tslm_state = self.base_lm.prefill(combined_embed)
+        tslm_hidden_seq = self.tslm_adapter(barbet_hidden_seq)
+
+        enc_outputs = (
+            self.fsq_layer(tslm_hidden_seq) * feat_mask.unsqueeze(-1) + tslm_hidden_seq * text_mask.unsqueeze(-1)
+        )
+        lm_hidden = enc_outputs[:, -1, :]
+
+        # --- RALM prefill (MiniCPM static KV cache) --- #
+        residual_enc_inputs = self.fusion_concat_proj(
+            torch.cat((enc_outputs, feat_mask.unsqueeze(-1) * feat_embed_lm), dim=-1)
+        )
+        residual_enc_outputs, residual_kv_cache_tuple = self.residual_lm(
+            inputs_embeds=residual_enc_inputs,
+            is_causal=True,
+        )
+        self.residual_lm.kv_cache.fill_caches(residual_kv_cache_tuple)
+        residual_hidden = residual_enc_outputs[:, -1, :]
+
+        for i in tqdm(range(max_len)):
+            dit_hidden_1 = self.lm_to_dit_proj(lm_hidden)  # [b, h_dit]
+            dit_hidden_2 = self.res_to_dit_proj(residual_hidden)  # [b, h_dit]
+            dit_hidden = torch.cat((dit_hidden_1, dit_hidden_2), dim=-1)
+
+            pred_feat = self.feat_decoder(
+                mu=dit_hidden,
+                patch_size=self.patch_size,
+                cond=prefix_feat_cond.transpose(1, 2).contiguous(),
+                n_timesteps=inference_timesteps,
+                cfg_value=cfg_value,
+            ).transpose(1, 2)  # [b, p, d]
+
+            curr_locenc = self.feat_encoder(pred_feat.unsqueeze(1))  # b, 1, h_enc
+            curr_embed_tslm = self.enc_to_tslm_proj(curr_locenc)
+            curr_embed_lm = self.enc_to_lm_proj(curr_locenc)
+            curr_embed = curr_embed_tslm  # naming parity with VoxCPM2
+
+            pred_feat_seq.append(pred_feat.unsqueeze(1))  # b, 1, p, d
+            prefix_feat_cond = pred_feat
+
+            if streaming:
+                feat_pred = rearrange(pred_feat.unsqueeze(1), "b t p d -> b d (t p)", b=B, p=self.patch_size)
+                yield feat_pred, pred_feat_seq, context_len
+                if len(pred_feat_seq) > streaming_prefix_len:
+                    pred_feat_seq = pred_feat_seq[-streaming_prefix_len:]
+
+            stop_flag = self.stop_head(self.stop_actn(self.stop_proj(lm_hidden))).argmax(dim=-1)[0].cpu().item()
+            if i > min_len and stop_flag == 1:
+                break
+
+            barbet_step_hidden = self.base_lm.forward_step(curr_embed[:, 0, :], tslm_state)
+            lm_hidden = self.fsq_layer(self.tslm_adapter(barbet_step_hidden))
+
+            curr_residual_input = self.fusion_concat_proj(torch.cat((lm_hidden, curr_embed_lm[:, 0, :]), dim=-1))
+            residual_hidden = self.residual_lm.forward_step(
+                curr_residual_input,
+                torch.tensor([self.residual_lm.kv_cache.step()], device=curr_residual_input.device),
+            ).clone()
+
+        if not streaming:
+            pred_feat_seq = torch.cat(pred_feat_seq, dim=1)  # b, t, p, d
+            feat_pred = rearrange(pred_feat_seq, "b t p d -> b d (t p)", b=B, p=self.patch_size)
+            generated_feat = pred_feat_seq[:, context_len:, :, :].squeeze(0).cpu()
+            yield feat_pred, generated_feat, context_len
+
+    def inference(self, *args, **kwargs) -> Tuple[torch.Tensor, torch.Tensor]:
+        feat_pred, generated_feat, _ = next_and_close(self._inference(*args, streaming=False, **kwargs))
+        return feat_pred, generated_feat
+
+    # ------------------------------------------------------------------ #
+    # Checkpoint I/O (BlueMagpie's own format)
+    # ------------------------------------------------------------------ #
+    def save_pretrained(self, path: str):
+        os.makedirs(path, exist_ok=True)
+        with open(os.path.join(path, "config.json"), "w", encoding="utf-8") as f:
+            f.write(self.config.model_dump_json(indent=2))
+        state = {k: v for k, v in self.state_dict().items() if not k.startswith("audio_vae.")}
+        torch.save(state, os.path.join(path, "pytorch_model.bin"))
+        if self.audio_vae is not None:
+            torch.save(self.audio_vae.state_dict(), os.path.join(path, "audiovae.pth"))
+        if self.text_tokenizer is not None and hasattr(self.text_tokenizer, "save_pretrained"):
+            self.text_tokenizer.save_pretrained(path)
+
+    @classmethod
+    def from_local(
+        cls,
+        path: str,
+        tokenizer=None,
+        training: bool = False,
+        device: str | None = None,
+    ) -> "BlueMagpieModel":
+        from bluemagpie._vendor.voxcpm.modules.audiovae import AudioVAEV2
+
+        with open(os.path.join(path, "config.json"), "r", encoding="utf-8") as f:
+            config = BlueMagpieConfig.model_validate_json(f.read())
+
+        if tokenizer is None:
+            try:
+                from transformers import AutoTokenizer
+
+                tokenizer = AutoTokenizer.from_pretrained(path)
+            except Exception:
+                tokenizer = None
+
+        audio_vae = AudioVAEV2(config=config.audio_vae_config) if config.audio_vae_config else AudioVAEV2()
+        vae_path = os.path.join(path, "audiovae.pth")
+        if os.path.exists(vae_path):
+            vae_state = torch.load(vae_path, map_location="cpu", weights_only=True)
+            audio_vae.load_state_dict(vae_state.get("state_dict", vae_state))
+
+        model = cls(config, tokenizer, audio_vae, device=device)
+        # mmap keeps the checkpoint memory-mapped instead of materializing a
+        # second full copy in RAM (load peak ~halved; matters on Colab-class hosts).
+        state = torch.load(
+            os.path.join(path, "pytorch_model.bin"), map_location="cpu", weights_only=True, mmap=True
+        )
+        missing, unexpected = model.load_state_dict(state, strict=False)
+        missing = [k for k in missing if not k.startswith("audio_vae.")]
+        if missing or unexpected:
+            print(f"[bluemagpie] missing keys: {missing[:8]}... unexpected: {unexpected[:8]}...", file=sys.stderr)
+
+        if not training:
+            model = model.to(get_dtype(model.config.dtype))
+        model.audio_vae = model.audio_vae.to(torch.float32)
+        return model.to(model.device).eval() if not training else model.to(model.device)

bluemagpie/tslm.py

@@ -0,0 +1,98 @@
+"""Barbet as the Text-Semantic LM (TSLM), with incremental decoding.
+
+VoxCPM2's generation loop decodes one latent patch at a time, so the TSLM needs
+a stateful single-step path. As of the open_formosa R2 architecture, Barbet
+ships exactly that: a real **Mamba2** mixer and a native ``BarbetCache`` holding
+
+- attention K/V states (rolling window for sliding-window layers),
+- the trailing ``d_conv - 1`` causal-conv inputs per Mamba layer, and
+- the Mamba2 selective-scan SSM state per Mamba layer.
+
+We delegate stepwise decoding to it rather than re-deriving a cache by hand:
+the selective-scan state cannot be reconstructed from a conv ring buffer alone,
+so the official cache is the only correct option (and it stays in lockstep with
+upstream). ``tests/test_step_equivalence.py`` asserts prefill + forward_step
+reproduces the full-sequence forward across all layer types.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+import torch
+from torch import nn
+
+from barbet import BarbetConfig, BarbetModel
+from barbet.modeling_barbet import BarbetCache
+
+
+@dataclass
+class BarbetStepState:
+    """Decoding state: Barbet's native hybrid cache (tracks position itself)."""
+
+    cache: BarbetCache
+
+    @property
+    def pos(self) -> int:
+        return self.cache.seen_tokens
+
+
+class BarbetTSLM(nn.Module):
+    """Barbet backbone with embeddings, full forward, and cached stepwise decode."""
+
+    def __init__(self, config: BarbetConfig) -> None:
+        super().__init__()
+        self.config = config
+        self.backbone = BarbetModel(config)
+
+    @property
+    def embed_tokens(self) -> nn.Embedding:
+        return self.backbone.embed_tokens
+
+    @property
+    def hidden_size(self) -> int:
+        return self.config.hidden_size
+
+    def forward(
+        self,
+        inputs_embeds: torch.Tensor,
+        attention_mask: torch.Tensor | None = None,
+    ) -> torch.Tensor:
+        """Full-sequence forward (training / teacher-forcing), no cache."""
+        out = self.backbone(
+            inputs_embeds=inputs_embeds,
+            attention_mask=attention_mask,
+            use_cache=False,
+            return_dict=True,
+        )
+        return out.last_hidden_state
+
+    # ------------------------------------------------------------------ #
+    # Stepwise decoding (delegated to BarbetCache)
+    # ------------------------------------------------------------------ #
+    @torch.no_grad()
+    def prefill(self, inputs_embeds: torch.Tensor) -> tuple[torch.Tensor, BarbetStepState]:
+        """Run the prompt and return (hidden_states, state) with a warm cache."""
+        cache = BarbetCache(self.config)
+        out = self.backbone(
+            inputs_embeds=inputs_embeds,
+            past_key_values=cache,
+            use_cache=True,
+            return_dict=True,
+        )
+        return out.last_hidden_state, BarbetStepState(cache=out.past_key_values or cache)
+
+    @torch.no_grad()
+    def forward_step(self, x: torch.Tensor, state: BarbetStepState) -> torch.Tensor:
+        """Decode one position. ``x``: [B, H] input embedding -> [B, H] hidden.
+
+        The cache tracks the running position (``seen_tokens``), so RoPE offsets
+        and the sliding-window / conv / SSM states advance automatically.
+        """
+        out = self.backbone(
+            inputs_embeds=x.unsqueeze(1),
+            past_key_values=state.cache,
+            use_cache=True,
+            return_dict=True,
+        )
+        return out.last_hidden_state[:, 0, :]

production.py

@@ -6,9 +6,13 @@ import math
 import random
 import re
 import unicodedata
+from dataclasses import dataclass
+from datetime import date
+from typing import Sequence
 
 import numpy as np
 import torch
+from opencc import OpenCC
 from torch import nn
 
 
@@ -16,6 +20,18 @@ _SPACE_RE = re.compile(r"\s+")
 _PUNCT_NO_LEFT_SPACE_RE = re.compile(r"\s+([，。！？；：、,.!?;:])")
 _CJK_PUNCT_RIGHT_SPACE_RE = re.compile(r"([，。！？；：、])\s+")
 _BOPOMOFO_TONES = {"ˊ", "ˇ", "ˋ", "˙"}
+_BOPOMOFO_ASR_POPO_RE = re.compile(
+    r"(?i)(?<![a-z0-9])(?:po|bo)\s*(?:po|bo)\s*(?:mo|摸)(?![a-z0-9])"
+)
+_ASR_HOMOPHONE_TRANSLATION = str.maketrans(
+    {
+        "她": "他",
+        "它": "他",
+        "牠": "他",
+        "祂": "他",
+        "妳": "你",
+    }
+)
 _TERMINAL_PUNCTUATION = frozenset("。！？.!?")
 _NONTERMINAL_TRAILING_PUNCTUATION = frozenset("，,、；;：:")
 _TRAILING_CLOSERS = frozenset("\"'”’」』】）)]}")
@@ -30,6 +46,127 @@ _BOPOMOFO_READINGS = {
     "ㄨ": "烏", "ㄩ": "迂",
 }
 
+_ZH_DIGITS = "零一二三四五六七八九"
+_ZH_SMALL_UNITS = ("", "十", "百", "千")
+_ZH_LARGE_UNITS = ("", "萬", "億", "兆")
+_SPOKEN_DATE_RE = re.compile(
+    r"(?<![A-Za-z0-9/])(\d{4})([/-])(\d{1,2})\2(\d{1,2})(?![A-Za-z0-9/])"
+)
+_SPOKEN_ZH_DATE_RE = re.compile(
+    r"(?<![A-Za-z0-9])(\d{4})年(\d{1,2})月(\d{1,2})(?:日|號)(?![A-Za-z0-9])"
+)
+_SPOKEN_TIME_RE = re.compile(
+    r"(?<![A-Za-z0-9:])([01]?\d|2[0-3]):([0-5]\d)(?![A-Za-z0-9:])"
+)
+_SPOKEN_PERCENT_RE = re.compile(
+    r"(?<![A-Za-z0-9.])([+-]?\d+(?:\.\d+)?)\s*[%％](?![A-Za-z0-9])"
+)
+_SPOKEN_UNIT_RE = re.compile(
+    r"(?<![A-Za-z0-9.])([+-]?\d+(?:\.\d+)?)\s*"
+    r"(km/h|m/s|kHz|MHz|GHz|Hz|km|cm|mm|kg|mg|mL|ml|kW|°\s*[Cc]|℃|m|g|L|l|W|V)"
+    r"(?![A-Za-z])"
+)
+_SPOKEN_ZH_MEASURE_RE = re.compile(
+    r"(?<![A-Za-z0-9.])([+-]?\d+(?:\.\d+)?)\s*"
+    r"(公里|公尺|公分|公厘|厘米|毫米|公斤|千克|公克|毫克|公升|毫升|"
+    r"攝氏度|千赫茲|兆赫茲|吉赫茲|赫茲|小時|分鐘|米|克|升|度|秒|天|"
+    r"個|人|次|張|台|件|份|元)"
+    r"(?![A-Za-z0-9])"
+)
+_SPOKEN_URL_RE = re.compile(
+    r"(?i)(?<![A-Za-z0-9])(?:https?://|ftp://|www\.)[^\s<>\"'，。！？；]+"
+)
+_SPOKEN_EMAIL_RE = re.compile(
+    r"(?i)(?<![A-Za-z0-9.!#$%&'*+/=?^_`{|}~-])"
+    r"[A-Za-z0-9.!#$%&'*+/=?^_`{|}~-]+@"
+    r"[A-Za-z0-9-]+(?:\.[A-Za-z0-9-]+)+"
+    r"(?![A-Za-z0-9-])"
+)
+_SPOKEN_SEMVER_RE = re.compile(
+    r"(?i)(?<![A-Za-z0-9.])(?:v|version\s+|版本\s*(?:v\s*)?)?"
+    r"(\d+)\.(\d+)\.(\d+)"
+    r"(?:-([0-9A-Za-z]+(?:[.-][0-9A-Za-z]+)*))?"
+    r"(?:\+([0-9A-Za-z]+(?:[.-][0-9A-Za-z]+)*))?"
+    r"(?![A-Za-z0-9.])"
+)
+_CURRENCY_NUMBER_PATTERN = r"[+-]?(?:\d{1,3}(?:,\d{3})+|\d+)(?:\.\d+)?"
+_CURRENCY_CODE_PATTERN = (
+    r"NT\$|TWD|NTD|US\$|USD|HK\$|HKD|CN¥|CNY|RMB|JPY|EUR|GBP|KRW|"
+    r"\$|€|¥|￥|£|₩"
+)
+_SPOKEN_CURRENCY_PREFIX_RE = re.compile(
+    rf"(?i)(?<![A-Za-z0-9])({_CURRENCY_CODE_PATTERN})\s*"
+    rf"({_CURRENCY_NUMBER_PATTERN})(?![\d,.])"
+)
+_SPOKEN_CURRENCY_SUFFIX_RE = re.compile(
+    rf"(?i)(?<![A-Za-z0-9.])({_CURRENCY_NUMBER_PATTERN})\s*"
+    rf"(TWD|NTD|USD|HKD|CNY|RMB|JPY|EUR|GBP|KRW)(?![A-Za-z])"
+)
+_MODEL_CODE_RE = re.compile(
+    r"(?<![A-Za-z0-9])([A-Z]{2,8})[ -]*(\d{1,8})(?![A-Za-z0-9.])"
+)
+_UPPERCASE_ACRONYM_RE = re.compile(r"(?<![A-Za-z0-9])([A-Z]{2,8})(?![A-Za-z0-9])")
+_ZH_UNIT_READINGS = {
+    "km/h": "公里每小時",
+    "m/s": "公尺每秒",
+    "khz": "千赫茲",
+    "mhz": "兆赫茲",
+    "ghz": "吉赫茲",
+    "hz": "赫茲",
+    "km": "公里",
+    "cm": "公分",
+    "mm": "毫米",
+    "kg": "公斤",
+    "mg": "毫克",
+    "ml": "毫升",
+    "kw": "千瓦",
+    "°c": "攝氏度",
+    "℃": "攝氏度",
+    "m": "公尺",
+    "g": "公克",
+    "l": "公升",
+    "w": "瓦",
+    "v": "伏特",
+}
+_ZH_CURRENCY_READINGS = {
+    "NT$": "新台幣",
+    "TWD": "新台幣",
+    "NTD": "新台幣",
+    "US$": "美元",
+    "USD": "美元",
+    "$": "美元",
+    "HK$": "港幣",
+    "HKD": "港幣",
+    "CN¥": "人民幣",
+    "CNY": "人民幣",
+    "RMB": "人民幣",
+    "JPY": "日圓",
+    "¥": "日圓",
+    "￥": "日圓",
+    "EUR": "歐元",
+    "€": "歐元",
+    "GBP": "英鎊",
+    "£": "英鎊",
+    "KRW": "韓元",
+    "₩": "韓元",
+}
+_ZH_NETWORK_SYMBOL_READINGS = {
+    ".": "點",
+    "/": "斜線",
+    ":": "冒號",
+    "?": "問號",
+    "=": "等於",
+    "&": "和",
+    "#": "井號",
+    "@": "小老鼠",
+    "-": "橫線",
+    "_": "底線",
+    "%": "百分號",
+    "+": "加號",
+    "~": "波浪號",
+}
+_T2S_CONVERTER = OpenCC("t2s")
+
 
 class StopHysteresisController(nn.Module):
     """Apply probability-threshold hysteresis around a legacy stop head.
@@ -152,6 +289,262 @@ def _is_bopomofo(char: str) -> bool:
     return 0x3100 <= codepoint <= 0x312F or 0x31A0 <= codepoint <= 0x31BF or char in _BOPOMOFO_TONES
 
 
+def _zh_four_digit_section(value: int, *, suppress_leading_one: bool = True) -> str:
+    """Read one non-negative, at-most-four-digit integer in Mandarin."""
+
+    output: list[str] = []
+    pending_zero = False
+    for position in range(3, -1, -1):
+        divisor = 10**position
+        digit = value // divisor
+        value %= divisor
+        if digit:
+            if pending_zero and output:
+                output.append(_ZH_DIGITS[0])
+            if not (digit == 1 and position == 1 and not output and suppress_leading_one):
+                output.append(_ZH_DIGITS[digit])
+            output.append(_ZH_SMALL_UNITS[position])
+            pending_zero = False
+        elif output and value:
+            pending_zero = True
+    return "".join(output)
+
+
+def _zh_integer(value: int) -> str:
+    if value == 0:
+        return _ZH_DIGITS[0]
+    if value < 0 or value >= 10**16:
+        raise ValueError("Mandarin integer normalizer supports values from 0 to 10^16 - 1")
+
+    sections: list[int] = []
+    while value:
+        sections.append(value % 10000)
+        value //= 10000
+
+    output: list[str] = []
+    pending_zero = False
+    for section_index in range(len(sections) - 1, -1, -1):
+        section = sections[section_index]
+        if section == 0:
+            if output and any(sections[:section_index]):
+                pending_zero = True
+            continue
+        if output and (pending_zero or section < 1000):
+            output.append(_ZH_DIGITS[0])
+        output.append(_zh_four_digit_section(section, suppress_leading_one=not output))
+        output.append(_ZH_LARGE_UNITS[section_index])
+        pending_zero = False
+    return "".join(output)
+
+
+def _zh_number(number: str) -> str:
+    """Read a validated decimal string as a Mandarin cardinal number."""
+
+    sign = ""
+    if number.startswith(("+", "-")):
+        sign = "正" if number[0] == "+" else "負"
+        number = number[1:]
+    integer, separator, fraction = number.partition(".")
+    if not integer or not integer.isdigit() or separator and (not fraction or not fraction.isdigit()):
+        raise ValueError("invalid decimal number")
+    if len(integer) > 16:
+        raise ValueError("number is too large for conservative normalization")
+    spoken = _zh_integer(int(integer))
+    if separator:
+        spoken += "點" + "".join(_ZH_DIGITS[int(digit)] for digit in fraction)
+    return sign + spoken
+
+
+def _zh_digit_sequence(digits: str) -> str:
+    if not digits or not digits.isdigit():
+        raise ValueError("expected a non-empty digit sequence")
+    return "".join(_ZH_DIGITS[int(digit)] for digit in digits)
+
+
+def _zh_network_text(value: str) -> str:
+    """Spell network identifiers while making separators audible."""
+
+    output: list[str] = []
+    buffer: list[str] = []
+
+    def flush() -> None:
+        if not buffer:
+            return
+        token = "".join(buffer)
+        if token.isdigit():
+            output.append(_zh_digit_sequence(token))
+        elif token.lower() == "www" or len(token) > 1 and token.isupper():
+            output.extend(token.upper())
+        else:
+            output.append(token)
+        buffer.clear()
+
+    for character in value:
+        if character.isascii() and character.isalnum():
+            if buffer and character.isdigit() != buffer[-1].isdigit():
+                flush()
+            buffer.append(character)
+            continue
+        flush()
+        reading = _ZH_NETWORK_SYMBOL_READINGS.get(character)
+        if reading:
+            output.append(reading)
+        elif not character.isspace():
+            output.append(character)
+    flush()
+    return " ".join(output)
+
+
+def _zh_url(value: str) -> str:
+    scheme, separator, remainder = value.partition("://")
+    if separator:
+        protocol = " ".join(scheme.upper())
+        return f"{protocol} 冒號 斜線 斜線 {_zh_network_text(remainder)}"
+    return _zh_network_text(value)
+
+
+def _zh_email(value: str) -> str:
+    local, domain = value.rsplit("@", 1)
+    return f"{_zh_network_text(local)} 小老鼠 {_zh_network_text(domain)}"
+
+
+def _zh_semantic_version(match: re.Match[str]) -> str:
+    major, minor, patch, prerelease, build = match.groups()
+    try:
+        core = "點".join(_zh_integer(int(part)) for part in (major, minor, patch))
+    except ValueError:
+        return match.group(0)
+    output = f"版本{core}"
+    if prerelease:
+        output += f" 預發布 {_zh_network_text(prerelease)}"
+    if build:
+        output += f" 建置 {_zh_network_text(build)}"
+    return output
+
+
+def _zh_currency(code: str, number: str, original: str) -> str:
+    reading = _ZH_CURRENCY_READINGS.get(code.upper(), _ZH_CURRENCY_READINGS.get(code))
+    if reading is None:
+        return original
+    try:
+        spoken_number = _zh_number(number.replace(",", ""))
+    except ValueError:
+        return original
+    return reading + spoken_number
+
+
+def normalize_spoken_forms(text: str, *, locale: str = "zh-TW") -> str:
+    """Conservatively expand common written forms for TTS.
+
+    Chinese locales expand validated dates, network addresses, semantic
+    versions, currencies, percentages, numeric units, and standalone
+    all-uppercase acronyms/model identifiers. Ordinary English words are
+    deliberately left untouched. English locales only receive the existing
+    Unicode/punctuation normalization. Unknown locales raise instead of
+    silently applying the wrong pronunciation rules.
+    """
+
+    raw = unicodedata.normalize("NFC", str(text or ""))
+    locale_key = str(locale or "").replace("_", "-").lower()
+    if locale_key in {"en", "en-us", "en-gb"}:
+        return normalize_tts_text(raw)
+    if locale_key not in {"zh", "zh-tw", "zh-hant", "zh-cn", "zh-hans"}:
+        raise ValueError(f"unsupported spoken-form locale: {locale!r}")
+
+    protected: list[str] = []
+
+    def protect(spoken: str) -> str:
+        marker = f"\uf000{len(protected)}\uf001"
+        protected.append(spoken)
+        return marker
+
+    def replace_url(match: re.Match[str]) -> str:
+        matched = match.group(0)
+        core = matched.rstrip(".,!?;:")
+        trailing = matched[len(core) :]
+        return protect(_zh_url(core)) + trailing
+
+    def replace_email(match: re.Match[str]) -> str:
+        return protect(_zh_email(match.group(0)))
+
+    def replace_date(match: re.Match[str]) -> str:
+        year_text, _, month_text, day_text = match.groups()
+        try:
+            date(int(year_text), int(month_text), int(day_text))
+        except ValueError:
+            return match.group(0)
+        return (
+            f"{_zh_digit_sequence(year_text)}年"
+            f"{_zh_integer(int(month_text))}月{_zh_integer(int(day_text))}日"
+        )
+
+    def replace_zh_date(match: re.Match[str]) -> str:
+        year_text, month_text, day_text = match.groups()
+        try:
+            date(int(year_text), int(month_text), int(day_text))
+        except ValueError:
+            return match.group(0)
+        return (
+            f"{_zh_digit_sequence(year_text)}年"
+            f"{_zh_integer(int(month_text))}月{_zh_integer(int(day_text))}日"
+        )
+
+    def replace_number(match: re.Match[str], suffix: str) -> str:
+        try:
+            return _zh_number(match.group(1)) + suffix
+        except ValueError:
+            return match.group(0)
+
+    def replace_time(match: re.Match[str]) -> str:
+        hour = int(match.group(1))
+        minute = int(match.group(2))
+        minute_text = "整" if minute == 0 else f"{_zh_integer(minute)}分"
+        return f"{_zh_integer(hour)}點{minute_text}"
+
+    def replace_percent(match: re.Match[str]) -> str:
+        try:
+            return "百分之" + _zh_number(match.group(1))
+        except ValueError:
+            return match.group(0)
+
+    def replace_unit(match: re.Match[str]) -> str:
+        unit_key = re.sub(r"\s+", "", match.group(2)).lower()
+        reading = _ZH_UNIT_READINGS.get(unit_key)
+        if reading is None:
+            return match.group(0)
+        return replace_number(match, reading)
+
+    def replace_model_code(match: re.Match[str]) -> str:
+        letters, digits = match.groups()
+        return f"{' '.join(letters)} {_zh_digit_sequence(digits)}"
+
+    output = _SPOKEN_URL_RE.sub(replace_url, raw)
+    output = _SPOKEN_EMAIL_RE.sub(replace_email, output)
+    output = _SPOKEN_ZH_DATE_RE.sub(replace_zh_date, output)
+    output = _SPOKEN_DATE_RE.sub(replace_date, output)
+    output = _SPOKEN_TIME_RE.sub(replace_time, output)
+    output = _SPOKEN_PERCENT_RE.sub(replace_percent, output)
+    output = _SPOKEN_UNIT_RE.sub(replace_unit, output)
+    output = _SPOKEN_ZH_MEASURE_RE.sub(
+        lambda match: replace_number(match, match.group(2)),
+        output,
+    )
+    output = _SPOKEN_SEMVER_RE.sub(_zh_semantic_version, output)
+    output = _SPOKEN_CURRENCY_PREFIX_RE.sub(
+        lambda match: _zh_currency(match.group(1), match.group(2), match.group(0)),
+        output,
+    )
+    output = _SPOKEN_CURRENCY_SUFFIX_RE.sub(
+        lambda match: _zh_currency(match.group(2), match.group(1), match.group(0)),
+        output,
+    )
+    output = _MODEL_CODE_RE.sub(replace_model_code, output)
+    output = _UPPERCASE_ACRONYM_RE.sub(lambda match: " ".join(match.group(1)), output)
+    for index, spoken in enumerate(protected):
+        output = output.replace(f"\uf000{index}\uf001", spoken)
+    return normalize_tts_text(output)
+
+
 def normalize_tts_text(text: str) -> str:
     """Normalize common punctuation and pronounce standalone Bopomofo symbols."""
 
@@ -205,7 +598,13 @@ def normalize_tts_text(text: str) -> str:
         elif char in {";", "；"}:
             output.append("；" if has_cjk else ";")
         elif char in {":", "："}:
-            output.append(":" if following == "/" or previous.isdigit() and following.isdigit() else ("：" if has_cjk else ":"))
+            ascii_context = (
+                previous.isascii()
+                and previous.isalnum()
+                and following.isascii()
+                and following.isalnum()
+            )
+            output.append(":" if following == "/" or ascii_context else ("：" if has_cjk else ":"))
         elif char in {"“", "”", "„", "＂"}:
             output.append('"')
         elif char in {"‘", "’", "＇"}:
@@ -224,6 +623,21 @@ def normalize_tts_text(text: str) -> str:
     return normalized.strip()
 
 
+def normalize_tts_eval_text(text: str) -> str:
+    """Normalize only verified ASR-equivalent forms for semantic scoring.
+
+    This remains separate from model-input normalization: Mandarin pronoun
+    homophones are acoustically indistinguishable, but their written forms must
+    stay untouched in the text sent to the TTS model.
+    """
+
+    normalized = normalize_tts_text(text)
+    normalized = normalized.translate(_ASR_HOMOPHONE_TRANSLATION)
+    if "注音" in normalized or "符號" in normalized:
+        normalized = _BOPOMOFO_ASR_POPO_RE.sub("波坡摸", normalized)
+    return normalized
+
+
 def ensure_terminal_punctuation(text: str) -> str:
     """Give the acoustic model an explicit endpoint cue without changing words."""
 
@@ -605,6 +1019,406 @@ def apply_loudness_floor(
     return output
 
 
+def _comparison_text(text: str, *, locale: str) -> str:
+    normalized = normalize_tts_eval_text(
+        normalize_spoken_forms(text, locale=locale)
+    ).casefold()
+    locale_key = str(locale or "").replace("_", "-").lower()
+    if locale_key in {"zh", "zh-tw", "zh-hant", "zh-cn", "zh-hans"}:
+        normalized = _T2S_CONVERTER.convert(normalized)
+    return "".join(char for char in normalized if char.isalnum())
+
+
+def _levenshtein_alignment(source: str, hypothesis: str) -> list[tuple[str, int, int]]:
+    """Return deterministic edit operations as ``(op, source_index, hyp_index)``."""
+
+    rows = len(source) + 1
+    columns = len(hypothesis) + 1
+    distance = [[0] * columns for _ in range(rows)]
+    for source_index in range(rows):
+        distance[source_index][0] = source_index
+    for hyp_index in range(columns):
+        distance[0][hyp_index] = hyp_index
+    for source_index in range(1, rows):
+        for hyp_index in range(1, columns):
+            substitution = distance[source_index - 1][hyp_index - 1] + (
+                source[source_index - 1] != hypothesis[hyp_index - 1]
+            )
+            deletion = distance[source_index - 1][hyp_index] + 1
+            insertion = distance[source_index][hyp_index - 1] + 1
+            distance[source_index][hyp_index] = min(substitution, deletion, insertion)
+
+    operations: list[tuple[str, int, int]] = []
+    source_index = len(source)
+    hyp_index = len(hypothesis)
+    while source_index or hyp_index:
+        if source_index and hyp_index:
+            cost = source[source_index - 1] != hypothesis[hyp_index - 1]
+            if distance[source_index][hyp_index] == distance[source_index - 1][hyp_index - 1] + cost:
+                operations.append(
+                    (
+                        "substitute" if cost else "equal",
+                        source_index - 1,
+                        hyp_index - 1,
+                    )
+                )
+                source_index -= 1
+                hyp_index -= 1
+                continue
+        if source_index and distance[source_index][hyp_index] == distance[source_index - 1][hyp_index] + 1:
+            operations.append(("delete", source_index - 1, hyp_index))
+            source_index -= 1
+            continue
+        operations.append(("insert", source_index, hyp_index - 1))
+        hyp_index -= 1
+    operations.reverse()
+    return operations
+
+
+@dataclass(frozen=True)
+class AsrComparison:
+    """Character-level semantic completion evidence for one TTS candidate."""
+
+    target_text: str
+    transcript_text: str
+    edit_distance: int
+    cer: float
+    prefix_cer: float
+    suffix_cer: float
+    prefix_deletions: int
+    suffix_deletions: int
+    extra_tail: str
+    extra_tail_units: int
+    passed: bool
+
+
+def compare_asr_text(
+    target: str,
+    transcript: str,
+    *,
+    locale: str = "zh-TW",
+    prefix_units: int = 6,
+    suffix_units: int = 6,
+    max_cer: float = 0.10,
+    max_prefix_cer: float = 0.0,
+    max_suffix_cer: float = 0.0,
+    max_extra_tail_units: int = 0,
+) -> AsrComparison:
+    """Compare an ASR transcript with strict onset, completion, and tail gates.
+
+    Punctuation and whitespace are ignored, while written dates/numbers are
+    normalized before alignment. Empty targets, unsupported locales, invalid
+    limits, and non-finite limits all fail closed via ``passed=False``.
+    """
+
+    try:
+        target_text = _comparison_text(target, locale=locale)
+        transcript_text = _comparison_text(transcript, locale=locale)
+    except (TypeError, ValueError):
+        target_text = ""
+        transcript_text = ""
+
+    try:
+        prefix_count = int(prefix_units)
+        suffix_count = int(suffix_units)
+    except (TypeError, ValueError, OverflowError):
+        prefix_count = 0
+        suffix_count = 0
+    prefix_limit = min(max(0, prefix_count), len(target_text))
+    suffix_start = max(0, len(target_text) - max(0, suffix_count))
+
+    operations = _levenshtein_alignment(target_text, transcript_text)
+    edit_distance = sum(operation != "equal" for operation, _, _ in operations)
+    cer = edit_distance / len(target_text) if target_text else math.inf
+    prefix_errors = 0
+    suffix_errors = 0
+    prefix_deletions = 0
+    suffix_deletions = 0
+    tail_characters: list[str] = []
+    for operation, source_index, hyp_index in operations:
+        if operation == "equal":
+            continue
+        if operation == "insert":
+            if source_index < prefix_limit:
+                prefix_errors += 1
+            if source_index >= suffix_start:
+                suffix_errors += 1
+            if source_index >= len(target_text) and 0 <= hyp_index < len(transcript_text):
+                tail_characters.append(transcript_text[hyp_index])
+            continue
+        if source_index < prefix_limit:
+            prefix_errors += 1
+            prefix_deletions += operation == "delete"
+        if source_index >= suffix_start:
+            suffix_errors += 1
+            suffix_deletions += operation == "delete"
+
+    prefix_cer = prefix_errors / prefix_limit if prefix_limit else math.inf
+    suffix_length = len(target_text) - suffix_start
+    suffix_cer = suffix_errors / suffix_length if suffix_length else math.inf
+    extra_tail = "".join(tail_characters)
+
+    limits: list[float] = []
+    for value in (max_cer, max_prefix_cer, max_suffix_cer, max_extra_tail_units):
+        try:
+            limits.append(float(value))
+        except (TypeError, ValueError, OverflowError):
+            limits.append(math.nan)
+    valid_limits = all(math.isfinite(value) and value >= 0.0 for value in limits)
+    passed = bool(
+        target_text
+        and transcript_text
+        and prefix_limit > 0
+        and suffix_length > 0
+        and valid_limits
+        and cer <= limits[0]
+        and prefix_cer <= limits[1]
+        and suffix_cer <= limits[2]
+        and len(extra_tail) <= limits[3]
+    )
+    return AsrComparison(
+        target_text=target_text,
+        transcript_text=transcript_text,
+        edit_distance=edit_distance,
+        cer=cer,
+        prefix_cer=prefix_cer,
+        suffix_cer=suffix_cer,
+        prefix_deletions=prefix_deletions,
+        suffix_deletions=suffix_deletions,
+        extra_tail=extra_tail,
+        extra_tail_units=len(extra_tail),
+        passed=passed,
+    )
+
+
+def _finite_number(value, *, minimum: float | None = None, maximum: float | None = None) -> float | None:
+    if isinstance(value, (bool, np.bool_)):
+        return None
+    try:
+        number = float(value)
+    except (TypeError, ValueError, OverflowError):
+        return None
+    if not math.isfinite(number):
+        return None
+    if minimum is not None and number < minimum:
+        return None
+    if maximum is not None and number > maximum:
+        return None
+    return number
+
+
+def candidate_local_score(
+    *,
+    cer: float,
+    speaker_similarity: float,
+    boundary_speaker_drop: float,
+    prefix_cer: float = 0.0,
+    suffix_cer: float = 0.0,
+    pace_penalty: float = 0.0,
+    style_penalty: float = 0.0,
+    asr_passed: bool = True,
+    truncated: bool = False,
+    extra_tail: bool = False,
+    speaker_weight: float = 0.05,
+    boundary_weight: float = 0.10,
+    prefix_weight: float = 0.50,
+    suffix_weight: float = 0.50,
+    pace_weight: float = 1.0,
+    style_weight: float = 1.0,
+    max_cer: float | None = None,
+    min_speaker_similarity: float | None = None,
+    max_boundary_speaker_drop: float | None = None,
+) -> float:
+    """Return a finite local candidate cost or ``inf`` for unsafe evidence."""
+
+    if asr_passed is not True or truncated is not False or extra_tail is not False:
+        return math.inf
+    metric_values = [
+        _finite_number(cer, minimum=0.0),
+        _finite_number(speaker_similarity, minimum=-1.0, maximum=1.0),
+        _finite_number(boundary_speaker_drop, minimum=0.0),
+        _finite_number(prefix_cer, minimum=0.0),
+        _finite_number(suffix_cer, minimum=0.0),
+        _finite_number(pace_penalty, minimum=0.0),
+        _finite_number(style_penalty, minimum=0.0),
+    ]
+    weights = [
+        _finite_number(speaker_weight, minimum=0.0),
+        _finite_number(boundary_weight, minimum=0.0),
+        _finite_number(prefix_weight, minimum=0.0),
+        _finite_number(suffix_weight, minimum=0.0),
+        _finite_number(pace_weight, minimum=0.0),
+        _finite_number(style_weight, minimum=0.0),
+    ]
+    if any(value is None for value in metric_values + weights):
+        return math.inf
+    candidate_cer, similarity, boundary_drop, onset_cer, ending_cer, pace, style = metric_values
+
+    gate_max_cer = None if max_cer is None else _finite_number(max_cer, minimum=0.0)
+    gate_min_similarity = (
+        None
+        if min_speaker_similarity is None
+        else _finite_number(min_speaker_similarity, minimum=-1.0, maximum=1.0)
+    )
+    gate_max_boundary = (
+        None
+        if max_boundary_speaker_drop is None
+        else _finite_number(max_boundary_speaker_drop, minimum=0.0)
+    )
+    if (
+        max_cer is not None and gate_max_cer is None
+        or min_speaker_similarity is not None and gate_min_similarity is None
+        or max_boundary_speaker_drop is not None and gate_max_boundary is None
+    ):
+        return math.inf
+    if gate_max_cer is not None and candidate_cer > gate_max_cer:
+        return math.inf
+    if gate_min_similarity is not None and similarity < gate_min_similarity:
+        return math.inf
+    if gate_max_boundary is not None and boundary_drop > gate_max_boundary:
+        return math.inf
+
+    speaker_w, boundary_w, prefix_w, suffix_w, pace_w, style_w = weights
+    score = (
+        candidate_cer
+        + speaker_w * (1.0 - similarity)
+        + boundary_w * boundary_drop
+        + prefix_w * onset_cer
+        + suffix_w * ending_cer
+        + pace_w * pace
+        + style_w * style
+    )
+    return score if math.isfinite(score) and score >= 0.0 else math.inf
+
+
+def candidate_transition_score(
+    *,
+    speaker_similarity: float,
+    f0_delta: float,
+    rms_delta: float,
+    speaker_weight: float = 1.0,
+    f0_weight: float = 1.0,
+    rms_weight: float = 1.0,
+) -> float:
+    """Score continuity between adjacent candidates with finite-only inputs."""
+
+    values = [
+        _finite_number(speaker_similarity, minimum=-1.0, maximum=1.0),
+        _finite_number(f0_delta, minimum=0.0),
+        _finite_number(rms_delta, minimum=0.0),
+        _finite_number(speaker_weight, minimum=0.0),
+        _finite_number(f0_weight, minimum=0.0),
+        _finite_number(rms_weight, minimum=0.0),
+    ]
+    if any(value is None for value in values):
+        return math.inf
+    similarity, pitch_delta, loudness_delta, speaker_w, pitch_w, loudness_w = values
+    score = (
+        speaker_w * (1.0 - similarity)
+        + pitch_w * pitch_delta
+        + loudness_w * loudness_delta
+    )
+    return score if math.isfinite(score) and score >= 0.0 else math.inf
+
+
+@dataclass(frozen=True)
+class CandidateSequenceSelection:
+    candidate_indices: tuple[int, ...]
+    total_score: float
+
+
+def select_candidate_sequence(
+    local_scores: Sequence[Sequence[float]],
+    transition_scores: Sequence[Sequence[Sequence[float]]] = (),
+) -> CandidateSequenceSelection | None:
+    """Select the minimum-cost candidate path in ``O(N K^2)``.
+
+    Non-finite/negative scores remove only their candidate or edge. Malformed
+    matrices and graphs with no complete finite path return ``None`` so callers
+    cannot accidentally fall back to an unverified candidate.
+    """
+
+    try:
+        local_rows = [list(row) for row in local_scores]
+    except TypeError:
+        return None
+    if not local_rows or any(not row for row in local_rows):
+        return None
+    safe_local = [
+        [
+            score if score is not None else math.inf
+            for score in (_finite_number(value, minimum=0.0) for value in row)
+        ]
+        for row in local_rows
+    ]
+
+    if len(safe_local) == 1:
+        try:
+            if len(transition_scores) != 0:
+                return None
+        except TypeError:
+            return None
+        best_index = min(range(len(safe_local[0])), key=safe_local[0].__getitem__)
+        best_score = safe_local[0][best_index]
+        if not math.isfinite(best_score):
+            return None
+        return CandidateSequenceSelection((best_index,), best_score)
+
+    try:
+        transitions = [[list(row) for row in matrix] for matrix in transition_scores]
+    except TypeError:
+        return None
+    if len(transitions) != len(safe_local) - 1:
+        return None
+    safe_transitions: list[list[list[float]]] = []
+    for index, matrix in enumerate(transitions):
+        previous_count = len(safe_local[index])
+        current_count = len(safe_local[index + 1])
+        if len(matrix) != previous_count or any(len(row) != current_count for row in matrix):
+            return None
+        safe_transitions.append(
+            [
+                [
+                    score if score is not None else math.inf
+                    for score in (_finite_number(value, minimum=0.0) for value in row)
+                ]
+                for row in matrix
+            ]
+        )
+
+    previous_costs = safe_local[0]
+    backpointers: list[list[int]] = []
+    for step in range(1, len(safe_local)):
+        current_costs = [math.inf] * len(safe_local[step])
+        current_backpointers = [-1] * len(safe_local[step])
+        for current_index, local_score in enumerate(safe_local[step]):
+            if not math.isfinite(local_score):
+                continue
+            for previous_index, previous_score in enumerate(previous_costs):
+                edge_score = safe_transitions[step - 1][previous_index][current_index]
+                if not math.isfinite(previous_score) or not math.isfinite(edge_score):
+                    continue
+                total = previous_score + edge_score + local_score
+                if math.isfinite(total) and total < current_costs[current_index]:
+                    current_costs[current_index] = total
+                    current_backpointers[current_index] = previous_index
+        previous_costs = current_costs
+        backpointers.append(current_backpointers)
+
+    final_index = min(range(len(previous_costs)), key=previous_costs.__getitem__)
+    total_score = previous_costs[final_index]
+    if not math.isfinite(total_score):
+        return None
+    indices = [final_index]
+    for pointers in reversed(backpointers):
+        final_index = pointers[final_index]
+        if final_index < 0:
+            return None
+        indices.append(final_index)
+    indices.reverse()
+    return CandidateSequenceSelection(tuple(indices), total_score)
+
+
 @torch.no_grad()
 def extract_windowed_speaker_embedding(
     wav_path: str,

quality_runtime.py

@@ -0,0 +1,1610 @@
+"""Fail-closed quality runtime helpers for the BlueMagpie-TTS Space.
+
+The module deliberately has no import-time model downloads.  Both Whisper and
+ECAPA are supplied through small injectable boundaries so the online Space can
+load the real models lazily while unit tests remain deterministic and offline.
+"""
+
+from __future__ import annotations
+
+import math
+import operator
+import threading
+from dataclasses import dataclass
+from math import gcd
+from typing import Any, Callable, Sequence
+
+import numpy as np
+import torch
+import torch.nn.functional as torch_functional
+
+from production import AsrComparison, CandidateSequenceSelection, compare_asr_text
+
+
+WHISPER_MODEL_ID = "openai/whisper-large-v3-turbo"
+WHISPER_REVISION = "41f01f3fe87f28c78e2fbf8b568835947dd65ed9"
+WHISPER_SAMPLE_RATE = 16_000
+WHISPER_MAX_SEGMENT_SECONDS = 28.0
+REQUEST_SEED_LIMIT = 2**31
+ACTIVE_VOICE_TOP_DB = 35.0
+ACTIVE_VOICE_FRAME_MS = 25.0
+ACTIVE_VOICE_HOP_MS = 10.0
+ACTIVE_VOICE_MIN_RMS = 1.0e-4
+
+
+@dataclass(frozen=True)
+class GenerationPolicy:
+    """Candidate-specific endpoint duration estimate used by the Space."""
+
+    name: str
+    cjk_cps: float
+    ascii_cps: float
+    hard_stop_margin_steps: int
+
+
+BASE_GENERATION_POLICY = GenerationPolicy(
+    name="base",
+    cjk_cps=5.2,
+    ascii_cps=4.6,
+    hard_stop_margin_steps=1,
+)
+SAFE_DURATION_GENERATION_POLICY = GenerationPolicy(
+    name="safe_duration",
+    cjk_cps=4.6,
+    ascii_cps=4.0,
+    hard_stop_margin_steps=1,
+)
+
+
+def generation_policy_for_candidate_offset(candidate_offset: int) -> GenerationPolicy:
+    """Map candidate zero to base and every retry to the safe estimate.
+
+    The policy changes only the native-duration endpoint estimate.  It is not a
+    minimum-length policy and therefore never holds the generation loop open to
+    enforce playback pace.
+    """
+
+    if isinstance(candidate_offset, (bool, np.bool_)):
+        raise ValueError("candidate_offset must be a non-negative integer")
+    try:
+        offset = operator.index(candidate_offset)
+    except (TypeError, ValueError, OverflowError) as error:
+        raise ValueError("candidate_offset must be a non-negative integer") from error
+    if offset < 0:
+        raise ValueError("candidate_offset must be a non-negative integer")
+    return BASE_GENERATION_POLICY if offset == 0 else SAFE_DURATION_GENERATION_POLICY
+
+
+def resolve_request_seed(
+    request_seed: int | None,
+    random_seed_factory: Callable[[int], int],
+) -> int:
+    """Return a validated root seed, drawing randomness only for ``None``."""
+
+    candidate = (
+        random_seed_factory(REQUEST_SEED_LIMIT)
+        if request_seed is None
+        else request_seed
+    )
+    if isinstance(candidate, (bool, np.bool_)):
+        raise ValueError(f"request_seed must be an integer in [0, {REQUEST_SEED_LIMIT})")
+    try:
+        # ``operator.index`` semantics reject floats and numeric strings while
+        # accepting Python and NumPy integer scalars.
+        seed = operator.index(candidate)
+    except (AttributeError, TypeError, ValueError, OverflowError) as error:
+        raise ValueError(
+            f"request_seed must be an integer in [0, {REQUEST_SEED_LIMIT})"
+        ) from error
+    seed = int(seed)
+    if not 0 <= seed < REQUEST_SEED_LIMIT:
+        raise ValueError(f"request_seed must be an integer in [0, {REQUEST_SEED_LIMIT})")
+    return seed
+
+
+def _finite_float(
+    value: Any,
+    *,
+    minimum: float | None = None,
+    maximum: float | None = None,
+) -> float | None:
+    if isinstance(value, (bool, np.bool_)):
+        return None
+    try:
+        result = float(value)
+    except (TypeError, ValueError, OverflowError):
+        return None
+    if not math.isfinite(result):
+        return None
+    if minimum is not None and result < minimum:
+        return None
+    if maximum is not None and result > maximum:
+        return None
+    return result
+
+
+def _mono_audio(audio: np.ndarray | Sequence[float]) -> np.ndarray:
+    """Return contiguous mono float32 audio, rejecting ambiguous/bad inputs."""
+
+    waveform = np.asarray(audio)
+    if waveform.ndim == 1:
+        pass
+    elif waveform.ndim == 2:
+        first, second = waveform.shape
+        if first <= 8 and second > first:
+            waveform = waveform.mean(axis=0)
+        elif second <= 8 and first > second:
+            waveform = waveform.mean(axis=1)
+        else:
+            raise ValueError("2-D audio must have an identifiable channel axis (at most 8 channels)")
+    else:
+        raise ValueError("audio must be a one- or two-dimensional array")
+    waveform = np.asarray(waveform, dtype=np.float32).reshape(-1)
+    if waveform.size == 0:
+        raise ValueError("audio is empty")
+    if not np.isfinite(waveform).all():
+        raise ValueError("audio contains non-finite samples")
+    return np.ascontiguousarray(waveform)
+
+
+def _resample_audio(audio: np.ndarray, sample_rate: int, target_rate: int) -> np.ndarray:
+    try:
+        source_rate = int(sample_rate)
+        destination_rate = int(target_rate)
+    except (TypeError, ValueError, OverflowError) as error:
+        raise ValueError("sample rates must be positive integers") from error
+    if source_rate <= 0 or destination_rate <= 0:
+        raise ValueError("sample rates must be positive integers")
+    if source_rate == destination_rate:
+        return np.ascontiguousarray(audio, dtype=np.float32)
+
+    # SpeechBrain already depends on SciPy.  ``resample_poly`` avoids an
+    # undeclared optional ``librosa`` resampler dependency in the Space image.
+    from scipy.signal import resample_poly
+
+    common_divisor = gcd(source_rate, destination_rate)
+    output = resample_poly(
+        np.asarray(audio, dtype=np.float32),
+        destination_rate // common_divisor,
+        source_rate // common_divisor,
+    )
+    output = np.asarray(output, dtype=np.float32).reshape(-1)
+    if output.size == 0 or not np.isfinite(output).all():
+        raise ValueError("resampling produced invalid audio")
+    return np.ascontiguousarray(output)
+
+
+def _trim_active_speech(
+    audio: np.ndarray,
+    *,
+    top_db: float = 35.0,
+    frame_length: int = 512,
+    hop_length: int = 128,
+) -> np.ndarray:
+    threshold_db = _finite_float(top_db, minimum=0.0)
+    if threshold_db is None:
+        raise ValueError("top_db must be finite and non-negative")
+    if float(np.max(np.abs(audio))) <= 1.0e-7:
+        raise ValueError("audio contains no active speech")
+
+    import librosa
+
+    intervals = librosa.effects.split(
+        audio,
+        top_db=threshold_db,
+        frame_length=max(32, int(frame_length)),
+        hop_length=max(1, int(hop_length)),
+    )
+    if intervals.size == 0:
+        raise ValueError("audio contains no active speech")
+    start = int(intervals[0, 0])
+    stop = int(intervals[-1, 1])
+    active = np.asarray(audio[start:stop], dtype=np.float32)
+    if active.size == 0 or float(np.max(np.abs(active))) <= 1.0e-7:
+        raise ValueError("audio contains no active speech")
+    return np.ascontiguousarray(active)
+
+
+def active_voiced_intervals(
+    audio: np.ndarray | Sequence[float],
+    sample_rate: int,
+    *,
+    top_db: float = ACTIVE_VOICE_TOP_DB,
+    frame_ms: float = ACTIVE_VOICE_FRAME_MS,
+    hop_ms: float = ACTIVE_VOICE_HOP_MS,
+    min_rms: float = ACTIVE_VOICE_MIN_RMS,
+) -> tuple[tuple[int, int], ...]:
+    """Return the deterministic union of active RMS-frame intervals.
+
+    This is the same 25 ms / 10 ms, peak-minus-35 dB, 1e-4 floor contract
+    used by the independent hosted evaluator.  Unlike first-to-last trimming,
+    the interval union excludes internal punctuation and joining pauses from
+    both online pace evidence and the speaker-gate duration threshold.
+    """
+
+    signal = _mono_audio(audio)
+    if isinstance(sample_rate, (bool, np.bool_)):
+        raise ValueError("sample rate must be positive")
+    try:
+        source_rate = operator.index(sample_rate)
+    except (TypeError, ValueError, OverflowError) as error:
+        raise ValueError("sample rate must be positive") from error
+    if source_rate <= 0:
+        raise ValueError("sample rate must be positive")
+    frame_duration = _finite_float(frame_ms, minimum=0.0)
+    hop_duration = _finite_float(hop_ms, minimum=0.0)
+    rms_floor = _finite_float(min_rms, minimum=0.0)
+    relative_db = _finite_float(top_db, minimum=0.0)
+    if (
+        frame_duration is None
+        or frame_duration <= 0.0
+        or hop_duration is None
+        or hop_duration <= 0.0
+        or rms_floor is None
+        or rms_floor <= 0.0
+        or relative_db is None
+    ):
+        raise ValueError("active-voice detector settings are invalid")
+
+    frame = max(1, int(round(frame_duration * source_rate / 1000.0)))
+    hop = max(1, int(round(hop_duration * source_rate / 1000.0)))
+    if signal.size <= frame:
+        starts = np.asarray([0], dtype=np.int64)
+    else:
+        starts = np.arange(0, signal.size - frame + 1, hop, dtype=np.int64)
+        final_start = signal.size - frame
+        if int(starts[-1]) != final_start:
+            starts = np.append(starts, final_start)
+    rms = np.asarray(
+        [
+            float(
+                np.sqrt(
+                    np.mean(
+                        np.square(
+                            signal[int(start) : int(start) + frame],
+                            dtype=np.float64,
+                        )
+                    )
+                )
+            )
+            for start in starts
+        ],
+        dtype=np.float64,
+    )
+    peak = float(rms.max(initial=0.0))
+    threshold = max(rms_floor, peak * 10.0 ** (-relative_db / 20.0))
+    active_starts = starts[rms >= threshold]
+    intervals: list[list[int]] = []
+    for raw_start in active_starts:
+        start = int(raw_start)
+        end = min(signal.size, start + frame)
+        if intervals and start <= intervals[-1][1]:
+            intervals[-1][1] = max(intervals[-1][1], end)
+        else:
+            intervals.append([start, end])
+    return tuple((start, end) for start, end in intervals)
+
+
+def active_voiced_duration_seconds(
+    audio: np.ndarray | Sequence[float],
+    sample_rate: int,
+    **detector_kwargs: Any,
+) -> float:
+    """Measure active interval-union duration under the hosted gate contract."""
+
+    intervals = active_voiced_intervals(audio, sample_rate, **detector_kwargs)
+    active_samples = sum(end - start for start, end in intervals)
+    return active_samples / float(operator.index(sample_rate))
+
+
+@dataclass(frozen=True)
+class WhisperRuntime:
+    """Loaded processor/model pair for deterministic Whisper transcription."""
+
+    processor: Any
+    model: Any
+    device: torch.device
+    dtype: torch.dtype
+
+
+def load_pinned_whisper_runtime(
+    *,
+    device: str | torch.device | None = None,
+    processor_factory: Any | None = None,
+    model_factory: Any | None = None,
+) -> WhisperRuntime:
+    """Load the exact ASR revision used by the quality gate.
+
+    Factory injection exists for offline tests.  The default imports
+    ``transformers`` only when this function is first called.
+    """
+
+    if processor_factory is None or model_factory is None:
+        from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor
+
+        processor_factory = processor_factory or AutoProcessor
+        model_factory = model_factory or AutoModelForSpeechSeq2Seq
+
+    selected_device = torch.device(
+        device if device is not None else ("cuda" if torch.cuda.is_available() else "cpu")
+    )
+    dtype = torch.float16 if selected_device.type == "cuda" else torch.float32
+    processor = processor_factory.from_pretrained(
+        WHISPER_MODEL_ID,
+        revision=WHISPER_REVISION,
+    )
+    model = model_factory.from_pretrained(
+        WHISPER_MODEL_ID,
+        revision=WHISPER_REVISION,
+        torch_dtype=dtype,
+        low_cpu_mem_usage=True,
+        use_safetensors=True,
+    )
+    model = model.to(selected_device)
+    model.eval()
+    return WhisperRuntime(
+        processor=processor,
+        model=model,
+        device=selected_device,
+        dtype=dtype,
+    )
+
+
+class LazyWhisperASR:
+    """Thread-safe one-shot lazy loader with an injectable runtime factory."""
+
+    def __init__(self, runtime_loader: Callable[[], WhisperRuntime] | None = None) -> None:
+        self._runtime_loader = runtime_loader or load_pinned_whisper_runtime
+        self._runtime: WhisperRuntime | None = None
+        self._lock = threading.Lock()
+
+    def get_runtime(self) -> WhisperRuntime:
+        runtime = self._runtime
+        if runtime is not None:
+            return runtime
+        with self._lock:
+            if self._runtime is None:
+                self._runtime = self._runtime_loader()
+            return self._runtime
+
+
+_DEFAULT_WHISPER = LazyWhisperASR()
+
+
+def _split_whisper_audio(
+    waveform: np.ndarray,
+    *,
+    sample_rate: int = WHISPER_SAMPLE_RATE,
+    max_segment_seconds: float = WHISPER_MAX_SEGMENT_SECONDS,
+    boundary_search_seconds: float = 1.5,
+) -> tuple[np.ndarray, ...]:
+    """Split long audio near low-energy boundaries below Whisper's 30s cap."""
+
+    maximum_seconds = _finite_float(max_segment_seconds, minimum=1.0)
+    search_seconds = _finite_float(boundary_search_seconds, minimum=0.0)
+    if maximum_seconds is None or search_seconds is None or sample_rate <= 0:
+        raise ValueError("invalid Whisper segmentation settings")
+    maximum_samples = max(1, int(round(maximum_seconds * sample_rate)))
+    if waveform.size <= maximum_samples:
+        return (waveform,)
+
+    segment_count = int(math.ceil(waveform.size / maximum_samples))
+    ideal_samples = waveform.size / float(segment_count)
+    search_samples = int(round(search_seconds * sample_rate))
+    probe_radius = max(1, int(round(0.02 * sample_rate)))
+    probe_hop = max(1, int(round(0.02 * sample_rate)))
+    boundaries = [0]
+    for boundary_index in range(1, segment_count):
+        target = int(round(boundary_index * ideal_samples))
+        lower = max(boundaries[-1] + probe_radius, target - search_samples)
+        upper = min(waveform.size - probe_radius, target + search_samples)
+        if lower >= upper:
+            boundary = target
+        else:
+            probes = range(lower, upper + 1, probe_hop)
+            boundary = min(
+                probes,
+                key=lambda index: float(
+                    np.mean(
+                        np.square(
+                            waveform[index - probe_radius : index + probe_radius],
+                            dtype=np.float64,
+                        )
+                    )
+                ),
+            )
+        boundaries.append(boundary)
+    boundaries.append(waveform.size)
+    segments = tuple(
+        np.ascontiguousarray(waveform[start:stop], dtype=np.float32)
+        for start, stop in zip(boundaries, boundaries[1:])
+    )
+    if (
+        not segments
+        or any(segment.size == 0 for segment in segments)
+        or any(segment.size > int(round(30.0 * sample_rate)) for segment in segments)
+    ):
+        raise ValueError("failed to split audio within Whisper's segment limit")
+    return segments
+
+
+@torch.inference_mode()
+def transcribe_whisper(
+    audio: np.ndarray | Sequence[float],
+    sample_rate: int,
+    *,
+    lazy_asr: LazyWhisperASR | None = None,
+    runtime: WhisperRuntime | None = None,
+    language: str = "zh",
+    task: str = "transcribe",
+    max_new_tokens: int = 128,
+) -> str:
+    """Transcribe ndarray audio using deterministic decoding.
+
+    ``runtime`` and ``lazy_asr`` are mutually exclusive injection points.  An
+    empty decoded string is returned as-is; the semantic verifier will reject
+    it rather than accepting an arbitrary TTS fallback.
+    """
+
+    if runtime is not None and lazy_asr is not None:
+        raise ValueError("pass either runtime or lazy_asr, not both")
+    token_limit = int(max_new_tokens)
+    if token_limit <= 0:
+        raise ValueError("max_new_tokens must be positive")
+    waveform = _resample_audio(_mono_audio(audio), int(sample_rate), WHISPER_SAMPLE_RATE)
+    segments = _split_whisper_audio(waveform)
+    selected_runtime = runtime or (lazy_asr or _DEFAULT_WHISPER).get_runtime()
+    processor_input: np.ndarray | list[np.ndarray]
+    processor_input = segments[0] if len(segments) == 1 else list(segments)
+    features = selected_runtime.processor(
+        processor_input,
+        sampling_rate=WHISPER_SAMPLE_RATE,
+        return_tensors="pt",
+    ).input_features
+    features = features.to(device=selected_runtime.device, dtype=selected_runtime.dtype)
+    token_ids = selected_runtime.model.generate(
+        features,
+        language=language,
+        task=task,
+        do_sample=False,
+        num_beams=1,
+        max_new_tokens=token_limit,
+    )
+    decoded = selected_runtime.processor.batch_decode(token_ids, skip_special_tokens=True)
+    if not decoded or len(decoded) != len(segments):
+        return ""
+    return " ".join(str(text).strip() for text in decoded if str(text).strip())
+
+
+@dataclass(frozen=True)
+class PreparedCandidateAudio:
+    """Validated candidate waveform and its ASR transcript."""
+
+    waveform: np.ndarray
+    duration_seconds: float
+    transcript_text: str
+
+
+def prepare_candidate_audio(
+    audio: np.ndarray | Sequence[float],
+    sample_rate: int,
+    *,
+    transcriber: Callable[[np.ndarray, int], str] | None = None,
+) -> PreparedCandidateAudio | None:
+    """Prepare one candidate, returning ``None`` for candidate-data errors.
+
+    Invalid/empty/non-finite audio and a transcriber's ``ValueError`` describe
+    an unusable candidate, not a service outage.  They therefore become a
+    normal gate rejection so the cascade can try the next seed.  Runtime and
+    I/O failures deliberately propagate and abort the request fail-closed.
+    """
+
+    try:
+        selected_sample_rate = int(sample_rate)
+        if selected_sample_rate <= 0:
+            raise ValueError("sample_rate must be positive")
+        waveform = _mono_audio(audio)
+        transcript = (transcriber or transcribe_whisper)(
+            waveform,
+            selected_sample_rate,
+        )
+        if not isinstance(transcript, str):
+            raise ValueError("ASR transcript must be a string")
+    except (TypeError, ValueError, OverflowError):
+        return None
+    return PreparedCandidateAudio(
+        waveform=waveform,
+        duration_seconds=waveform.size / float(selected_sample_rate),
+        transcript_text=transcript.strip(),
+    )
+
+
+@torch.inference_mode()
+def _encode_speaker_segments(
+    segments: Sequence[np.ndarray],
+    encoder: Any,
+    *,
+    device: str | torch.device,
+) -> np.ndarray:
+    """Encode a variable-length segment batch in one ECAPA forward pass."""
+
+    waveforms = tuple(np.asarray(segment, dtype=np.float32).reshape(-1) for segment in segments)
+    if not waveforms or any(
+        waveform.size == 0 or not np.isfinite(waveform).all()
+        for waveform in waveforms
+    ):
+        raise ValueError("speaker segments must be non-empty and finite")
+    maximum_length = max(waveform.size for waveform in waveforms)
+    batch = np.zeros((len(waveforms), maximum_length), dtype=np.float32)
+    relative_lengths = np.empty(len(waveforms), dtype=np.float32)
+    for index, waveform in enumerate(waveforms):
+        batch[index, : waveform.size] = waveform
+        relative_lengths[index] = waveform.size / float(maximum_length)
+
+    selected_device = torch.device(device)
+    tensor = torch.from_numpy(batch).to(selected_device)
+    wav_lens = torch.from_numpy(relative_lengths).to(selected_device)
+    embeddings = encoder.encode_batch(tensor, wav_lens=wav_lens)
+    embeddings = torch.as_tensor(embeddings).detach().float()
+    if embeddings.ndim == 0 or embeddings.shape[0] != len(waveforms):
+        raise ValueError("speaker encoder returned an invalid batch size")
+    embeddings = embeddings.reshape(len(waveforms), -1)
+    if embeddings.shape[1] == 0 or not torch.isfinite(embeddings).all():
+        raise ValueError("speaker encoder returned an invalid embedding")
+    norms = torch.linalg.vector_norm(embeddings, dim=1)
+    if not torch.isfinite(norms).all() or bool(torch.any(norms <= 1.0e-8)):
+        raise ValueError("speaker encoder returned a zero-norm embedding")
+    normalized = torch_functional.normalize(embeddings, dim=1).cpu().numpy().astype(np.float32)
+    if not np.isfinite(normalized).all():
+        raise ValueError("speaker encoder returned a non-finite embedding")
+    return normalized
+
+
+@torch.inference_mode()
+def speaker_embedding_from_audio(
+    audio: np.ndarray | Sequence[float],
+    sample_rate: int,
+    encoder: Any,
+    *,
+    device: str | torch.device = "cpu",
+    target_sample_rate: int = 16_000,
+    active_top_db: float = 35.0,
+) -> np.ndarray:
+    """Extract one normalized ECAPA embedding from active ndarray speech."""
+
+    waveform = _mono_audio(audio)
+    waveform = _resample_audio(waveform, int(sample_rate), int(target_sample_rate))
+    waveform = _trim_active_speech(waveform, top_db=active_top_db)
+    return _encode_speaker_segments((waveform,), encoder, device=device)[0]
+
+
+def cosine_similarity(left: np.ndarray | Sequence[float], right: np.ndarray | Sequence[float]) -> float:
+    """Return a finite cosine similarity, raising on unusable embeddings."""
+
+    left_array = np.asarray(left, dtype=np.float64).reshape(-1)
+    right_array = np.asarray(right, dtype=np.float64).reshape(-1)
+    if left_array.size == 0 or left_array.shape != right_array.shape:
+        raise ValueError("speaker embeddings must have equal non-empty shapes")
+    if not np.isfinite(left_array).all() or not np.isfinite(right_array).all():
+        raise ValueError("speaker embeddings must be finite")
+    denominator = float(np.linalg.norm(left_array) * np.linalg.norm(right_array))
+    if not math.isfinite(denominator) or denominator <= 1.0e-12:
+        raise ValueError("speaker embeddings must have non-zero norm")
+    similarity = float(np.dot(left_array, right_array) / denominator)
+    if not math.isfinite(similarity):
+        raise ValueError("speaker cosine similarity is non-finite")
+    return float(np.clip(similarity, -1.0, 1.0))
+
+
+@dataclass(frozen=True)
+class SpeakerEvidence:
+    similarity: float
+    begin_similarity: float
+    end_similarity: float
+    boundary_drop: float
+    active_duration_seconds: float
+    speaker_embedding: np.ndarray
+    active_rms_db: float
+
+
+def speaker_evidence_from_audio(
+    audio: np.ndarray | Sequence[float],
+    sample_rate: int,
+    encoder: Any,
+    anchor_embedding: np.ndarray | Sequence[float],
+    *,
+    device: str | torch.device = "cpu",
+    edge_seconds: float = 1.5,
+    whole_window_seconds: float = 3.0,
+    whole_max_windows: int = 4,
+    active_top_db: float = 35.0,
+) -> SpeakerEvidence:
+    """Measure whole/begin/end anchor similarity on trimmed active speech."""
+
+    edge_duration = _finite_float(edge_seconds, minimum=0.01)
+    window_duration = _finite_float(whole_window_seconds, minimum=0.01)
+    try:
+        max_windows = int(whole_max_windows)
+    except (TypeError, ValueError, OverflowError):
+        max_windows = 0
+    if edge_duration is None or window_duration is None or max_windows <= 0:
+        raise ValueError("speaker window settings must be finite and positive")
+    waveform = _resample_audio(_mono_audio(audio), int(sample_rate), 16_000)
+    active_duration_seconds = active_voiced_duration_seconds(
+        waveform,
+        16_000,
+        top_db=active_top_db,
+    )
+    active = _trim_active_speech(waveform, top_db=active_top_db)
+    edge_samples = max(1, int(round(edge_duration * 16_000)))
+    whole_window_samples = max(1, int(round(window_duration * 16_000)))
+    begin = active[:edge_samples]
+    end = active[-edge_samples:]
+    if active.size <= whole_window_samples:
+        whole_segments = [active]
+    else:
+        starts = np.linspace(
+            0,
+            active.size - whole_window_samples,
+            num=max_windows,
+        ).round().astype(int)
+        whole_segments = [
+            active[start : start + whole_window_samples]
+            for start in dict.fromkeys(starts.tolist())
+        ]
+    embeddings = _encode_speaker_segments(
+        (*whole_segments, begin, end),
+        encoder,
+        device=device,
+    )
+    whole_embeddings = embeddings[: len(whole_segments)]
+    whole_embedding = np.mean(whole_embeddings, axis=0, dtype=np.float64)
+    whole_norm = float(np.linalg.norm(whole_embedding))
+    if not math.isfinite(whole_norm) or whole_norm <= 1.0e-8:
+        raise ValueError("speaker windows produced a zero-norm embedding")
+    whole_embedding = np.asarray(whole_embedding / whole_norm, dtype=np.float32)
+    begin_embedding, end_embedding = embeddings[-2:]
+    similarity = cosine_similarity(whole_embedding, anchor_embedding)
+    begin_similarity = cosine_similarity(begin_embedding, anchor_embedding)
+    end_similarity = cosine_similarity(end_embedding, anchor_embedding)
+    boundary_drop = max(0.0, begin_similarity - end_similarity)
+    active_rms = float(np.sqrt(np.mean(np.square(active, dtype=np.float64))))
+    if not math.isfinite(active_rms) or active_rms <= 1.0e-8:
+        raise ValueError("active speech has invalid RMS")
+    return SpeakerEvidence(
+        similarity=similarity,
+        begin_similarity=begin_similarity,
+        end_similarity=end_similarity,
+        boundary_drop=boundary_drop,
+        active_duration_seconds=active_duration_seconds,
+        speaker_embedding=whole_embedding.copy(),
+        active_rms_db=20.0 * math.log10(active_rms),
+    )
+
+
+def active_audio_rms_db(audio: np.ndarray | Sequence[float], *, top_db: float = 35.0) -> float:
+    """Measure finite RMS dB on the active region of candidate audio."""
+
+    active = _trim_active_speech(_mono_audio(audio), top_db=top_db)
+    rms = float(np.sqrt(np.mean(np.square(active, dtype=np.float64))))
+    if not math.isfinite(rms) or rms <= 1.0e-8:
+        raise ValueError("active speech has invalid RMS")
+    return 20.0 * math.log10(rms)
+
+
+@dataclass(frozen=True)
+class CandidateObservation:
+    target_text: str
+    transcript_text: str
+    audio_duration_seconds: float
+    speaker_similarity: float | None = None
+    begin_speaker_similarity: float | None = None
+    end_speaker_similarity: float | None = None
+    pace_cps: float | None = None
+    truncated: bool = False
+
+
+@dataclass(frozen=True)
+class CandidateGateResult:
+    passed: bool
+    comparison: AsrComparison
+    speaker_gate_applied: bool
+    speaker_similarity: float | None
+    boundary_speaker_drop: float | None
+    pace_cps: float | None
+    score: float
+    rejection_reasons: tuple[str, ...]
+
+
+@dataclass(frozen=True)
+class ChunkCandidateArtifact:
+    """Acoustic evidence retained for sequence-level candidate selection."""
+
+    speaker_embedding: np.ndarray | None = None
+    rms_db: float | None = None
+    median_f0_hz: float | None = None
+
+
+def verify_candidate(
+    observation: CandidateObservation,
+    *,
+    locale: str = "zh-TW",
+    short_text_units: int = 6,
+    short_text_max_cer: float = 0.0,
+    max_cer: float = 0.20,
+    prefix_units: int = 6,
+    suffix_units: int = 6,
+    max_prefix_cer: float = 0.0,
+    max_suffix_cer: float = 0.0,
+    max_extra_tail_units: int = 0,
+    short_audio_seconds: float = 1.5,
+    min_speaker_similarity: float = 0.10,
+    max_boundary_speaker_drop: float = 0.03,
+    max_pace_cps: float | None = None,
+    speaker_weight: float = 0.05,
+    boundary_weight: float = 0.10,
+) -> CandidateGateResult:
+    """Apply strict semantic and duration-aware speaker gates to a candidate."""
+
+    duration = _finite_float(observation.audio_duration_seconds, minimum=0.0)
+    short_duration_limit = _finite_float(short_audio_seconds, minimum=0.0)
+    general_cer_limit = _finite_float(max_cer, minimum=0.0)
+    exact_cer_limit = _finite_float(short_text_max_cer, minimum=0.0)
+    min_similarity = _finite_float(min_speaker_similarity, minimum=-1.0, maximum=1.0)
+    max_boundary = _finite_float(max_boundary_speaker_drop, minimum=0.0)
+    max_pace = None if max_pace_cps is None else _finite_float(max_pace_cps, minimum=0.0)
+    speaker_cost_weight = _finite_float(speaker_weight, minimum=0.0)
+    boundary_cost_weight = _finite_float(boundary_weight, minimum=0.0)
+    try:
+        short_unit_limit = max(0, int(short_text_units))
+    except (TypeError, ValueError, OverflowError):
+        short_unit_limit = -1
+
+    # First normalize with a permissive finite limit to determine target units.
+    preliminary = compare_asr_text(
+        observation.target_text,
+        observation.transcript_text,
+        locale=locale,
+        prefix_units=prefix_units,
+        suffix_units=suffix_units,
+        max_cer=general_cer_limit if general_cer_limit is not None else math.nan,
+        max_prefix_cer=max_prefix_cer,
+        max_suffix_cer=max_suffix_cer,
+        max_extra_tail_units=max_extra_tail_units,
+    )
+    selected_cer_limit = general_cer_limit
+    if short_unit_limit >= 0 and len(preliminary.target_text) <= short_unit_limit:
+        selected_cer_limit = exact_cer_limit
+    comparison = compare_asr_text(
+        observation.target_text,
+        observation.transcript_text,
+        locale=locale,
+        prefix_units=prefix_units,
+        suffix_units=suffix_units,
+        max_cer=selected_cer_limit if selected_cer_limit is not None else math.nan,
+        max_prefix_cer=max_prefix_cer,
+        max_suffix_cer=max_suffix_cer,
+        max_extra_tail_units=max_extra_tail_units,
+    )
+
+    reasons: list[str] = []
+    if duration is None or duration <= 0.0:
+        reasons.append("invalid_audio_duration")
+    if observation.truncated is not False:
+        reasons.append("truncated")
+    if not comparison.passed:
+        reasons.append("semantic_gate")
+    pace = None
+    if max_pace_cps is not None:
+        pace = _finite_float(observation.pace_cps, minimum=0.0)
+        if max_pace is None:
+            reasons.append("invalid_gate_config")
+        elif pace is None:
+            reasons.append("missing_pace_evidence")
+        elif pace > max_pace:
+            reasons.append("pace_too_fast")
+
+    valid_common_config = all(
+        value is not None
+        for value in (
+            short_duration_limit,
+            general_cer_limit,
+            exact_cer_limit,
+            min_similarity,
+            max_boundary,
+            speaker_cost_weight,
+            boundary_cost_weight,
+        )
+    ) and short_unit_limit >= 0
+    if not valid_common_config:
+        reasons.append("invalid_gate_config")
+
+    speaker_gate_applied = bool(
+        duration is not None
+        and short_duration_limit is not None
+        and duration >= short_duration_limit
+    )
+    similarity: float | None = None
+    boundary_drop: float | None = None
+    if speaker_gate_applied:
+        similarity = _finite_float(
+            observation.speaker_similarity,
+            minimum=-1.0,
+            maximum=1.0,
+        )
+        begin_similarity = _finite_float(
+            observation.begin_speaker_similarity,
+            minimum=-1.0,
+            maximum=1.0,
+        )
+        end_similarity = _finite_float(
+            observation.end_speaker_similarity,
+            minimum=-1.0,
+            maximum=1.0,
+        )
+        if similarity is None or begin_similarity is None or end_similarity is None:
+            reasons.append("missing_speaker_evidence")
+        else:
+            boundary_drop = max(0.0, begin_similarity - end_similarity)
+            if min_similarity is None or similarity < min_similarity:
+                reasons.append("speaker_similarity")
+            if max_boundary is None or boundary_drop > max_boundary:
+                reasons.append("boundary_speaker_drop")
+
+    score = math.inf
+    if not reasons:
+        score = comparison.cer
+        if speaker_gate_applied:
+            assert similarity is not None and boundary_drop is not None
+            assert speaker_cost_weight is not None and boundary_cost_weight is not None
+            score += speaker_cost_weight * (1.0 - similarity)
+            score += boundary_cost_weight * boundary_drop
+        if not math.isfinite(score) or score < 0.0:
+            reasons.append("nonfinite_score")
+            score = math.inf
+
+    return CandidateGateResult(
+        passed=not reasons,
+        comparison=comparison,
+        speaker_gate_applied=speaker_gate_applied,
+        speaker_similarity=similarity,
+        boundary_speaker_drop=boundary_drop,
+        pace_cps=pace,
+        score=score,
+        rejection_reasons=tuple(reasons),
+    )
+
+
+@dataclass(frozen=True)
+class TrajectoryGateResult:
+    passed: bool
+    candidate_results: tuple[CandidateGateResult, ...]
+    score: float
+    rejection_reasons: tuple[str, ...]
+    chunk_artifacts: tuple[ChunkCandidateArtifact, ...] = ()
+
+
+def verify_trajectory(
+    observations: Sequence[CandidateObservation],
+    *,
+    chunk_artifacts: Sequence[ChunkCandidateArtifact] = (),
+    **candidate_gate_kwargs: Any,
+) -> TrajectoryGateResult:
+    """Require every chunk in a non-empty trajectory to pass all hard gates."""
+
+    try:
+        candidates = tuple(observations)
+    except TypeError:
+        candidates = ()
+    if not candidates:
+        return TrajectoryGateResult(False, (), math.inf, ("empty_trajectory",))
+
+    try:
+        artifacts = tuple(chunk_artifacts)
+    except TypeError:
+        artifacts = ()
+    if artifacts and (
+        len(artifacts) != len(candidates)
+        or any(not isinstance(artifact, ChunkCandidateArtifact) for artifact in artifacts)
+    ):
+        return TrajectoryGateResult(
+            False,
+            (),
+            math.inf,
+            ("invalid_chunk_artifacts",),
+        )
+
+    results: list[CandidateGateResult] = []
+    rejection_reasons: list[str] = []
+    for index, observation in enumerate(candidates):
+        try:
+            result = verify_candidate(observation, **candidate_gate_kwargs)
+        except (TypeError, ValueError, OverflowError):
+            # A malformed observation must reject the entire trajectory.
+            comparison = compare_asr_text("", "")
+            result = CandidateGateResult(
+                passed=False,
+                comparison=comparison,
+                speaker_gate_applied=False,
+                speaker_similarity=None,
+                boundary_speaker_drop=None,
+                pace_cps=None,
+                score=math.inf,
+                rejection_reasons=("malformed_observation",),
+            )
+        results.append(result)
+        rejection_reasons.extend(f"chunk_{index}:{reason}" for reason in result.rejection_reasons)
+    passed = bool(results) and all(result.passed for result in results)
+    score = sum(result.score for result in results) if passed else math.inf
+    if not math.isfinite(score):
+        passed = False
+        score = math.inf
+        if not rejection_reasons:
+            rejection_reasons.append("nonfinite_trajectory_score")
+    return TrajectoryGateResult(
+        passed=passed,
+        candidate_results=tuple(results),
+        score=score,
+        rejection_reasons=tuple(rejection_reasons),
+        chunk_artifacts=artifacts,
+    )
+
+
+def qualify_trajectory_with_joined_output(
+    local_verification: TrajectoryGateResult,
+    joined_verification: TrajectoryGateResult,
+) -> TrajectoryGateResult:
+    """Require joined-output safety without discarding DP-local evidence.
+
+    ``candidate_results`` and ``chunk_artifacts`` always stay local to the
+    generated chunks.  This lets sequence DP reuse individually safe chunks
+    when RMS matching, fades, pauses or crossfade make the same-seed joined
+    waveform fail its whole-output gate.
+    """
+
+    if not isinstance(local_verification, TrajectoryGateResult):
+        raise TypeError("local_verification must be a TrajectoryGateResult")
+    if local_verification.passed is not True or not math.isfinite(
+        local_verification.score
+    ):
+        return local_verification
+    if not isinstance(joined_verification, TrajectoryGateResult):
+        raise TypeError("joined_verification must be a TrajectoryGateResult")
+
+    joined_passed = bool(
+        joined_verification.passed is True
+        and math.isfinite(joined_verification.score)
+        and len(joined_verification.candidate_results) == 1
+    )
+    if joined_passed:
+        return local_verification
+
+    joined_reasons = joined_verification.rejection_reasons or (
+        "invalid_joined_verification",
+    )
+    return TrajectoryGateResult(
+        passed=False,
+        candidate_results=local_verification.candidate_results,
+        score=math.inf,
+        rejection_reasons=tuple(
+            f"joined_output:{reason}" for reason in joined_reasons
+        ),
+        chunk_artifacts=local_verification.chunk_artifacts,
+    )
+
+
+class NoQualifiedCandidateError(RuntimeError):
+    """Raised when the full adaptive cascade has no verified trajectory."""
+
+
+class FinalOutputRejectedError(RuntimeError):
+    """Raised when post-join output fails the final whole-waveform gate."""
+
+
+def require_verified_final_output(
+    verification: TrajectoryGateResult,
+) -> TrajectoryGateResult:
+    """Return verified final evidence or reject without an audio fallback."""
+
+    if not isinstance(verification, TrajectoryGateResult):
+        raise FinalOutputRejectedError("final verifier returned an invalid result")
+    if (
+        verification.passed is not True
+        or not verification.candidate_results
+        or not all(result.passed for result in verification.candidate_results)
+        or not math.isfinite(verification.score)
+    ):
+        reasons = ",".join(verification.rejection_reasons) or "unsafe_final_output"
+        raise FinalOutputRejectedError(f"final output rejected: {reasons}")
+    return verification
+
+
+@dataclass(frozen=True)
+class CascadeResult:
+    trajectory: Any
+    verification: TrajectoryGateResult
+    seed: int | None
+    candidate_index: int | None
+    attempted_seeds: tuple[int, ...]
+    chunk_candidate_indices: tuple[int, ...] = ()
+    chunk_seeds: tuple[int, ...] = ()
+    selection_mode: str = "whole_trajectory"
+    sequence_path_rank: int | None = None
+    sequence_paths_checked: int = 0
+
+
+def select_k_candidate_sequences(
+    local_scores: Sequence[Sequence[float]],
+    transition_scores: Sequence[Sequence[Sequence[float]]] = (),
+    *,
+    max_paths: int = 3,
+) -> tuple[CandidateSequenceSelection, ...]:
+    """Return up to three distinct finite paths in stable cost order.
+
+    Each DP state retains only its ``max_paths`` best prefixes.  This keeps the
+    search bounded at ``O(N K² max_paths)`` while still producing exact k-best
+    paths for the requested small bound.  Single-chunk requests intentionally
+    return no sequence fallback.
+    """
+
+    if isinstance(max_paths, (bool, np.bool_)):
+        raise ValueError("max_paths must be an integer between 1 and 3")
+    try:
+        path_limit = operator.index(max_paths)
+    except (TypeError, ValueError, OverflowError) as error:
+        raise ValueError("max_paths must be an integer between 1 and 3") from error
+    if not 1 <= path_limit <= 3:
+        raise ValueError("max_paths must be an integer between 1 and 3")
+
+    try:
+        raw_local = [list(row) for row in local_scores]
+    except TypeError:
+        return ()
+    if len(raw_local) <= 1 or any(not row for row in raw_local):
+        return ()
+    safe_local = [
+        [
+            score if score is not None else math.inf
+            for score in (_finite_float(value, minimum=0.0) for value in row)
+        ]
+        for row in raw_local
+    ]
+    try:
+        raw_transitions = [
+            [list(row) for row in matrix]
+            for matrix in transition_scores
+        ]
+    except TypeError:
+        return ()
+    if len(raw_transitions) != len(safe_local) - 1:
+        return ()
+
+    safe_transitions: list[list[list[float]]] = []
+    for step, matrix in enumerate(raw_transitions):
+        previous_count = len(safe_local[step])
+        current_count = len(safe_local[step + 1])
+        if len(matrix) != previous_count or any(
+            len(row) != current_count for row in matrix
+        ):
+            return ()
+        safe_transitions.append(
+            [
+                [
+                    score if score is not None else math.inf
+                    for score in (
+                        _finite_float(value, minimum=0.0)
+                        for value in row
+                    )
+                ]
+                for row in matrix
+            ]
+        )
+
+    # One list of (cost, path) prefixes for each current candidate position.
+    states: list[list[tuple[float, tuple[int, ...]]]] = []
+    for candidate_index, score in enumerate(safe_local[0]):
+        states.append(
+            [(score, (candidate_index,))] if math.isfinite(score) else []
+        )
+
+    for step in range(1, len(safe_local)):
+        next_states: list[list[tuple[float, tuple[int, ...]]]] = []
+        for current_index, local_score in enumerate(safe_local[step]):
+            options: dict[tuple[int, ...], float] = {}
+            if math.isfinite(local_score):
+                for previous_index, prefixes in enumerate(states):
+                    edge_score = safe_transitions[step - 1][previous_index][
+                        current_index
+                    ]
+                    if not math.isfinite(edge_score):
+                        continue
+                    for previous_score, prefix in prefixes:
+                        total = previous_score + edge_score + local_score
+                        path = prefix + (current_index,)
+                        if math.isfinite(total):
+                            old_score = options.get(path, math.inf)
+                            if total < old_score:
+                                options[path] = total
+            ranked = sorted(
+                ((score, path) for path, score in options.items()),
+                key=lambda item: (item[0], item[1]),
+            )[:path_limit]
+            next_states.append(ranked)
+        states = next_states
+
+    complete: dict[tuple[int, ...], float] = {}
+    for prefixes in states:
+        for score, path in prefixes:
+            old_score = complete.get(path, math.inf)
+            if score < old_score:
+                complete[path] = score
+    ranked_complete = sorted(
+        ((score, path) for path, score in complete.items()),
+        key=lambda item: (item[0], item[1]),
+    )[:path_limit]
+    return tuple(
+        CandidateSequenceSelection(candidate_indices=path, total_score=score)
+        for score, path in ranked_complete
+    )
+
+
+def candidate_chunk_transition_score(
+    previous_result: CandidateGateResult,
+    previous_artifact: ChunkCandidateArtifact,
+    current_result: CandidateGateResult,
+    current_artifact: ChunkCandidateArtifact,
+    *,
+    speaker_weight: float = 1.0,
+    rms_db_weight: float = 0.05,
+    median_f0_weight: float = 0.10,
+) -> float:
+    """Return a finite adjacent-chunk cost or ``inf`` for an unsafe edge."""
+
+    if previous_result.passed is not True or current_result.passed is not True:
+        return math.inf
+    speaker_w = _finite_float(speaker_weight, minimum=0.0)
+    rms_w = _finite_float(rms_db_weight, minimum=0.0)
+    f0_w = _finite_float(median_f0_weight, minimum=0.0)
+    previous_rms = _finite_float(previous_artifact.rms_db)
+    current_rms = _finite_float(current_artifact.rms_db)
+    if None in (speaker_w, rms_w, f0_w, previous_rms, current_rms):
+        return math.inf
+
+    previous_embedding = previous_artifact.speaker_embedding
+    current_embedding = current_artifact.speaker_embedding
+    speaker_cost = 0.0
+    if previous_result.speaker_gate_applied and previous_embedding is None:
+        return math.inf
+    if current_result.speaker_gate_applied and current_embedding is None:
+        return math.inf
+    if previous_embedding is not None and current_embedding is not None:
+        try:
+            speaker_cost = 1.0 - cosine_similarity(previous_embedding, current_embedding)
+        except ValueError:
+            return math.inf
+
+    assert previous_rms is not None and current_rms is not None
+    rms_cost = abs(previous_rms - current_rms)
+    f0_cost = 0.0
+    previous_f0 = previous_artifact.median_f0_hz
+    current_f0 = current_artifact.median_f0_hz
+    if previous_f0 is not None or current_f0 is not None:
+        previous_pitch = _finite_float(previous_f0, minimum=1.0)
+        current_pitch = _finite_float(current_f0, minimum=1.0)
+        if previous_pitch is None or current_pitch is None:
+            return math.inf
+        f0_cost = abs(math.log2(current_pitch / previous_pitch))
+
+    assert speaker_w is not None and rms_w is not None and f0_w is not None
+    score = speaker_w * speaker_cost + rms_w * rms_cost + f0_w * f0_cost
+    return score if math.isfinite(score) and score >= 0.0 else math.inf
+
+
+def candidate_limit_for_chunk_budget(
+    chunk_count: int,
+    *,
+    max_candidates: int = 10,
+    max_generated_chunks: int = 20,
+) -> int:
+    """Return a candidate cap that never exceeds the generated-chunk budget."""
+
+    if isinstance(chunk_count, (bool, np.bool_)):
+        raise ValueError("chunk_count must be a positive integer")
+    try:
+        chunks = int(chunk_count)
+        candidates = int(max_candidates)
+        generated_chunks = int(max_generated_chunks)
+    except (TypeError, ValueError, OverflowError) as error:
+        raise ValueError("candidate budget values must be integers") from error
+    if chunks <= 0:
+        raise ValueError("chunk_count must be a positive integer")
+    if candidates <= 0 or candidates > 10:
+        raise ValueError("max_candidates must be between 1 and 10")
+    if generated_chunks <= 0:
+        raise ValueError("max_generated_chunks must be positive")
+    if chunks > generated_chunks:
+        raise ValueError("one trajectory exceeds the generated-chunk budget")
+    return min(candidates, generated_chunks // chunks)
+
+
+@dataclass(frozen=True)
+class _VerifiedTrajectoryCandidate:
+    candidate_index: int
+    seed: int
+    trajectory: Any
+    verification: TrajectoryGateResult
+
+
+def _whole_trajectory_result(
+    candidate: _VerifiedTrajectoryCandidate,
+    attempted_seeds: Sequence[int],
+    chunk_count: int,
+) -> CascadeResult:
+    return CascadeResult(
+        trajectory=candidate.trajectory,
+        verification=candidate.verification,
+        seed=candidate.seed,
+        candidate_index=candidate.candidate_index,
+        attempted_seeds=tuple(attempted_seeds),
+        chunk_candidate_indices=(candidate.candidate_index,) * chunk_count,
+        chunk_seeds=(candidate.seed,) * chunk_count,
+        selection_mode="whole_trajectory",
+    )
+
+
+def _sequence_fallback_results(
+    candidates: Sequence[_VerifiedTrajectoryCandidate],
+    attempted_seeds: Sequence[int],
+    chunk_count: int,
+    *,
+    max_paths: int,
+) -> tuple[CascadeResult, ...]:
+    """Rank mixed-seed paths using only independently gate-safe chunks."""
+
+    if chunk_count <= 1 or not candidates:
+        return ()
+    local_scores: list[list[float]] = [[] for _ in range(chunk_count)]
+    usable: list[bool] = []
+    for candidate in candidates:
+        verification = candidate.verification
+        try:
+            trajectory_length = len(candidate.trajectory)
+        except TypeError:
+            trajectory_length = -1
+        candidate_usable = bool(
+            trajectory_length == chunk_count
+            and len(verification.candidate_results) == chunk_count
+            and len(verification.chunk_artifacts) == chunk_count
+        )
+        usable.append(candidate_usable)
+        for chunk_index in range(chunk_count):
+            score = math.inf
+            if candidate_usable:
+                result = verification.candidate_results[chunk_index]
+                artifact = verification.chunk_artifacts[chunk_index]
+                if result.passed and math.isfinite(result.score) and result.score >= 0.0:
+                    speaker_artifact_valid = True
+                    if result.speaker_gate_applied:
+                        try:
+                            speaker_artifact_valid = bool(
+                                artifact.speaker_embedding is not None
+                                and cosine_similarity(
+                                    artifact.speaker_embedding,
+                                    artifact.speaker_embedding,
+                                ) >= 1.0 - 1.0e-6
+                            )
+                        except ValueError:
+                            speaker_artifact_valid = False
+                    if speaker_artifact_valid:
+                        score = result.score
+            local_scores[chunk_index].append(score)
+
+    transitions: list[list[list[float]]] = []
+    for chunk_index in range(1, chunk_count):
+        matrix: list[list[float]] = []
+        for previous_position, previous_candidate in enumerate(candidates):
+            row: list[float] = []
+            for current_position, current_candidate in enumerate(candidates):
+                score = math.inf
+                if usable[previous_position] and usable[current_position]:
+                    score = candidate_chunk_transition_score(
+                        previous_candidate.verification.candidate_results[chunk_index - 1],
+                        previous_candidate.verification.chunk_artifacts[chunk_index - 1],
+                        current_candidate.verification.candidate_results[chunk_index],
+                        current_candidate.verification.chunk_artifacts[chunk_index],
+                    )
+                row.append(score)
+            matrix.append(row)
+        transitions.append(matrix)
+
+    selections = select_k_candidate_sequences(
+        local_scores,
+        transitions,
+        max_paths=max_paths,
+    )
+    output: list[CascadeResult] = []
+    for rank, selection in enumerate(selections, 1):
+        selected_candidates = tuple(
+            candidates[position]
+            for position in selection.candidate_indices
+        )
+        selected_trajectory = tuple(
+            candidate.trajectory[chunk_index]
+            for chunk_index, candidate in enumerate(selected_candidates)
+        )
+        selected_results = tuple(
+            candidate.verification.candidate_results[chunk_index]
+            for chunk_index, candidate in enumerate(selected_candidates)
+        )
+        selected_artifacts = tuple(
+            candidate.verification.chunk_artifacts[chunk_index]
+            for chunk_index, candidate in enumerate(selected_candidates)
+        )
+        verification = TrajectoryGateResult(
+            passed=True,
+            candidate_results=selected_results,
+            score=selection.total_score,
+            rejection_reasons=(),
+            chunk_artifacts=selected_artifacts,
+        )
+        output.append(
+            CascadeResult(
+                trajectory=selected_trajectory,
+                verification=verification,
+                seed=None,
+                candidate_index=None,
+                attempted_seeds=tuple(attempted_seeds),
+                chunk_candidate_indices=tuple(
+                    candidate.candidate_index for candidate in selected_candidates
+                ),
+                chunk_seeds=tuple(
+                    candidate.seed for candidate in selected_candidates
+                ),
+                selection_mode="sequence_dp",
+                sequence_path_rank=rank,
+            )
+        )
+    return tuple(output)
+
+
+def _preferred_speaker_verification(
+    verification: TrajectoryGateResult,
+    *,
+    min_similarity: float,
+    max_boundary_drop: float,
+) -> bool:
+    if verification.passed is not True or not math.isfinite(verification.score):
+        return False
+    for result in verification.candidate_results:
+        if not result.speaker_gate_applied:
+            continue
+        similarity = _finite_float(result.speaker_similarity, minimum=-1.0, maximum=1.0)
+        boundary_drop = _finite_float(result.boundary_speaker_drop, minimum=0.0)
+        if (
+            similarity is None
+            or boundary_drop is None
+            or similarity < min_similarity
+            or boundary_drop > max_boundary_drop
+        ):
+            return False
+    return True
+
+
+def run_adaptive_cascade(
+    chunks: Sequence[str],
+    root_seed: int,
+    candidate_generator: Callable[[tuple[str, ...], int], Any],
+    candidate_verifier: Callable[[Any, tuple[str, ...], int], TrajectoryGateResult],
+    *,
+    initial_candidates: int = 1,
+    max_candidates: int = 5,
+    preferred_min_speaker_similarity: float = 0.25,
+    preferred_max_boundary_speaker_drop: float = 0.05,
+    sequence_final_verifier: (
+        Callable[[CascadeResult, tuple[str, ...]], TrajectoryGateResult] | None
+    ) = None,
+    max_sequence_paths: int = 3,
+) -> CascadeResult:
+    """Run a deterministic 1-to-5-to-10 fail-closed trajectory cascade.
+
+    The generator is called once per trajectory with ``root_seed + offset``.
+    It receives all chunks in one call, making the shared per-trajectory seed
+    contract explicit and preventing accidental per-chunk seed drift.
+    """
+
+    try:
+        chunk_tuple = tuple(str(chunk) for chunk in chunks)
+        base_seed = int(root_seed)
+        first_stage = int(initial_candidates)
+        limit = int(max_candidates)
+    except (TypeError, ValueError, OverflowError) as error:
+        raise ValueError("invalid adaptive cascade arguments") from error
+    if not chunk_tuple or any(not chunk for chunk in chunk_tuple):
+        raise ValueError("adaptive cascade requires non-empty text chunks")
+    if first_stage != 1:
+        raise ValueError("online adaptive cascade must start with exactly one candidate")
+    if limit < first_stage or limit > 10:
+        raise ValueError("adaptive cascade supports between 1 and 10 candidates")
+    if isinstance(max_sequence_paths, (bool, np.bool_)):
+        raise ValueError("max_sequence_paths must be an integer between 1 and 3")
+    try:
+        sequence_path_limit = operator.index(max_sequence_paths)
+    except (TypeError, ValueError, OverflowError) as error:
+        raise ValueError(
+            "max_sequence_paths must be an integer between 1 and 3"
+        ) from error
+    if not 1 <= sequence_path_limit <= 3:
+        raise ValueError("max_sequence_paths must be an integer between 1 and 3")
+    if sequence_final_verifier is not None and not callable(sequence_final_verifier):
+        raise ValueError("sequence_final_verifier must be callable")
+    preferred_similarity = _finite_float(
+        preferred_min_speaker_similarity,
+        minimum=-1.0,
+        maximum=1.0,
+    )
+    preferred_boundary = _finite_float(
+        preferred_max_boundary_speaker_drop,
+        minimum=0.0,
+    )
+    if preferred_similarity is None or preferred_boundary is None:
+        raise ValueError("preferred speaker thresholds must be finite")
+
+    attempted_seeds: list[int] = []
+    candidates: list[_VerifiedTrajectoryCandidate] = []
+
+    first_seed = base_seed
+    first_trajectory = candidate_generator(chunk_tuple, first_seed)
+    first_verification = candidate_verifier(first_trajectory, chunk_tuple, first_seed)
+    attempted_seeds.append(first_seed)
+    if not isinstance(first_verification, TrajectoryGateResult):
+        raise TypeError("candidate_verifier must return TrajectoryGateResult")
+    first_candidate = _VerifiedTrajectoryCandidate(
+        candidate_index=0,
+        seed=first_seed,
+        trajectory=first_trajectory,
+        verification=first_verification,
+    )
+    candidates.append(first_candidate)
+    if (
+        first_verification.passed
+        and math.isfinite(first_verification.score)
+        and (
+            limit == 1
+            or _preferred_speaker_verification(
+                first_verification,
+                min_similarity=preferred_similarity,
+                max_boundary_drop=preferred_boundary,
+            )
+        )
+    ):
+        return _whole_trajectory_result(
+            first_candidate,
+            attempted_seeds,
+            len(chunk_tuple),
+        )
+
+    stages = [min(5, limit)]
+    if limit > 5:
+        stages.append(limit)
+    next_candidate = first_stage
+    for stage_size in stages:
+        for candidate_index in range(next_candidate, stage_size):
+            seed = base_seed + candidate_index
+            trajectory = candidate_generator(chunk_tuple, seed)
+            verification = candidate_verifier(trajectory, chunk_tuple, seed)
+            attempted_seeds.append(seed)
+            if not isinstance(verification, TrajectoryGateResult):
+                raise TypeError("candidate_verifier must return TrajectoryGateResult")
+            candidates.append(
+                _VerifiedTrajectoryCandidate(
+                    candidate_index=candidate_index,
+                    seed=seed,
+                    trajectory=trajectory,
+                    verification=verification,
+                )
+            )
+        qualified = [
+            candidate
+            for candidate in candidates
+            if candidate.verification.passed
+            and math.isfinite(candidate.verification.score)
+        ]
+        final_stage = stage_size == limit
+        if qualified:
+            preferred_qualified = [
+                candidate
+                for candidate in qualified
+                if _preferred_speaker_verification(
+                    candidate.verification,
+                    min_similarity=preferred_similarity,
+                    max_boundary_drop=preferred_boundary,
+                )
+            ]
+            selectable = qualified if final_stage else preferred_qualified
+            if selectable:
+                selected_whole = min(
+                    selectable,
+                    key=lambda candidate: (
+                        candidate.verification.score,
+                        candidate.candidate_index,
+                    ),
+                )
+                return _whole_trajectory_result(
+                    selected_whole,
+                    attempted_seeds,
+                    len(chunk_tuple),
+                )
+            next_candidate = stage_size
+            continue
+        # With a final-aware callback, defer DP until all whole-trajectory
+        # candidates in the request budget have been exhausted.  This keeps
+        # whole trajectories globally preferred and bounds joined checks to
+        # at most ``max_sequence_paths`` once per request.
+        if sequence_final_verifier is not None and not final_stage:
+            next_candidate = stage_size
+            continue
+        sequence_results = _sequence_fallback_results(
+            candidates,
+            attempted_seeds,
+            len(chunk_tuple),
+            max_paths=(sequence_path_limit if sequence_final_verifier else 1),
+        )
+        if sequence_final_verifier is None:
+            sequence_result = sequence_results[0] if sequence_results else None
+            if sequence_result is not None and (
+                final_stage
+                or _preferred_speaker_verification(
+                    sequence_result.verification,
+                    min_similarity=preferred_similarity,
+                    max_boundary_drop=preferred_boundary,
+                )
+            ):
+                return sequence_result
+        else:
+            for checked_count, sequence_result in enumerate(sequence_results, 1):
+                try:
+                    final_verification = sequence_final_verifier(
+                        sequence_result,
+                        chunk_tuple,
+                    )
+                except Exception as error:
+                    raise RuntimeError(
+                        "sequence final verification failed; refusing unverified audio"
+                    ) from error
+                if not isinstance(final_verification, TrajectoryGateResult):
+                    raise RuntimeError(
+                        "sequence final verifier returned an invalid result"
+                    )
+                if (
+                    final_verification.passed is True
+                    and math.isfinite(final_verification.score)
+                    and len(final_verification.candidate_results) == 1
+                    and isinstance(
+                        final_verification.candidate_results[0],
+                        CandidateGateResult,
+                    )
+                    and final_verification.candidate_results[0].passed is True
+                    and math.isfinite(
+                        final_verification.candidate_results[0].score
+                    )
+                    and not final_verification.rejection_reasons
+                ):
+                    return CascadeResult(
+                        trajectory=sequence_result.trajectory,
+                        verification=sequence_result.verification,
+                        seed=sequence_result.seed,
+                        candidate_index=sequence_result.candidate_index,
+                        attempted_seeds=sequence_result.attempted_seeds,
+                        chunk_candidate_indices=(
+                            sequence_result.chunk_candidate_indices
+                        ),
+                        chunk_seeds=sequence_result.chunk_seeds,
+                        selection_mode=sequence_result.selection_mode,
+                        sequence_path_rank=sequence_result.sequence_path_rank,
+                        sequence_paths_checked=checked_count,
+                    )
+        next_candidate = stage_size
+
+    raise NoQualifiedCandidateError(
+        f"no verified TTS trajectory after {len(attempted_seeds)} candidates"
+    )

requirements.txt

@@ -1,6 +1,17 @@
 # torch / gradio / spaces are provided by the ZeroGPU image.
-git+https://github.com/OpenFormosa/Barbet.git
-git+https://github.com/OpenFormosa/BlueMagpie-TTS.git@ce384c8cc54efea1aaba7b9f1d7ded6c1c99aa9a
+git+https://github.com/OpenFormosa/Barbet.git@6fcd7ce4aa37f2250a3242995bef0fbc3b026ba8
 huggingface_hub==0.36.0
-soundfile
-speechbrain>=1.0.3,<1.1
+opencc-python-reimplemented==0.1.7
+transformers==4.57.6
+accelerate==1.12.0
+einops==0.8.2
+pydantic==2.11.10
+numpy==2.3.5
+scipy==1.17.1
+numexpr==2.14.1
+bottleneck==1.6.0
+tqdm==4.68.2
+safetensors==0.8.0
+librosa==0.11.0
+soundfile==0.14.0
+speechbrain==1.0.3

tests/test_production.py

@@ -6,12 +6,19 @@ from torch import nn
 
 from production import (
     StopHysteresisController,
+    candidate_local_score,
+    candidate_transition_score,
+    compare_asr_text,
     count_speech_units,
     duration_hard_stop_steps,
     effective_generation_cfg,
     endpoint_generation_plan,
     ensure_terminal_punctuation,
     finish_audio,
+    normalize_spoken_forms,
+    normalize_tts_eval_text,
+    normalize_tts_text,
+    select_candidate_sequence,
     select_generation_cps,
     split_leading_clause,
     split_text_for_tts,
@@ -21,6 +28,28 @@ from production import (
 )
 
 
+def test_eval_only_pronoun_homophones_do_not_change_model_input_text():
+    text = "她提醒妳，它、牠和祂都在這裡，再把她的東西放得穩穩地。"
+
+    assert normalize_tts_text(text) == text
+    assert normalize_tts_eval_text("她提醒妳，它、牠和祂都在這裡。") == (
+        "他提醒你，他、他和他都在這裡。"
+    )
+    assert normalize_tts_eval_text("的地得在再") == "的地得在再"
+
+
+def test_asr_comparison_accepts_only_verified_pronoun_homophones():
+    comparison = compare_asr_text("她提醒我", "他提醒我")
+
+    assert comparison.passed
+    assert comparison.cer == 0.0
+    assert comparison.prefix_cer == 0.0
+    assert comparison.suffix_cer == 0.0
+    assert comparison.extra_tail_units == 0
+
+    assert not compare_asr_text("他在這裡", "他再這裡").passed
+
+
 class _SequenceStopHead(nn.Module):
     def __init__(self, probabilities):
         super().__init__()
@@ -206,3 +235,191 @@ def test_finish_audio_fades_endpoint_and_appends_silence():
     assert output.shape == (13,)
     np.testing.assert_allclose(output[-4:], 0.0)
     assert output[-5] < output[-6] < output[-7]
+
+
+def test_spoken_form_normalizer_expands_common_zh_tw_forms():
+    text = "日期 2026/07/15，成長 12.5%，距離 10 km，顯卡 RTX 5090。"
+    assert normalize_spoken_forms(text) == (
+        "日期 二零二六年七月十五日，成長 百分之十二點五，"
+        "距離 十公里，顯卡 R T X 五零九零。"
+    )
+    assert normalize_spoken_forms("2026年7月16日，RTX5090") == (
+        "二零二六年七月十六日，R T X 五零九零"
+    )
+
+
+def test_spoken_form_normalizer_expands_clock_time_and_spells_urls():
+    assert normalize_spoken_forms("時間是 15:30，午夜是 00:00。") == (
+        "時間是 十五點三十分，午夜是 零點整。"
+    )
+    assert normalize_spoken_forms("無效時間 25:30") == "無效時間 25:30"
+    assert normalize_spoken_forms("IP 192.168.1.1，網址 https://example.test:30/path") == (
+        "I P 192.168.1.1，網址 H T T P S 冒號 斜線 斜線 "
+        "example 點 test 冒號 三零 斜線 path"
+    )
+
+
+def test_asr_comparison_unifies_traditional_and_simplified_chinese():
+    comparison = compare_asr_text("今天天氣真好。", "今天天气真好")
+
+    assert comparison.target_text == comparison.transcript_text == "今天天气真好"
+    assert comparison.cer == 0.0
+    assert comparison.passed is True
+    assert normalize_spoken_forms("溫度 -3.5 °C，頻率 3.2 GHz。") == (
+        "溫度 負三點五攝氏度，頻率 三點二吉赫茲。"
+    )
+
+
+def test_spoken_form_normalizer_canonicalizes_zh_measures_for_asr_comparison():
+    assert normalize_spoken_forms("成長 12.5%，距離 10公里，重量 1.5公斤。") == (
+        "成長 百分之十二點五，距離 十公里，重量 一點五公斤。"
+    )
+    comparison = compare_asr_text("距離十公里", "距離10公里")
+    assert comparison.passed
+    assert comparison.edit_distance == 0
+
+
+def test_spoken_form_normalizer_expands_network_identifiers_and_versions():
+    assert normalize_spoken_forms(
+        "網址 https://api.example.com/v1/items?q=RTX-5090&n=2。"
+    ) == (
+        "網址 H T T P S 冒號 斜線 斜線 api 點 example 點 com 斜線 v 一 "
+        "斜線 items 問號 q 等於 R T X 橫線 五零九零 和 n 等於 二。"
+    )
+    assert normalize_spoken_forms("信箱 USER.name+tts@example.com。") == (
+        "信箱 U S E R 點 name 加號 tts 小老鼠 example 點 com。"
+    )
+    assert normalize_spoken_forms(
+        "版本 v1.2.3，候選 version 10.4.0-beta.1+build.5。"
+    ) == (
+        "版本一點二點三，候選 版本十點四點零 預發布 beta 點 一 "
+        "建置 build 點 五。"
+    )
+
+
+def test_spoken_form_normalizer_expands_currency_and_preserves_ip_and_invalid_date():
+    assert normalize_spoken_forms("售價 NT$1,299.50、USD 12.50、100 JPY。") == (
+        "售價 新台幣一千二百九十九點五零、美元十二點五零、日圓一百。"
+    )
+    assert normalize_spoken_forms("IP 192.168.1.1") == "I P 192.168.1.1"
+    assert normalize_spoken_forms("無效日期 2026年2月30日") == (
+        "無效日期 2026年2月30日"
+    )
+
+
+def test_spoken_form_normalizer_is_conservative_and_locale_aware():
+    text = "Use OpenAI for stable English on 2026/07/15."
+    assert normalize_spoken_forms(text, locale="en-US") == text
+    assert normalize_spoken_forms("OpenAI stable diffusion") == "OpenAI stable diffusion"
+    assert normalize_spoken_forms("日期 2026/02/30") == "日期 2026/02/30"
+    with np.testing.assert_raises(ValueError):
+        normalize_spoken_forms("10 km", locale="fr-FR")
+
+
+def test_asr_comparison_accepts_normalized_semantic_match():
+    comparison = compare_asr_text(
+        "成長 12.5%，使用 RTX 5090。",
+        "成長百分之十二點五使用 R T X 五零九零",
+    )
+    assert comparison.passed
+    assert comparison.edit_distance == 0
+    assert comparison.extra_tail == ""
+
+
+def test_asr_comparison_rejects_prefix_suffix_and_extra_tail_errors():
+    missing_prefix = compare_asr_text("第一個字不能遺漏", "一個字不能遺漏")
+    assert not missing_prefix.passed
+    assert missing_prefix.prefix_deletions == 1
+    assert missing_prefix.prefix_cer > 0.0
+
+    missing_suffix = compare_asr_text("句尾內容必須完整", "句尾內容必須")
+    assert not missing_suffix.passed
+    assert missing_suffix.suffix_deletions == 2
+    assert missing_suffix.suffix_cer > 0.0
+
+    extra_tail = compare_asr_text("內容完成", "內容完成多講兩字")
+    assert not extra_tail.passed
+    assert extra_tail.extra_tail == "多讲两字"
+    assert extra_tail.extra_tail_units == 4
+
+
+def test_asr_comparison_fails_closed_for_empty_or_nonfinite_limits():
+    assert not compare_asr_text("", "任意輸出").passed
+    assert not compare_asr_text("目標內容", "", max_cer=1.0).passed
+    assert not compare_asr_text("目標內容", "目標內容", max_cer=math.nan).passed
+    assert not compare_asr_text("目標內容", "目標內容", prefix_units=0).passed
+
+
+def test_candidate_scores_are_finite_safe_and_gate_unsafe_audio():
+    score = candidate_local_score(
+        cer=0.02,
+        speaker_similarity=0.8,
+        boundary_speaker_drop=0.01,
+        prefix_cer=0.0,
+        suffix_cer=0.0,
+        pace_penalty=0.03,
+        style_penalty=0.02,
+        max_cer=0.10,
+        min_speaker_similarity=0.5,
+        max_boundary_speaker_drop=0.03,
+    )
+    assert math.isfinite(score)
+    assert math.isclose(score, 0.081)
+
+    base = dict(cer=0.02, speaker_similarity=0.8, boundary_speaker_drop=0.01)
+    assert math.isinf(candidate_local_score(**base, asr_passed=False))
+    assert math.isinf(candidate_local_score(**base, truncated=True))
+    assert math.isinf(candidate_local_score(**base, extra_tail=True))
+    assert math.isinf(candidate_local_score(**{**base, "cer": math.nan}))
+    assert math.isinf(candidate_local_score(**{**base, "boundary_speaker_drop": -0.01}))
+    assert math.isinf(candidate_local_score(**base, max_cer=math.nan))
+
+
+def test_transition_score_rejects_nan_and_invalid_similarity():
+    assert candidate_transition_score(
+        speaker_similarity=0.9,
+        f0_delta=0.1,
+        rms_delta=0.2,
+    ) == 0.4
+    assert math.isinf(
+        candidate_transition_score(
+            speaker_similarity=math.nan,
+            f0_delta=0.1,
+            rms_delta=0.2,
+        )
+    )
+    assert math.isinf(
+        candidate_transition_score(
+            speaker_similarity=1.1,
+            f0_delta=0.1,
+            rms_delta=0.2,
+        )
+    )
+
+
+def test_sequence_selector_finds_stable_global_path():
+    selection = select_candidate_sequence(
+        local_scores=[[0.0, 0.4], [0.0, 0.1], [0.2, 0.0]],
+        transition_scores=[
+            [[2.0, 0.0], [0.0, 2.0]],
+            [[2.0, 0.0], [0.0, 2.0]],
+        ],
+    )
+    assert selection is not None
+    assert selection.candidate_indices == (0, 1, 0)
+    assert math.isclose(selection.total_score, 0.3)
+
+
+def test_sequence_selector_is_deterministic_and_fail_closed():
+    tied = select_candidate_sequence([[0.0, 0.0]])
+    assert tied is not None
+    assert tied.candidate_indices == (0,)
+
+    skips_nan = select_candidate_sequence([[math.nan, 0.2]])
+    assert skips_nan is not None
+    assert skips_nan.candidate_indices == (1,)
+
+    assert select_candidate_sequence([[math.nan], [0.0]], [[[0.0]]]) is None
+    assert select_candidate_sequence([[0.0], [0.0]], [[[math.nan]]]) is None
+    assert select_candidate_sequence([[0.0], [0.0]], [[[0.0, 1.0]]]) is None
+    assert select_candidate_sequence([]) is None

tests/test_quality_runtime.py

@@ -0,0 +1,1450 @@
+import math
+from types import SimpleNamespace
+
+import numpy as np
+import pytest
+import torch
+
+from production import split_leading_clause, split_text_for_tts
+from quality_runtime import (
+    BASE_GENERATION_POLICY,
+    SAFE_DURATION_GENERATION_POLICY,
+    WHISPER_MODEL_ID,
+    WHISPER_REVISION,
+    CandidateObservation,
+    CascadeResult,
+    ChunkCandidateArtifact,
+    FinalOutputRejectedError,
+    LazyWhisperASR,
+    NoQualifiedCandidateError,
+    TrajectoryGateResult,
+    WhisperRuntime,
+    active_voiced_duration_seconds,
+    candidate_chunk_transition_score,
+    candidate_limit_for_chunk_budget,
+    cosine_similarity,
+    generation_policy_for_candidate_offset,
+    load_pinned_whisper_runtime,
+    prepare_candidate_audio,
+    qualify_trajectory_with_joined_output,
+    require_verified_final_output,
+    resolve_request_seed,
+    run_adaptive_cascade,
+    select_k_candidate_sequences,
+    speaker_embedding_from_audio,
+    speaker_evidence_from_audio,
+    transcribe_whisper,
+    verify_candidate,
+    verify_trajectory,
+)
+
+
+class _Factory:
+    def __init__(self, value):
+        self.value = value
+        self.calls = []
+
+    def from_pretrained(self, *args, **kwargs):
+        self.calls.append((args, kwargs))
+        return self.value
+
+
+def test_candidate_generation_policy_mapping_uses_safe_duration_after_offset_zero():
+    base = generation_policy_for_candidate_offset(0)
+    assert base is BASE_GENERATION_POLICY
+    assert (base.name, base.cjk_cps, base.ascii_cps, base.hard_stop_margin_steps) == (
+        "base",
+        5.2,
+        4.6,
+        1,
+    )
+
+    for offset in (1, 2, 5, 9):
+        safe = generation_policy_for_candidate_offset(offset)
+        assert safe is SAFE_DURATION_GENERATION_POLICY
+        assert (
+            safe.name,
+            safe.cjk_cps,
+            safe.ascii_cps,
+            safe.hard_stop_margin_steps,
+        ) == ("safe_duration", 4.6, 4.0, 1)
+
+
+@pytest.mark.parametrize("offset", [-1, 0.5, 1.0, True, None, "one"])
+def test_candidate_generation_policy_mapping_rejects_invalid_offsets(offset):
+    with pytest.raises(ValueError, match="non-negative integer"):
+        generation_policy_for_candidate_offset(offset)
+
+
+def test_explicit_request_seed_is_forwarded_without_calling_random_factory():
+    calls = []
+
+    def factory(limit):
+        calls.append(limit)
+        return 456
+
+    assert resolve_request_seed(123, factory) == 123
+    assert calls == []
+    assert resolve_request_seed(None, factory) == 456
+    assert calls == [2**31]
+
+
+@pytest.mark.parametrize("seed", [-1, 2**31, 1.0, True, "123", None])
+def test_request_seed_validation_fails_closed(seed):
+    factory = lambda _limit: 2**31 if seed is None else 0
+    with pytest.raises(ValueError, match=r"\[0, 2147483648\)"):
+        resolve_request_seed(seed, factory)
+
+
+def test_quality_gate_accepts_only_eval_canonicalized_pronoun_homophones():
+    result = verify_candidate(
+        CandidateObservation(
+            target_text="她提醒我",
+            transcript_text="他提醒我",
+            audio_duration_seconds=1.0,
+        )
+    )
+
+    assert result.passed
+    assert result.comparison.cer == 0.0
+    assert result.comparison.prefix_cer == 0.0
+    assert result.comparison.suffix_cer == 0.0
+
+
+class _FakeWhisperModel:
+    def __init__(self):
+        self.device = None
+        self.evaluated = False
+        self.generate_calls = []
+
+    def to(self, device):
+        self.device = torch.device(device)
+        return self
+
+    def eval(self):
+        self.evaluated = True
+        return self
+
+    def generate(self, features, **kwargs):
+        self.generate_calls.append((features, kwargs))
+        return torch.tensor([[1, 2, 3]]).repeat(features.shape[0], 1)
+
+
+class _FakeProcessor:
+    def __init__(self):
+        self.calls = []
+
+    def __call__(self, waveform, **kwargs):
+        copied = (
+            [segment.copy() for segment in waveform]
+            if isinstance(waveform, list)
+            else waveform.copy()
+        )
+        self.calls.append((copied, kwargs))
+        batch_size = len(waveform) if isinstance(waveform, list) else 1
+        return SimpleNamespace(input_features=torch.ones(batch_size, 80, 10))
+
+    def batch_decode(self, token_ids, **kwargs):
+        assert all(row == [1, 2, 3] for row in token_ids.tolist())
+        assert kwargs == {"skip_special_tokens": True}
+        return ["  合成內容完整。  "] * token_ids.shape[0]
+
+
+class _StatsEncoder:
+    def __init__(self, *, invalid=False):
+        self.lengths = []
+        self.wav_lens = []
+        self.invalid = invalid
+
+    def encode_batch(self, tensor, wav_lens=None):
+        self.lengths.append(tensor.shape[-1])
+        if wav_lens is None:
+            wav_lens = torch.ones(tensor.shape[0], device=tensor.device)
+        self.wav_lens.append(wav_lens.detach().cpu().numpy())
+        if self.invalid:
+            return torch.full(
+                (tensor.shape[0], 1, 2),
+                math.nan,
+                device=tensor.device,
+            )
+        rows = []
+        for row, relative_length in zip(tensor, wav_lens, strict=True):
+            sample_count = max(1, int(round(float(relative_length) * tensor.shape[-1])))
+            active = row[:sample_count]
+            rows.append(
+                torch.stack(
+                    (
+                        active.mean(),
+                        active.std(unbiased=False),
+                        active.abs().amax(),
+                    )
+                )
+            )
+        return torch.stack(rows)
+
+
+def _tone(sample_rate=16_000, seconds=1.0, amplitude=0.2, frequency=220.0):
+    timeline = np.arange(round(sample_rate * seconds), dtype=np.float32) / sample_rate
+    return amplitude * np.sin(2.0 * np.pi * frequency * timeline).astype(np.float32)
+
+
+def _gate_result(passed, score=0.0):
+    return TrajectoryGateResult(
+        passed=passed,
+        candidate_results=(),
+        score=score if passed else math.inf,
+        rejection_reasons=() if passed else ("rejected",),
+    )
+
+
+def _chunk_verification(target, transcript, *, embedding, rms_db):
+    observation = CandidateObservation(
+        target_text=target,
+        transcript_text=transcript,
+        audio_duration_seconds=2.0,
+        speaker_similarity=0.8,
+        begin_speaker_similarity=0.8,
+        end_speaker_similarity=0.8,
+    )
+    return observation, ChunkCandidateArtifact(
+        speaker_embedding=(
+            None if embedding is None else np.asarray(embedding, dtype=np.float32)
+        ),
+        rms_db=rms_db,
+    )
+
+
+def _whole_speaker_verification(*, similarity, boundary_drop, passed=True):
+    target = "完整而且穩定的候選內容"
+    observation = CandidateObservation(
+        target_text=target,
+        transcript_text=target if passed else "錯誤內容",
+        audio_duration_seconds=2.5,
+        speaker_similarity=similarity,
+        begin_speaker_similarity=0.60,
+        end_speaker_similarity=0.60 - boundary_drop,
+    )
+    artifact = ChunkCandidateArtifact(
+        speaker_embedding=np.array([1.0, 0.0], dtype=np.float32),
+        rms_db=-20.0,
+    )
+    return verify_trajectory(
+        [observation],
+        chunk_artifacts=[artifact],
+        min_speaker_similarity=0.10,
+        max_boundary_speaker_drop=0.10,
+    )
+
+
+def _joined_verification(target, transcript):
+    return verify_trajectory(
+        [
+            CandidateObservation(
+                target_text=target,
+                transcript_text=transcript,
+                audio_duration_seconds=3.0,
+                speaker_similarity=0.8,
+                begin_speaker_similarity=0.8,
+                end_speaker_similarity=0.8,
+            )
+        ]
+    )
+
+
+def test_pinned_whisper_loader_uses_exact_revision_without_global_download():
+    processor = _FakeProcessor()
+    model = _FakeWhisperModel()
+    processor_factory = _Factory(processor)
+    model_factory = _Factory(model)
+
+    runtime = load_pinned_whisper_runtime(
+        device="cpu",
+        processor_factory=processor_factory,
+        model_factory=model_factory,
+    )
+
+    assert runtime.processor is processor
+    assert runtime.model is model
+    assert runtime.device == torch.device("cpu")
+    assert runtime.dtype == torch.float32
+    assert model.evaluated
+    assert processor_factory.calls == [
+        ((WHISPER_MODEL_ID,), {"revision": WHISPER_REVISION})
+    ]
+    model_args, model_kwargs = model_factory.calls[0]
+    assert model_args == (WHISPER_MODEL_ID,)
+    assert model_kwargs["revision"] == WHISPER_REVISION
+    assert model_kwargs["torch_dtype"] == torch.float32
+    assert model_kwargs["use_safetensors"] is True
+
+
+def test_lazy_whisper_runtime_loads_once_and_transcription_is_deterministic():
+    processor = _FakeProcessor()
+    model = _FakeWhisperModel()
+    runtime = WhisperRuntime(processor, model, torch.device("cpu"), torch.float32)
+    loads = []
+    lazy_asr = LazyWhisperASR(lambda: loads.append("load") or runtime)
+    audio_8khz = _tone(sample_rate=8_000, seconds=0.5)
+
+    first = transcribe_whisper(audio_8khz, 8_000, lazy_asr=lazy_asr)
+    second = transcribe_whisper(audio_8khz, 8_000, lazy_asr=lazy_asr)
+
+    assert first == second == "合成內容完整。"
+    assert loads == ["load"]
+    assert processor.calls[0][0].shape == (8_000,)
+    assert processor.calls[0][1] == {
+        "sampling_rate": 16_000,
+        "return_tensors": "pt",
+    }
+    _, generation_kwargs = model.generate_calls[0]
+    assert generation_kwargs == {
+        "language": "zh",
+        "task": "transcribe",
+        "do_sample": False,
+        "num_beams": 1,
+        "max_new_tokens": 128,
+    }
+
+
+def test_long_whisper_audio_is_batched_below_thirty_second_limit():
+    processor = _FakeProcessor()
+    model = _FakeWhisperModel()
+    runtime = WhisperRuntime(processor, model, torch.device("cpu"), torch.float32)
+    audio = np.concatenate(
+        (
+            _tone(seconds=21.0),
+            np.zeros(16_000, dtype=np.float32),
+            _tone(seconds=21.0, frequency=240.0),
+            np.zeros(16_000, dtype=np.float32),
+            _tone(seconds=21.0, frequency=260.0),
+        )
+    )
+
+    transcript = transcribe_whisper(audio, 16_000, runtime=runtime, max_new_tokens=440)
+
+    segments = processor.calls[0][0]
+    assert isinstance(segments, list)
+    assert len(segments) == 3
+    assert all(segment.size <= 30 * 16_000 for segment in segments)
+    assert transcript == "合成內容完整。 合成內容完整。 合成內容完整。"
+    assert len(model.generate_calls) == 1
+    assert model.generate_calls[0][1]["max_new_tokens"] == 440
+
+
+def test_transcription_rejects_nonfinite_audio_and_conflicting_injection():
+    processor = _FakeProcessor()
+    model = _FakeWhisperModel()
+    runtime = WhisperRuntime(processor, model, torch.device("cpu"), torch.float32)
+    with pytest.raises(ValueError, match="non-finite"):
+        transcribe_whisper(np.array([0.0, math.nan]), 16_000, runtime=runtime)
+    with pytest.raises(ValueError, match="either runtime or lazy_asr"):
+        transcribe_whisper(
+            np.ones(20),
+            16_000,
+            runtime=runtime,
+            lazy_asr=LazyWhisperASR(lambda: runtime),
+        )
+
+
+def test_candidate_audio_data_or_asr_value_error_becomes_rejection_evidence():
+    calls = []
+
+    def transcriber(waveform, sample_rate):
+        calls.append((waveform.copy(), sample_rate))
+        return "  內容完整  "
+
+    prepared = prepare_candidate_audio(_tone(seconds=0.25), 16_000, transcriber=transcriber)
+    assert prepared is not None
+    assert prepared.transcript_text == "內容完整"
+    assert prepared.duration_seconds == pytest.approx(0.25)
+    assert len(calls) == 1
+
+    assert prepare_candidate_audio(
+        [0.0, math.nan],
+        16_000,
+        transcriber=transcriber,
+    ) is None
+    assert prepare_candidate_audio(
+        _tone(seconds=0.25),
+        16_000,
+        transcriber=lambda *_: (_ for _ in ()).throw(ValueError("bad candidate")),
+    ) is None
+    assert len(calls) == 1
+
+
+def test_candidate_audio_infrastructure_error_propagates_fail_closed():
+    with pytest.raises(RuntimeError, match="ASR unavailable"):
+        prepare_candidate_audio(
+            _tone(seconds=0.25),
+            16_000,
+            transcriber=lambda *_: (_ for _ in ()).throw(RuntimeError("ASR unavailable")),
+        )
+
+
+def test_ecapa_ndarray_helper_trims_resamples_normalizes_and_handles_channels():
+    encoder = _StatsEncoder()
+    tone = _tone(sample_rate=8_000, seconds=1.0)
+    padded = np.concatenate((np.zeros(4_000), tone, np.zeros(4_000))).astype(np.float32)
+    stereo = np.stack((padded, padded), axis=0)
+
+    embedding = speaker_embedding_from_audio(stereo, 8_000, encoder)
+
+    assert embedding.shape == (3,)
+    assert np.isfinite(embedding).all()
+    assert np.linalg.norm(embedding) == pytest.approx(1.0, abs=1e-6)
+    assert 16_000 <= encoder.lengths[0] < 32_000
+    assert len(encoder.lengths) == 1
+
+
+def test_ecapa_ndarray_helper_fails_closed_on_silence_nan_and_bad_embedding():
+    with pytest.raises(ValueError, match="active speech"):
+        speaker_embedding_from_audio(np.zeros(16_000), 16_000, _StatsEncoder())
+    with pytest.raises(ValueError, match="non-finite samples"):
+        speaker_embedding_from_audio(
+            np.array([0.0, math.nan], dtype=np.float32),
+            16_000,
+            _StatsEncoder(),
+        )
+    with pytest.raises(ValueError, match="invalid embedding"):
+        speaker_embedding_from_audio(_tone(), 16_000, _StatsEncoder(invalid=True))
+
+
+def test_cosine_and_speaker_evidence_are_finite_and_directional():
+    encoder = _StatsEncoder()
+    audio = np.concatenate((_tone(seconds=1.0), _tone(seconds=1.0, amplitude=0.1)))
+    anchor = speaker_embedding_from_audio(audio, 16_000, encoder)
+
+    evidence = speaker_evidence_from_audio(
+        audio,
+        16_000,
+        encoder,
+        anchor,
+        edge_seconds=0.75,
+    )
+
+    # Anchor extraction is one call; whole/begin/end evidence is one batched
+    # call instead of three repeated ECAPA forwards for the same candidate.
+    assert len(encoder.lengths) == 2
+    assert encoder.wav_lens[-1].shape == (3,)
+    assert evidence.similarity == pytest.approx(1.0, abs=1e-5)
+    assert -1.0 <= evidence.begin_similarity <= 1.0
+    assert -1.0 <= evidence.end_similarity <= 1.0
+    assert evidence.boundary_drop == max(
+        0.0,
+        evidence.begin_similarity - evidence.end_similarity,
+    )
+    assert evidence.active_duration_seconds > 1.5
+    assert cosine_similarity(anchor, anchor) == pytest.approx(1.0)
+    with pytest.raises(ValueError):
+        cosine_similarity([0.0, 0.0], [0.0, 0.0])
+
+
+def test_long_speaker_evidence_uses_one_bounded_window_batch():
+    encoder = _StatsEncoder()
+    audio = _tone(seconds=20.0)
+    anchor = np.array([0.0, 0.0, 1.0], dtype=np.float32)
+
+    evidence = speaker_evidence_from_audio(audio, 16_000, encoder, anchor)
+
+    assert len(encoder.lengths) == 1
+    assert encoder.lengths[0] <= 3 * 16_000
+    assert encoder.wav_lens[0].shape == (6,)
+    assert evidence.active_duration_seconds == pytest.approx(20.0, abs=0.05)
+    assert evidence.speaker_embedding.shape == (3,)
+    assert math.isfinite(evidence.active_rms_db)
+
+
+def test_active_union_pace_ignores_one_or_three_second_internal_silence():
+    def waveform(pause_seconds):
+        return np.concatenate(
+            (
+                _tone(seconds=0.75),
+                np.zeros(round(pause_seconds * 16_000), dtype=np.float32),
+                _tone(seconds=0.75, frequency=260.0),
+            )
+        )
+
+    one_second = active_voiced_duration_seconds(waveform(1.0), 16_000)
+    three_seconds = active_voiced_duration_seconds(waveform(3.0), 16_000)
+
+    assert one_second == pytest.approx(three_seconds, abs=1.0e-9)
+    assert 8 / one_second == pytest.approx(8 / three_seconds, abs=1.0e-9)
+    assert one_second < 1.6
+
+
+def test_active_union_duration_controls_speaker_eligibility_at_1_5_seconds():
+    below = active_voiced_duration_seconds(_tone(seconds=1.49), 16_000)
+    boundary = active_voiced_duration_seconds(_tone(seconds=1.50), 16_000)
+    assert below == pytest.approx(1.49, abs=1.0e-9)
+    assert boundary == pytest.approx(1.50, abs=1.0e-9)
+
+    common = {
+        "target_text": "這是一段完整而且穩定的合成內容",
+        "transcript_text": "這是一段完整而且穩定的合成內容",
+    }
+    short_result = verify_candidate(
+        CandidateObservation(**common, audio_duration_seconds=below)
+    )
+    boundary_result = verify_candidate(
+        CandidateObservation(**common, audio_duration_seconds=boundary)
+    )
+    assert short_result.passed
+    assert not short_result.speaker_gate_applied
+    assert not boundary_result.passed
+    assert boundary_result.speaker_gate_applied
+    assert "missing_speaker_evidence" in boundary_result.rejection_reasons
+
+
+def test_short_candidate_uses_exact_semantics_but_bypasses_speaker_hard_gate():
+    passed = verify_candidate(
+        CandidateObservation(
+            target_text="謝謝你",
+            transcript_text="謝謝你",
+            audio_duration_seconds=1.49,
+        )
+    )
+    failed = verify_candidate(
+        CandidateObservation(
+            target_text="謝謝你",
+            transcript_text="謝謝",
+            audio_duration_seconds=1.0,
+        )
+    )
+
+    assert passed.passed
+    assert not passed.speaker_gate_applied
+    assert passed.speaker_similarity is None
+    assert passed.score == 0.0
+    assert not failed.passed
+    assert failed.rejection_reasons == ("semantic_gate",)
+
+
+def test_candidate_pace_gate_is_optional_and_fails_closed_when_enabled():
+    base = CandidateObservation(
+        target_text="內容完整",
+        transcript_text="內容完整",
+        audio_duration_seconds=1.0,
+    )
+
+    assert verify_candidate(base).passed
+    assert verify_candidate(
+        CandidateObservation(**{**base.__dict__, "pace_cps": 4.2}),
+        max_pace_cps=4.3,
+    ).passed
+    missing = verify_candidate(base, max_pace_cps=4.3)
+    fast = verify_candidate(
+        CandidateObservation(**{**base.__dict__, "pace_cps": 4.31}),
+        max_pace_cps=4.3,
+    )
+
+    assert missing.rejection_reasons == ("missing_pace_evidence",)
+    assert fast.rejection_reasons == ("pace_too_fast",)
+
+
+def test_long_candidate_requires_speaker_evidence_and_clamps_negative_drop():
+    observation = CandidateObservation(
+        target_text="這是一段完整而且穩定的合成內容",
+        transcript_text="這是一段完整而且穩定的合成內容",
+        audio_duration_seconds=2.0,
+        speaker_similarity=0.70,
+        begin_speaker_similarity=0.50,
+        end_speaker_similarity=0.60,
+    )
+    result = verify_candidate(observation)
+
+    assert result.passed
+    assert result.speaker_gate_applied
+    assert result.boundary_speaker_drop == 0.0
+    assert result.score == pytest.approx(0.05 * 0.30)
+
+    missing = verify_candidate(
+        CandidateObservation(
+            target_text=observation.target_text,
+            transcript_text=observation.transcript_text,
+            audio_duration_seconds=2.0,
+        )
+    )
+    assert not missing.passed
+    assert "missing_speaker_evidence" in missing.rejection_reasons
+
+
+def test_candidate_rejects_boundary_drop_low_similarity_truncation_and_tail():
+    base = dict(
+        target_text="這是一段完整而且穩定的合成內容",
+        transcript_text="這是一段完整而且穩定的合成內容",
+        audio_duration_seconds=2.0,
+        speaker_similarity=0.7,
+        begin_speaker_similarity=0.5,
+        end_speaker_similarity=0.5,
+    )
+    boundary = verify_candidate(
+        CandidateObservation(**{**base, "begin_speaker_similarity": 0.8})
+    )
+    similarity = verify_candidate(
+        CandidateObservation(**{**base, "speaker_similarity": 0.09})
+    )
+    truncation = verify_candidate(CandidateObservation(**{**base, "truncated": True}))
+    tail = verify_candidate(
+        CandidateObservation(**{**base, "transcript_text": base["target_text"] + "多講"})
+    )
+
+    assert boundary.rejection_reasons == ("boundary_speaker_drop",)
+    assert similarity.rejection_reasons == ("speaker_similarity",)
+    assert truncation.rejection_reasons == ("truncated",)
+    assert "semantic_gate" in tail.rejection_reasons
+    # ASR comparison canonicalizes both sides to Simplified Chinese.
+    assert tail.comparison.extra_tail == "多讲"
+
+
+def test_candidate_and_trajectory_fail_closed_for_bad_values_or_any_bad_chunk():
+    good_short = CandidateObservation("內容完整", "內容完整", 1.0)
+    bad_short = CandidateObservation("句尾完整", "句尾", 1.0)
+    invalid = verify_candidate(
+        CandidateObservation("內容完整", "內容完整", math.nan),
+        max_cer=math.nan,
+    )
+    passed_trajectory = verify_trajectory([good_short, good_short])
+    failed_trajectory = verify_trajectory([good_short, bad_short])
+
+    assert not invalid.passed
+    assert "invalid_audio_duration" in invalid.rejection_reasons
+    assert "invalid_gate_config" in invalid.rejection_reasons
+    assert passed_trajectory.passed
+    assert passed_trajectory.score == 0.0
+    assert not failed_trajectory.passed
+    assert math.isinf(failed_trajectory.score)
+    assert failed_trajectory.rejection_reasons == ("chunk_1:semantic_gate",)
+    assert verify_trajectory([]).rejection_reasons == ("empty_trajectory",)
+
+
+def test_final_whole_output_gate_rejects_post_join_regression_without_fallback():
+    passed = verify_trajectory(
+        [CandidateObservation("整段內容完整", "整段內容完整", 1.0)]
+    )
+    rejected = verify_trajectory(
+        [CandidateObservation("整段內容完整", "整段內容", 1.0)]
+    )
+
+    assert require_verified_final_output(passed) is passed
+    with pytest.raises(FinalOutputRejectedError, match="final output rejected"):
+        require_verified_final_output(rejected)
+    with pytest.raises(FinalOutputRejectedError, match="invalid result"):
+        require_verified_final_output(None)
+
+
+def test_joined_output_rejection_preserves_local_results_for_sequence_dp():
+    observations = []
+    artifacts = []
+    for target in ("第一段完整", "第二段完整"):
+        observation, artifact = _chunk_verification(
+            target,
+            target,
+            embedding=[1.0, 0.0],
+            rms_db=-20.0,
+        )
+        observations.append(observation)
+        artifacts.append(artifact)
+    local = verify_trajectory(observations, chunk_artifacts=artifacts)
+    joined_failed = _joined_verification("第一段完整第二段完整", "第一段完整")
+
+    qualified = qualify_trajectory_with_joined_output(local, joined_failed)
+
+    assert local.passed
+    assert not qualified.passed
+    assert math.isinf(qualified.score)
+    assert qualified.candidate_results is local.candidate_results
+    assert qualified.chunk_artifacts is local.chunk_artifacts
+    assert qualified.rejection_reasons == ("joined_output:chunk_0:semantic_gate",)
+    assert all(result.passed for result in qualified.candidate_results)
+
+
+def test_joined_output_qualification_returns_local_result_only_when_joined_passes():
+    local = verify_trajectory(
+        [CandidateObservation("第一段完整", "第一段完整", 1.0)]
+    )
+    joined = verify_trajectory(
+        [CandidateObservation("第一段完整", "第一段完整", 1.0)]
+    )
+
+    assert qualify_trajectory_with_joined_output(local, joined) is local
+
+
+def test_k_best_sequence_paths_are_cost_ranked_stable_distinct_and_bounded():
+    ranked = select_k_candidate_sequences(
+        [[0.0, 0.2], [0.0, 0.1]],
+        [[[0.4, 0.0], [0.0, 0.5]]],
+        max_paths=3,
+    )
+
+    assert [selection.candidate_indices for selection in ranked] == [
+        (0, 1),
+        (1, 0),
+        (0, 0),
+    ]
+    assert [selection.total_score for selection in ranked] == pytest.approx(
+        [0.1, 0.2, 0.4]
+    )
+    assert len({selection.candidate_indices for selection in ranked}) == 3
+
+    tied = select_k_candidate_sequences(
+        [[0.0, 0.0], [0.0, 0.0]],
+        [[[0.0, 0.0], [0.0, 0.0]]],
+        max_paths=3,
+    )
+    assert [selection.candidate_indices for selection in tied] == [
+        (0, 0),
+        (0, 1),
+        (1, 0),
+    ]
+
+
+def test_k_best_sequence_selection_fails_closed_for_malformed_or_single_chunk_graphs():
+    assert select_k_candidate_sequences([[0.0, 0.1]], (), max_paths=3) == ()
+    assert select_k_candidate_sequences([[0.0], [0.0]], (), max_paths=3) == ()
+    assert (
+        select_k_candidate_sequences(
+            [[0.0, 0.1], [0.0, 0.1]],
+            [[[0.0]]],
+            max_paths=3,
+        )
+        == ()
+    )
+    for invalid in (0, 4, 1.0, True):
+        with pytest.raises(ValueError, match="between 1 and 3"):
+            select_k_candidate_sequences(
+                [[0.0], [0.0]],
+                [[[0.0]]],
+                max_paths=invalid,
+            )
+
+
+def _locally_safe_join_rejected_verification(chunks, seed):
+    # Crossed embeddings make the two mixed paths cheaper than either
+    # same-candidate path, giving deterministic k-best callback order.
+    first_candidate = seed % 2 == 0
+    similarities = (0.9, 0.2) if first_candidate else (0.2, 0.9)
+    embeddings = (
+        ([1.0, 0.0], [0.0, 1.0])
+        if first_candidate
+        else ([0.0, 1.0], [1.0, 0.0])
+    )
+    observations = []
+    artifacts = []
+    for target, similarity, embedding in zip(
+        chunks,
+        similarities,
+        embeddings,
+        strict=True,
+    ):
+        observations.append(
+            CandidateObservation(
+                target_text=target,
+                transcript_text=target,
+                audio_duration_seconds=2.0,
+                speaker_similarity=similarity,
+                begin_speaker_similarity=similarity,
+                end_speaker_similarity=similarity,
+            )
+        )
+        artifacts.append(
+            ChunkCandidateArtifact(
+                speaker_embedding=np.asarray(embedding, dtype=np.float32),
+                rms_db=-20.0,
+            )
+        )
+    local = verify_trajectory(observations, chunk_artifacts=artifacts)
+    joined_failed = _joined_verification("".join(chunks), chunks[0])
+    return qualify_trajectory_with_joined_output(local, joined_failed)
+
+
+def test_k_best_sequence_final_callback_uses_rank_order_and_first_passing_path():
+    chunks = ("第一段完整", "第二段完整")
+    callback_paths = []
+
+    def generator(candidate_chunks, seed):
+        return tuple(f"seed-{seed}-chunk-{index}" for index in range(len(candidate_chunks)))
+
+    def verifier(trajectory, candidate_chunks, seed):
+        return _locally_safe_join_rejected_verification(candidate_chunks, seed)
+
+    def final_verifier(sequence_result, candidate_chunks):
+        callback_paths.append(sequence_result.chunk_candidate_indices)
+        transcript = (
+            "".join(candidate_chunks)
+            if sequence_result.chunk_candidate_indices == (1, 0)
+            else candidate_chunks[0]
+        )
+        return _joined_verification("".join(candidate_chunks), transcript)
+
+    result = run_adaptive_cascade(
+        chunks,
+        20,
+        generator,
+        verifier,
+        max_candidates=2,
+        sequence_final_verifier=final_verifier,
+        max_sequence_paths=3,
+    )
+
+    assert callback_paths == [(0, 1), (1, 0)]
+    assert result.selection_mode == "sequence_dp"
+    assert result.chunk_candidate_indices == (1, 0)
+    assert result.sequence_path_rank == 2
+    assert result.sequence_paths_checked == 2
+
+
+def test_k_best_sequence_all_paths_rejected_is_no_qualified_candidate():
+    chunks = ("第一段完整", "第二段完整")
+    callback_paths = []
+
+    def generator(candidate_chunks, seed):
+        return tuple(f"seed-{seed}-chunk-{index}" for index in range(len(candidate_chunks)))
+
+    def final_verifier(sequence_result, candidate_chunks):
+        callback_paths.append(sequence_result.chunk_candidate_indices)
+        return _joined_verification("".join(candidate_chunks), candidate_chunks[0])
+
+    with pytest.raises(NoQualifiedCandidateError, match="after 2 candidates"):
+        run_adaptive_cascade(
+            chunks,
+            20,
+            generator,
+            lambda trajectory, candidate_chunks, seed: (
+                _locally_safe_join_rejected_verification(candidate_chunks, seed)
+            ),
+            max_candidates=2,
+            sequence_final_verifier=final_verifier,
+            max_sequence_paths=3,
+        )
+
+    assert callback_paths == [(0, 1), (1, 0), (0, 0)]
+    assert len(set(callback_paths)) == 3
+
+
+def test_k_best_sequence_callback_exception_aborts_fail_closed():
+    chunks = ("第一段完整", "第二段完整")
+    callback_paths = []
+
+    def callback(sequence_result, candidate_chunks):
+        callback_paths.append(sequence_result.chunk_candidate_indices)
+        raise ValueError("ASR backend disappeared")
+
+    with pytest.raises(RuntimeError, match="refusing unverified audio"):
+        run_adaptive_cascade(
+            chunks,
+            20,
+            lambda candidate_chunks, seed: tuple(candidate_chunks),
+            lambda trajectory, candidate_chunks, seed: (
+                _locally_safe_join_rejected_verification(candidate_chunks, seed)
+            ),
+            max_candidates=2,
+            sequence_final_verifier=callback,
+        )
+
+    assert callback_paths == [(0, 1)]
+
+
+def test_k_best_sequence_invalid_callback_result_aborts_fail_closed():
+    chunks = ("第一段完整", "第二段完整")
+
+    with pytest.raises(RuntimeError, match="invalid result"):
+        run_adaptive_cascade(
+            chunks,
+            20,
+            lambda candidate_chunks, seed: tuple(candidate_chunks),
+            lambda trajectory, candidate_chunks, seed: (
+                _locally_safe_join_rejected_verification(candidate_chunks, seed)
+            ),
+            max_candidates=2,
+            sequence_final_verifier=lambda *args: None,
+        )
+
+
+def test_k_best_sequence_inconsistent_callback_evidence_never_passes():
+    chunks = ("第一段完整", "第二段完整")
+    failed_candidate = verify_candidate(
+        CandidateObservation("完整內容", "錯誤內容", 1.0)
+    )
+
+    with pytest.raises(NoQualifiedCandidateError, match="after 2 candidates"):
+        run_adaptive_cascade(
+            chunks,
+            20,
+            lambda candidate_chunks, seed: tuple(candidate_chunks),
+            lambda trajectory, candidate_chunks, seed: (
+                _locally_safe_join_rejected_verification(candidate_chunks, seed)
+            ),
+            max_candidates=2,
+            sequence_final_verifier=lambda *args: TrajectoryGateResult(
+                passed=True,
+                candidate_results=(failed_candidate,),
+                score=0.0,
+                rejection_reasons=(),
+            ),
+        )
+
+
+def test_sequence_final_callback_is_not_used_for_single_chunk_or_safe_whole():
+    callback_calls = []
+
+    with pytest.raises(NoQualifiedCandidateError):
+        run_adaptive_cascade(
+            ("單一完整段落",),
+            20,
+            lambda candidate_chunks, seed: tuple(candidate_chunks),
+            lambda trajectory, candidate_chunks, seed: TrajectoryGateResult(
+                passed=False,
+                candidate_results=(
+                    verify_candidate(
+                        CandidateObservation(
+                            candidate_chunks[0],
+                            candidate_chunks[0],
+                            1.0,
+                        )
+                    ),
+                ),
+                score=math.inf,
+                rejection_reasons=("joined_output:semantic_gate",),
+                chunk_artifacts=(ChunkCandidateArtifact(rms_db=-20.0),),
+            ),
+            max_candidates=2,
+            sequence_final_verifier=lambda *args: callback_calls.append(args),
+        )
+    assert callback_calls == []
+
+    result = run_adaptive_cascade(
+        ("第一段", "第二段"),
+        30,
+        lambda candidate_chunks, seed: tuple(candidate_chunks),
+        lambda *args: _gate_result(True, 0.1),
+        max_candidates=2,
+        sequence_final_verifier=lambda *args: callback_calls.append(args),
+    )
+    assert result.selection_mode == "whole_trajectory"
+    assert callback_calls == []
+
+
+def test_local_pass_join_fail_expands_and_selects_next_joined_safe_whole():
+    calls = []
+
+    def generator(chunks, seed):
+        calls.append(seed)
+        return tuple(f"seed-{seed}-{chunk}" for chunk in chunks)
+
+    def verifier(trajectory, chunks, seed):
+        observations = []
+        artifacts = []
+        for target in chunks:
+            observation, artifact = _chunk_verification(
+                target,
+                target,
+                embedding=[1.0, 0.0],
+                rms_db=-20.0,
+            )
+            observations.append(observation)
+            artifacts.append(artifact)
+        local = verify_trajectory(observations, chunk_artifacts=artifacts)
+        joined_transcript = "".join(chunks) if seed == 11 else chunks[0]
+        joined = _joined_verification("".join(chunks), joined_transcript)
+        return qualify_trajectory_with_joined_output(local, joined)
+
+    result = run_adaptive_cascade(
+        ("第一段完整", "第二段完整"),
+        10,
+        generator,
+        verifier,
+        max_candidates=2,
+    )
+
+    assert calls == [10, 11]
+    assert result.selection_mode == "whole_trajectory"
+    assert result.candidate_index == 1
+    assert result.seed == 11
+
+
+def test_all_local_pass_join_fail_can_only_return_as_sequence_dp():
+    chunks = ("第一段完整", "第二段完整")
+
+    def generator(candidate_chunks, seed):
+        return tuple(f"seed-{seed}-{chunk}" for chunk in candidate_chunks)
+
+    def verifier(trajectory, candidate_chunks, seed):
+        similarities = (0.9, 0.2) if seed == 20 else (0.2, 0.9)
+        observations = []
+        artifacts = []
+        for target, similarity in zip(candidate_chunks, similarities, strict=True):
+            observations.append(
+                CandidateObservation(
+                    target_text=target,
+                    transcript_text=target,
+                    audio_duration_seconds=2.0,
+                    speaker_similarity=similarity,
+                    begin_speaker_similarity=similarity,
+                    end_speaker_similarity=similarity,
+                )
+            )
+            artifacts.append(
+                ChunkCandidateArtifact(
+                    speaker_embedding=np.array([1.0, 0.0], dtype=np.float32),
+                    rms_db=-20.0,
+                )
+            )
+        local = verify_trajectory(observations, chunk_artifacts=artifacts)
+        joined = _joined_verification("".join(candidate_chunks), candidate_chunks[0])
+        return qualify_trajectory_with_joined_output(local, joined)
+
+    result = run_adaptive_cascade(
+        chunks,
+        20,
+        generator,
+        verifier,
+        max_candidates=2,
+    )
+
+    assert result.selection_mode == "sequence_dp"
+    assert result.candidate_index is None
+    assert result.seed is None
+    assert result.chunk_candidate_indices == (0, 1)
+    assert result.attempted_seeds == (20, 21)
+
+
+def test_adaptive_cascade_returns_first_candidate_immediately():
+    generation_calls = []
+    verification_calls = []
+
+    def generator(chunks, seed):
+        generation_calls.append((chunks, seed))
+        return tuple((chunk, seed) for chunk in chunks)
+
+    def verifier(trajectory, chunks, seed):
+        verification_calls.append((trajectory, chunks, seed))
+        assert all(chunk_seed == seed for _, chunk_seed in trajectory)
+        return _gate_result(True, 0.2)
+
+    result = run_adaptive_cascade(
+        ["第一段", "第二段"],
+        12345,
+        generator,
+        verifier,
+    )
+
+    assert isinstance(result, CascadeResult)
+    assert result.seed == 12345
+    assert result.candidate_index == 0
+    assert result.attempted_seeds == (12345,)
+    assert generation_calls == [(("第一段", "第二段"), 12345)]
+    assert len(verification_calls) == 1
+
+
+def test_marginal_stage_one_speaker_pass_expands_to_preferred_candidate():
+    calls = []
+
+    def generator(chunks, seed):
+        calls.append(seed)
+        return (chunks[0], seed)
+
+    def verifier(trajectory, chunks, seed):
+        if seed == 100:
+            return _whole_speaker_verification(
+                similarity=0.20,
+                boundary_drop=0.04,
+            )
+        if seed == 105:
+            return _whole_speaker_verification(
+                similarity=0.35,
+                boundary_drop=0.02,
+            )
+        return _whole_speaker_verification(
+            similarity=0.30,
+            boundary_drop=0.02,
+            passed=False,
+        )
+
+    result = run_adaptive_cascade(
+        ["完整而且穩定的候選內容"],
+        100,
+        generator,
+        verifier,
+        max_candidates=10,
+    )
+
+    assert calls == list(range(100, 110))
+    assert result.selection_mode == "whole_trajectory"
+    assert result.candidate_index == 5
+    assert result.seed == 105
+
+
+def test_final_stage_can_return_marginal_hard_pass_to_preserve_availability():
+    calls = []
+
+    def generator(chunks, seed):
+        calls.append(seed)
+        return (chunks[0], seed)
+
+    def verifier(trajectory, chunks, seed):
+        return _whole_speaker_verification(
+            similarity=0.20,
+            boundary_drop=0.08,
+            passed=seed == 200,
+        )
+
+    result = run_adaptive_cascade(
+        ["完整而且穩定的候選內容"],
+        200,
+        generator,
+        verifier,
+        max_candidates=2,
+    )
+
+    assert calls == [200, 201]
+    assert result.seed == 200
+    assert result.candidate_index == 0
+    assert result.verification.passed
+
+
+def test_adaptive_cascade_expands_to_five_and_selects_lowest_verified_score():
+    calls = []
+    scores = {
+        100: None,
+        101: 0.4,
+        102: None,
+        103: 0.1,
+        104: 0.1,
+    }
+
+    def generator(chunks, seed):
+        calls.append((chunks, seed))
+        return {"chunks": tuple((chunk, seed) for chunk in chunks), "seed": seed}
+
+    def verifier(trajectory, chunks, seed):
+        assert trajectory["seed"] == seed
+        assert all(chunk_seed == seed for _, chunk_seed in trajectory["chunks"])
+        score = scores[seed]
+        return _gate_result(score is not None, score or 0.0)
+
+    result = run_adaptive_cascade(
+        ["第一段", "第二段"],
+        100,
+        generator,
+        verifier,
+    )
+
+    assert [seed for _, seed in calls] == [100, 101, 102, 103, 104]
+    assert result.seed == 103
+    assert result.candidate_index == 3
+    assert result.verification.score == 0.1
+    assert result.attempted_seeds == (100, 101, 102, 103, 104)
+
+
+def test_adaptive_cascade_uses_ten_only_when_stage_five_has_no_verified_candidate():
+    calls = []
+
+    def generator(chunks, seed):
+        calls.append(seed)
+        return (chunks, seed)
+
+    def verifier(trajectory, chunks, seed):
+        return _gate_result(seed in {107, 109}, score=float(110 - seed))
+
+    result = run_adaptive_cascade(
+        ["完整內容"],
+        100,
+        generator,
+        verifier,
+        max_candidates=10,
+    )
+
+    assert result.seed == 109
+    assert result.candidate_index == 9
+    assert result.attempted_seeds == tuple(range(100, 110))
+    assert calls == list(range(100, 110))
+
+
+def test_sequence_fallback_can_take_a_prefix_from_a_and_suffix_from_b():
+    chunks = ("第一段完整", "第二段完整")
+    trajectories = {
+        10: ("audio-a0", "audio-a1"),
+        11: ("audio-b0", "audio-b1"),
+    }
+
+    def generator(candidate_chunks, seed):
+        assert candidate_chunks == chunks
+        return trajectories[seed]
+
+    def verifier(trajectory, candidate_chunks, seed):
+        transcripts = (
+            candidate_chunks
+            if seed == 999
+            else (
+                (candidate_chunks[0], "錯誤內容")
+                if seed == 10
+                else ("錯誤內容", candidate_chunks[1])
+            )
+        )
+        observations = []
+        artifacts = []
+        for target, transcript in zip(candidate_chunks, transcripts, strict=True):
+            observation, artifact = _chunk_verification(
+                target,
+                transcript,
+                embedding=[1.0, 0.0],
+                rms_db=-20.0,
+            )
+            observations.append(observation)
+            artifacts.append(artifact)
+        return verify_trajectory(observations, chunk_artifacts=artifacts)
+
+    result = run_adaptive_cascade(
+        chunks,
+        10,
+        generator,
+        verifier,
+        max_candidates=2,
+    )
+
+    assert result.selection_mode == "sequence_dp"
+    assert result.seed is None
+    assert result.candidate_index is None
+    assert result.trajectory == ("audio-a0", "audio-b1")
+    assert result.chunk_candidate_indices == (0, 1)
+    assert result.chunk_seeds == (10, 11)
+    assert result.attempted_seeds == (10, 11)
+    assert result.verification.passed
+    assert len(result.verification.candidate_results) == 2
+    assert len(result.verification.chunk_artifacts) == 2
+
+
+def test_same_seed_whole_trajectory_wins_before_sequence_fallback():
+    chunks = ("第一段完整", "第二段完整")
+
+    def generator(candidate_chunks, seed):
+        return tuple(f"seed-{seed}-chunk-{index}" for index in range(len(candidate_chunks)))
+
+    def verifier(trajectory, candidate_chunks, seed):
+        transcripts = (
+            (candidate_chunks[0], "錯誤內容")
+            if seed == 20
+            else candidate_chunks
+        )
+        observations = []
+        artifacts = []
+        for target, transcript in zip(candidate_chunks, transcripts, strict=True):
+            observation, artifact = _chunk_verification(
+                target,
+                transcript,
+                embedding=[1.0, 0.0],
+                rms_db=-20.0,
+            )
+            observations.append(observation)
+            artifacts.append(artifact)
+        return verify_trajectory(observations, chunk_artifacts=artifacts)
+
+    result = run_adaptive_cascade(
+        chunks,
+        20,
+        generator,
+        verifier,
+        max_candidates=2,
+    )
+
+    assert result.selection_mode == "whole_trajectory"
+    assert result.seed == 21
+    assert result.candidate_index == 1
+    assert result.chunk_candidate_indices == (1, 1)
+    assert result.chunk_seeds == (21, 21)
+    assert result.trajectory == ("seed-21-chunk-0", "seed-21-chunk-1")
+
+
+def test_sequence_fallback_rejects_unsafe_transition_edge():
+    chunks = ("第一段完整", "第二段完整")
+
+    def generator(candidate_chunks, seed):
+        return tuple(f"seed-{seed}-chunk-{index}" for index in range(len(candidate_chunks)))
+
+    def verifier(trajectory, candidate_chunks, seed):
+        transcripts = (
+            (candidate_chunks[0], "錯誤內容")
+            if seed == 30
+            else ("錯誤內容", candidate_chunks[1])
+        )
+        embeddings = (
+            ([1.0, 0.0], [1.0, 0.0])
+            if seed == 30
+            else ([1.0, 0.0], [1.0, 0.0, 0.0])
+        )
+        observations = []
+        artifacts = []
+        for target, transcript, embedding in zip(
+            candidate_chunks,
+            transcripts,
+            embeddings,
+            strict=True,
+        ):
+            observation, artifact = _chunk_verification(
+                target,
+                transcript,
+                embedding=embedding,
+                rms_db=-20.0,
+            )
+            observations.append(observation)
+            artifacts.append(artifact)
+        return verify_trajectory(observations, chunk_artifacts=artifacts)
+
+    with pytest.raises(NoQualifiedCandidateError, match="after 2 candidates"):
+        run_adaptive_cascade(
+            chunks,
+            30,
+            generator,
+            verifier,
+            max_candidates=2,
+        )
+
+
+def test_transition_score_requires_safe_evidence_and_penalizes_rms_delta():
+    first_observation, first_artifact = _chunk_verification(
+        "第一段完整",
+        "第一段完整",
+        embedding=[1.0, 0.0],
+        rms_db=-20.0,
+    )
+    second_observation, second_artifact = _chunk_verification(
+        "第二段完整",
+        "第二段完整",
+        embedding=[1.0, 0.0],
+        rms_db=-16.0,
+    )
+    first = verify_trajectory([first_observation], chunk_artifacts=[first_artifact])
+    second = verify_trajectory([second_observation], chunk_artifacts=[second_artifact])
+
+    score = candidate_chunk_transition_score(
+        first.candidate_results[0],
+        first.chunk_artifacts[0],
+        second.candidate_results[0],
+        second.chunk_artifacts[0],
+    )
+    assert score == pytest.approx(0.20)
+
+    unsafe = ChunkCandidateArtifact(speaker_embedding=None, rms_db=-16.0)
+    assert math.isinf(
+        candidate_chunk_transition_score(
+            first.candidate_results[0],
+            first.chunk_artifacts[0],
+            second.candidate_results[0],
+            unsafe,
+        )
+    )
+
+
+@pytest.mark.parametrize(
+    ("chunk_count", "expected_limit"),
+    [(1, 10), (2, 10), (3, 6), (4, 5), (5, 4), (6, 3), (10, 2), (20, 1)],
+)
+def test_candidate_limit_obeys_twenty_generated_chunk_budget(chunk_count, expected_limit):
+    limit = candidate_limit_for_chunk_budget(chunk_count)
+    assert limit == expected_limit
+    assert limit * chunk_count <= 20
+
+
+def test_candidate_limit_rejects_a_trajectory_that_already_exceeds_budget():
+    with pytest.raises(ValueError, match="one trajectory exceeds"):
+        candidate_limit_for_chunk_budget(21)
+    with pytest.raises(ValueError, match="between 1 and 10"):
+        candidate_limit_for_chunk_budget(1, max_candidates=11)
+
+
+def test_360_character_profiles_stay_inside_generated_chunk_budget():
+    profiles = [
+        "甲" * 360,
+        ("甲" * 11 + "。") * 30,
+        ("甲" * 40 + "。") * 8 + "乙" * 32,
+    ]
+    for text in profiles:
+        chunks = split_text_for_tts(text, max_chars=80, min_chunk_chars=12)
+        chunks = split_leading_clause(
+            chunks[0],
+            search_chars=40,
+            min_chunk_chars=12,
+        ) + chunks[1:]
+        limit = candidate_limit_for_chunk_budget(len(chunks))
+        assert limit * len(chunks) <= 20
+
+
+def test_candidate_asr_value_error_rejects_seed_and_cascade_continues():
+    calls = []
+
+    def generator(chunks, seed):
+        calls.append(seed)
+        return _tone(seconds=0.25)
+
+    def verifier(trajectory, chunks, seed):
+        prepared = prepare_candidate_audio(
+            trajectory,
+            16_000,
+            transcriber=(
+                (lambda *_: (_ for _ in ()).throw(ValueError("candidate ASR failure")))
+                if seed == 50
+                else (lambda *_: chunks[0])
+            ),
+        )
+        if prepared is None:
+            return _gate_result(False)
+        return verify_trajectory(
+            [
+                CandidateObservation(
+                    target_text=chunks[0],
+                    transcript_text=prepared.transcript_text,
+                    audio_duration_seconds=prepared.duration_seconds,
+                )
+            ]
+        )
+
+    result = run_adaptive_cascade(
+        ["完整內容"],
+        50,
+        generator,
+        verifier,
+        max_candidates=2,
+    )
+
+    assert result.seed == 51
+    assert result.candidate_index == 1
+    assert calls == [50, 51]
+
+
+def test_adaptive_cascade_raises_without_qualified_candidate_and_never_falls_back():
+    generated = []
+
+    def generator(chunks, seed):
+        generated.append(seed)
+        return (chunks, seed)
+
+    with pytest.raises(NoQualifiedCandidateError, match="after 5 candidates"):
+        run_adaptive_cascade(
+            ["必須完整"],
+            7,
+            generator,
+            lambda trajectory, chunks, seed: _gate_result(False),
+        )
+    assert generated == [7, 8, 9, 10, 11]
+
+
+def test_adaptive_cascade_validates_contract_and_rejects_non_gate_verifier():
+    generator = lambda chunks, seed: (chunks, seed)
+    with pytest.raises(ValueError, match="start with exactly one"):
+        run_adaptive_cascade(
+            ["內容"],
+            1,
+            generator,
+            lambda *args: _gate_result(True),
+            initial_candidates=2,
+        )
+    with pytest.raises(ValueError, match="between 1 and 10"):
+        run_adaptive_cascade(
+            ["內容"],
+            1,
+            generator,
+            lambda *args: _gate_result(True),
+            max_candidates=11,
+        )
+    with pytest.raises(TypeError, match="TrajectoryGateResult"):
+        run_adaptive_cascade(["內容"], 1, generator, lambda *args: True)

tests/test_release_pins.py

@@ -1,4 +1,5 @@
 import ast
+import hashlib
 from pathlib import Path
 
 
@@ -26,12 +27,203 @@ def test_remote_model_and_speaker_encoder_are_revision_pinned():
     assert constants["MODEL_REVISION"] == "aaf1a0878e37875382bb0e5c8a3a2ba43be67297"
     assert constants["ECAPA_REVISION"] == "0f99f2d0ebe89ac095bcc5903c4dd8f72b367286"
     assert "snapshot_download(REPO_ID, revision=MODEL_REVISION)" in source
-    assert "revision=ECAPA_REVISION" in source
+    assert "snapshot_download(ECAPA_REPO_ID, revision=ECAPA_REVISION)" in source
+    assert "source=ECAPA_DIR" in source
+    assert 'overrides={"pretrained_path": ECAPA_DIR}' in source
 
 
-def test_tts_runtime_dependency_is_commit_pinned():
+def test_tts_runtime_is_vendored_from_the_frozen_commit():
     requirements = (ROOT / "requirements.txt").read_text(encoding="utf-8").splitlines()
-    tts_lines = [line.strip() for line in requirements if "BlueMagpie-TTS.git" in line]
+    provenance = (ROOT / "bluemagpie" / "UPSTREAM_RUNTIME.md").read_text(
+        encoding="utf-8"
+    )
 
-    assert len(tts_lines) == 1
-    assert "@" in tts_lines[0].rsplit("/", 1)[-1]
+    assert not any("BlueMagpie-TTS.git" in line for line in requirements)
+    assert "ce384c8cc54efea1aaba7b9f1d7ded6c1c99aa9a" in provenance
+    assert (ROOT / "bluemagpie" / "LICENSE.upstream").is_file()
+    assert (ROOT / "bluemagpie" / "_vendor" / "voxcpm" / "LICENSE").is_file()
+    pinned_hashes = {
+        "model.py": "91810524212b34f727880154d90653fab4ae1b75eb3471b86cafd92c75514fef",
+        "loading.py": "e3407544e9bc888018fe5771edc01d954469e2af548b566873e0ef7f2afe6dca",
+        "_vendor/voxcpm/model/utils.py": (
+            "cea16e1ab57f15129a7f5dec13c428bd14a771221abcf17dd0a90b3d65e763a2"
+        ),
+    }
+    for relative_path, expected_hash in pinned_hashes.items():
+        payload = (ROOT / "bluemagpie" / relative_path).read_bytes()
+        assert hashlib.sha256(payload).hexdigest() == expected_hash
+
+
+def test_barbet_runtime_dependency_is_commit_pinned():
+    requirements = (ROOT / "requirements.txt").read_text(encoding="utf-8").splitlines()
+    barbet_lines = [line for line in requirements if "OpenFormosa/Barbet.git" in line]
+
+    assert barbet_lines == [
+        "git+https://github.com/OpenFormosa/Barbet.git@"
+        "6fcd7ce4aa37f2250a3242995bef0fbc3b026ba8"
+    ]
+
+
+def test_quality_asr_is_revision_pinned():
+    constants = _string_constants(ROOT / "quality_runtime.py")
+    app_source = (ROOT / "app.py").read_text(encoding="utf-8")
+
+    assert constants["WHISPER_MODEL_ID"] == "openai/whisper-large-v3-turbo"
+    assert constants["WHISPER_REVISION"] == "41f01f3fe87f28c78e2fbf8b568835947dd65ed9"
+    assert "snapshot_download(WHISPER_MODEL_ID, revision=WHISPER_REVISION)" in app_source
+
+
+def test_quality_runtime_dependencies_are_version_pinned():
+    requirements = set(
+        (ROOT / "requirements.txt").read_text(encoding="utf-8").splitlines()
+    )
+
+    assert "huggingface_hub==0.36.0" in requirements
+    assert "opencc-python-reimplemented==0.1.7" in requirements
+    assert "transformers==4.57.6" in requirements
+    assert "accelerate==1.12.0" in requirements
+    assert "einops==0.8.2" in requirements
+    assert "pydantic==2.11.10" in requirements
+    assert "numpy==2.3.5" in requirements
+    assert "scipy==1.17.1" in requirements
+    assert "numexpr==2.14.1" in requirements
+    assert "bottleneck==1.6.0" in requirements
+    assert "tqdm==4.68.2" in requirements
+    assert "safetensors==0.8.0" in requirements
+    assert "librosa==0.11.0" in requirements
+    assert "soundfile==0.14.0" in requirements
+    assert "speechbrain==1.0.3" in requirements
+
+
+def test_readme_describes_conditional_cascade_and_unwired_experimental_scores():
+    readme = (ROOT / "README.md").read_text(encoding="utf-8")
+
+    assert "same-seed whole trajectory first" in readme
+    assert "1→5→10 sequence fallback" in readme
+    assert "floor(20 / chunk count)" in readme
+    assert "at most 20 generated TTS chunks" in readme
+    assert "不使用 reference-style distribution score" in readme
+    assert "尚未啟用 normalized median-F0 transition" in readme
+    assert "candidate 0" in readme
+    assert "5.2 CJK / 4.6 ASCII" in readme
+    assert "candidate 1–9" in readme
+    assert "4.6 CJK / 4.0 ASCII" in readme
+
+
+def test_app_wires_candidate_offset_to_explicit_generation_policy_and_logs_it():
+    source = (ROOT / "app.py").read_text(encoding="utf-8")
+
+    assert source.count("policy: GenerationPolicy") == 2
+    assert "policy=generation_policy_for_candidate_offset(seed - request_seed)" in source
+    assert 'f"name={policy.name}' in source
+    assert "chunk_policies={selected_policies}" in source
+    assert '"min_len": min_len' in source
+
+
+def test_internal_synthesize_accepts_only_a_keyword_seed_while_ui_stays_unchanged():
+    source = (ROOT / "app.py").read_text(encoding="utf-8")
+    tree = ast.parse(source)
+    functions = {
+        node.name: node
+        for node in tree.body
+        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef))
+    }
+
+    synthesize = functions["_synthesize"]
+    assert "request_seed" not in [argument.arg for argument in synthesize.args.args]
+    assert [argument.arg for argument in synthesize.args.kwonlyargs][-1] == "request_seed"
+    assert isinstance(synthesize.args.kw_defaults[-1], ast.Constant)
+    assert synthesize.args.kw_defaults[-1].value is None
+    assert "request_seed = resolve_request_seed(request_seed, secrets.randbelow)" in source
+    assert "run_adaptive_cascade(\n                chunks,\n                request_seed," in source
+    for wrapper_name in ("tts_speaker", "tts_reference", "tts_longform"):
+        wrapper = functions[wrapper_name]
+        arguments = wrapper.args.args + wrapper.args.kwonlyargs
+        assert "request_seed" not in [argument.arg for argument in arguments]
+
+
+def test_app_reverifies_the_post_join_speed_adjusted_whole_waveform():
+    source = (ROOT / "app.py").read_text(encoding="utf-8")
+    tree = ast.parse(source)
+    functions = {
+        node.name: node
+        for node in tree.body
+        if isinstance(node, ast.FunctionDef)
+    }
+    assemble_source = ast.get_source_segment(
+        source,
+        functions["_assemble_trajectory_audio"],
+    )
+    synthesize_source = ast.get_source_segment(source, functions["_synthesize"])
+
+    assert assemble_source is not None
+    assert synthesize_source is not None
+    speed_index = assemble_source.index(
+        "waveform = _apply_speed(waveform, playback_speed)"
+    )
+    finish_index = assemble_source.index("return finish_audio(waveform, SR)")
+    assert speed_index < finish_index
+    assemble_index = synthesize_source.index(
+        "waveform = _assemble_trajectory_audio(cascade.trajectory, chunks, speed)"
+    )
+    verify_index = synthesize_source.index("final_verification = _verify_trajectory_audio(")
+    require_index = synthesize_source.index(
+        "require_verified_final_output(final_verification)"
+    )
+    return_index = synthesize_source.index("return SR, waveform")
+    assert assemble_index < verify_index < require_index < return_index
+    assert "(text,)" in synthesize_source[verify_index:require_index]
+    assert "        1.0," in synthesize_source[verify_index:require_index]
+    assert "QUALITY_FINAL_ASR_MAX_NEW_TOKENS" in synthesize_source[
+        verify_index:require_index
+    ]
+
+
+def test_whole_candidate_qualification_uses_the_exact_return_assembler_after_local_pass():
+    source = (ROOT / "app.py").read_text(encoding="utf-8")
+    tree = ast.parse(source)
+    functions = {
+        node.name: node
+        for node in tree.body
+        if isinstance(node, ast.FunctionDef)
+    }
+    qualify_source = ast.get_source_segment(
+        source,
+        functions["_qualify_candidate_trajectory_audio"],
+    )
+    synthesize_source = ast.get_source_segment(source, functions["_synthesize"])
+
+    assert qualify_source is not None
+    assert synthesize_source is not None
+    local_index = qualify_source.index("local_verification = _verify_trajectory_audio(")
+    local_fail_index = qualify_source.index("if not local_verification.passed:")
+    assemble_index = qualify_source.index("waveform = _assemble_trajectory_audio(")
+    joined_index = qualify_source.index("joined_verification = _verify_trajectory_audio(")
+    assert local_index < local_fail_index < assemble_index < joined_index
+    assert "QUALITY_FINAL_ASR_MAX_NEW_TOKENS" in qualify_source[joined_index:]
+    assert "qualify_trajectory_with_joined_output(" in qualify_source[joined_index:]
+    assert (
+        "waveform = _assemble_trajectory_audio(cascade.trajectory, chunks, speed)"
+        in synthesize_source
+    )
+    assert "require_verified_final_output(final_verification)" in synthesize_source
+
+
+def test_space_wires_bounded_k_best_paths_to_exact_assembled_whole_gate():
+    source = (ROOT / "app.py").read_text(encoding="utf-8")
+    readme = (ROOT / "README.md").read_text(encoding="utf-8")
+
+    assert "sequence_final_verifier=lambda sequence_result, candidate_chunks:" in source
+    assert "_verify_sequence_trajectory_audio(" in source
+    assert "max_sequence_paths=3" in source
+    assert "waveform = _assemble_trajectory_audio(" in source
+    assert "QUALITY_FINAL_ASR_MAX_NEW_TOKENS" in source
+    assert "sequence_rank={cascade.sequence_path_rank}" in source
+    assert "sequence_paths_checked={cascade.sequence_paths_checked}" in source
+    assert "cer={comparison.cer:.6f}" in source
+    assert "prefix_cer={comparison.prefix_cer:.6f}" in source
+    assert "suffix_cer={comparison.suffix_cer:.6f}" in source
+    assert "tail_units={comparison.extra_tail_units}" in source
+    assert "最多 3 條不同完整路徑" in readme
+    assert "不增加 TTS generation 數量" in readme
+    assert "不套用於單 chunk request" in readme