Stage1: Mixed LibriSpeech + Seamless Interact (Top-4 Encoder + CTC + Speaker Activity)
Speech-LLM checkpoint for ASR alignment, trained on a mix of LibriSpeech and Seamless Interact conversational data.
Architecture
| Component | Model | Parameters |
|---|---|---|
| Encoder | nvidia/parakeet-ctc-0.6b (FastConformer CTC) |
600M |
| Projector | Conv1d + Transformer (1024 → 896, stride=2) | trainable |
| LLM | Qwen/Qwen2.5-0.5B |
500M |
- Freeze strategy:
projector_and_encoder_top— projector fully trainable, top-4 encoder layers unfrozen - CTC auxiliary loss: λ=0.3
- Speaker kernel: SortFormer-based speaker activity injected into encoder
Training Data
| Dataset | Samples | Description |
|---|---|---|
| LibriSpeech 960h | ~276k | Read English speech (standard ASR benchmark) |
| Seamless Interact | ~2.17M | Conversational English speech |
Seamless Interact Data Pipeline
- Transcripts: Canary-Qwen-2.5B ASR (preserves disfluencies: uh, um, partial words)
- Timestamps: Parakeet CTC 0.6B forced alignment (word-level)
- Speaker activity: SortFormer diarization on mixed audio (per-speaker probability at 100 Hz)
Training Configuration
- Optimizer: AdamW (lr=5e-5, weight_decay=0.01)
- Scheduler: warmup (2000 steps) + cosine decay (min_lr=1e-6)
- Effective batch size: 128 (4 GPUs × batch_size=8 × grad_accum=4)
- Total steps: 100,000
- DeepSpeed ZeRO-2, AMP bfloat16
- Gradient clipping: 5.0
Usage
import torch
checkpoint = torch.load("final.pt", map_location="cpu")
# Load into SpeechLLM model — see the parakeet-turntaking repository for details
Files
final.pt— Full model checkpoint (encoder + projector + LLM state dicts)config.yaml— Training configuration
- Downloads last month
- -
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support