# Echo 840M - Continued Pre-Training (300M samples)

Continued pre-training of the Echo 840M DiT text-to-speech model.

## Base Checkpoint
- From: [TTS-AGI/echo-800m-v1-ckpts](https://huggingface.co/TTS-AGI/echo-800m-v1-ckpts) step 300K
- Original training: ~76.8M samples

## Training Configuration
- **Model**: 840M params (BlockDiT, flow-matching)
- **Hardware**: TPU v4-32 (16 chips, 4 workers)
- **Batch Size**: 256 (16/device)
- **LR Schedule**: Cosine annealing, peak=8e-5 (80% of original 1e-4), 2% warmup
- **Total New Samples**: 300M
- **Optimizer**: AdamW (b1=0.9, b2=0.99, wd=1e-2, grad_clip=1.0)
- **Checkpoint Interval**: Every 500K samples (~1953 steps)

## Datasets
| Ratio | Dataset |
|-------|---------|
| 5%  | TTS-AGI/emotion-attribute-conditioning-dacvae |
| 10% | TTS-AGI/podcast-tokenized-bg3.5-enj5 |
| 35% | TTS-AGI/emolia-hq-tokenized |
| 40% | TTS-AGI/podcast-tokenized-bg2.5-enj4.5 |

## Checkpoint Format
Each checkpoint contains:
- `params`: model parameters (float32)
- `opt_state`: Adam optimizer state (momentum + variance) for safe resume
- `step`: training step (in this continued run)
- `samples_seen`: cumulative samples (including original pre-training)
- `global_bs`: batch size used
- `config`: identifier string

## Architecture (840M)
- Backbone: dim=1280, FFN=3520, 24 layers, 10 heads (head_dim=128)
- Encoder: dim=768, FFN=2048, 8 layers, 6 heads
- Text: dim=768, FFN=2048, 8 layers, 6 heads, vocab=256 (byte-level)
- AdaLN rank=256, RoPE half-head, SwiGLU, QK-norm