README.md

---
license: apache-2.0
language:
- en
tags:
- audio
- speech
- next-token-prediction
- research
- hierarchical
---

# p5-hier-decay100-9984b218

One of 13 runs from the **SODA-extension flat-vs-hier campaign**: a
compute-matched comparison of the flattened **SODA** architecture
([paper](https://arxiv.org/abs/2602.16687)) against a CSM/Moshi-style **hierarchical** factorization
(backbone over steps + small depth transformer over the 8 Mimi RVQ codebook
slots), trained on the same interleaved text+audio corpus with a shared
id-space so held-out NLL is directly comparable across arms.

- 🌐 **Project page:** [https://soda-audio.github.io](https://soda-audio.github.io)
- 💻 **Code + full study:** [https://github.com/potsawee/marin/tree/soda-extension/experiments/audio](https://github.com/potsawee/marin/tree/soda-extension/experiments/audio) — results and analysis in
  [report/FINDINGS.md](https://github.com/potsawee/marin/tree/soda-extension/experiments/audio/report/FINDINGS.md)
- 🗂️ **All models:** [soda-research](https://huggingface.co/soda-research)

## Model details

| | |
|---|---|
| **Architecture** | hierarchical (backbone over steps + depth transformer over codebook slots) |
| **Loss recipe** | per-codebook geometric decay, `w_k = 100^(1-k/7)` over the 7 acoustic codebooks (text/semantic = 100) |
| **Compute budget (3× forward FLOPs)** | 3e18 |
| **Backbone** | d=768, L=8 |
| **Depth transformer** | d=384, L=4 |
| **Window** | 1024 steps |
| **Total parameters (incl. embeddings)** | 308M |
| **Training step** | 56570 |
| **Audio** | Mimi RVQ, 1 semantic + 7 acoustic codebooks, 12.5 Hz |

Trained on the 42k-hour campaign corpus (YODAS + Emilia EN, interleaved text+audio, single pass).

## Loading

The hierarchical model ships its own modeling code
(`modeling_soda_hier.py`, `configuration_soda_hier.py`) and loads with
`trust_remote_code`:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("soda-research/p5-hier-decay100-9984b218", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("soda-research/p5-hier-decay100-9984b218", trust_remote_code=True)
```