---
license: apache-2.0
language:
- en
tags:
- audio
- speech
- next-token-prediction
- research
- hierarchical
---

# soda-hier-1.1b-trunk-step27729

Mid-training **stable-phase snapshot** (learning rate NOT yet decayed) of the
SODA-Hier trunk run `soda-hier-1b-08d907e0` at step 27,729 (~6.2e19 FLOPs,
3× forward). Published to reproduce the decay-leg analysis in Part 3 of
[report/FINDINGS.md](https://github.com/potsawee/marin/tree/soda-extension/experiments/audio/report/FINDINGS.md). For the final model see
[soda-hier-1.1b](https://huggingface.co/soda-research/soda-hier-1.1b).

## Model details

| | |
|---|---|
| **Architecture** | hierarchical (backbone over steps + depth transformer over codebook slots) |
| **Loss recipe** | per-codebook geometric decay, `w_k = 100^(1-k/7)` over the 7 acoustic codebooks (text/semantic = 100) |
| **Compute budget (3× forward FLOPs)** | ~6.2e19 |
| **Backbone** | d=1536, L=15 |
| **Depth transformer** | d=1152, L=4 |
| **Window** | 1024 steps |
| **Total parameters (incl. embeddings)** | 1111M |
| **Training step** | 27,729 |
| **Audio** | Mimi RVQ, 1 semantic + 7 acoustic codebooks, 12.5 Hz |

## Loading

The hierarchical model ships its own modeling code
(`modeling_soda_hier.py`, `configuration_soda_hier.py`) and loads with
`trust_remote_code`:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("soda-research/soda-hier-1.1b-trunk-step27729", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("soda-research/soda-hier-1.1b-trunk-step27729", trust_remote_code=True)
```