File size: 1,606 Bytes
9030742 a7aaf25 9030742 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 | ---
license: apache-2.0
language:
- en
tags:
- audio
- speech
- next-token-prediction
- research
- hierarchical
---
# soda-hier-1.1b-trunk-step27729
Mid-training **stable-phase snapshot** (learning rate NOT yet decayed) of the
SODA-Hier trunk run `soda-hier-1b-08d907e0` at step 27,729 (~6.2e19 FLOPs,
3× forward). Published to reproduce the decay-leg analysis in Part 3 of
[report/FINDINGS.md](https://github.com/potsawee/marin/tree/soda-extension/experiments/audio/report/FINDINGS.md). For the final model see
[soda-hier-1.1b](https://huggingface.co/soda-research/soda-hier-1.1b).
## Model details
| | |
|---|---|
| **Architecture** | hierarchical (backbone over steps + depth transformer over codebook slots) |
| **Loss recipe** | per-codebook geometric decay, `w_k = 100^(1-k/7)` over the 7 acoustic codebooks (text/semantic = 100) |
| **Compute budget (3× forward FLOPs)** | ~6.2e19 |
| **Backbone** | d=1536, L=15 |
| **Depth transformer** | d=1152, L=4 |
| **Window** | 1024 steps |
| **Total parameters (incl. embeddings)** | 1111M |
| **Training step** | 27,729 |
| **Audio** | Mimi RVQ, 1 semantic + 7 acoustic codebooks, 12.5 Hz |
## Loading
The hierarchical model ships its own modeling code
(`modeling_soda_hier.py`, `configuration_soda_hier.py`) and loads with
`trust_remote_code`:
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("soda-research/soda-hier-1.1b-trunk-step27729", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("soda-research/soda-hier-1.1b-trunk-step27729", trust_remote_code=True)
```
|