p1b-hier-uniform-ddf05167

One of 13 runs from the SODA-extension flat-vs-hier campaign: a compute-matched comparison of the flattened SODA architecture (paper) against a CSM/Moshi-style hierarchical factorization (backbone over steps + small depth transformer over the 8 Mimi RVQ codebook slots), trained on the same interleaved text+audio corpus with a shared id-space so held-out NLL is directly comparable across arms.

• 🌐 Project page: https://soda-audio.github.io
• 💻 Code + full study: https://github.com/potsawee/marin/tree/soda-extension/experiments/audio — results and analysis in report/FINDINGS.md
• 🗂️ All models: soda-research

Model details

Architecture	hierarchical (backbone over steps + depth transformer over codebook slots)
Loss recipe	uniform cross-entropy over all tokens
Compute budget (3× forward FLOPs)	3e18
Backbone	d=768, L=8
Depth transformer	d=384, L=4
Window	1024 steps
Total parameters (incl. embeddings)	308M
Training step	56570
Audio	Mimi RVQ, 1 semantic + 7 acoustic codebooks, 12.5 Hz

Trained on the 42k-hour campaign corpus (YODAS + Emilia EN, interleaved text+audio, single pass).

Loading

The hierarchical model ships its own modeling code (modeling_soda_hier.py, configuration_soda_hier.py) and loads with trust_remote_code:

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("soda-research/p1b-hier-uniform-ddf05167", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("soda-research/p1b-hier-uniform-ddf05167", trust_remote_code=True)