soda-hier-1.1b-trunk-step27729

Mid-training stable-phase snapshot (learning rate NOT yet decayed) of the SODA-Hier trunk run soda-hier-1b-08d907e0 at step 27,729 (~6.2e19 FLOPs, 3× forward). Published to reproduce the decay-leg analysis in Part 3 of report/FINDINGS.md. For the final model see soda-hier-1.1b.

Model details

Architecture	hierarchical (backbone over steps + depth transformer over codebook slots)
Loss recipe	per-codebook geometric decay, w_k = 100^(1-k/7) over the 7 acoustic codebooks (text/semantic = 100)
Compute budget (3× forward FLOPs)	~6.2e19
Backbone	d=1536, L=15
Depth transformer	d=1152, L=4
Window	1024 steps
Total parameters (incl. embeddings)	1111M
Training step	27,729
Audio	Mimi RVQ, 1 semantic + 7 acoustic codebooks, 12.5 Hz

Loading

The hierarchical model ships its own modeling code (modeling_soda_hier.py, configuration_soda_hier.py) and loads with trust_remote_code:

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("soda-research/soda-hier-1.1b-trunk-step27729", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("soda-research/soda-hier-1.1b-trunk-step27729", trust_remote_code=True)