rvq-encoder-v3-hiddens-best.pt
Hey, nice writeup and thanks for the credit!
You can start from my v3 to test: I dropped the 8 classification heads (what you're running now, that was my v2, same wall: ~41% sem / 7% ac top-1). v3 regresses the LM per-frame hidden state h[4096] instead, and the published lm_head + greedy depth chain decode the codes closed-form. Same 41M trunk, zero overfitting in 100 epochs, and the e2e replay judge jumps from 0.85 to 0.93-0.94 STFT cossim on held-out tracks. That's trained on only 10h of dataset generated overnight by my GGML port in Q8, nothing more.
The training code is 2 commits ahead of the one you linked, head here:
https://github.com/ServeurpersoCom/minimaxmusic.cpp/tree/master/training
It has the full loop (gen-corpus -> encode latents -> dump hiddens -> train -> e2e replay eval), plus a live mirror on my server:
https://www.serveurperso.com/ia/ssd/workspace/git/minimaxmusic.cpp/training/
The 10h pilot dataset is browsable here if you want to compare or reuse it:
https://www.serveurperso.com/ia/ssd/workspace/git/minimaxmusic.cpp/training/datasets/pilot/
And the checkpoints (v3 hiddens route + v2 heads route, torch state dicts loading straight into training/model.py) plus real-world reconstruction examples are here:
https://www.serveurperso.com/ia/ssd/workspace/rvq-eval/
Grab whatever you want to continue.
And don't trust token top-1 too much: 4-6% acoustic exact match already replays at 0.83+ STFT, the codes have huge equivalence classes. The replay judge is the only metric that means something here.
https://huggingface.co/collections/SimpleTuner/open-rvq-encoder-for-minimax-music-3
see my v4 approach and the general repo which has a comfyUI node to use it
Interesting. The performance is near identical (within the margin of measurement noise) to my v3 model, which has 5.5 times fewer parameters and was trained on only 10 hours of self-generated audio. I tested this on out-of-distribution music and on data from both datasets. The difference is what we regress: instead of classifying the codes, my encoder predicts the language model hidden state per frame, and the published lm_head and depth decoder turn it into the 8 codes in closed form. It's time to test training on the two datasets to see how it turns out, but I think it converged to the same result. I'm going to integrate it directly into the vocoder's GGUF to turn it into functions within the C++ application itself.