Any interests in collaborating with Speculators?
Hey! I'm a maintainer of https://github.com/vllm-project/speculators and was excited to see this model is trained with it! Great results too, +35% on math reasoning over the official DFlash assistant is a big jump.
Two things I'm curious about:
Your notes mention you used patched dev versions of speculators and vllm. Would you be open to upstreaming those patches? Happy to help get them in.
The differential learning rate for the warm started body vs the newly initialized Markov and confidence heads is a nice finding. Have you tried it on other warm starts?
We're in the vLLM slack under #speculators if you'd like to chat there. Would love to collaborate more!
@shanjiaz Thanks! Yes, I’d be happy to work on upstreaming the patches. One of the goals of releasing the model, dataset, and recipe is to make it easier for others to either continue training from this checkpoint or train their own DSpark drafter from scratch for different workloads and larger training distributions.
I haven’t tried the differential LR on other warm starts yet. I introduced it partway through this run after seeing that using the same LR for the warm-started DFlash body and newly initialized heads hurt the early proposal positions, while the lower body LR worked better.