A brief history of distillation in AI
This is a warm-up for Class 2 of our Training an Agent series, where we distill open models with TRL. First, a short history.
Distillation is a hot topic in post-training. The latest GLM-5.2 release uses it, Qwen3 and DeepSeek lean on it, and Anthropic just accused several companies of copying its models with it. It is also one of the strongest tools for turning small models into surprisingly good ones.
You may have seen distillation in open models without even noticing. Remember names like DeepSeek-R1-Distill-Qwen-7B? Here is the short version of the story. And if you want to know more, come to the live on July 7th (Ben's announcement).
The beginning: compression
Model Compression (Bucilă, Caruana, Niculescu-Mizil, 2006): train one small model to reproduce a big, slow model's predictions. Same behavior, a fraction of the cost. No new capability, just a cheaper model.
Predictably, Schmidhuber appears in the history: he says he did it first, in 1991.
The idea clicked in 2015 with Hinton, Vinyals and Dean, Distilling the Knowledge in a Neural Network. This is the teacher-student setup: a big teacher, a small student. Instead of learning only the single hard label, the student learns the teacher's full distribution, softened with a temperature. That extra signal, which other options were plausible and how close they were, is the "dark knowledge", and it is why a student can learn more from a teacher than from the raw data.
From there it spread quietly. Through vision (FitNets, attention transfer) and through NLP, where DistilBERT (2019) kept most of BERT's quality at 40% less size and put a distilled model in millions of hands. For almost a decade, distillation meant one thing: make an existing model smaller and faster.
Then the teacher became an LLM: from compression to transfer
Around 2023 the goal changed, and the teacher was now an LLM. With a strong instruction model to query, you could train a small model not to compress a model you already had, but to copy behavior it did not have. Alpaca and Self-Instruct pulled capability out of a bigger model's text for a few hundred dollars. Distillation stopped being about compression and became about transfer.
The most visible case is DeepSeek-R1-Distill (2025). Instead of training a giant reasoning model from scratch, the team distilled reasoning traces from a strong teacher into compact Qwen and Llama students, and shipped them openly. A small model plus a good teacher, and suddenly it can reason. That is exactly the model from the top of this article.
Now it post-trains the frontier
Distillation did not stay a shortcut for small models. At the frontier, its current form is on-policy distillation: the student generates, and a stronger teacher scores the student's own tokens, so it learns to fix the mistakes it actually makes. GLM-5.2 uses it in post-training, Qwen3 leans on distillation too, and the line between distillation, supervised fine-tuning, reinforcement learning and synthetic data is getting blurry. They are increasingly the same post-training pipeline with different names. TRL, Hugging Face's open post-training library, implements the on-policy form as GKD (Agarwal et al., 2024), with a reverse-KL variant in MiniLLM (Gu et al., 2023).
So the whole arc is one shift in ambition: compress a model (2006), speed it up, copy another model's behavior, and now transfer its reasoning. Same mechanic, bigger goals.
The recent controversy
Distillation is also at the center of an ongoing dispute. This year Anthropic accused several labs (DeepSeek, Moonshot and MiniMax, and more recently Alibaba) of distilling Claude's outputs at scale through large numbers of accounts, which it says breaks its terms of service (Anthropic, CNBC). OpenAI has said similar. Since model outputs are not copyrightable in the US, the argument is about terms of service and access, not copyright.
For a history of distillation, the takeaway is simple: learning from a stronger model's outputs is both how open models get built and what these fights are about. Permission and access are what separate the tool from the attack.
It is all open
The techniques at the heart of this story, from logit distillation to the on-policy form the frontier uses, are implemented in TRL. You do not need frontier infrastructure to try them.
That is Class 2 (July 7th), where we keep training our small coding agent: soft labels, off-policy vs on-policy, self-distillation, hands-on in TRL, in code, with practical cases. See you there (Ben's announcement).
