Architecture trends over 5 years

Some of the most consequential ideas in model architecture spread quietly. A technique appears in one paper, sits on the shelf for years, and then suddenly every new release ships with it. Because hfviewer builds its architecture graphs from each model’s real weights, we can actually measure this: for every concept, the share of new models in the catalog that include it, quarter by quarter.

We recently launched the Glossary - illustrated pages for the key concepts inside modern models, from attention variants to MoE routing, each with its adoption curve. Here are five curves that tell the story of the last five years.

Multi-head latent attention: the overnight success, eighteen months in the making

Invented for DeepSeek‑V2 (May 2024), MLA compresses keys and values into a small latent vector to shrink the KV cache. It stayed a DeepSeek specialty for a year and a half. Then this quarter it jumped to 23% of new models: DeepSeek‑V4, GLM‑5.2, Kimi K2.7, LongCat 2.0, and even Mistral’s newest MoE all use it.

QK-Norm: six years from paper to default

Normalizing queries and keys before the attention dot product was proposed by Henry et al. in October 2020 and then mostly forgotten. It came back when labs hit attention instabilities while scaling up, and after Qwen3 and Gemma 3 adopted it, it went mainstream. Now over half of new models ship with it.

Grouped-query attention: no waiting period at all

The opposite story. Ainslie et al. published GQA in May 2023, Llama 2 and Mistral 7B shipped it within months, and it has climbed to two thirds of new models. Probably the fastest paper‑to‑default sprint on this list.

RMSNorm vs LayerNorm: the quiet takeover

Zhang & Sennrich simplified LayerNorm in 2019 by dropping the mean‑centering, Llama made RMSNorm the open‑source default in 2023, and the curves crossed in early 2024. Today 9 out of 10 new models use RMSNorm. Nobody announced this switch. It just happened.

Linear attention: one to watch

Linear attention dates back to 2020 (“Transformers are RNNs”, Katharopoulos et al.) and spent five years on the shelf. The new wave of hybrid models, like the Qwen3.5 and Qwen3.6 generation, interleaves linear‑attention layers with full attention, and suddenly it’s in nearly a third of new models.

How these curves are made

Every graph on hfviewer is traced from the model’s actual weights, not its model card, so a concept only counts when the operation is really in the network. The curves show the share of new models in the hfviewer catalog (2,400+ traced models) that contain each concept, grouped by the quarter the model appeared.

There are 40 concepts in the Glossary so far, each with an illustrated explainer, its adoption curve, the key papers, and the models that use it. They also pop up right in the viewer when you hover a node in any model’s graph. If there’s a concept or comparison you’d like to see charted, come tell me in the #hfviewer Discord.