---
license: apache-2.0
base_model:
- allenai/Olmo-3-1025-7B
language:
- en
library_name: transformers
tags:
- long-context
- sparse-attention
- efficient-attention
- pretraining
- olmo3
- hils-attention
arxiv: 2607.02980
---
HiLS-Attention-7B
**HiLS-Attention** is a chunk-wise sparse attention mechanism that learns chunk selection end-to-end under the language-modeling loss, enabling native sparse training for efficient long-context modeling. This repository hosts the **7B** checkpoint continued-trained on top of an OLMo3-style backbone.
Model introduced in the paper [Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling](https://arxiv.org/abs/2607.02980).
## Model Description
Naive block sparse attention selects top-k chunks by their exact chunk mass, but computing all chunk masses requires full QK computation. HiLS-Attention instead uses **compressed chunk keys** to estimate a chunk-mass surrogate and **factorizes attention into inter-chunk and intra-chunk softmax**, enabling end-to-end learning from the next-token prediction loss.
*Overview of HiLS-Attention. Naive block sparse attention selects top-k chunks by their exact chunk mass, but computing all chunk masses requires full QK computation. HiLS-Attention instead uses compressed chunk keys to estimate a chunk-mass surrogate and factorizes attention into inter-chunk and intra-chunk softmax, enabling end-to-end learning from the next-token prediction loss.*
- **Parameters:** ~7B
- **Base architecture:** OLMo3-7B
- **Paper:** https://arxiv.org/abs/2607.02980
- **Code:** https://github.com/Tencent-Hunyuan/HiLS-Attention
## Highlights
After only **50B continued-training tokens**, HiLS-Attention inherits the capability of full attention while bringing two key advantages:
- **Strong ultra-long context extrapolation** beyond the YaRN-extended 4× length.
- **Faster inference** for long sequences.
Meanwhile, it preserves comparable performance for short- and medium-context tasks, within both the original training length and the YaRN-extrapolated range.
*After only 50B continued-training tokens, HiLS-Attention inherits the capability of full attention while bringing two key advantages: **strong ultra-long context extrapolation** beyond the YaRN-extended 4× length (a) and **faster inference** (b). Meanwhile, it preserves comparable performance for short- and medium-context tasks, within both the original training length and the YaRN-extrapolated range (c & d).*
## Intended Uses
- Long-context language modeling and inference (retrieval, QA, summarization over long documents).
- Research on native sparse-attention training and long-context extrapolation.
### Out-of-Scope
Not aligned/instruction-tuned for safety-critical deployment. Outputs may be inaccurate or biased; validate before production use.
## How to Use
HiLS-Attention cannot be loaded directly via the standard `transformers` `AutoModel` API — the custom hierarchical sparse attention must be registered through the **HiLS-Attention** codebase. Please set up the environment and run the model following the [GitHub repository](https://github.com/Tencent-Hunyuan/HiLS-Attention), in particular the **Evaluation** section (see `eval/` and `scripts/eval/`), which shows how to load the checkpoint with the corresponding HiLS config and run generation / evaluation.
> For efficient long-context serving, SGLang inference support is on the roadmap — see the [GitHub repo](https://github.com/Tencent-Hunyuan/HiLS-Attention) for updates.
## Evaluation
The model is evaluated across short-, medium-, and long-context benchmarks. Full scripts are available in the [GitHub repository](https://github.com/Tencent-Hunyuan/HiLS-Attention).
### Long-context
| Benchmark | Description |
| -------------- | --------------------------------------------- |
| LongBench v1 | Long-context QA (21 tasks, up to 64K input) |
| RULER | Synthetic probing at 8K / 16K / 32K / 128K |
| PPL | Perplexity across 64 … 256K sequence lengths |
### Short-context (OpenCompass, 11 benchmarks)
| Benchmark | Type |
| ----------------------------------------- | --------------------- |
| MMLU, GPQA, HellaSwag, ARC-c, BoolQ, RACE | Few-shot PPL |
| GSM8K, CMath | Math generation (CoT) |
| HumanEval+, MBPP+, CRUXEval-O | Code generation |
## Training
- **Method:** Continued pre-training with native sparse-attention (HiLS-Attention) on top of an OLMo3-7B backbone.
- **Tokens:** ~50B continued-training tokens.
- **Positional encoding / long-context:** HoPE with YaRN-based length extrapolation.
Training and continue-pre-training scripts are documented in the [GitHub repository](https://github.com/Tencent-Hunyuan/HiLS-Attention).
## Limitations and Bias
This is a pretrained base model without alignment or safety tuning. It may reflect biases present in the training corpus and can produce inaccurate or unsafe content. Users are responsible for evaluating suitability for their use case.
## Citation
```bibtex
@misc{hu2026hierarchicalsparseattentionright,
title={Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling},
author={Xiang Hu and Xinyu Wei and Hao Gu and Minshen Zhang and Tian Liang and Huayang Li and Lei Zhu and Yan Wang and Sirui Han and Yushi Bai and Kewei Tu and Haitao Mi and Leo Liang},
year={2026},
eprint={2607.02980},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2607.02980},
}
```