---
datasets:
- cerebras/SlimPajama-627B
language:
- en
library_name: transformers
license: mit
pipeline_tag: text-generation
tags:
- text-generation
- gla
---

# GLA 1.3B-100B

This repository contains the `gla-1.3B-100B` model, a 1.3B parameter variant trained on 100B tokens, which was presented in the paper [Gated Linear Attention Transformers with Hardware-Efficient Training](https://huggingface.co/papers/2312.06635).

## Usage

This model can be easily loaded and used for text generation tasks with the Hugging Face `transformers` library:

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

# Load the tokenizer and model
model_id = "fla-hub/gla-1.3B-100B"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)

# Example for text generation
prompt = "Hello, my name is"
inputs = tokenizer(prompt, return_tensors="pt")

# Generate text
outputs = model.generate(**inputs, max_new_tokens=50, do_sample=True, top_k=50, top_p=0.95, temperature=0.7)
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)

print(generated_text)
```

## Paper and Citation

If you find this work useful, please consider citing the original paper:

[Gated Linear Attention Transformers with Hardware-Efficient Training](https://huggingface.co/papers/2507.06457)

```bibtex
@article{li2025systematic,
  title={Gated Linear Attention Transformers with Hardware-Efficient Training},
  author={Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, Yoon Kim},
  journal={arXiv preprint arXiv:2312.06635},
  year={2023},
}
```

## Code

The official codebase for the models and research, including training scripts and other checkpoints, can be found on GitHub:

[https://github.com/fla-org/flash-linear-attention](https://github.com/fla-org/flash-linear-attention)