---
license: apache-2.0
tags:
- tiny
- lumia
- custom-pytorch
- alibi
- gelu
---

# Lumia-Tiny

1,017,088 parameter causal language model with ALiBi (Attention with Linear Biases) and GELU activation.

## Architecture

| Config | Value |
|--------|-------|
| Parameters | 1,017,088 |
| Hidden size | 128 |
| Intermediate | 512 |
| Layers | 2 |
| Heads | 8 (query), 4 (key/value) |
| Max seq len | 2048 |
| Vocab | 4096 (BPE, byte fallback) |
| Position | ALiBi (no learned embeddings) |
| Activation | GELU |
| Tie embeddings | Yes |

## Usage

```python
import torch
from model_tiny import TinyModel

model = TinyModel.from_pretrained("samcheng0/lumia-tiny")
tokenizer = ...

inputs = tokenizer("Hello world", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0]))
```

### GGUF (llama.cpp)

The model is also available in GGUF Q4_K_M format:

```
tiny.gguf
```

## Training

Trained on [trl-lib/Capybara](https://huggingface.co/datasets/trl-lib/Capybara) via SFT.