Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,54 @@
+---
+license: apache-2.0
+tags:
+- tiny
+- lumia
+- custom-pytorch
+- alibi
+- gelu
+---
+
+# Lumia-Tiny
+
+1,017,088 parameter causal language model with ALiBi (Attention with Linear Biases) and GELU activation.
+
+## Architecture
+
+| Config | Value |
+|--------|-------|
+| Parameters | 1,017,088 |
+| Hidden size | 128 |
+| Intermediate | 512 |
+| Layers | 2 |
+| Heads | 8 (query), 4 (key/value) |
+| Max seq len | 2048 |
+| Vocab | 4096 (BPE, byte fallback) |
+| Position | ALiBi (no learned embeddings) |
+| Activation | GELU |
+| Tie embeddings | Yes |
+
+## Usage
+
+```python
+import torch
+from model_tiny import TinyModel
+
+model = TinyModel.from_pretrained("samcheng0/lumia-tiny")
+tokenizer = ...
+
+inputs = tokenizer("Hello world", return_tensors="pt")
+outputs = model.generate(**inputs, max_new_tokens=50)
+print(tokenizer.decode(outputs[0]))
+```
+
+### GGUF (llama.cpp)
+
+The model is also available in GGUF Q4_K_M format:
+
+```
+tiny.gguf
+```
+
+## Training
+
+Trained on [trl-lib/Capybara](https://huggingface.co/datasets/trl-lib/Capybara) via SFT.