---
language: en
tags:
  - tiny
  - pct-v3
  - rpw
  - gpp
  - vcr
---

# Lumia Tiny (PCT-V3)

Custom PyTorch LM ~969K params. Arsitektur dari first principles.

## Architecture
- **RPW** — Relative Positional Warp (learned Fourier additive bias)
- **GQA** — 8 query / 4 KV heads
- **GPP** — Gated Principal Projection (96-dim bottleneck)
- **VCR** — Variance-Controlled Residual (R² gating)
- **ECI** — Entropy-Calibrated Initialization
- Vocab: 4096, RMSNorm, tied embeddings, 6 layers

## Files
| File | Description |
|---|---|
| `model_tiny.py` | Arsitektur PCT-V3 (RPW, GQA, GPP, VCR, TinyModel) |
| `train_tiny.py` | Training loop (IterableDataset, cosine LR, checkpoint) |
| `train_tiny.yaml` | Konfigurasi training (LR 3e-4, batch 8, GA 4, 50k steps) |
| `config.json` | HF config auto-map untuk TinyModel |
| `tokenizer.json` | BPE tokenizer 4096 vocab |
| `gen_tokenizer.py` | Generator tokenizer |

## Usage
```python
from model_tiny import create_model
model = create_model()  # PCT-V3 defaults
```