---
license: apache-2.0
datasets:
- openbmb/Ultra-FineWeb-L3
- openbmb/UltraData-Math
- HuggingFaceFW/fineweb-edu
- HuggingFaceTB/dclm-edu
- HuggingFaceTB/smollm-corpus
- nampdn-ai/mini-peS2o
- Geralt-Targaryen/books3
- omarkamali/wikipedia-monthly
- khtsly/roblox_docs_corpus_text
- khtsly/luau-stack-hq
- khtsly/devforum-roblox-text
- khtsly/luau-repo-docs-text
- khtsly/luau-org-web-text
library_name: transformers
language:
- en
tags:
- text-generation
- roblox
- luau
- coder
- on-device
base_model:
- khtsly/Luau-Coder-1.0-0.5B-Base
pipeline_tag: text-generation
---

# Luau Coder 1.0 - 0.5B (Base) 🦭

## Model Overview

- Type: Text Causal Language Model
- Training Stage: Pre-training
- Language Model
    - Number of Parameters: 0.5B
    - Hidden Dimension: 1024
    - Token Embedding: 32768 (Padded)
    - Number of Layers: 32
    - Hidden Layout: 8 × (3 × (Kimi DeltaNet → FFN) → 1 × (Multi-head Latent Attention → FFN))
    - Kimi DeltaNet:
        - Number of Linear Attention Heads: 16
        - Head Dimension: 64
    - Multi-head Latent Attention:
        - Number of Attention Heads: 16
        - Head Dimension: 64 or 128
        - No Positional Embedding Dimension: 64
        - Use Output Gate: Yes
    - Attention Residuals:
        - Granularity: Sub-Layer
        - Block Size: 4
    - Feed Forward Network:
        - Intermediate Dimension: 2816
        - Activation: SiTU-GLU
    - LM Output: 32768 (Tied to token embedding)
- Context Length: 65,536 natively.

## License

This repository and Luau-Coder model weights are released under the Apache-2.0 License.