---
language:
- en
tags:
- mlp-mixer
- causal-lm
- byte-level
- attention-free
datasets:
- daily_dialog
- roneneldan/TinyStories
license: apache-2.0
---
---
## 📋 Overview
**MicroMixer-1-300K** is a medium-sized model with 2.4x more parameters than the 100K variant. It can process 128 token sequences and begins to learn basic word patterns.
---
## 🏗️ Architecture
```mermaid
graph TD
A[Byte Input] --> B[Token Embedding]
B --> C[RoPE Position Encoding]
C --> D[ImprovedMixerLayer ×3]
D --> E[LayerNorm]
E --> F[LM Head]
F --> G[Byte Output]
style A fill:#007BFF,color:#fff
style G fill:#00D620,color:#fff
style D fill:#AE00FF,color:#fff
```
### Model Configuration
| Parameter |
Value |
| Total Parameters | 331,680 |
| Hidden Dimension | 128 |
| Channel MLP Dimension | 288 |
| Number of Layers | 3 |
| Max Sequence Length | 128 |
| Vocabulary Size | 256 (Byte-level) |
### Core Components
```
┌─────────────────────────────────────────────┐
│ ImprovedMixerLayer │
│ ┌─────────────────────────────────────┐ │
│ │ LayerNorm → HyperMixing → Residual │ │ ← Token Mixing
│ ├─────────────────────────────────────┤ │
│ │ LayerNorm → MlpBlock → Residual │ │ ← Channel Mixing
│ └─────────────────────────────────────┘ │
└─────────────────────────────────────────────┘
```
#### 1️⃣ RoPE (Rotary Position Embedding)
- Encodes positions via **rotation transformations**
- Enables length extrapolation beyond training sequences
#### 2️⃣ HyperMixing (Token Mixing)
- Compresses past context via **cumulative average pooling**
- Hypernetwork generates adaptive weights
- O(S) complexity token mixing without attention
#### 3️⃣ MlpBlock (Channel Mixing)
- Non-linear transformation of feature dimensions
- Structure: `Linear → GELU → Linear`
---
## 📈 Key Differences from 100K
| Metric | 100K | 300K | Change |
|--------|------|------|--------|
| Parameters | 136,908 | 331,680 | **2.4x** |
| Hidden Dim | 84 | 128 | 1.5x |
| Channel MLP | 128 | 288 | 2.3x |
| Sequence Length | 64 | 128 | **2x** |
---
## ⚠️ Limitations
| Limitation | Description |
|------------|-------------|
| **Limited Context** | 128 tokens still insufficient for complex context |
| **Unstable Generation** | Word patterns appear but sentence completion is poor |
| **Vocabulary Gaps** | Rare characters handled poorly despite byte-level encoding |
| **Repetitive Output** | Repeats patterns like "little girl named Timmy" |
---
## 📊 Training Data
**Dataset**: [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories)
- Simple children's stories dataset
- Learns basic grammar and vocabulary
- Contains many patterns like "Once upon a time", "little girl/boy"
---
## 🔧 Usage
```python
import torch
from huggingface_hub import hf_hub_download
from src.model import MicroMixerV2, MicroMixerV2Config
from src.tokenizer import ByteTokenizer
# Clone the repository first:
# git clone https://github.com/llaa33219/MicroMixer-1.git
# cd MicroMixer-1
config = MicroMixerV2Config(
max_seq_len=128,
hidden_dim=128,
channel_mlp_dim=288,
num_layers=3,
use_hyper=True,
)
model = MicroMixerV2(config)
weights_path = hf_hub_download("llaa33219/MicroMixer-1-300K-TinyStories", "model.pt")
model.load_state_dict(torch.load(weights_path, map_location="cpu"))
model.eval()
tokenizer = ByteTokenizer()
input_ids = torch.tensor([tokenizer.encode("Once upon a time")])
with torch.no_grad():
output = model.generate(input_ids, max_new_tokens=64, temperature=0.8, top_k=40)
print(tokenizer.decode(output[0].tolist()))
```
---
---
[](https://github.com/llaa33219/MicroMixer-1)
Part of the MicroMixer-1 research project
# MicroMixer-1-300K-TinyStories
