---
license: apache-2.0
tags:
- slm
- custom-code
- micro-model
- in-context-learning
pipeline_tag: text-generation
---

# ApproxDumb (560-Parameter Micro-SLM)

`ApproxDumb` is an ultra-micro language model built with just **560 parameters**.  
It serves as a proof-of-concept for **In-Context Learning (ICL)** in micro-architectures, demonstrating how a model can infer underlying rules (such as dynamic addition patterns) on-the-fly from a single provided example.

---

## ⚠️ Input & Output Constraints

Due to the extreme parameter budget, this model **cannot** process natural language text or complex multi-digit arithmetic. Please strictly adhere to the following input and output specifications.

### 1. Input Constraints
- **Allowed Vocabulary**:
  - **Single Digits**: `0`, `1`, `2`, `3`, `4`, `5`, `6`, `7`, `8`, `9`
  - **Separator**: `->`
  - **Special Tokens**: `<bos>`, `<eos>`, `<pad>`
  - *Note: Words, letters, multi-digit numbers (e.g., 10, 100), or unsupported punctuation are **NOT** allowed.*
- **Formatting**:
  - Tokens **must be separated by half-width spaces** (e.g., `"1 3 -> 4"`).
  - Recommended Prompt Structure: `[Example Input] [Example Output] -> [Query Input]` (Total of 4 tokens).
- **Max Sequence Length**: **8 tokens** (4 to 5 tokens recommended).

### 2. Output Constraints
- **Output Format**: The model predicts the probability distribution (logits) for the **next single digit token (`0`–`9`)**.
- **Limitations**:
  - Cannot generate multi-digit numbers ($\ge 10$) or negative numbers.
  - Not designed for multi-token free-form text generation (specialized for single next-token prediction).

---

## 💡 Prompt Examples

By providing a single "Example", the model infers the rule and predicts the answer for the "Query".

| Prompt (`input_text`) | Inferred Rule | Expected Prediction |
| :--- | :--- | :--- |
| `"1 2 -> 4"` | $+1$ rule | **`5`** |
| `"1 3 -> 4"` | $+2$ rule | **`6`** |
| `"1 4 -> 2"` | $+3$ rule | **`5`** |
| `"0 4 -> 1"` | $+4$ rule | **`5`** |

---

## 🚀 How to Use

Make sure to pass `trust_remote_code=True` when loading the model and tokenizer.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load model and tokenizer
model_id = "56m/ApproxDumb"  # Replace with your Hugging Face repository
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)

# Prepare prompt: Example "1 -> 3" (+2 rule), Query "4 -> ?"
prompt = "1 3 -> 4"
inputs = tokenizer(prompt, return_tensors="pt")

# Inference
model.eval()
with torch.no_grad():
    outputs = model(**inputs)
    
# Extract the most probable next digit token from the final position
next_token_logits = outputs.logits[0, -1, :]
predicted_token_id = torch.argmax(next_token_logits).item()
predicted_symbol = tokenizer.decode([predicted_token_id])

print(f"Input: '{prompt}'")
print(f"Predicted Next Digit: {predicted_symbol}")
```

---

## 🏗️ Model Architecture

- **Architecture**: Decoder-only Micro-Transformer
- **Total Parameters**: 560
- **Embedding Dimension ($d_{model}$)**: 6
- **Transformer Layers**: 1
- **Attention Heads**: 1
- **Vocabulary Size**: 14