Update README.md

README.md

@@ -1,3 +1,95 @@
+
 ---
 license: apache-2.0
+tags:
+- slm
+- custom-code
+- micro-model
+- in-context-learning
+pipeline_tag: text-generation
 ---
+
+# ApproxDumb (560-Parameter Micro-SLM)
+
+`ApproxDumb` is an ultra-micro language model built with just **560 parameters**.  
+It serves as a proof-of-concept for **In-Context Learning (ICL)** in micro-architectures, demonstrating how a model can infer underlying rules (such as dynamic addition patterns) on-the-fly from a single provided example.
+
+---
+
+## ⚠️ Input & Output Constraints
+
+Due to the extreme parameter budget, this model **cannot** process natural language text or complex multi-digit arithmetic. Please strictly adhere to the following input and output specifications.
+
+### 1. Input Constraints
+- **Allowed Vocabulary**:
+  - **Single Digits**: `0`, `1`, `2`, `3`, `4`, `5`, `6`, `7`, `8`, `9`
+  - **Separator**: `->`
+  - **Special Tokens**: `<bos>`, `<eos>`, `<pad>`
+  - *Note: Words, letters, multi-digit numbers (e.g., 10, 100), or unsupported punctuation are **NOT** allowed.*
+- **Formatting**:
+  - Tokens **must be separated by half-width spaces** (e.g., `"1 3 -> 4"`).
+  - Recommended Prompt Structure: `[Example Input] [Example Output] -> [Query Input]` (Total of 4 tokens).
+- **Max Sequence Length**: **8 tokens** (4 to 5 tokens recommended).
+
+### 2. Output Constraints
+- **Output Format**: The model predicts the probability distribution (logits) for the **next single digit token (`0`–`9`)**.
+- **Limitations**:
+  - Cannot generate multi-digit numbers ($\ge 10$) or negative numbers.
+  - Not designed for multi-token free-form text generation (specialized for single next-token prediction).
+
+---
+
+## 💡 Prompt Examples
+
+By providing a single "Example", the model infers the rule and predicts the answer for the "Query".
+
+| Prompt (`input_text`) | Inferred Rule | Expected Prediction |
+| :--- | :--- | :--- |
+| `"1 2 -> 4"` | $+1$ rule | **`5`** |
+| `"1 3 -> 4"` | $+2$ rule | **`6`** |
+| `"1 4 -> 2"` | $+3$ rule | **`5`** |
+| `"0 4 -> 1"` | $+4$ rule | **`5`** |
+
+---
+
+## 🚀 How to Use
+
+Make sure to pass `trust_remote_code=True` when loading the model and tokenizer.
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+# Load model and tokenizer
+model_id = "56m/ApproxDumb"  # Replace with your Hugging Face repository
+tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
+
+# Prepare prompt: Example "1 -> 3" (+2 rule), Query "4 -> ?"
+prompt = "1 3 -> 4"
+inputs = tokenizer(prompt, return_tensors="pt")
+
+# Inference
+model.eval()
+with torch.no_grad():
+    outputs = model(**inputs)
+    
+# Extract the most probable next digit token from the final position
+next_token_logits = outputs.logits[0, -1, :]
+predicted_token_id = torch.argmax(next_token_logits).item()
+predicted_symbol = tokenizer.decode([predicted_token_id])
+
+print(f"Input: '{prompt}'")
+print(f"Predicted Next Digit: {predicted_symbol}")
+```
+
+---
+
+## 🏗️ Model Architecture
+
+- **Architecture**: Decoder-only Micro-Transformer
+- **Total Parameters**: 560
+- **Embedding Dimension ($d_{model}$)**: 6
+- **Transformer Layers**: 1
+- **Attention Heads**: 1
+- **Vocabulary Size**: 14