Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,159 @@
+---
+language: en
+license: apache-2.0
+tags:
+  - pytorch
+  - moe
+  - sparse-moe
+  - bitnet
+  - 1-bit
+  - 4-bit
+  - scratch
+  - turbowarp
+  - instruct
+base_model: brulee-1/SSMoELM-Base
+---
+
+# SSMoELM-it
+
+**Scratch Small MoE Language Model — Instruct** — instruction-tuned version of [SSMoELM-Base](https://huggingface.co/brulee-1/SSMoELM-Base).
+
+- **47M total / 25.8M active parameters** (top-2 sparse routing)
+- **12.1 MB** packed weights (1-bit routed experts, 4-bit attention & embedding)
+- Fine-tuned on [Dolly-15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k) + [oasst1 EN](https://huggingface.co/datasets/OpenAssistant/oasst1)
+
+> **Note:** The HuggingFace model card may display ~12M parameters and an "8-bit" quantization badge. Both are artifacts of reading the packed `model.safetensors` directly. The actual model has **47M parameters** quantized to **1-bit and 4-bit**.
+
+> "Scratch" carries two meanings: built *for Scratch*, trained *from scratch*.
+
+---
+
+## Model Details
+
+| | |
+|---|---|
+| Architecture | Decoder-only Transformer + Sparse MoE FFN |
+| Total params | 47.04M |
+| Active params | 25.80M (per forward pass) |
+| d_model | 768 |
+| Layers | 6 |
+| Attention | GQA — 12 heads, kv_heads=3, head_dim=64 |
+| Positional encoding | RoPE |
+| Normalization | RMSNorm |
+| Activation | SwiGLU |
+| MoE | 8 routed experts + 1 shared expert, top-2 routing |
+| d_ff (per expert) | 256 |
+| Vocabulary | 8,192 (BPE, byte-fallback, English-optimized) |
+| Context length | 2,048 tokens |
+| Base model | SSMoELM-Base (900M token pretrain) |
+| Framework | MLX (training) / PyTorch (inference) |
+
+---
+
+## Quantization Scheme
+
+Same as SSMoELM-Base. See [SSMoELM-Base](https://huggingface.co/brulee-1/SSMoELM-Base) for details.
+
+---
+
+## Training
+
+### Pretraining
+| | |
+|---|---|
+| Dataset | FineWeb-Edu-score-2 (60%) + FineWeb (40%) |
+| Tokens | 900M |
+
+### Instruction Tuning (SFT)
+| | |
+|---|---|
+| Base checkpoint | SSMoELM-Base (step 013734) |
+| Dataset | Dolly-15k (CC BY-SA 3.0) + oasst1 EN (Apache 2.0) |
+| Samples | ~39K (14.8K Dolly + 24K oasst1) |
+| Steps | 20,000 |
+| Learning rate | 1e-5 (constant) |
+| Loss | Assistant tokens only |
+
+---
+
+## Benchmark Results (0-shot, 500 samples)
+
+| Task | Shot | Metric | Samples | Random | Base | **Instruct** | Δ |
+|---|---|---|---|---|---|---|---|
+| HellaSwag | 0-shot | acc_norm | 500 | 25% | 33.4% | **33.2%** | -0.2% |
+| LAMBADA | 0-shot | acc | 500 | N/A | 13.8% | **14.8%** | +1.0% |
+| PIQA | 0-shot | acc_norm | 500 | 50% | 53.2% | **55.4%** | +2.2% |
+| WinoGrande | 0-shot | acc | 500 | 50% | 49.6% | **49.6%** | 0% |
+| ARC-Easy | 0-shot | acc_norm | 500 | 25% | 35.0% | **35.2%** | +0.2% |
+| ARC-Challenge | 0-shot | acc_norm | 500 | 25% | 21.0% | **24.0%** | +3.0% |
+| BoolQ | 0-shot | acc | 500 | 50% | 36.2% | **44.4%** | +8.2% |
+| MMLU (57 tasks avg) | 0-shot | acc | up to 500/task | 25% | 23.4% | **23.2%** | -0.2% |
+
+---
+
+## Tokenizer
+
+- BPE, vocabulary size = 8,192
+- Byte fallback enabled (no `<unk>`)
+- ASCII/English-optimized segmentation
+
+### Special Tokens
+
+| Token | ID | Role |
+|---|---|---|
+| `<bos>` | 0 | sequence start |
+| `<eos>` | 1 | end of sequence |
+| `<pad>` | 2 | padding |
+| `<\|system\|>` | 3 | system turn |
+| `<\|user\|>` | 4 | user turn |
+| `<\|assistant\|>` | 5 | assistant turn |
+| `<\|eot\|>` | 6 | end of turn |
+
+### Chat Template
+
+```
+<bos><|user|>
+{user}<|eot|>
+<|assistant|>
+{response}<|eot|><eos>
+```
+
+---
+
+## Usage
+
+Download `inference.py` and `tokenizer.json` from this repo. Requires: `torch`, `safetensors`, `tokenizers`.
+
+```bash
+pip install torch safetensors tokenizers
+```
+
+```python
+from inference import load_packed_model
+from tokenizers import Tokenizer
+
+model = load_packed_model("model.safetensors")
+tok   = Tokenizer.from_file("tokenizer.json")
+
+# Build prompt manually
+BOS, USER, ASSISTANT, EOT = 0, 4, 5, 6
+ids = [BOS] + [USER] + tok.encode("\nHello, how are you?").ids + [EOT, ASSISTANT]
+out = model.generate(ids, max_new_tokens=100, temperature=0.8)
+print(tok.decode(out))
+```
+
+CLI:
+```bash
+python inference.py \
+  --ckpt model.safetensors \
+  --prompt "What is the capital of France?" \
+  --max-tokens 100
+```
+
+> **Memory:** Weights stay in packed uint8 format (12.1 MB). Peak RAM ~18 MB during inference.
+
+---
+
+## License
+
+Apache 2.0