Upload README.md with huggingface_hub

README.md

@@ -8,51 +8,125 @@ tags: [hobbylm, mixture-of-experts, moe, sparse-moe]
 
 # HobbyLM-Omni (500M MoE, text + image + audio)
 
-Multimodal (omni) variant: a TinyLLaVA-style VLM over the HobbyLM MoE core, with vision and speech projectors.
+HobbyLM-Omni is the multimodal core: **one** 500M MoE model that handles text, image, video, audio, and speech — plus tool use, OCR, and UI grounding — folded into a single checkpoint across 18 training paths (TinyLLaVA-style projectors over frozen SigLIP2 / Whisper / CLAP front-ends). The headline isn't any single score; it's the **breadth** in one small model.
 
-Part of the **HobbyLM** family — a from-scratch 500M sparse-MoE model trained on consumer-scale budgets.
+It's part of the **HobbyLM** family — a 500M sparse-MoE model (and its variants) built from scratch on a
+hobby budget: FineWeb, a handful of Modal H100 hours, a lot of ablations, and a from-scratch Rust engine
+([`hobby-rs`](https://github.com/harishsg993010/HobbyLM)) to run it on a laptop CPU.
+
+## Intended use
+
+Vision-language and audio-language tasks: captioning, visual QA, OCR, sound/speech understanding, spoken-question answering, and tool calling. Image/audio/speech features are projected and spliced at the `[IMAGE]`/`[AUDIO]`/`[SPEECH]` sentinel tokens (ids 50257–50262).
 
 ## Architecture
 
-HobbyLM is a **sparse Mixture-of-Experts (MoE)** transformer (DeepSeek-V3 / Ling-style):
+Every HobbyLM variant shares one core: a **sparse Mixture-of-Experts (MoE)** decoder in the modern
+small-MoE style (DeepSeek-V3 / OLMoE lineage), where each design choice was picked by ablation rather
+than by guesswork.
 
 | Component | Value |
 |---|---|
-| Total parameters | ~500M (≈ a fraction active per token) |
-| Hidden size / layers | 768 / 16 (1 dense FFN layer, 15 MoE) |
+| Total parameters | ~500M (only a fraction is active per token) |
+| Hidden size / layers | 768 / 16 (first FFN dense, the rest MoE) |
 | Routed experts / active | 36 / top-6 (+ 1 always-on shared expert) |
-| Attention | GQA, 12 query / 3 KV heads, head-dim 128, per-head QK-norm |
-| Router | sigmoid gating, aux-loss-free balancing bias, no top-k renorm |
-| Positional | RoPE |
+| Attention | GQA, 12 query / 3 KV heads, decoupled head-dim 128, per-head QK-norm |
+| Router | sigmoid gating, DeepSeek-V3 aux-loss-free load balancing, no top-k renorm |
+| Positional | RoPE (θ up to 1e6 for the 8k-context checkpoints) |
 | Tokenizer | GPT-2 byte-level BPE (50,304 vocab, sentinel-padded) |
+| Optimizer | Muon on the 2-D + per-expert matrices, AdamW on everything else |
 
+The full ablation log (QK-norm is the single biggest lever; aux-loss-free beats classic aux-loss;
+≥32 experts and top-6 help; embedding-scaling hurt) lives in the project's architecture notes.
 
 ## Multimodal use
 
-This repo also ships the projector weights:
-`vision_projector.safetensors` (SigLIP2 → LLM) and `speech_projector.safetensors` (Whisper-mel → LLM), plus `melfilters.bytes`. Image/audio are encoded by the (frozen) SigLIP2 / mel front-ends, projected, and spliced in at the `[IMAGE]`/`[AUDIO]`/`[SPEECH]` sentinel tokens (ids 50257–50262).
+This repo also ships the projector weights — `vision_projector.safetensors` (SigLIP2 → LLM) and `speech_projector.safetensors` (Whisper-mel → LLM), plus `melfilters.bytes`. The frozen front-ends encode the raw image/audio, the projectors map those features into the LLM embedding space, and they're spliced in at the modality sentinel tokens.
+
+## Benchmarks
+
+Visual QA is scored with **containment** (the model is chat-trained and answers in full sentences, so strict
+single-word exact-match badly under-scores it):
+
+| Task | Score |
+|---|---|
+| VQAv2 (val) | 47.0 |
+| GQA | 39.2 |
+| POPE — accuracy / F1 | 50.0 / 66.7 |
+| Tool calling — Needle (JSON-parse / Name-F1 / param-halluc) | 93.8 / 77.7 / 0.0 |
+| BFCL (forced-call: simple / multiple) | 21.7 / 18.3 |
+| Text — lm-eval 9-task avg | 0.432 |
+
+POPE at 50/66.7 is a **real** ceiling — object-presence hallucination ("yes" to everything) is the known
+small-VLM weakness, quantified. On function calling, Omni *can* call as well as the dedicated tool model
+(forced-call simple 21.7 ≈ the specialist's 22.7); left to itself it prefers to abstain (irrelevance 86.7),
+a safer agent failure mode. Speech does spoken-QA and commands rather than verbatim transcription.
+
+> **How these were measured.** All language-model scores are **0-shot** through our own port of
+> EleutherAI's `lm-evaluation-harness` (a custom `MoELMWrapper` that runs log-likelihood scoring over the
+> HobbyLM MoE + GPT-2 tokenizer). Reference models in the comparison table were run through the **identical
+> harness and task set**, so the numbers are apples-to-apples with ours — they are *not* copied from other
+> model cards. We validated the harness against published cards (e.g. TinyLlama 52.75 vs card 52.99). These
+> are small research models: read the numbers in context, not as leaderboard claims.
 
-## Files
+## Usage
 
-- `model.safetensors` — the model weights (fp32).
-- `config.json` — architecture / hyperparameters.
-- GGUF builds (arch `hobbylm`) live in [`rootxhacker/HobbyLM-gguf`](https://huggingface.co/rootxhacker/HobbyLM-gguf).
+### Python (PyTorch reference implementation)
 
-## Loading (safetensors)
+HobbyLM is a custom sparse-MoE architecture — there's no `transformers` `AutoModel` for it, so load it with
+the small reference implementation from the [GitHub repo](https://github.com/harishsg993010/HobbyLM):
 
 ```python
-import json, torch
+# HobbyLM is a CUSTOM sparse-MoE architecture, so load it with the reference implementation —
+# NOT transformers.AutoModelForCausalLM (there is no AutoModel mapping for this arch).
+# pip install torch safetensors tiktoken huggingface_hub
+# git clone https://github.com/harishsg993010/HobbyLM && cd HobbyLM
+
+import json, torch, tiktoken
+from huggingface_hub import hf_hub_download
 from safetensors.torch import load_file
-sd  = load_file("model.safetensors")
-cfg = json.load(open("config.json"))
-# rebuild the HobbyLM nn.Module from `cfg` and `load_state_dict(sd)`.
+from hobbylm.config import ModelConfig
+from hobbylm.model import MoETransformer
+from hobbylm.generate import generate
+
+repo = "rootxhacker/HobbyLM-Omni"
+cfg = ModelConfig(**{k: v for k, v in json.load(open(hf_hub_download(repo, "config.json"))).items() if k != "preset"})
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+cfg.expert_backend = "grouped" if device.type == "cuda" else "bmm"
+
+model = MoETransformer(cfg).to(device).eval()
+model.load_state_dict(load_file(hf_hub_download(repo, "model.safetensors")))
+
+enc = tiktoken.get_encoding("gpt2")
+prompt = "USER: Explain a mixture-of-experts model in one sentence.\nASSISTANT:"
+ids = torch.tensor([enc.encode_ordinary(prompt)], device=device)
+out = generate(model, ids, max_new_tokens=64, temperature=0.7, top_k=0, device=device,
+               repetition_penalty=1.3)               # temperature=0.0 for greedy
+print(enc.decode(out[0].tolist()))
+```
+
+> The snippet above is the **text** path. For image / audio / speech, encode the input with the (frozen) SigLIP2 / Whisper / CLAP front-end, project it with the bundled projectors, and splice it at the modality sentinel token — see `hobbylm/multimodal.py`, or just pass `--image` / `--speech` to `hobby-rs`.
+
+### GGUF + hobby-rs (CPU)
+
+GGUF builds (architecture `hobbylm`) live in [`rootxhacker/HobbyLM-gguf`](https://huggingface.co/rootxhacker/HobbyLM-gguf). They load
+directly in the from-scratch `hobby-rs` CPU engine — **stock llama.cpp won't load them** without registering
+the `hobbylm` architecture first.
+
+```bash
+hobby-rs --model HobbyLM-Omni.gguf --prompt "..." --n 64
 ```
 
-## Notes & limitations
+## Training
+
+Built in stages on the context-extended (8k, θ 1e6) backbone with a 512px SigLIP2 vision tower: projector alignment → multimodal SFT → a joint 18-path co-training cycle (image / video / audio / speech / text / tools / OCR / UI-grounding) that keeps every modality from drifting.
+
+## Limitations
 
-- Research model at the ~500M scale: fluent but with the capability ceiling of a small model.
-- The GGUF uses a custom `hobbylm` architecture (see the GGUF repo) and needs `moe-rs` or a patched llama.cpp.
+- Breadth over depth: strong **in-distribution** (VQA, JSON tool calls with 0 hallucination, OCR, grounding) but below specialist sub-1B models on hard text reasoning (GSM8K, multi-hop QA).
+- Object-presence hallucination on POPE-style probes.
+- Verbose by default — ask for short answers explicitly, or score with containment, not exact-match.
 
 ## License
 
-Apache-2.0.
+Apache-2.0. Weights aren't a substitute for judgement — this is a research / hobby model at the 500M scale,
+not a production system.