---
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
language:
  - en
tags:
  - deepseek
  - mixture-of-experts
  - moe
  - multi-head-latent-attention
  - from-scratch
  - small-language-model
  - instruct
base_model: whoashish115/Moonfrost-777M
datasets:
  - HuggingFaceFW/fineweb-edu
  - HuggingFaceTB/smol-smoltalk
  - whoashish115/Moonfrost-Persona-SFT
model-index:
  - name: Moonfrost-777M-Instruct-v2
    results:
      - task: { type: text-generation, name: Multiple-choice science questions }
        dataset: { name: ARC-Easy, type: allenai/ai2_arc, config: ARC-Easy, split: test }
        metrics:
          - { type: accuracy, value: 44.4, name: "accuracy (5-shot, 250 examples)" }
      - task: { type: text-generation, name: Multiple-choice science questions }
        dataset: { name: ARC-Challenge, type: allenai/ai2_arc, config: ARC-Challenge, split: test }
        metrics:
          - { type: accuracy, value: 24.4, name: "accuracy (5-shot, 250 examples)" }
      - task: { type: text-generation, name: Commonsense sentence completion }
        dataset: { name: HellaSwag, type: Rowan/hellaswag, split: validation }
        metrics:
          - { type: accuracy, value: 37.2, name: "accuracy (5-shot, 250 examples)" }
      - task: { type: text-generation, name: Pronoun coreference }
        dataset: { name: WinoGrande, type: allenai/winogrande, config: winogrande_xs, split: validation }
        metrics:
          - { type: accuracy, value: 54.0, name: "accuracy (5-shot, 250 examples)" }
      - task: { type: text-generation, name: Yes/no reading comprehension }
        dataset: { name: BoolQ, type: google/boolq, split: validation }
        metrics:
          - { type: accuracy, value: 58.8, name: "accuracy (5-shot, 250 examples)" }
      - task: { type: text-generation, name: Multitask knowledge }
        dataset: { name: MMLU, type: cais/mmlu, config: all, split: test }
        metrics:
          - { type: accuracy, value: 30.8, name: "accuracy (5-shot, 250 examples)" }
---

# Moonfrost-777M-Instruct-v2

**[Code](https://github.com/whoashish115/moonfrost-ai)** · **[Site](https://moonfrost-ai.vercel.app)** · **[Training runs](https://wandb.ai/whoashish115-base/moonfrost-777m)**

**A supervised fine-tune of [Moonfrost-777M](https://huggingface.co/whoashish115/Moonfrost-777M),
not a separate model.** It starts from that base and trains for 1.7 epochs on instruction
data. Architecture, tokenizer and parameter count are identical; only the weights differ.
Fine-tuning changed how the model answers, not what it knows, so the benchmark table below
describes the family rather than this tune alone; the figures in it were measured on these
weights, which is the checkpoint the column is named after.

Two tunes of that base are published and both are usable; they differ in purpose rather
than in kind. **This one is for conversation:** it includes the
[persona set](https://huggingface.co/datasets/whoashish115/Moonfrost-Persona-SFT), so it
answers as Moonfrost, keeps speaker roles straight and holds a thread.
[Instruct-v1](https://huggingface.co/whoashish115/Moonfrost-777M-Instruct-v1) has no
identity data at all, which makes it the neutral starting point if you would rather tune
your own behaviour onto it.

The base was pretrained from nothing: its own byte-level tokenizer, its own attention and
routing code, its own training loop. No existing model was adapted and no weights were
borrowed. Pretraining and both fine-tunes together cost about **$55** of rented H100 time
across roughly twelve GPU-hours.

The architecture follows DeepSeek-V2 closely, because two ideas from that paper are what
make a model this size behave larger than its compute budget. **Multi-head Latent
Attention** compresses everything the model caches into one shared 320-number latent per
token instead of a key and a value for each of fourteen heads, with position carried
separately on a small decoupled rotary key, since a position-rotated key cannot be rebuilt
from an unrotated latent. That takes the cache from 1,792 numbers per token down to 352.
**DeepSeekMoE** then makes the feed-forward layers sparse: thirty-two routed experts sit
in each layer, a router picks three, one shared expert always runs, and so a token touches
four of thirty-three. Seventy per cent of the parameters are idle for any given token,
which is why 777M total costs about as much to run as **161M active**.

What that budget bought is behaviour, not knowledge. The model holds a conversation,
tracks what you told it earlier in the same chat, writes short working code, and knows its
own story. It does not know very much about the world, and the section on limitations
below is not boilerplate.

| Property | Value |
|---|---|
| **Parameters** | 777,148,032 total, **161,036,224 active per token** |
| **Layers** | 14, of which layer 0 is dense and 1-13 are Mixture-of-Experts |
| **Hidden size / heads** | 896 / 14 |
| **Experts** | 32 routed with top-3 routing, plus 1 shared expert |
| **Attention** | Multi-head Latent Attention, 320 KV latent + 32 decoupled rotary key |
| **Context** | 1,024 tokens |
| **Vocabulary** | 32,768, byte-level BPE trained from scratch on the same corpus |
| **Pretraining** | ~6B tokens of FineWeb-Edu, val loss 2.976 |
| **Chat fine-tune** | 1.7 epochs, best at step 7,800 of 9,201, **val loss 1.2484** |
| **Peak / min LR** | pretraining 6e-4 / 6e-5, fine-tune 2e-4 / 2e-5 |
| **Batch** | micro-batch 24, accumulation 3, 73,728 tokens per step |
| **Precision** | bf16 autocast with fp32 master weights |
| **Throughput** | 179,000 training tokens/sec on one H100 |
| **Inference** | 16.8 tokens/sec, fp32, batch 1, on an RTX 3050 |
| **Compute and cost** | 1x H100, ~12 GPU-hours, about $55 |

## Usage

The architecture is not part of `transformers`, so the repository ships its own modelling
code and needs `trust_remote_code=True`.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "whoashish115/Moonfrost-777M-Instruct-v2"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, trust_remote_code=True, torch_dtype=torch.float32
).eval()

prompt = "<|system|>You are Moonfrost, a helpful assistant.<|user|>Why is the sky blue?<|assistant|>"
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=120, do_sample=True,
                        temperature=0.7, top_p=0.9)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```

Conversations are formatted `<|system|>…<|user|>…<|assistant|>…<|endoftext|>`, and you
must stop generation on **both `<|endoftext|>` and `<|user|>`**. That second stop token is
not optional: a model trained this briefly will cheerfully carry on past its own answer and
write your next message for you. Temperature around 0.7 with top-p 0.92 and a repetition
penalty of about 1.15 is what the serving code uses; much hotter and the factual errors get
more creative without the prose getting better.

## Benchmarks

![Benchmark comparison](charts/benchmarks.png)

Every number here was measured on one machine with one harness, five-shot, 200 examples per
benchmark and 250 for MMLU, scored by which answer option the model finds most likely. The reference models
were run through **that same harness on those same examples** rather than quoted from their
cards, because prompt wording and length normalisation move these scores by several points
and a table that mixes harnesses is not a comparison. Qwen2.5-0.5B makes the point
concretely: its published MMLU is 47.5, and it scores 34.4 here.

| Benchmark | Chance | Moonfrost Base | Moonfrost Instruct v1 | Moonfrost Instruct v2 | SmolLM2-135M | SmolLM2-360M | Qwen2.5-0.5B |
|---|---|---|---|---|---|---|---|
| ARC-Easy | 25.0 | 54.8 | 52.4 | 44.4 | 62.8 | **68.4** | 64.4 |
| ARC-Challenge | 25.0 | 25.2 | 24.4 | 24.4 | 27.6 | **37.2** | 34.8 |
| HellaSwag | 25.0 | 36.0 | 38.4 | 37.2 | 40.0 | **43.6** | 42.4 |
| WinoGrande | 50.0 | 51.2 | 53.2 | 54.0 | 54.0 | 56.0 | **56.8** |
| BoolQ | 50.0 | 62.4 | 61.2 | 58.8 | 62.0 | 63.6 | **65.2** |
| MMLU | 25.0 | 28.8 | 30.0 | 30.8 | 32.4 | **36.8** | 34.4 |

Moonfrost is last or near-last on most rows, and the reason is not the architecture.

The tokens-per-parameter chart is on the [base model card](https://huggingface.co/whoashish115/Moonfrost-777M), where the
pretraining budget it describes belongs.

Six billion tokens for 777 million parameters is about **eight tokens per parameter**.
Chinchilla put the compute-optimal ratio near twenty, and everything released since has
gone far past even that, because text is cheap compared with serving a bigger model
forever. SmolLM2-360M read roughly two trillion tokens; Qwen2.5-0.5B read eighteen
trillion. Those are three hundred and three thousand times more text than this model has
seen, from models half its size. Nothing in the design closes a gap like that, and no
amount of further chat fine-tuning will either, because fine-tuning teaches format and not
facts.

Read the three Moonfrost columns down each row. Almost every difference between them is
noise: at 250 examples the 95% interval on a single score is roughly **±6 points**, and
fifteen of the eighteen gaps are under three. One benchmark moves, and it moves in one
direction. **ARC-Easy falls 54.8, 52.4, 44.4** across the base, the half-epoch tune and the
1.7-epoch tune. Ten points is what it costs to teach the model to answer in a chat format
instead of continuing a multiple-choice stem, and the cost grows with how long you tune.

A caveat that applies to the whole table, and that this project ran into directly. The
harness draws its five in-context examples from the rows immediately after the evaluation
slice, so changing the number of examples also changes the prompt. An earlier run of these
same weights over 200 examples put BoolQ at 44.5; the run over 250 puts it at 58.8. Same
model, same harness, same questions for the first two hundred of them, **14.3 points
apart**. Treat any single number here as a measurement with a wide interval, and treat
scores copied from a different harness as not comparable at all.

## Training

The pretraining loss curves are on the [base model card](https://huggingface.co/whoashish115/Moonfrost-777M).

Pretraining ran in two phases on two separate machines, reading disjoint
shards of FineWeb-Edu `sample/10BT` so no document was seen twice. Splitting a single
annealing schedule across two machines works because the learning rate is parameterised by
**elapsed fraction of training rather than by step**: phase 2 started at fraction 0.5227,
exactly where phase 1 stopped, and the cosine curve continued instead of restarting. Only
the weights crossed the boundary, which is why phase 2 re-warms for 150 steps.

The chat fine-tune mixes smol-smoltalk at 84% of rows with smoltalk's
`everyday-conversations` at 12%, repeated twenty-five times, and the
[Moonfrost-Persona-SFT](https://huggingface.co/datasets/whoashish115/Moonfrost-Persona-SFT)
identity set at 3.3%. The repeats exist because small talk is a rounding error in
smol-smoltalk and without them the model answered "hi" with a lecture on quantum physics.
Conversations are packed several to a 1,024-token row rather than padded, which took useful
tokens per batch from roughly 40% to 61%, and loss is masked on system and user turns so
only assistant text is ever supervised.

A note on the charts: both pretraining logs are partial, and the panels say where. Phase 1
is two fragments, because the attempt that dropped its connection at step 1,730 logged
locally and the run that finished never had its log downloaded; what sits to the right of
the gap was read back out of retained console output. Phase 2 logged its first 105 minutes
of 340. The step counts and losses in the table above come from checkpoint metadata, which
is complete, as is the chat fine-tune log. All three runs are on
[Weights & Biases](https://wandb.ai/whoashish115-base/moonfrost-777m).

## Versions

Two chat tunes were run on the same pretrained base and both are published. They differ in
exposure and in whether they carry identity data, and the behavioural gap between them is
the clearest thing in this project about what supervised fine-tuning actually buys. Asked
who made it, v1 answers that it is a retired professor of English named Jack Harris and
then asks the question back; v2 answers in one line and stops. Asked for hello world in
Java, v1 emits a class with two `main` methods that will not compile; v2 emits the correct
four lines and stops cleanly.

| Version | Val loss | SFT epochs | Identity data | Behaviour |
|---|---|---|---|---|
| [v1](https://huggingface.co/whoashish115/Moonfrost-777M-Instruct-v1) | 1.3251 | 0.42 | none | confuses speaker roles, invents an identity |
| **v2** (this model) | **1.2484** | **1.7** | 3.3% of rows | stable roles, stable identity, usable code |

## Intended use

Use these weights for short conversational turns, for questions with definitional answers,
and for reading what a 777M Mixture-of-Experts model trained on six billion tokens can and
cannot do. It follows a chat template, answers as Moonfrost, and stops cleanly on
`<|endoftext|>` and `<|user|>`.

To build on it, start from [Moonfrost-777M](https://huggingface.co/whoashish115/Moonfrost-777M)
rather than from here: tuning a tune compounds the format of the first pass. If you want an
instruction-following starting point with no identity baked in,
[Instruct-v1](https://huggingface.co/whoashish115/Moonfrost-777M-Instruct-v1) is that.

Do not put it in front of users, in a product, or anywhere an answer is acted on, and do
not use it for anything touching medicine, law, finance or safety. It has no safety tuning
and no content filtering, and its confident wrong answers read exactly like its correct
ones.

## Limits

It invents facts with complete confidence, and it does so most often on exactly the topics
that a corpus of educational web text does not cover. Ask it about cryptocurrency and it
will produce fluent paragraphs about binomial coefficients. It cannot do arithmetic or
multi-step reasoning, has no tools and no internet, keeps no memory between conversations,
handles English only, and sees at most 1,024 tokens at a time. Open-ended "how do I"
questions tend to run to the token limit without ever concluding, while short definitional
questions stop cleanly.

There is no safety tuning, no RLHF, and no content filtering at either the data or the
output stage. Treat nothing it says as factual and do not put it anywhere a wrong answer
costs something.

## Citation

```bibtex
@misc{moonfrost2026,
  title  = {Moonfrost: a 777M-parameter Mixture-of-Experts language model trained from scratch},
  author = {Ashish Kumar},
  year   = {2026},
  url    = {https://huggingface.co/whoashish115/Moonfrost-777M}
}
```

Apache 2.0.