---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- 1pp
- one-persona-pretraining
- sft
- asst
---

# 1pp-0.5b-asst-sft

**One Persona Pretraining (1PP)** experiment model: 0.58B parameters, pretraining condition **rewritten conversations, assistant-turn loss**, followed by supervised fine-tuning (SFT).

Part of a 3 × 3 study: three sizes (0.5B, 1B, 1.7B) × three pretraining conditions on the same 47.8M source documents in the same order (original documents; rewritten conversations with loss on assistant turns; rewritten conversations with loss on user and assistant turns). Every run saw the identical batch sequence, so the conditions differ only in the document text and the loss mask. Models are grouped in the [1pp collection](https://huggingface.co/collections/Raghav-Singhal/1pp-6a999df54bfcf9335355a649).

## Architecture

Llama-style decoder, 24 layers, hidden 1,152, FFN 4,608 (SwiGLU), attention heads / KV heads 9 / 3 (head dim 128), RMSNorm, RoPE base 10,000, untied embeddings, no biases, no QK-norm, sequence length 4,096. Tokenizer: SmolLM2 vocabulary (49,152) plus `<|pad|>`; `<|endoftext|>` is the end-of-document token.

## Pretraining

Data: the 1PP conversations rewritten from those documents; loss only on assistant turns (no loss on user turns or `<|endoftext|>`). One pass over 47.8M documents (66.2B tokens of original documents; 63.0B tokens as conversations), 31,777 steps at global batch 512 × 4,096 tokens, cross-document attention masking, best-fit packing with step-aligned document assignment. Optimizer: Muon (shape scaling, matrix LR 0.005) with Adam for embeddings and norms, warmup 2,000 steps, constant, linear decay over the last 10% to 1/100, weight decay 0.1, bf16.

Validation loss (per token, 2,433 held-out documents, final checkpoint):

| assistant text | user text | document text |
|---|---|---|
| 1.579 | 6.878 | 3.372 |

## Supervised fine-tuning

One epoch over a 400k-conversation mix: `jkminder/model-raising-pb-100k-3c-mt-sft` (98.5k multi-turn, constitution-cited track), `dlab-spp/sp-sft-normal-300k` minus prompts duplicated in the first set (271.6k), and a 30k sample of `dlab-spp/sp-sft-safety-180k`. Same stack as pretraining (Megatron, Muon, ChatML without a system turn, loss on assistant turns only). Matrix LR 0.002 selected per model from {0.0005, 0.001, 0.002, 0.005} by held-out loss; global batch 128 × 4,096, linear decay to 1/10 after 3% warmup.

Held-out SFT loss (assistant tokens, 1,998 held-out conversations): 2.023

## Chat format

ChatML **without a system turn** (the models never saw one):

```
<|im_start|>user\n{message}<|im_end|>\n<|im_start|>assistant\n{reply}<|im_end|>\n
```

The bundled `chat_template` renders exactly this. Generation stops at `<|im_end|>`.

## Verification

The HF weights were checked against the Megatron checkpoint by recomputing validation losses with this model:

| set | HF loss | Megatron reference | abs. diff |
|---|---|---|---|
| sft_val segments [3, 4] | 2.0234 | 2.0234 | 0.0000 |

## Links

- Training logs: wandb projects [1pp-training](https://wandb.ai/raghav_singhal/1pp-training) and [1pp-sft](https://wandb.ai/raghav_singhal/1pp-sft)
- Research artifact from the 1PP project (EPFL DLAB); not a general-purpose assistant.