---
library_name: transformers
license: apache-2.0
language:
  - ko
  - en
model_type: llama
tags:
  - 3b
  - korean
  - from-scratch
  - orpo
  - instruction-tuned
  - preference-aligned
  - fp8
  - b200
  - gguf
datasets:
  - cc100
  - allenai/c4
  - heegyu/orca-math-korean-preference-cleaned
  - nayohan/preference-collection-ko-full
  - maywell/ko_Ultrafeedback_binarized
  - HuggingFaceTB/cosmopedia
  - wikimedia/wikipedia
pipeline_tag: text-generation
model-index:
  - name: FRANKENSTALLM-3B
    results:
      - task:
          type: text-generation
        dataset:
          type: kobest
          name: KoBEST (0-shot)
        metrics:
          - name: Average
            type: accuracy
            value: 52.75
          - name: COPA
            type: accuracy
            value: 63.9
          - name: HellaSwag-KO
            type: accuracy
            value: 38.0
          - name: SentiNeg
            type: accuracy
            value: 62.5
          - name: BoolQ
            type: accuracy
            value: 50.6
          - name: WiC
            type: accuracy
            value: 48.8
      - task:
          type: text-generation
        dataset:
          type: haerae
          name: HAE-RAE (0-shot)
        metrics:
          - name: Average
            type: accuracy
            value: 21.81
      - task:
          type: text-generation
        dataset:
          type: piqa
          name: PIQA (0-shot)
        metrics:
          - name: Accuracy
            type: accuracy
            value: 59.9
      - task:
          type: text-generation
        dataset:
          type: ai2_arc
          name: ARC-Easy (0-shot)
        metrics:
          - name: Accuracy
            type: accuracy
            value: 36.0
---

# FRANKENSTALLM 3B

> **A Korean 3B LLM built entirely from scratch — tokenizer, pretraining, SFT, and ORPO — on 8× NVIDIA B200 GPUs.**

| | |
|---|---|
| **Developer** | [pathcosmos](https://huggingface.co/pathcosmos) |
| **Parameters** | ~2.4B (3B-class with weight tying) |
| **Languages** | Korean (primary), English (secondary) |
| **License** | Apache 2.0 |
| **Training** | 3-phase: Pretrain → SFT → ORPO |
| **Hardware** | 8× NVIDIA B200 (FP8), ~86 hours total |

---

## Quick Start

### Transformers

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "pathcosmos/frankenstallm"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, device_map="auto"
)

inputs = tokenizer(
    "한국의 전통 음식 중 김치에 대해 설명해주세요.",
    return_tensors="pt"
).to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        do_sample=True,
        temperature=0.7,
        repetition_penalty=1.2,  # recommended
        top_p=0.9,
        max_new_tokens=512,
        pad_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

### Ollama (GGUF)

```bash
# Download GGUF + Modelfile
huggingface-cli download pathcosmos/frankenstallm \
  gguf/frankenstallm-3b-v2-Q4_K_M.gguf \
  gguf/Modelfile.3b-v2-Q4_K_M \
  --local-dir ./frankenstallm

# Fix FROM path in Modelfile, then create
ollama create frankenstallm -f ./frankenstallm/gguf/Modelfile.3b-v2-Q4_K_M

# Run
ollama run frankenstallm
```

---

## Model Highlights

- **From-scratch Korean tokenizer**: SentencePiece Unigram, 64K vocab, 99.95% Korean character coverage
- **3-phase training pipeline**: Pretrain (57K steps, ~60B tokens) → SFT (25.5K steps, 2.4M samples) → ORPO (10K steps, 630K preference pairs)
- **B200 FP8 native training**: TransformerEngine MXFP8 on NVIDIA B200 — 2× theoretical throughput vs BF16
- **GGUF deployment ready**: Q4_K_M (757MB), Q8_0 (1.2GB), F16 (2.3GB) with optimized Ollama Modelfiles

---

## Architecture

| Component | Value |
|-----------|-------|
| Type | Decoder-only Transformer (LLaMA-style) |
| Hidden size | 3,072 |
| Layers | 28 |
| Attention heads | 24 |
| KV heads | 8 (GQA 3:1) |
| FFN dim | 8,192 (SwiGLU) |
| Vocab size | 64,000 |
| Context length | 4,096 (trained at 2,048) |
| Position encoding | RoPE (θ=500,000) |
| Normalization | Pre-norm RMSNorm |
| Attention impl | FlashAttention-2 |
| Precision | FP8 (MXFP8 via TransformerEngine) |
| Weight tying | Yes (embedding ↔ lm_head) |

---

## Training Pipeline

### Phase 1: Pretraining

| Detail | Value |
|--------|-------|
| Steps | 57,000 |
| Final loss | 1.466 |
| Tokens seen | ~60B (38.5B unique × ~1.5 epochs) |
| Duration | ~63 hours |
| Data | CC-100 KO, HPLT KO, C4 KO, NamuWiki, Wikipedia KO, Cosmopedia (EN) |
| Batch size | 5 × 8 GPU × 8 accum × 2,048 seq = ~655K tok/step |

### Phase 2: Supervised Fine-Tuning (SFT)

| Detail | Value |
|--------|-------|
| Steps | 25,500 (early stop at 77.3%) |
| Best val_loss | 1.8851 (step 23,000) |
| Duration | ~15.5 hours |
| Data | 2,439,397 samples from 24 sources (7.48 GB) |
| Mix | 70% SFT + 30% pretrain replay (catastrophic forgetting prevention) |
| Knowledge forgetting | 0.9% (19 datasets) |

### Phase 3: ORPO (Odds Ratio Preference Optimization)

| Detail | Value |
|--------|-------|
| Steps | 9,997 (early convergence) |
| Best eval_loss | 1.625 |
| Preference accuracy | 76.02% |
| Reward margin | 0.6100 |
| Duration | ~7 hours |
| Data | ~630K preference pairs from 7 Korean HF datasets |
| Hyperparams | beta=0.25, lr=1.2e-5, eff_batch=128 |

**Total training time: ~86 hours on 8× B200**

---

## Benchmarks

### Training Phase Progression (Base → SFT → ORPO)

| Benchmark | Base | SFT | ORPO | Δ (Base→ORPO) |
|-----------|:----:|:---:|:----:|:---:|
| **KoBEST Avg (0-shot)** | 43.7% | 43.3% | **52.8%** | **+9.1pp** |
| KoBEST COPA | 49.3% | 48.6% | **63.9%** | +14.6pp |
| KoBEST HellaSwag-KO | 21.6% | 19.8% | **38.0%** | +16.4pp |
| KoBEST SentiNeg | 48.6% | 49.1% | **62.5%** | +13.9pp |
| KoBEST BoolQ | 50.3% | 50.1% | 50.6% | +0.3pp |
| PIQA | 52.5% | 52.6% | **59.9%** | +7.3pp |
| ARC-Easy | 25.6% | 25.9% | **36.0%** | +10.4pp |
| HAE-RAE | 19.7% | 19.9% | 21.8% | +2.1pp |
| HellaSwag EN | 26.2% | 26.1% | 29.2% | +3.0pp |
| Greedy 3-gram repetition | 61.0% | 73.0% | **30.9%** | -30.1pp |
| EOS termination rate | 0% | 60% | **67%** | +67pp |
| PPL forgetting | — | 0.9% | 4.1% | within 15% ✅ |

### 3B-class Model Comparison (Ollama, 35 tests)

| Model | Params | Korean NLU | Knowledge | Instruction | Reasoning | Avg Score |
|-------|:------:|:----------:|:---------:|:-----------:|:---------:|:---------:|
| Qwen 2.5 3B | 3B | 100.0 | 20.8 | 55.6 | 62.5 | **63.4** |
| Phi-4 Mini | 3.8B | 66.7 | 29.2 | 33.3 | **87.5** | 60.6 |
| **FRANKENSTALLM 3B** | **3B** | **100.0** | **75.0** | **66.7** | 50.0 | 46.7 |

> FRANKENSTALLM leads in **Korean NLU** (tied with Qwen), **Korean Knowledge** (75 vs 20.8/29.2), and **Instruction Following** (66.7 vs 55.6/33.3).

### Inference Speed (Ollama, Q4_K_M)

| Model | Avg TTFT | TPS | Note |
|-------|:--------:|:---:|------|
| **FRANKENSTALLM 3B** | **16.7ms** | **142.5** | Fastest |
| Phi-4 Mini 3.8B | 25.6ms | 100.4 | |
| Qwen 2.5 3B | 28.2ms | 93.8 | |

### Perplexity Preservation (ORPO Knowledge Retention)

| Dataset | Base PPL | ORPO PPL | Forgetting |
|---------|:--------:|:--------:|:----------:|
| Korean C4 | 5.72 | 5.87 | +2.7% |
| Korean Wiki | 11.84 | 12.21 | +3.2% |
| Max forgetting | — | — | 4.1% ✅ |

---

## Training Data

### Pretraining (~38.5B tokens)

| Category | Sources | Est. Tokens |
|----------|---------|:-----------:|
| Korean Web Crawl | C4 KO, CC-100 KO, HPLT KO | ~17.2B |
| Korean Encyclopedia | Wikipedia KO, NamuWiki (2 versions) | ~2.8B |
| English Educational | Cosmopedia (Stories, Web, Stanford, WikiHow, OpenStax, Khan) | ~5.7B |
| English Math/Science | AutoMathText, OpenWebMath, Proof-Pile-2 | ~8.5B |
| Code | StarCoder (filtered) | ~4.3B |

### SFT (2.4M samples, 24 sources)

| Domain | Share | Key Datasets |
|--------|:-----:|-------------|
| Reasoning/CoT | 38% | reasoning_r1_1.4m, magpie_reasoning |
| Korean Instructions | 23% | korean_instruction_mix, open_korean_instructions, kullm_v2 |
| English General | 16% | openhermes_2.5, ultrachat_200k |
| Math | 12% | NuminaMath-CoT, orca-math-ko |
| Dialog/Code/Other | 11% | smol-koreantalk, Evol-Instruct-Code-80k-ko |

### ORPO (~630K preference pairs, 7 sources)

| Dataset | Size | Domain |
|---------|:----:|--------|
| nayohan/preference-collection-ko-full | 4.9GB | General preference |
| heegyu/orca-math-korean-preference-cleaned | 1.6GB | Math reasoning |
| kuotient/orca-math-korean-dpo-pairs | 750MB | Math DPO |
| maywell/ko_Ultrafeedback_binarized | 394MB | Feedback alignment |
| tellang/yeji-preference-ko-v1 | 171MB | General preference |
| jojo0217/korean_rlhf_dataset | 137MB | RLHF pairs |
| lemon-mint/korean-realqa-reasoning-v01-preference | 58MB | QA reasoning |

---

## GGUF & Ollama

### Available Quantizations

| File | Size | Description |
|------|:----:|-------------|
| `gguf/frankenstallm-3b-v2-Q4_K_M.gguf` | 757MB | **Recommended** — best size/quality balance |
| `gguf/frankenstallm-3b-v2-Q8_0.gguf` | 1.2GB | Higher quality |
| `gguf/frankenstallm-3b-v2-f16.gguf` | 2.3GB | Full precision |
| `model.safetensors` | 4.76GB | Transformers native (ORPO best, byte-fallback fixed) |

### Recommended Sampling Parameters

| Parameter | Value | Notes |
|-----------|:-----:|-------|
| `temperature` | 0.7 | Optimal for Korean generation quality |
| `repeat_penalty` | 1.2 | **Required** — without it, greedy repetition is 30.9% |
| `top_p` | 0.9 | Nucleus sampling |
| `top_k` | 50 | Top-k candidates |
| `max_tokens` | 512 | Max generation length |
| `num_ctx` | 4096 | Context window (do not exceed) |

> ⚠️ Always use `repeat_penalty >= 1.2`. With it, repetition drops to **0%**. Without it, greedy decoding produces ~31% 3-gram repetition.

---

## Limitations

- **English performance is limited**: MMLU-EN ~23%, HellaSwag-EN ~29% — this is a Korean-focused model
- **Code generation**: Near zero capability (limited code in training data)
- **Greedy repetition**: 30.9% 3-gram repetition without `repeat_penalty` — always use sampling with `repeat_penalty >= 1.2`
- **Safety**: Safety alignment data was not included in training; use with appropriate guardrails
- **Scale gap**: Compared to commercial 3B models trained on trillions of tokens, this model was trained on ~60B tokens — expect lower overall benchmark scores

---

## Hardware & Training Environment

| Component | Specification |
|-----------|---------------|
| GPU | 8× NVIDIA B200 (183GB HBM3e each, ~1.47TB total) |
| FP8 Compute | 2,250 TFLOPS/GPU (18,000 TFLOPS total) |
| Interconnect | NVLink 5.0, NVSwitch all-to-all mesh |
| CPU | 2× AMD EPYC 9365 (72 cores, Zen 5) |
| RAM | 2.21 TB DDR5 |
| PyTorch | 2.10.0a0+b4e4ee81d3.nv25.12 (NVIDIA custom) |
| TransformerEngine | 2.10.0 |
| FlashAttention | 2.7.4 |
| NCCL | 2.28.9 |
| CUDA | 13.1 |
| Total training | ~86 hours (Pretrain 63h + SFT 15.5h + ORPO 7h) |

---

## Citation

```bibtex
@misc{frankenstallm2026,
  title={FRANKENSTALLM: A Korean 3B LLM Built From Scratch on B200 GPUs},
  author={pathcosmos},
  year={2026},
  url={https://huggingface.co/pathcosmos/frankenstallm},
  note={3-phase training (Pretrain, SFT, ORPO) with FP8 on 8x NVIDIA B200}
}
```

---

## Links

- **GitHub**: [pathcosmos/FRANKENSTALLM](https://github.com/pathcosmos/FRANKENSTALLM) — Full source code, training scripts, and builder's log
- **HuggingFace**: [pathcosmos/frankenstallm](https://huggingface.co/pathcosmos/frankenstallm)