---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- historical
- 19th-century
- from-scratch
- llama
- time-capsule
datasets:
- haykgrigorian/english-historical-corpus-1800-1875-15GB-sample
---

# TimeCapsuleLLM — English 1800–1875 (v3 mini eval, 500M)

## Model Overview

**v3 mini-eval** model, trained from scratch on a ~20GB (5B-token) sample of 1800–1875 English texts using a Llama-based Causal Language Model.

| Detail | Value |
| :--- | :--- |
| **Model Architecture** | LlamaForCausalLM (Decoder-Only Transformer) |
| **Parameter Count** | **~500M** |
| **Training Type** | Trained **from Scratch** (Random Initialization) |
| **Tokenizer** | Custom BPE, Vocab Size 32,000 |
| **Sequence Length** | 4096 tokens |
| **Attention Type** | Grouped Query Attention (GQA) |

## Configuration Details

This model is a custom size and configuration based on Llama:

| Parameter | Value |
| :--- | :--- |
| **Number of Layers** | 24 |
| **Hidden Size (d)** | 1280 |
| **Intermediate Size ($\text{d}_{\text{ff}}$)** | 3456 |
| **Attention Heads** | 20 (Query) / 5 (Key/Value) |
| **Activation Function** | SiLU (`silu`) |
| **Normalization** | RMS Norm (`rms_norm_eps`: 1e-06) |
| **Position Embeddings** | Rotary Positional Embeddings (RoPE, `theta`: 10000) |


## Training data

- Corpus: english-historical-corpus-1800-1875-15GB-sample (`sample_5b_tokens`)
- 863,087 documents, ~5.71B tokens, packed into 1,393,977 sequences of length 4096

## Training

- From scratch, 13,000 steps (~0.9 epoch), effective batch 96, LR 3e-4 (500 warmup, linear decay), AdamW fused, bf16+tf32, FlashAttention-2
- 1x H100 SXM 80GB, ~21h, final train loss ~2.9

## Evaluation notes

- Strong, consistent period style across registers (newspaper, sermon, diary, parliamentary, scientific).
- No date leakage surfaced; date-seeded prompts stayed within 1800–1875.
- OCR residue is the main corpus cleanup item: end-of-line hyphenation (`ap-\npear`) and character-confusion errors (`deatli`, `lie` for "he").
- Weak on procedural/recipe text; topic drift over long generations (expected at this scale).

## Usage

Model weights and tokenizer are in the `final/` subfolder.

    import torch
    from transformers import AutoModelForCausalLM, AutoTokenizer
    REPO = "haykgrigorian/TimeCapsuleLLM-English-1800-1875-v3-mini-eval"
    tok = AutoTokenizer.from_pretrained(REPO, subfolder="final")
    model = AutoModelForCausalLM.from_pretrained(REPO, subfolder="final", dtype=torch.bfloat16).eval()

## Cost

This model was trained on an H100 SXM from RunPod

Total: $85