---
license: other
language:
- pl
datasets:
- SlayerLab/gollem-corpus-16b-pl
library_name: pytorch
tags:
- polish
- language-model
- research
- gollem
pipeline_tag: text-generation
---

# GoLLeM v4 — 250M (Polish, research base model — final checkpoint + training trajectory)

> ⚠️ **STATUS: TRAINING COMPLETE — RESEARCH BASE MODEL.** This repository contains the **final checkpoint** (`ckpt_48828`, 100% of the planned budget) together with the **full series of training snapshots** (roughly 4–100% of training), released **for research purposes only**. This is a **base (pretrained) model, not instruction-tuned**, and not a production system. It is a 250M research model trained on a single GPU — do not treat it as a general measure of "how good a Polish LM can be".

GoLLeM v4 is a 250M-parameter decoder-only (GPT-style) Polish language model, trained from scratch on a Polish corpus. It is a Slayer Lab project. We publish **the full series of snapshots from successive stages of training** (through the final checkpoint) so that others can study how the model learns over time.

**Author:** Arkadiusz Słota (Slayer Lab).

## What it is for (and what it is not)

**Intended use:** research on scaling and training dynamics (comparing successive snapshots), analysis of a Polish language model, and tests of fluency and text continuation in Polish.

**Not an instruction-following model.** This is a **base** (pretrained) model, **not** instruction-tuned: it continues the text you give it, it does not follow commands. It understands **Polish only** — by design, it performs poorly in English and when asked questions without context. It **does not reliably end its output** (it does not consistently emit the `<|endoftext|>` token; at the final checkpoint the measured end-of-sequence rate is ~0, so it generates up to the length limit). This is expected for a base model of this size, not a bug.

## Training snapshots (trajectory)

| Snapshot | Step | % of training | Validation loss | BPB (held-out) |
|---|---|---|---|---|
| `ckpt_2000` | 2000 | ~4% | 3.416 | 1.135 |
| `ckpt_4000` (metric only) | 4000 | ~8% | — | 1.042 |
| `ckpt_6000` | 6000 | ~12% | 2.966 | 1.001 |
| `ckpt_8000` | 8000 | ~16% | 2.937 | 0.978 |
| `ckpt_10000` | 10000 | ~20% | 2.878 | 0.961 |
| `ckpt_12000` | 12000 | ~25% | 2.846 | 0.947 |
| `ckpt_14000` | 14000 | ~29% | 2.849 | 0.938 |
| `ckpt_16000` | 16000 | ~33% | 2.789 | 0.932 |
| `ckpt_18000` | 18000 | ~37% | 2.799 | 0.924 |
| `ckpt_20000` | 20000 | ~41% | 2.757 | 0.915 |
| `ckpt_22000` | 22000 | ~45% | 2.756 | 0.910 |
| `ckpt_24000` | 24000 | ~49% | 2.689 | 0.903 |
| `ckpt_26000` | 26000 | ~53% | 2.670 | 0.901 |
| `ckpt_28000` | 28000 | ~57% | 2.652 | 0.897 |
| `ckpt_30000` | 30000 | ~61% | 2.672 | 0.891 |
| `ckpt_32000` | 32000 | ~66% | 2.692 | 0.888 |
| `ckpt_34000` | 34000 | ~70% | 2.686 | 0.884 |
| `ckpt_36000` | 36000 | ~74% | 2.639 | 0.881 |
| `ckpt_38000` | 38000 | ~78% | 2.650 | 0.877 |
| `ckpt_40000` | 40000 | ~82% | 2.643 | 0.8746 |
| `ckpt_42000` | 42000 | ~86% | 2.608 | 0.8731 |
| `ckpt_44000` (metric only) | 44000 | ~90% | 2.625 | 0.8712 |
| `ckpt_46000` | 46000 | ~94% | 2.612 | 0.8700 |
| `ckpt_48000` | 48000 | ~98% | 2.568 | 0.8696 |
| **`ckpt_48828` (final)** | 48828 | 100% | — | **0.8700** |

> **Note on metric-only rows (`ckpt_4000`, `ckpt_44000`):** their held-out BPB was measured and is shown for trajectory continuity, but **their weights are not published** — `ckpt_4000` is an early snapshot that was never saved to the release series, and `ckpt_44000` was rotated out during training. Every other listed snapshot is downloadable.

Training is **complete**: the final checkpoint is `ckpt_48828` (48,828 steps, ~71 h on a single RTX 5090). BPB falls steadily across the run (0.978 → 0.870), flattening to a plateau in the final steps; validation loss tracks training loss closely, so there is no sign of overfitting. The small bumps in validation loss are single-batch noise, not a change in trend. Over the final steps (46000–48828) the held-out BPB has **converged to a plateau of ~0.8696–0.8700** — the sub-0.001 variation between these points is evaluation noise (16-window eval), not continued improvement; the model has effectively converged at this token budget. (The final checkpoint fell between evaluation points, so it has no separate validation-loss reading; its held-out BPB was measured directly.)

**How we validated the forecast (out of sample).** Throughout training we tested a power-law forecast on steps that were not part of the fit:
- The prediction for step 18000 was on target. Between steps 20000 and 26000 the model came in *below* the extrapolation — it learned faster than the power law predicted.
- From step 26000 onward, results matched the refreshed fit within Δ ≈ −0.002 to −0.004 BPB. The forward check held **twelve times in a row** (steps 26000–48000), including every check through the learning-rate annealing phase that began at step 30000.
- An anti-collapse diversity monitor (distinct-n on generated text) flagged a transient dip at step 38000 (distinct-3 fell to 0.930). This was a **benign annealing transient, not memorization**: the direct held-out test excluded memorization — held-out BPB kept falling through the flagged window (0.8807 → 0.8772 → 0.8746) — and distinct-n rebounded. Across the whole run the model is **over-determined benign on both axes** — the direct held-out discriminator (primary) and the distinct-n proxy (secondary, 0.9564 at the endpoint) — with memorization directly disconfirmed throughout.

**End of training (measured, not forecast).** The endpoint is no longer an estimate. At the final checkpoint the held-out **BPB = 0.8700 (bits-per-byte), measured at step 48828 and recompute-verified independently** (two separate evaluation runs — the trainer's and an independent re-computation — produced the same number; the evaluation path was validated by reproducing the step-48000 value exactly). Across the deep tail (steps 46000–48828) the held-out BPB plateaus at ~0.8696–0.8700, essentially flat within evaluation noise. For the record, the earlier power-law forecast (~0.869) was **mildly optimistic**: the real measurement came in at 0.8700, consistent with the observed flattening. The theoretical floor for the language is around 0.83 BPB.

## Architecture

| | |
|---|---|
| Parameters | 247,954,432 (~250M, with tied embeddings) |
| Layers (n_layer) | 17 |
| Model width (d_model) | 1024 |
| Attention heads (n_head) | 16 (head dim 64) |
| Context length (block) | 1024 |
| Vocabulary | 32,000 (V32k tokenizer, Polish BPE) |
| Positional encoding | learned positional embeddings (`nn.Embedding`, not RoPE) |
| Block | pre-LayerNorm, causal SDPA attention, 4× GELU MLP |
| Weights | input embedding and output head are **tied** (weight tying, as in GPT-2) |

The architecture is **custom** and **does not load via `transformers`/`AutoModel`**. Use the included loader `modeling_gollem.py` (standalone; requires only `torch`, `tokenizers` and `safetensors`).

## Training data

Corpus: **[`SlayerLab/gollem-corpus-16b-pl`](https://huggingface.co/datasets/SlayerLab/gollem-corpus-16b-pl)** — **16.58B tokens, 100% Polish**, cleaned (deduplication, PII filtering, decontamination):

| Register | Tokens | Share |
|---|---|---|
| web (HPLT v3 PL, cleaned) | 14.62B | 88.2% |
| encyclopedic (Polish Wikipedia) | 984M | 5.9% |
| science | 522M | 3.1% |
| news | 211M | 1.3% |
| legal | 176M | 1.1% |
| literary | 60M | 0.4% |
| mixed | 1.6M | 0.01% |

Training target: 24B tokens (~1.45 epochs). This is deliberate overtraining — roughly 5× the Chinchilla-optimal token count — which is optimal for the cost of later inference rather than for training itself. Hardware: a single RTX 5090.

## Usage

```bash
pip install torch tokenizers safetensors
# final checkpoint (default):
python generate.py "Polska to kraj" --n 100 --temp 0.8 --topk 50
# a specific snapshot from the trajectory:
python generate.py "Polska to kraj" --ckpt ckpt_6000.safetensors --n 100
```
```python
from modeling_gollem import GollemGPT
m = GollemGPT.from_pretrained("./")                          # final checkpoint
# m = GollemGPT.from_pretrained("./", ckpt="ckpt_6000.safetensors")  # a specific stage
print(m.generate("Stolica Polski to", max_new_tokens=100, temperature=0.8, top_k=50))
```

Files in the repository: `ckpt_*.safetensors` (weights of each snapshot, pickle-free format), `config.json` (architecture and list of snapshots), `tokenizer.json` (V32k), `modeling_gollem.py` and `generate.py` (standalone loader and generation script).

> **Note on format:** weights are stored as **`safetensors`** (pure tensors, no executable code), which is safer for public release than raw `.pt` (pickle) files. Full training checkpoints (with optimizer state, for resuming training) remain internal.

## Limitations and ethical notes

- This is a **250M research base model**; although training is complete, it is small and is **not** a production or instruction-following system.
- It was trained on web data; despite filtering, it may reproduce **biases, errors and content present in the corpus**.
- A PII filter was applied (residual rate ≈ 0.0018%), but this **does not guarantee** the complete absence of personal data.
- **Research use only.** Slayer Lab accepts no responsibility for production use.

## License and data provenance

The data comes from, among other sources, the HPLT v3 (PL) corpus and CC-BY-SA resources (the encyclopedic register and parts of the legal and literary registers). The provenance of each register is documented in the corpus manifest. The weights are released for research purposes.

---
*GoLLeM v4 — Slayer Lab. **Author: Arkadiusz Słota.** Research base model (final checkpoint + full training trajectory); card maintained by the Slayer team.*