SlayerLab/gollem-v4-250m-pl

GoLLeM v4 — 250M (Polish, research base model — final checkpoint + training trajectory)

⚠️ STATUS: TRAINING COMPLETE — RESEARCH BASE MODEL. This repository contains the final checkpoint (ckpt_48828, 100% of the planned budget) together with the full series of training snapshots (roughly 4–100% of training), released for research purposes only. This is a base (pretrained) model, not instruction-tuned, and not a production system. It is a 250M research model trained on a single GPU — do not treat it as a general measure of "how good a Polish LM can be".

GoLLeM v4 is a 250M-parameter decoder-only (GPT-style) Polish language model, trained from scratch on a Polish corpus. It is a Slayer Lab project. We publish the full series of snapshots from successive stages of training (through the final checkpoint) so that others can study how the model learns over time.

Author: Arkadiusz Słota (Slayer Lab).

What it is for (and what it is not)

Intended use: research on scaling and training dynamics (comparing successive snapshots), analysis of a Polish language model, and tests of fluency and text continuation in Polish.

Not an instruction-following model. This is a base (pretrained) model, not instruction-tuned: it continues the text you give it, it does not follow commands. It understands Polish only — by design, it performs poorly in English and when asked questions without context. It does not reliably end its output (it does not consistently emit the <|endoftext|> token; at the final checkpoint the measured end-of-sequence rate is ~0, so it generates up to the length limit). This is expected for a base model of this size, not a bug.

Training snapshots (trajectory)

Snapshot	Step	% of training	Validation loss	BPB (held-out)
ckpt_2000	2000	~4%	3.416	1.135
ckpt_4000 (metric only)	4000	~8%	—	1.042
ckpt_6000	6000	~12%	2.966	1.001
ckpt_8000	8000	~16%	2.937	0.978
ckpt_10000	10000	~20%	2.878	0.961
ckpt_12000	12000	~25%	2.846	0.947
ckpt_14000	14000	~29%	2.849	0.938
ckpt_16000	16000	~33%	2.789	0.932
ckpt_18000	18000	~37%	2.799	0.924
ckpt_20000	20000	~41%	2.757	0.915
ckpt_22000	22000	~45%	2.756	0.910
ckpt_24000	24000	~49%	2.689	0.903
ckpt_26000	26000	~53%	2.670	0.901
ckpt_28000	28000	~57%	2.652	0.897
ckpt_30000	30000	~61%	2.672	0.891
ckpt_32000	32000	~66%	2.692	0.888
ckpt_34000	34000	~70%	2.686	0.884
ckpt_36000	36000	~74%	2.639	0.881
ckpt_38000	38000	~78%	2.650	0.877
ckpt_40000	40000	~82%	2.643	0.8746
ckpt_42000	42000	~86%	2.608	0.8731
ckpt_44000 (metric only)	44000	~90%	2.625	0.8712
ckpt_46000	46000	~94%	2.612	0.8700
ckpt_48000	48000	~98%	2.568	0.8696
ckpt_48828 (final)	48828	100%	—	0.8700

Note on metric-only rows (ckpt_4000, ckpt_44000): their held-out BPB was measured and is shown for trajectory continuity, but their weights are not published — ckpt_4000 is an early snapshot that was never saved to the release series, and ckpt_44000 was rotated out during training. Every other listed snapshot is downloadable.

Training is complete: the final checkpoint is ckpt_48828 (48,828 steps, ~71 h on a single RTX 5090). BPB falls steadily across the run (0.978 → 0.870), flattening to a plateau in the final steps; validation loss tracks training loss closely, so there is no sign of overfitting. The small bumps in validation loss are single-batch noise, not a change in trend. Over the final steps (46000–48828) the held-out BPB has converged to a plateau of ~0.8696–0.8700 — the sub-0.001 variation between these points is evaluation noise (16-window eval), not continued improvement; the model has effectively converged at this token budget. (The final checkpoint fell between evaluation points, so it has no separate validation-loss reading; its held-out BPB was measured directly.)

How we validated the forecast (out of sample). Throughout training we tested a power-law forecast on steps that were not part of the fit:

• The prediction for step 18000 was on target. Between steps 20000 and 26000 the model came in below the extrapolation — it learned faster than the power law predicted.
• From step 26000 onward, results matched the refreshed fit within Δ ≈ −0.002 to −0.004 BPB. The forward check held twelve times in a row (steps 26000–48000), including every check through the learning-rate annealing phase that began at step 30000.
• An anti-collapse diversity monitor (distinct-n on generated text) flagged a transient dip at step 38000 (distinct-3 fell to 0.930). This was a benign annealing transient, not memorization: the direct held-out test excluded memorization — held-out BPB kept falling through the flagged window (0.8807 → 0.8772 → 0.8746) — and distinct-n rebounded. Across the whole run the model is over-determined benign on both axes — the direct held-out discriminator (primary) and the distinct-n proxy (secondary, 0.9564 at the endpoint) — with memorization directly disconfirmed throughout.

End of training (measured, not forecast). The endpoint is no longer an estimate. At the final checkpoint the held-out BPB = 0.8700 (bits-per-byte), measured at step 48828 and recompute-verified independently (two separate evaluation runs — the trainer's and an independent re-computation — produced the same number; the evaluation path was validated by reproducing the step-48000 value exactly). Across the deep tail (steps 46000–48828) the held-out BPB plateaus at 0.8696–0.8700, essentially flat within evaluation noise. For the record, the earlier power-law forecast (0.869) was mildly optimistic: the real measurement came in at 0.8700, consistent with the observed flattening. The theoretical floor for the language is around 0.83 BPB.

Architecture

Parameters	247,954,432 (~250M, with tied embeddings)
Layers (n_layer)	17
Model width (d_model)	1024
Attention heads (n_head)	16 (head dim 64)
Context length (block)	1024
Vocabulary	32,000 (V32k tokenizer, Polish BPE)
Positional encoding	learned positional embeddings (nn.Embedding, not RoPE)
Block	pre-LayerNorm, causal SDPA attention, 4× GELU MLP
Weights	input embedding and output head are tied (weight tying, as in GPT-2)

The architecture is custom and does not load via transformers/AutoModel. Use the included loader modeling_gollem.py (standalone; requires only torch, tokenizers and safetensors).

Training data

Corpus: SlayerLab/gollem-corpus-16b-pl — 16.58B tokens, 100% Polish, cleaned (deduplication, PII filtering, decontamination):

Register	Tokens	Share
web (HPLT v3 PL, cleaned)	14.62B	88.2%
encyclopedic (Polish Wikipedia)	984M	5.9%
science	522M	3.1%
news	211M	1.3%
legal	176M	1.1%
literary	60M	0.4%
mixed	1.6M	0.01%

Training target: 24B tokens (~1.45 epochs). This is deliberate overtraining — roughly 5× the Chinchilla-optimal token count — which is optimal for the cost of later inference rather than for training itself. Hardware: a single RTX 5090.

Usage

pip install torch tokenizers safetensors
# final checkpoint (default):
python generate.py "Polska to kraj" --n 100 --temp 0.8 --topk 50
# a specific snapshot from the trajectory:
python generate.py "Polska to kraj" --ckpt ckpt_6000.safetensors --n 100

from modeling_gollem import GollemGPT
m = GollemGPT.from_pretrained("./")                          # final checkpoint
# m = GollemGPT.from_pretrained("./", ckpt="ckpt_6000.safetensors")  # a specific stage
print(m.generate("Stolica Polski to", max_new_tokens=100, temperature=0.8, top_k=50))

Files in the repository: ckpt_*.safetensors (weights of each snapshot, pickle-free format), config.json (architecture and list of snapshots), tokenizer.json (V32k), modeling_gollem.py and generate.py (standalone loader and generation script).

Note on format: weights are stored as safetensors (pure tensors, no executable code), which is safer for public release than raw .pt (pickle) files. Full training checkpoints (with optimizer state, for resuming training) remain internal.

Limitations and ethical notes

• This is a 250M research base model; although training is complete, it is small and is not a production or instruction-following system.
• It was trained on web data; despite filtering, it may reproduce biases, errors and content present in the corpus.
• A PII filter was applied (residual rate ≈ 0.0018%), but this does not guarantee the complete absence of personal data.
• Research use only. Slayer Lab accepts no responsibility for production use.

License and data provenance

The data comes from, among other sources, the HPLT v3 (PL) corpus and CC-BY-SA resources (the encyclopedic register and parts of the legal and literary registers). The provenance of each register is documented in the corpus manifest. The weights are released for research purposes.

---

GoLLeM v4 — Slayer Lab. Author: Arkadiusz Słota. Research base model (final checkpoint + full training trajectory); card maintained by the Slayer team.