xlr8harder/talkie-1930-13b-yarn-32k-tf

Talkie 1930 13B YaRN 32k

This is a 32k-context YaRN extension of xlr8harder/talkie-1930-13b-base-tf. It is the recommended long-context checkpoint from this experiment series.

The checkpoint was made by applying YaRN with a 16x extension from the 2,048-token configuration in the reference Talkie repository, then continuing pretraining for 500 steps at 32,768 tokens. The continued pretraining data was a Project Gutenberg split filtered to English public-domain books with publication years 1500-1930, for 265,080,702 Talkie tokens: xlr8harder/talkie-yarn-32k-gutenberg-pre1931-265m.

We originally used a 2k starting context because the public reference config advertised 2,048 positions. The Talkie team later clarified that the base model had been trained with 4k context. We also ran a 4k-start, 8x-extension variant; it was slightly stronger at short contexts but substantially weaker at 32k and collapsed on RULER variable tracking.

We selected step500 because it was more well rounded than the final step1000 checkpoint from the same 2k-start run.

Checkpoint Family

Checkpoint	Role
talkie-1930-13b-yarn-32k-tf	Recommended 2k-start step500 checkpoint
talkie-1930-13b-yarn-32k-step1000-tf	Final 2k-start checkpoint; included for comparison
talkie-1930-13b-yarn-32k-from4k-step500-tf	4k-start step500 comparison checkpoint
talkie-1930-13b-yarn-32k-from4k-step1000-tf	4k-start step1000 comparison checkpoint

Training Details

Continued pretraining used BF16 FSDP on one 8xA100 80GB node, with 8 FSDP ranks and one 32k sequence per GPU. This gives 262,144 tokens per optimizer step. The schedule used cosine LR decay from 1e-5 to 1e-6, 50 warmup steps, weight decay 0.01, validation every 100 steps, and exported model checkpoints every 100 steps.

License

This checkpoint inherits the upstream Talkie model license, Apache-2.0. See LICENSE. The continued-pretraining corpus has separate dataset provenance and licensing documented at xlr8harder/talkie-yarn-32k-gutenberg-pre1931-265m.

Usage

This model uses custom Talkie modeling/tokenization code, so load it with trust_remote_code=True.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "xlr8harder/talkie-1930-13b-yarn-32k-tf"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)

For vLLM, set --max-model-len 32768 and enable remote code.

RULER Results

Scores are aggregate RULER accuracy percentages from our harness, using 100 examples per task, greedy decoding, and the same prompt generation setup within each model family. Different tokenizers mean nominal context lengths are not byte-identical across unrelated model families, so use open-model rows as orientation rather than exact head-to-head leaderboard claims. It is also unclear how much RULER unintentionally penalizes Talkie because Talkie is intentionally limited to pre-1931 training data while some RULER tasks involve modern entities and facts; the effect is hard to quantify here, but it is likely non-zero.

Model / setup	2k	4k	8k	16k	32k
Talkie base, extrapolation only	85.86	77.71	23.40	n/a	n/a
Talkie YaRN 32k, 2k start, step500	80.78	79.50	73.15	70.05	61.83
Talkie YaRN 32k, 2k start, step1000	80.30	79.94	73.17	67.98	61.83
Talkie YaRN 32k, 4k start, step500	83.80	80.71	75.64	68.80	54.76
Talkie YaRN 32k, 4k start, step1000	84.18	80.98	76.17	68.45	55.01
Llama 3.1 8B base	97.12	94.25	92.34	91.61	88.54
Yarn-Llama-2 13B 64k	90.78	81.95	70.39	60.02	52.60
Qwen3 8B pretrain base	98.90	95.83	94.37	93.04	89.39

At 32k, the 2k-start step500 checkpoint was meaningfully stronger than the 4k-start checkpoints despite the 4k-start checkpoints being better at shorter lengths. The largest qualitative difference was variable tracking (vt), where the 4k-start run collapsed to near zero while this checkpoint retained partial ability.

Per-Task RULER Breakdown

The 2k run contains 12 benchmark groups; qa_2 exceeded the 2k context budget in this RULER setup and was excluded by the length constraint for that tier.

Task	2k	4k	8k	16k	32k
Overall	80.78	79.50	73.15	70.05	61.83
cwe	28.50	35.30	26.10	10.20	15.90
fwe	57.33	64.67	50.67	59.00	34.67
niah_multikey_1	100.00	100.00	99.00	99.00	97.00
niah_multikey_2	100.00	100.00	100.00	100.00	98.00
niah_multikey_3	87.00	86.00	77.00	41.00	16.00
niah_multiquery	96.75	98.00	95.25	97.00	92.25
niah_multivalue	92.00	92.50	71.50	81.50	54.75
niah_single_1	100.00	100.00	100.00	100.00	100.00
niah_single_2	100.00	100.00	100.00	100.00	100.00
niah_single_3	99.00	90.00	94.00	76.00	78.00
qa_1	74.00	75.00	64.00	61.00	49.00
qa_2	n/a	57.00	49.00	51.00	42.00
vt	34.80	35.00	24.40	35.00	26.20

Task shorthand: vt is variable tracking, cwe is common-word extraction, fwe is frequent/coded-word extraction, niah_* are needle-in-a-haystack retrieval variants, and qa_* are long-context question-answering tasks.

Notes

This is a research checkpoint for long-context experimentation. It improves Talkie's long-context RULER behavior relative to pure extrapolation, but it does not match stronger modern long-context baselines. Use normal evaluation for your target workload before relying on 32k behavior.