xlr8harder/talkie-1930-13b-yarn-32k-from4k-step1000-tf

Talkie 1930 13B YaRN 32k From 4k Step1000

This is an alternate 32k-context YaRN extension of xlr8harder/talkie-1930-13b-base-tf. It uses an 8x YaRN extension from a 4,096-token starting context and continued pretraining to step1000 on the same Project Gutenberg pre-1931 data recipe used for the main Talkie YaRN 32k checkpoint.

The recommended checkpoint from this experiment series is xlr8harder/talkie-1930-13b-yarn-32k-tf. That model used a 16x extension from the 2,048-token reference config and the step500 checkpoint. It performed better overall at 16k and 32k, and avoided the severe variable-tracking collapse seen in this 4k-start run.

License

This checkpoint inherits the upstream Talkie model license, Apache-2.0. See LICENSE. The continued-pretraining corpus has separate dataset provenance and licensing documented at xlr8harder/talkie-yarn-32k-gutenberg-pre1931-265m.

Checkpoint Family

Checkpoint	Role
talkie-1930-13b-yarn-32k-tf	Recommended 2k-start step500 checkpoint
talkie-1930-13b-yarn-32k-step1000-tf	Final 2k-start checkpoint
talkie-1930-13b-yarn-32k-from4k-step500-tf	4k-start step500 comparison checkpoint
talkie-1930-13b-yarn-32k-from4k-step1000-tf	This checkpoint

Usage

This model uses custom Talkie modeling/tokenization code, so load it with trust_remote_code=True.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "xlr8harder/talkie-1930-13b-yarn-32k-from4k-step1000-tf"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)

For vLLM, set --max-model-len 32768 and enable remote code.

RULER Comparison

Scores are aggregate RULER accuracy percentages from our harness, using 100 examples per task and greedy decoding. It is unclear how much RULER unintentionally penalizes Talkie because Talkie is intentionally limited to pre-1931 training data while some RULER tasks involve modern entities and facts; the effect is hard to quantify here, but it is likely non-zero.

Model / setup	2k	4k	8k	16k	32k
Talkie base, extrapolation only	85.86	77.71	23.40	n/a	n/a
Talkie YaRN 32k, 2k start, step500	80.78	79.50	73.15	70.05	61.83
Talkie YaRN 32k, 4k start, step500	83.80	80.71	75.64	68.80	54.76
Talkie YaRN 32k, 4k start, step1000	84.18	80.98	76.17	68.45	55.01

This checkpoint is included for comparison because it tests the later-discovered 4k starting context. It is better at short-context RULER tiers than the 2k-start checkpoint, but worse at the longest tiers. At 32k it scored 55.01 overall, with vt at 0.40, compared with 61.83 overall and vt at 26.20 for the recommended 2k-start step500 checkpoint.

Per-Task RULER Breakdown

The 2k run contains 12 benchmark groups; qa_2 exceeded the 2k context budget in this RULER setup and was excluded by the length constraint for that tier.

Task	2k	4k	8k	16k	32k
Overall	84.18	80.98	76.17	68.45	55.01
cwe	27.30	33.40	20.10	16.40	8.90
fwe	36.67	49.33	45.67	43.33	19.33
niah_multikey_1	100.00	100.00	99.00	95.00	92.00
niah_multikey_2	100.00	100.00	100.00	99.00	94.00
niah_multikey_3	92.00	84.00	79.00	39.00	7.00
niah_multiquery	99.25	99.00	96.75	91.25	88.50
niah_multivalue	98.00	93.75	81.75	85.50	47.00
niah_single_1	100.00	100.00	100.00	100.00	100.00
niah_single_2	100.00	100.00	100.00	97.00	100.00
niah_single_3	99.00	98.00	96.00	74.00	56.00
qa_1	70.00	76.00	66.00	65.00	52.00
qa_2	n/a	48.00	52.00	52.00	50.00
vt	88.00	71.20	54.00	32.40	0.40

Training Recipe

The training data was xlr8harder/talkie-yarn-32k-gutenberg-pre1931-265m, a Project Gutenberg split filtered to English public-domain books with publication years 1500-1930, totaling 265,080,702 Talkie tokens. Training used BF16 FSDP on one 8xA100 80GB node, 8 FSDP ranks, 32,768-token sequences, 262,144 tokens per step, 1000 max steps, cosine LR decay from 1e-5 to 1e-6, 50 warmup steps, and weight decay 0.01.