xlr8harder/talkie-1930-13b-yarn-32k-from4k-step500-tf

Talkie 1930 13B YaRN 32k From 4k Step500

This is the step500 checkpoint from the 4k-start Talkie YaRN 32k comparison run. It applies an 8x YaRN extension from a 4,096-token starting context, following the later clarification that the base Talkie model had been trained at 4k even though the public reference config advertised 2k.

The recommended checkpoint from this experiment series is xlr8harder/talkie-1930-13b-yarn-32k-tf, the 2k-start step500 checkpoint. The 4k-start checkpoints were stronger at short contexts but weaker at 16k and 32k, with a severe 32k collapse on variable tracking.

Training used xlr8harder/talkie-yarn-32k-gutenberg-pre1931-265m with BF16 FSDP on one 8xA100 80GB node, 8 FSDP ranks, one 32k sequence per GPU, cosine LR decay from 1e-5 to 1e-6, 50 warmup steps, and weight decay 0.01.

License

This checkpoint inherits the upstream Talkie model license, Apache-2.0. See LICENSE. The continued-pretraining corpus has separate dataset provenance and licensing documented at xlr8harder/talkie-yarn-32k-gutenberg-pre1931-265m.

Checkpoint Family

Checkpoint	Role
talkie-1930-13b-yarn-32k-tf	Recommended 2k-start step500 checkpoint
talkie-1930-13b-yarn-32k-step1000-tf	Final 2k-start checkpoint
talkie-1930-13b-yarn-32k-from4k-step500-tf	This checkpoint
talkie-1930-13b-yarn-32k-from4k-step1000-tf	4k-start step1000 comparison checkpoint

Usage

This model uses custom Talkie modeling/tokenization code, so load it with trust_remote_code=True.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "xlr8harder/talkie-1930-13b-yarn-32k-from4k-step500-tf"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)

For vLLM, set --max-model-len 32768 and enable remote code.

RULER Results

Scores are aggregate RULER accuracy percentages from our harness, using 100 examples per task and greedy decoding. It is unclear how much RULER unintentionally penalizes Talkie because Talkie is intentionally limited to pre-1931 training data while some RULER tasks involve modern entities and facts; the effect is hard to quantify here, but it is likely non-zero.

Model / setup	2k	4k	8k	16k	32k
Talkie YaRN 32k, 2k start, step500	80.78	79.50	73.15	70.05	61.83
Talkie YaRN 32k, 2k start, step1000	80.30	79.94	73.17	67.98	61.83
Talkie YaRN 32k, 4k start, step500	83.80	80.71	75.64	68.80	54.76
Talkie YaRN 32k, 4k start, step1000	84.18	80.98	76.17	68.45	55.01

Per-Task RULER Breakdown

The 2k run contains 12 benchmark groups; qa_2 exceeded the 2k context budget in this RULER setup and was excluded by the length constraint for that tier.

Task	2k	4k	8k	16k	32k
Overall	83.80	80.71	75.64	68.80	54.76
cwe	24.10	34.90	20.40	12.80	6.50
fwe	42.00	54.67	45.00	45.67	21.00
niah_multikey_1	100.00	100.00	99.00	94.00	92.00
niah_multikey_2	100.00	100.00	100.00	99.00	92.00
niah_multikey_3	88.00	73.00	80.00	31.00	9.00
niah_multiquery	99.25	99.25	97.25	97.00	89.75
niah_multivalue	98.00	96.75	83.25	88.75	33.25
niah_single_1	100.00	100.00	100.00	100.00	100.00
niah_single_2	100.00	100.00	100.00	99.00	100.00
niah_single_3	100.00	98.00	98.00	78.00	55.00
qa_1	71.00	77.00	62.00	64.00	62.00
qa_2	n/a	51.00	52.00	51.00	51.00
vt	83.20	64.60	46.40	34.20	0.40

Notes

This checkpoint is useful for comparing the 4k-start hypothesis. It is strong at 2k-8k but falls behind the recommended 2k-start step500 checkpoint at 16k and 32k.