xlr8harder/talkie-1930-13b-yarn-32k-step1000-tf

Talkie 1930 13B YaRN 32k Step1000

This is the final step1000 checkpoint from the 2k-start Talkie YaRN 32k run. The recommended checkpoint from this run is the earlier step500 export: xlr8harder/talkie-1930-13b-yarn-32k-tf.

The model uses a 16x YaRN extension from the 2,048-token reference config and was continued-pretrained at 32,768 tokens on xlr8harder/talkie-yarn-32k-gutenberg-pre1931-265m. Training used BF16 FSDP on one 8xA100 80GB node, 8 FSDP ranks, one 32k sequence per GPU, cosine LR decay from 1e-5 to 1e-6, 50 warmup steps, and weight decay 0.01.

License

This checkpoint inherits the upstream Talkie model license, Apache-2.0. See LICENSE. The continued-pretraining corpus has separate dataset provenance and licensing documented at xlr8harder/talkie-yarn-32k-gutenberg-pre1931-265m.

Checkpoint Family

Checkpoint	Role
talkie-1930-13b-yarn-32k-tf	Recommended 2k-start step500 checkpoint
talkie-1930-13b-yarn-32k-step1000-tf	This checkpoint
talkie-1930-13b-yarn-32k-from4k-step500-tf	4k-start step500 comparison checkpoint
talkie-1930-13b-yarn-32k-from4k-step1000-tf	4k-start step1000 comparison checkpoint

Usage

This model uses custom Talkie modeling/tokenization code, so load it with trust_remote_code=True.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "xlr8harder/talkie-1930-13b-yarn-32k-step1000-tf"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)

For vLLM, set --max-model-len 32768 and enable remote code.

RULER Results

Scores are aggregate RULER accuracy percentages from our harness, using 100 examples per task and greedy decoding. It is unclear how much RULER unintentionally penalizes Talkie because Talkie is intentionally limited to pre-1931 training data while some RULER tasks involve modern entities and facts; the effect is hard to quantify here, but it is likely non-zero.

Model / setup	2k	4k	8k	16k	32k
Talkie YaRN 32k, 2k start, step500	80.78	79.50	73.15	70.05	61.83
Talkie YaRN 32k, 2k start, step1000	80.30	79.94	73.17	67.98	61.83
Talkie YaRN 32k, 4k start, step500	83.80	80.71	75.64	68.80	54.76
Talkie YaRN 32k, 4k start, step1000	84.18	80.98	76.17	68.45	55.01

Per-Task RULER Breakdown

The 2k run contains 12 benchmark groups; qa_2 exceeded the 2k context budget in this RULER setup and was excluded by the length constraint for that tier.

Task	2k	4k	8k	16k	32k
Overall	80.30	79.94	73.17	67.98	61.83
cwe	29.70	36.40	27.40	15.00	15.20
fwe	47.00	55.00	50.67	48.00	30.00
niah_multikey_1	100.00	100.00	99.00	99.00	96.00
niah_multikey_2	100.00	100.00	100.00	100.00	97.00
niah_multikey_3	88.00	91.00	84.00	48.00	20.00
niah_multiquery	98.00	98.25	92.75	92.50	87.00
niah_multivalue	88.25	91.75	66.25	63.50	52.25
niah_single_1	100.00	100.00	100.00	100.00	100.00
niah_single_2	100.00	100.00	100.00	100.00	100.00
niah_single_3	99.00	90.00	93.00	68.00	67.00
qa_1	71.00	77.00	64.00	58.00	52.00
qa_2	n/a	56.00	47.00	50.00	48.00
vt	42.60	43.80	27.20	41.80	39.40

Notes

Step1000 preserved the same 32k aggregate score as step500, but was weaker at 16k and less well rounded overall. We therefore selected step500 as the main published checkpoint.