---
language:
- en
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
base_model: xlr8harder/talkie-1930-13b-base-tf
datasets:
- xlr8harder/talkie-yarn-32k-gutenberg-pre1931-265m
tags:
- transformers
- safetensors
- bfloat16
- custom_code
- text-generation
- talkie
- yarn
- long-context
- pre-1931
---

# Talkie 1930 13B YaRN 32k

This is a 32k-context YaRN extension of
[`xlr8harder/talkie-1930-13b-base-tf`](https://huggingface.co/xlr8harder/talkie-1930-13b-base-tf).
It is the recommended long-context checkpoint from this experiment series.

The checkpoint was made by applying YaRN with a 16x extension from the 2,048-token
configuration in the reference Talkie repository, then continuing pretraining for
500 steps at 32,768 tokens. The continued pretraining data was a Project Gutenberg
split filtered to English public-domain books with publication years 1500-1930,
for 265,080,702 Talkie tokens:
[`xlr8harder/talkie-yarn-32k-gutenberg-pre1931-265m`](https://huggingface.co/datasets/xlr8harder/talkie-yarn-32k-gutenberg-pre1931-265m).

We originally used a 2k starting context because the public reference config
advertised 2,048 positions. The Talkie team later clarified that the base model
had been trained with 4k context. We also ran a 4k-start, 8x-extension variant;
it was slightly stronger at short contexts but substantially weaker at 32k and
collapsed on RULER variable tracking.

We selected step500 because it was more well rounded than the final step1000
checkpoint from the same 2k-start run.

## Checkpoint Family

| Checkpoint | Role |
| --- | --- |
| [`talkie-1930-13b-yarn-32k-tf`](https://huggingface.co/xlr8harder/talkie-1930-13b-yarn-32k-tf) | Recommended 2k-start step500 checkpoint |
| [`talkie-1930-13b-yarn-32k-step1000-tf`](https://huggingface.co/xlr8harder/talkie-1930-13b-yarn-32k-step1000-tf) | Final 2k-start checkpoint; included for comparison |
| [`talkie-1930-13b-yarn-32k-from4k-step500-tf`](https://huggingface.co/xlr8harder/talkie-1930-13b-yarn-32k-from4k-step500-tf) | 4k-start step500 comparison checkpoint |
| [`talkie-1930-13b-yarn-32k-from4k-step1000-tf`](https://huggingface.co/xlr8harder/talkie-1930-13b-yarn-32k-from4k-step1000-tf) | 4k-start step1000 comparison checkpoint |

## Training Details

Continued pretraining used BF16 FSDP on one 8xA100 80GB node, with 8 FSDP ranks
and one 32k sequence per GPU. This gives 262,144 tokens per optimizer step.
The schedule used cosine LR decay from `1e-5` to `1e-6`, 50 warmup steps, weight
decay `0.01`, validation every 100 steps, and exported model checkpoints every
100 steps.

## License

This checkpoint inherits the upstream Talkie model license, Apache-2.0. See
[`LICENSE`](./LICENSE). The continued-pretraining corpus has separate dataset
provenance and licensing documented at
[`xlr8harder/talkie-yarn-32k-gutenberg-pre1931-265m`](https://huggingface.co/datasets/xlr8harder/talkie-yarn-32k-gutenberg-pre1931-265m).

## Usage

This model uses custom Talkie modeling/tokenization code, so load it with
`trust_remote_code=True`.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "xlr8harder/talkie-1930-13b-yarn-32k-tf"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)
```

For vLLM, set `--max-model-len 32768` and enable remote code.

## RULER Results

Scores are aggregate RULER accuracy percentages from our harness, using 100
examples per task, greedy decoding, and the same prompt generation setup within
each model family. Different tokenizers mean nominal context lengths are not
byte-identical across unrelated model families, so use open-model rows as
orientation rather than exact head-to-head leaderboard claims. It is also unclear
how much RULER unintentionally penalizes Talkie because Talkie is intentionally
limited to pre-1931 training data while some RULER tasks involve modern entities
and facts; the effect is hard to quantify here, but it is likely non-zero.

| Model / setup | 2k | 4k | 8k | 16k | 32k |
| --- | ---: | ---: | ---: | ---: | ---: |
| Talkie base, extrapolation only | 85.86 | 77.71 | 23.40 | n/a | n/a |
| Talkie YaRN 32k, 2k start, step500 | 80.78 | 79.50 | 73.15 | 70.05 | 61.83 |
| Talkie YaRN 32k, 2k start, step1000 | 80.30 | 79.94 | 73.17 | 67.98 | 61.83 |
| Talkie YaRN 32k, 4k start, step500 | 83.80 | 80.71 | 75.64 | 68.80 | 54.76 |
| Talkie YaRN 32k, 4k start, step1000 | 84.18 | 80.98 | 76.17 | 68.45 | 55.01 |
| Llama 3.1 8B base | 97.12 | 94.25 | 92.34 | 91.61 | 88.54 |
| Yarn-Llama-2 13B 64k | 90.78 | 81.95 | 70.39 | 60.02 | 52.60 |
| Qwen3 8B pretrain base | 98.90 | 95.83 | 94.37 | 93.04 | 89.39 |

At 32k, the 2k-start step500 checkpoint was meaningfully stronger than the
4k-start checkpoints despite the 4k-start checkpoints being better at shorter
lengths. The largest qualitative difference was variable tracking (`vt`), where
the 4k-start run collapsed to near zero while this checkpoint retained partial
ability.

## Per-Task RULER Breakdown

The 2k run contains 12 benchmark groups; `qa_2` exceeded the 2k context budget
in this RULER setup and was excluded by the length constraint for that tier.

| Task | 2k | 4k | 8k | 16k | 32k |
| --- | ---: | ---: | ---: | ---: | ---: |
| Overall | 80.78 | 79.50 | 73.15 | 70.05 | 61.83 |
| `cwe` | 28.50 | 35.30 | 26.10 | 10.20 | 15.90 |
| `fwe` | 57.33 | 64.67 | 50.67 | 59.00 | 34.67 |
| `niah_multikey_1` | 100.00 | 100.00 | 99.00 | 99.00 | 97.00 |
| `niah_multikey_2` | 100.00 | 100.00 | 100.00 | 100.00 | 98.00 |
| `niah_multikey_3` | 87.00 | 86.00 | 77.00 | 41.00 | 16.00 |
| `niah_multiquery` | 96.75 | 98.00 | 95.25 | 97.00 | 92.25 |
| `niah_multivalue` | 92.00 | 92.50 | 71.50 | 81.50 | 54.75 |
| `niah_single_1` | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 |
| `niah_single_2` | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 |
| `niah_single_3` | 99.00 | 90.00 | 94.00 | 76.00 | 78.00 |
| `qa_1` | 74.00 | 75.00 | 64.00 | 61.00 | 49.00 |
| `qa_2` | n/a | 57.00 | 49.00 | 51.00 | 42.00 |
| `vt` | 34.80 | 35.00 | 24.40 | 35.00 | 26.20 |

Task shorthand: `vt` is variable tracking, `cwe` is common-word extraction,
`fwe` is frequent/coded-word extraction, `niah_*` are needle-in-a-haystack
retrieval variants, and `qa_*` are long-context question-answering tasks.

## Notes

This is a research checkpoint for long-context experimentation. It improves
Talkie's long-context RULER behavior relative to pure extrapolation, but it does
not match stronger modern long-context baselines. Use normal evaluation for your
target workload before relying on 32k behavior.