---
language:
- en
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
base_model: xlr8harder/talkie-1930-13b-base-tf
datasets:
- xlr8harder/talkie-yarn-32k-gutenberg-pre1931-265m
tags:
- transformers
- safetensors
- bfloat16
- custom_code
- text-generation
- talkie
- yarn
- long-context
- pre-1931
- alternate-checkpoint
---

# Talkie 1930 13B YaRN 32k From 4k Step500

This is the step500 checkpoint from the 4k-start Talkie YaRN 32k comparison run.
It applies an 8x YaRN extension from a 4,096-token starting context, following
the later clarification that the base Talkie model had been trained at 4k even
though the public reference config advertised 2k.

The recommended checkpoint from this experiment series is
[`xlr8harder/talkie-1930-13b-yarn-32k-tf`](https://huggingface.co/xlr8harder/talkie-1930-13b-yarn-32k-tf),
the 2k-start step500 checkpoint. The 4k-start checkpoints were stronger at short
contexts but weaker at 16k and 32k, with a severe 32k collapse on variable
tracking.

Training used
[`xlr8harder/talkie-yarn-32k-gutenberg-pre1931-265m`](https://huggingface.co/datasets/xlr8harder/talkie-yarn-32k-gutenberg-pre1931-265m)
with BF16 FSDP on one 8xA100 80GB node, 8 FSDP ranks, one 32k sequence per GPU,
cosine LR decay from `1e-5` to `1e-6`, 50 warmup steps, and weight decay `0.01`.

## License

This checkpoint inherits the upstream Talkie model license, Apache-2.0. See
[`LICENSE`](./LICENSE). The continued-pretraining corpus has separate dataset
provenance and licensing documented at
[`xlr8harder/talkie-yarn-32k-gutenberg-pre1931-265m`](https://huggingface.co/datasets/xlr8harder/talkie-yarn-32k-gutenberg-pre1931-265m).

## Checkpoint Family

| Checkpoint | Role |
| --- | --- |
| [`talkie-1930-13b-yarn-32k-tf`](https://huggingface.co/xlr8harder/talkie-1930-13b-yarn-32k-tf) | Recommended 2k-start step500 checkpoint |
| [`talkie-1930-13b-yarn-32k-step1000-tf`](https://huggingface.co/xlr8harder/talkie-1930-13b-yarn-32k-step1000-tf) | Final 2k-start checkpoint |
| [`talkie-1930-13b-yarn-32k-from4k-step500-tf`](https://huggingface.co/xlr8harder/talkie-1930-13b-yarn-32k-from4k-step500-tf) | This checkpoint |
| [`talkie-1930-13b-yarn-32k-from4k-step1000-tf`](https://huggingface.co/xlr8harder/talkie-1930-13b-yarn-32k-from4k-step1000-tf) | 4k-start step1000 comparison checkpoint |

## Usage

This model uses custom Talkie modeling/tokenization code, so load it with
`trust_remote_code=True`.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "xlr8harder/talkie-1930-13b-yarn-32k-from4k-step500-tf"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)
```

For vLLM, set `--max-model-len 32768` and enable remote code.

## RULER Results

Scores are aggregate RULER accuracy percentages from our harness, using 100
examples per task and greedy decoding. It is unclear how much RULER
unintentionally penalizes Talkie because Talkie is intentionally limited to
pre-1931 training data while some RULER tasks involve modern entities and facts;
the effect is hard to quantify here, but it is likely non-zero.

| Model / setup | 2k | 4k | 8k | 16k | 32k |
| --- | ---: | ---: | ---: | ---: | ---: |
| Talkie YaRN 32k, 2k start, step500 | 80.78 | 79.50 | 73.15 | 70.05 | 61.83 |
| Talkie YaRN 32k, 2k start, step1000 | 80.30 | 79.94 | 73.17 | 67.98 | 61.83 |
| Talkie YaRN 32k, 4k start, step500 | 83.80 | 80.71 | 75.64 | 68.80 | 54.76 |
| Talkie YaRN 32k, 4k start, step1000 | 84.18 | 80.98 | 76.17 | 68.45 | 55.01 |

## Per-Task RULER Breakdown

The 2k run contains 12 benchmark groups; `qa_2` exceeded the 2k context budget
in this RULER setup and was excluded by the length constraint for that tier.

| Task | 2k | 4k | 8k | 16k | 32k |
| --- | ---: | ---: | ---: | ---: | ---: |
| Overall | 83.80 | 80.71 | 75.64 | 68.80 | 54.76 |
| `cwe` | 24.10 | 34.90 | 20.40 | 12.80 | 6.50 |
| `fwe` | 42.00 | 54.67 | 45.00 | 45.67 | 21.00 |
| `niah_multikey_1` | 100.00 | 100.00 | 99.00 | 94.00 | 92.00 |
| `niah_multikey_2` | 100.00 | 100.00 | 100.00 | 99.00 | 92.00 |
| `niah_multikey_3` | 88.00 | 73.00 | 80.00 | 31.00 | 9.00 |
| `niah_multiquery` | 99.25 | 99.25 | 97.25 | 97.00 | 89.75 |
| `niah_multivalue` | 98.00 | 96.75 | 83.25 | 88.75 | 33.25 |
| `niah_single_1` | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 |
| `niah_single_2` | 100.00 | 100.00 | 100.00 | 99.00 | 100.00 |
| `niah_single_3` | 100.00 | 98.00 | 98.00 | 78.00 | 55.00 |
| `qa_1` | 71.00 | 77.00 | 62.00 | 64.00 | 62.00 |
| `qa_2` | n/a | 51.00 | 52.00 | 51.00 | 51.00 |
| `vt` | 83.20 | 64.60 | 46.40 | 34.20 | 0.40 |

## Notes

This checkpoint is useful for comparing the 4k-start hypothesis. It is strong
at 2k-8k but falls behind the recommended 2k-start step500 checkpoint at 16k
and 32k.