---
language:
- en
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
model_name: talkie-1930-13b-base-tf
base_model:
- talkie-lm/talkie-1930-13b-base
tags:
- transformers
- safetensors
- bfloat16
- custom_code
- text-generation
- conversion
- talkie
- pre-1931
---

# talkie-1930-13b-base-tf (BF16 Transformers + safetensors conversion)

This repository is a Transformers-compatible conversion of
[`talkie-lm/talkie-1930-13b-base`](https://huggingface.co/talkie-lm/talkie-1930-13b-base), the original Talkie base completion model.

The upstream model is a 13B vintage language model trained on 260B tokens of pre-1931 English-language text, according to the original model card.

The original base checkpoint is FP32. This repository stores a BF16 conversion of those weights and packages them for Transformers with custom `trust_remote_code` modules and BF16 sharded safetensors.

This is not an official Talkie release; refer to the upstream model card for
the author-provided provenance and usage notes.

## Source Model

- Original model: [talkie-lm/talkie-1930-13b-base](https://huggingface.co/talkie-lm/talkie-1930-13b-base)
- Talkie report: [talkie-lm.com](https://talkie-lm.com/)
- Reference code: [github.com/talkie-lm/talkie](https://github.com/talkie-lm/talkie)

## Conversion Details

- Weight dtype: BF16
- Weight format: sharded safetensors
- Context length: 2048 tokens
- Architecture: custom Talkie code loaded with `trust_remote_code=True`
- Tokenizer: Talkie tiktoken-compatible tokenizer exposed through `AutoTokenizer`

## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

path = "xlr8harder/talkie-1930-13b-base-tf"
tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    path,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map={"": "cuda"},
    use_safetensors=True,
)
```

For base completions:

```python
inputs = tokenizer("The latest discoveries in physics suggest that", return_tensors="pt").to("cuda")
output = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```

## vLLM

The included remote-code model implements the Transformers attention-interface
hooks expected by vLLM's Transformers modeling backend. For compatibility with
that backend, the original single-scalar `lm_head_gain` is folded into
`lm_head.weight` during conversion; the other Talkie gain parameters remain
explicit model parameters. Using vLLM's `logit_scale`-style approach was not
used because it applies scaling after the output matmul, while Talkie applies
the gain to the head weight before the matmul. In BF16 this can introduce small
rounding differences and, in smoke tests, changed one near-tied top-token
ordering.

```bash
vllm serve xlr8harder/talkie-1930-13b-base-tf \
  --task generate \
  --model-impl transformers \
  --trust-remote-code \
  --dtype bfloat16 \
  --max-model-len 2048
```

## Validation

The BF16 checkpoint matched a runtime BF16 cast from the original FP32 checkpoint exactly on the tested forward pass. The Transformers safetensors model was also compared against the Talkie reference architecture; the top-10 next-token ordering matched exactly, with observed max absolute logit difference `0.03125`.