RWKV7-G1j-13.3B-20260831

RWKV-7 “Goose” in Flash Linear Attention format

This repository provides the RWKV-7 G1j 13.3B checkpoint in the flash-linear-attention (FLA) RWKV7 layout for Transformers-compatible inference.

This is a base language model, not a safety-aligned instruction-tuned assistant. The included chat template provides a conversational prompt format, but the model may not follow instructions consistently.

About RWKV-7

RWKV-7, also called Goose, is an attention-free recurrent language model. It maintains a constant-size recurrent state instead of an attention KV cache that grows with the preceding sequence. Training remains parallelizable, while recurrent decoding uses constant state size and constant work per generated token with respect to sequence length.

G1j identifies this checkpoint revision. The source checkpoint is available from BlinkDL/rwkv7-g1.

Model details

Property	Value
Architecture	RWKV-7 G1j
Parameters in this FLA checkpoint	13.269B
Layers	61
Hidden size	4,096
Heads	64 × 64
Feed-forward size	16,384
Vocabulary	RWKV World tokenizer, 65,536 tokens
Configured context length	16,384 tokens
Weight dtype	BF16
License	Apache-2.0

Run with FLA

Use a CUDA-capable NVIDIA GPU with BF16 support. Install the audited FLA revision with its CUDA dependency extra:

python -m pip install \
  "flash-linear-attention[cuda] @ git+https://github.com/fla-org/flash-linear-attention.git@8e84ed4a6727be082c34a3855c60623fd11411e9" \
  "transformers>=4.50.2"

Import fla before using the Auto classes so that the RWKV7 implementation is registered:

import fla
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, PreTrainedConfig

model_id = "fla-hub/RWKV7-G1j-13.3B-20260831"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    config=PreTrainedConfig(),
    trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
).to("cuda").eval()

messages = [
    {"role": "user", "content": "Explain recurrent language models in one paragraph."}
]
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    thinking=False,
)
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
    )

new_tokens = output_ids[0, inputs.input_ids.shape[1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))

Set thinking=True when applying the chat template to leave an open thinking prefix. This changes only the prompt format; it does not turn the base model into an instruction-tuned or safety-aligned assistant.

Tokenizer

This repository bundles tokenization_rwkv7.py, an exact linear-time trie implementation of the RWKV World tokenizer. It reads the self-contained tokenizer.json, preserves canonical token IDs, and avoids the poor scaling of the generic Unigram tokenizer on long repetitive inputs. Loading it requires trust_remote_code=True, as shown above.

Compatibility and validation

The package was checked with FLA commit 8e84ed4a6727be082c34a3855c60623fd11411e9 and Transformers 4.50.2. Its configuration, tokenizer, chat template, BF16 weights, and Transformers model loading were validated locally. All 2,016 model tensors load into RWKV7ForCausalLM without missing, unexpected, or mismatched keys.

CUDA/Triton generation was not executed in the local CPU-only validation environment. Runtime behavior can depend on the GPU, CUDA, PyTorch, Triton, and FLA versions. Cross-check benchmark or evaluation results against the official RWKV implementation before reporting them.

References

• RWKV-7 paper
• RWKV-LM
• Flash Linear Attention
• Source checkpoint repository