---
license: apache-2.0
library_name: rtpurbo
tags:
  - rtpurbo
  - indexer
  - stage1
  - qwen3.5-9b
---

# RTPurbo Stage-1 indexer — d_idx=32, Qwen3.5-9B, full 32-131K context

Trained by distilling the full self-attention's top-p=0.9 token set into a
32-dim retrieval index via forward KL (paper Eq.6), using a hybrid SVD
initialisation that seeds rows 0-15 from a prior d_idx=16 model and rows
16-31 from SVD principal components of Vq[16:32].

## Configuration
- base model: Qwen/Qwen3.5-9B (32 layers; 24 GDN + 8 GA at L3/7/11/15/19/23/27/31)
- GQA 16:4, **head_dim=256**, max_pos=262144
- seq_len=131072, chunked_teacher (cq=4096), teacher_query_sample=32768 (25%)
- teacher_query_min_pos=8192, reuse_k=5
- lr=1e-3 cosine, warmup=100, wd=0.01, batch=1, bf16
- init_from=hybrid_svd_dim32 (rows 0-15 from d_idx=16 ckpt, rows 16-31 SVD of Vq)
- steps=600, elapsed=341.4 min
- data: emozilla/pg19 train split, min_chars=120000 (docs roughly 30K-131K tokens)

## Final probe metrics (real source-code text @ seq=131072, block=64)

| top_p | teacher_tok | teacher_blk | indexer_blk | idx_recall |
|-------|-------------|--------------|--------------|------------|
| 0.50  | 0.001       | 0.009        | 0.032        | 0.487      |
| 0.70  | 0.003       | 0.023        | 0.075        | 0.700      |
| 0.80  | 0.007       | 0.041        | 0.117        | 0.784      |
| 0.90  | 0.018       | 0.083        | **0.200**    | **0.874**  |
| 0.95  | 0.037       | 0.140        | 0.288        | 0.918      |

- final_mean_kl = 1.7587
- final_max_kl  = 4.4431

## Compute reasoning (why d_idx=32, not 16)

Paper RTPurbo used head_dim=128 → d_idx=16 (1/8 ratio). Qwen3.5-9B has
head_dim=256, so the paper-equivalent compression is 256/8 = **d_idx=32**.
The earlier d_idx=16 run on this model was under-provisioned (equivalent to
paper's d_idx=8). Probe confirms recall is sufficient (87% at top_p=0.9,
92% at top_p=0.95), so the bottleneck for tightening the sparsity gap is the
training recipe (teacher-query-sample 25% vs paper's full-query), not d_idx.

## Files
- `indexer_final.pt`   — clean substate, 256 tensors of shape [32, 256] bf16
- `state_dict.pt`      — same content, raw state_dict format
- `stage1_summary.json`— full per-head KL breakdown for all 8 GA layers
- `loss_curve.csv`     — step,loss,lr,distill_kl,agree_kl,entropy,tok/s

## Loading
```python
import torch
from train.surgeries._rtpurbo_indexer import RetrievalIndexer

state = torch.load("indexer_final.pt", map_location="cpu")
# state is a flat dict of 256 tensors: q_heads.<layer_idx>.weight
# pass to RetrievalIndexer(d_idx=32, head_dim=256).load_state_dict(...)
```